Full Transcript for Ep 26: How To Build Resilient Distributed Systems with Bestselling Author Sam Newman

I I started with building res distributed systems because I'm old, I'm all I'm not that much older, but I'm I'm a I'm a fair chunk older than you, which meant that I was When I started writing systems, the hardware was terrible. And so if we wanted to write any kind of scalable, high performance application system, you had to use something which was effectively the same as a distributed system, but on a smaller scale on a couple of machines. because it's much more efficient. But it's incredibly, incredibly hard. and I know when I first heard about microservices, I thought of them as an operational tool. You know, well, microservices, you can scale up, you can scale out, you can. But it was it was only a few years later that I realized that actually microservices are a development tool. They're a developer tool, they're a way of making CI C D fast flow. work by dividing up work so that teams can work better in parallel. They they're really about making developers' lives easier. And to start with, they don't really hit the problems of high scale resilient systems. But I'm guessing that that eventually they do, which is the kind of where you come, your second wave comes in. Sam Newman: Well, I I got into microservices purely because of working with people doing kind of last mile optimization. So Anne: Mm. Sam Newman: going into organizations and they'd written some code, now it's getting to production. And so I do things like test automation, build pipeline modeling, know, path production type stuff, all theory of constraints, good old lead manufacturing stuff, right? And increasingly I found it was the architecture that was my main impediment. So my first talk on this topic was in 2011, I think. It was called designing for rapid release. was literally looking at architectural patterns about how you get software out of the door more easily. And that was the lens through which I kind of came at microservices. But ultimately semantic diffusion being a thing for many people, they've just become a replacement for the word for the name service oriented architecture. Although I think when done right, they're a bit, they're kind of more opinionated than general SLA. I also think there's a lot of, you know, I come back to a quote that So James Lewis said to me once years ago, so James Lewis, people that don't know was the person who kind of helped popularize microservices. Initially, he wrote the first paper of microservices with Martin Fowler, it actually on his website. And so we could friends, we spoke a lot about the topic at the time when he was working through these ideas. And I was working much more on the infrastructure space and he was working on the architecture space. And he said to me once that microservices buy you options. And that the idea is that you don't get these things by using the architecture. You get the option to buy these things. And so I do see a lot of people that assume that if they use a Microsoft architecture, that they'll have

a more autonomous team or that their system will be scalable or that their system will be more resilient. No, no, no, no, absolutely not. You have to do work to get them, but you get the option to do that work. And actually many, many cases I work with teams that have suffered significant degradation in resiliency by going to a Microsoft architecture because they don't understand things like network calls can fail. Anne: Mm. Sam Newman: So actually, I find that for a lot of teams, their resiliency can take a step back and not step forward by using these architectures. And so for me, was like, you I think writing a book is always a little bit I mean, I get, you know, it's a bit of a selfish ego trip, different thing, right. So to an extent, I wanted to write the book I wanted to write, but I felt that It was trying to make some of these ideas approachable was the important aspect of this. Because I think when we kind of talk about the endless list of things I cover in this book, I could have done the same thing about security, for example. Again, it just feels too daunting. And it's like, you got to try and meet people where they are a little bit. People are building these architectures, whether they should or not. Anne: Mm. Sam Newman: So what can I do to try and help like defang that a little bit? Anne: Yeah, yeah, it is it because distributive systems are really hard. And it's easy for people to and and it if if you get to it through microservices, it's quite easy to just get slowly kind of boiling a frog. It gets harder and harder and harder. It started quite straightforwardly. And then as as you're the irony is the more successful you become, the more scale, the more users you've got, then suddenly it becomes much, much harder. At an almost exponential rate. Sam Newman: I think it's partly that the problems of distributed systems are largely about the problems of failure. Anne: Mm. Sam Newman: But that when you don't have much load and you've got a small number of calls, failure is less likely to occur. Anne: Mm. Sam Newman: You're less likely to run out of resources. You're less likely to see partitions in the live. And so you're not as exposed to those things. So it's only really once you're already to the point where you're successful, where you're basically saturating resources or where you're starting, you've got a sheer enough volume of calls, you're starting to see component failure more regularly, you start to understand where you've got yourself. Anne: Mm-hmm. Sam Newman: And that's kind of why I try to distill down like why our digital systems complicated. Like, because I think I remember the eight. fallacies of distributed computing and I can never remember any things. As I've got older, I don't know if it's, you know, myself diagnosed ADHD brain or whatever else it might be, or just, you know, cognitive decline caused by age. I couldn't remember all eight. And so I kind of worked out that actually, if you explain to people, there

are three reasons why distributed systems are difficult. One, you can't make information move between two points instantly. Two, sometimes the thing you want to talk to isn't there. And three, resources run out. fundamentally underpins every single challenge that we experience at a technical level with distributed systems. It's just that as you do more of it, these things are much more like to bite you on your bum. I mean, technically speaking, if you build a single, if you build a, say a monolithic web application, you have a distributed system, right? Anne: Mm. Sam Newman: The clients are looking at your website. in a browser. So that's a program running on your device, looking at a server on the backend. The database is probably on a third machine, let alone the other pieces in the middle, like, know, CDNs, any in-house proxies or WAFs you might have and things. you have got to ditch me a system. It's just, it's less likely going to expose you to the challenges just because you've kept it, you you've not gone too far, right? So I think it was like about trying to recognize, help people recognize that that's kind of true. But as you wrap this stuff up, as you say, as you, and I think if you go microservices, you almost accept this premise that more is better, which is incorrect by the way. Anne: Yeah. Sam Newman: Right. And so I think you're just like, you're storing this pain up. so it's like, okay, well, if you're doing this stuff anyway, I don't want you to do it anyway. Let's try and hand hold you a little bit, but also. do so in a way that tries to bridge a little bit of that academic resilient engineering world in and also try and make concepts like socio-technical systems a little bit more understandable. Anne: Well talk talk me through socio technical systems. What do you mean by that and how does that how how does that work with the with the problems of a of distributed systems? Sam Newman: I mean, think you'd argue that any digital system is by definition a socio-technical system. So I'll start by that. we talked about the term is so socio and technical, socio-techny. It's basically people working together with technology. So any system where that occurs is described as a socio-technical system. It's based on some theory done by the Tavistock Institute during World War II, believe it or not, and looked into coal mining and stuff like that. But it's basically a study, the space of socio-technical systems is a study of how people and technology interact. When we think of technology, you know, that would be the hardware, that would be the software, that would be the infrastructure. But more broadly, you you throw in the other things, it's like your process, it's your organizational culture, it's the goals of the organization as well. And the reason this becomes really important when you're looking at this real resiliency mindset is that you've got to look at the components of the system. Who are the components of the system that can help

this system be resilient or not? Well, it's not just the software. It's not just the computers. It's not just the network equipment. It's also the people that operate the system. It's the goals. Those people are set. It's how that's how that organism, right? That organizational system reacts to things like failure, how it reacts to failure. How well can it do instant analysis? Can it do a surprise? Well, and so you can absolutely look at resiliency. through the lens of just the tech, the technology part of a socio-technical system, if you want to. But to do so means that you're looking probably at, it's a really, I think, inappropriate lens if you really care about resiliency. Because so many aspects of resiliency are actually about the operators and the users and the stakeholders in the system. And so I think you have to kind of understand the system as a whole. And so I like to share this like hexagonal model that I came across through some research around sort of break down that socio and the technical side of things. And so you can kind of think about like, not on the technology side of it, you've got the hardware, you've kind of got the infrastructural piece, you've got the software, you've got the processes involved with dealing with the human side of it. You've got the culture, the organization lives in, you've got the goals that that organization sets for people. So. you will quite often see failures or disasters caused by goal setting that encourages maintenance activity to not be dealt with. Some of the things I look at, for example, is a trail derailment that occurred. one of the main, one of the underpinning causes there was that the train company was trying to focus on modernization. That was an overt goal. And so they were deprioritizing, there was less incentive for anybody to focus on maintenance activity. So less maintenance was being done. And people were being penalized if they raised maintenance issues, because it was seen that they weren't doing their job. So these issues were getting raised. And so like you all start unpicking all that and you said, the root cause that they should have fixed a bit of track. Well, you can't look at it like that, right? It's way more stuff that goes on. So if you care about fixing those things or like improving... the residency of own organisation, you kind of have to take all those bits together. Anne: Yeah. It's so I mean yeah, the the the transport is always an amazing example of where they had a lot of failures and then learn from them. So, you know, true crew resource management, the the amazingly small number of aircraft craft accidents there are now based on how compared to how many they used to be, because they they deal with reporting error, they unearth issues much much better than they used to. But we're very good at learning from failure, but we we're not always so good at learning from success. And I always think that there's a really great example of success in the tech industry

that we don't talk anywhere near enough about, which is the internet, is an amazing example of socio technical success because everybody, literally all the way down to the people sitting in front of a TV at home, are part of the system. That keeps it alive because and it it runs on a massively variable resource. We think that, you know, solar solar and wind, we think of those as f as variably available resources. But bandwidth has been a variably available resource forever. And it's totally Sam Newman: I mean. Anne: unpredictable, and yet we manage. Sam Newman: But it's got to be the ancestry of the internet was based on the goals of developer and interconnected computer network that could withstand arbitrary loss of nodes in a military setting. I mean, it's sort of unsurprising in a way that it's worked and operated under those variable operating conditions when that was very much at its core. you start looking at these patterns that these application developers put into place to kind of deal with robustness issues. So a good example of is like a token bucket for like doing rate limiting. There's a client side, you can limit how many calls you make. I talk about the leaky button pattern in the book. And like, that's actually the TCP specification, right? Because it's there to do flow limiting and stuff like that. these things are built all the way kind of through the stack. And that variable condition thing is a really important part of that. There's this fundamental shift that goes from, say, safety critics, of traditional safety management to resilience engineering. Traditional safety management was all about saying, let's make sure that nothing bad happens. Right? Nothing bad's going to happen, and we can make sure nothing bad happens. The problem, of course, is if something bad does happen, you're not prepared for it. So resilience engineering is about making sure that as many things go right as possible. And I share quote in the book from Alec Kohlnagel, who's kind of quite well known in space. And he talks about the idea that a resilient system is one that should be able to vary its operations when things go wrong to keep working. So Anne: Mm-hmm. Mm-hmm. Sam Newman: we do also have to have this idea of what do we give up on? What do we accept can fail? Like in the internet, for example, when you're watching streaming at home, if you start suffering from less bandwidth, the streaming quality starts to degrade. We have graceful degradation happening at quite low levels in the protocols. And so taking those ideas into our own software, it's like, okay, well, I need to make sure as many things go right as possible, even when something wrong happens that goes wrong. So what are the things that are most important and what things can I afford to give up on? And what does good look like and what does good enough look like? And I think it lifts the debate a little bit to a more sensible level. Anne: Yeah, absolutely. It's it's it's the absolute foundation of resilience. I mean you could argue that the internet is, as you say, the internet is the most successful machine that humankind has ever developed and it's incredibly resilient. and as you say, it was built on a culture of not expecting everything to work. It's like expecting failure. If you expect failure, if your culture is expecting failure, you get an unbelievably resilient system. Sam Newman: Well, there's a comes down to Postel's law, right? John Postel, who wrote part of TCP spec, his law was, he said, be conservative in what you do and liberal in what you expect. He likes was talking about the need at the TCP spec to just expect nuts packets to arrive sometimes some kind of, you know, bit of networking equipment goes crazy on you just going to like, that's going to happen. How do I keep operating? And I think that's it moves us beyond that kind of naive mindset a little bit. And think it's also partly a facet that more of us are being exposed to large scale systems. I think it does come back to how frequently do you see an issue? Like if you've got a system that runs entirely on a single computer, if that single computer fails, really are, you you're screwed, right? So therefore it's probably beholden upon you to really make sure that that one computer has got a very, very low failure rate, right? You're going to run it. with like redundant power supplies, with redundant net rack, know, rack local networking, all of the, all of these, you know, probably a rack local UPS and all this other stuff going on, right. So try and reduce the chance that one component failing. When you've got a thousand computers operating, where's that tipping point where you stop caring about the health of the individual machine? Well, an example I use in the book is I share stories about, I'll an example of the early Google servers. which I got to see when I worked there. But there actually, I think it was a rack in a science museum and there's one I think in the computer, the computing museum in Silicon Valley, it was elsewhere as well. But their server racks, the old ones were like bare motherboards in racks, no cases on top. And that was to make it easy to get hold of components to replace them when they failed. The components that are most likely to fail were things with moving parts, which were the spinning platter hard drives. And the power supplies, the power supplies were just bolted on the back and held in place with Velcro. The hard drives were held in place with Velcro. They weren't screwed down because they'd switched their mindset, which is we've got so many computers that are going to fail, we'll build our systems to handle failure. And then what we're going to do instead is optimize for getting those computers up and running again. This is what David Woods calls rebound effectively, how quickly can we get up and running again? So literally, pull the rack out, rip out the Velcro, throw the hard drive in the bin, stick it on. kind of

wrap, you know, put it in the alkanes to get back in again, right? And so that that kind of, but that shift doesn't happen overnight. And it's, and I think comes back to that boiling frog problem we talked about a little bit as well. People start with a very, simple system. start with a topological, quite simple system, like say, you know, a monolithic single process, monolithic application, your network topology is simple, your infrastructure is simple. You can't see these issues as much. as you might otherwise do, but you start increasing the number of deployed units you've got and you'll start seeing these issues soon enough. Anne: And it becomes it's it's almost inhuman the scale at which once you reach a certain point, then probability, human human ideas of probability just cease to cease to mean anything anymore. It reminds me, it always reminds me of a I think I've talked about this in this podcast before. There's a Terry Pratchett joke where you know in fiction The million to one chance happens nine times out of ten. And i the the a a a sufficiently large system behaves more like a novel than reality. But co coincidence happen coincidences happen all the time. Sam Newman: Yeah, it comes back to the, you know, Murphy's law, aka, sods laws is called the UK, right, which is anything bad can happen, it will. And I just think you increase that certainty level to like close to one, right. Anne: Yeah, absolutely. Sam Newman: And I think that's And I don't know that that feels as real unless you've experienced it, but I don't think it's healthy or practical that we expect everybody to experience that failure before these things go wrong. that's kind of why I talk in the book quite a bit about, like the reason I talk about like the air traffic crashes and the rail trans crashes is because the stakes are so high in those domains that people do proper incident reports. And so we can learn from that. And that's kind of, don't often get the same level of detail from computer related disasters. companies aren't interested in sharing that stuff as publicly. But like I talk a lot about reading other people's incident reports, going and looking at the void database, for example, which is where there's a whole load of curated case study, know, instance that happened or doing your own game day exercises and things like that to at least try and put yourself in other people's shoes a little bit. I was saw a great talk at, I live at Lead, from incident.io. at LDX earlier this year, and they Anne: Okay. Sam Newman: said they did a game day exercise on what happens if their cloud region went down. That's in the wake of US East one crashing, like going down last year. They weren't actually directly themselves impacted by it. But some of their, although some of their sort of third parties were, but they thought, well, actually let's run through that scenario that happened to other people for ourselves. What would that do to us? How would that operate? And I think

that's a big part of being resilient. It's the So, Woods, I thought I outlined this, that Eric Woods is four concepts of Resilience Engineering, which is robustness, rebound, graceful extensibility, which is to do with surprise and sustained adaptability, which is kind of your ability to keep changing and adapting. And so you have to keep learning. You have to learn from what happened, what went wrong to you, from to you learn what happened, went wrong for other people to apply that, to change the system. The system being the people and the technology altogether, right? And if you don't continually learn and adapt, then you don't have a resilient system, you have a brittle system, right? So that's that kind of interesting challenge. I think absolutely, you know, never lose, never miss the opportunity to learn from a crisis and all that. But it doesn't mean you have to wait for your own crisis to learn from. So I do try and kind of weave through like ways of testing yourself. testing at a low level, things like game day exercises and things like that can be really useful to kind of make sure that you're happy with what you're doing or change something or whatever else. Anne: Yeah, yeah, that is a yes, maximizing your exposure to failure doesn't have to be your failure. Sam Newman: No, absolutely not. I came across a fantastic company was in my research called Uptime Labs and they do a very, very cool thing. They do incident handling training. So if the first time you've done incident response is during an incident, it's not a go well, right? You're going to be stressed, adrenaline is running high. You gonna do a great job. There's a lot to it. So they do like these online exercises, which is kind of deliberate practice because they've got like experts that give you guidance and feedback, but it's like set you up in the slack room with like monitor, you know, so you get observability data, give you information about the incident and you've got to work out how to respond to that incident. And so that's a fantastic way of you doing drills that don't have to be the real thing. I'd Anne: Mm. Sam Newman: spoken to some people and they're like, it's not the same. It's like, well, no, I it's not the same, but I remember England football team used to not practice penalties. Because the theory was that you couldn't reproduce what it was like to take a penalty in practice, in training, which of course, by logical extension means we shouldn't do any training. They should just go and play the game. It turned out that when England started actually practicing penalties, they won more penalty shootouts. And for me, like instant training, like something like Uptime Labs' ability to do that and give you good feedback on what you're learning from. And that was, was chatting to someone yesterday, he's got some junior P, he's got lots of, they haven't had many incidents. So says, that's great, but I'm worried because I haven't got that many people that know how to do

incident analysis. I said, so I just pointed at them and said, go run a few of those, just run people through them, get them to do like an hour a week or something. It's better than nothing. But you're also, you know, you're going to expose more people to that side of things. And again, that's directly a people element part of it, right? I mean, that's... There's obviously things we can do technically to make something like instant analysis, an instant response easier, like giving people good data to work with and tools that they know how to use to navigate that. But it's as much about people and behavior as anything else. Anne: And you do have to practice it because a lot of the behaviours when things go wrong are are not natural. You learn that things that you naturally want to do are not the right things to do. You know, it's like Sam Newman: Yeah, and they're not linear. They're not the same. Like I think you see all these, I've seen all these enterprise like, you know, playbooks for how you do incident analysis and stuff. And it's always like, well, when you have an incident, this will happen. And then you will do this and then you will do this and then you will do this and then profit. And it's like, no, I think, you know, incidents are messy and chaotic. And sometimes you don't realize there's an incident until halfway through. And some incidents that come up that the right person was in the right place has seen it before and goes, you'd do that sorted. Right. And that may never even get recorded as an incident. And so it's a whole messy set of different cognitive activities that happen during instant response. That don't happen in a nice, easy order. And Anne: Mm. Sam Newman: so, but all of the different aspects of it, you can break down and you can look at, and you can decide, you know, But how do I get better at this or that or the, know, whatever other part of that might be. And so that I found that was for me, really fascinating because I'd done instant response and done quite a lot of posts and reviews as well. But without ever really having any proper sort of formal training on that, if that makes sense. And so I've never had the fortune to be able to sort of step outside myself and look at what smart people say about how you should do these things and structure these things. So, you know, I think it's, that was again, selfishly the act of writing the book has been me having a lot of fun reading lots of research and stuff like that, which has been great. Anne: Yeah. But there's no there's no problem with having fun. That's it. Oddly enough, the even the the critical coping with with practice critical runs, it's stressful and it's scary, but it's also fun. But it's also team bonding, isn't it? There's nothing that bonds a team, like a bit of fear. And it's better if that fear isn't really like real fear, it's just like going to Sam Newman: Yeah. Anne: going to a theme park fear. Still Sam Newman: Yeah, but you will. mean, you swim fast if you drop a shark in the, in the swimming pool. Right. and that, was actually the nice thing. I thought the talk I saw from, for LDX from Luda was, you know, they talk, she talks about how they're there when they do their game day exercise, they're two parts, right? The first half of the day is the incident. And they have like people who are villains who are like people in the company that are going to run the instance. They, are the baddies. who are pretending to be either a really bad internal stakeholder that won't answer their email about the issue or customers who are complaining or whatever else. And they've got their own back channel and they're sort of running it. And then you've got the team trying to deal with the incident. then, but the second half of the day is them actually having a bit of a debrief and then they have pizzas coming in and then they actually have limited edition stickers that you get for coming to that game day. And so you try and collect as many stickers as you can, right? And so they've kind of made it fun and bonding exercise from that point of view. There's absolutely nothing wrong with that whatsoever. And, you know, I think it's... I think it also lifts that approach to things like that. Take those things away from what could otherwise be like a corporate box ticking exercise. People have to do some training and then now elevate it to being, this is a fun exercise. It's also for the company, incredibly valuable. Anne: Yeah, yeah. I mean, people people do escape rooms for for fun. Sam Newman: Yep. You Anne: The corporate equipment. Sam Newman: security people gets to capture the flag type stuff, Anne: No. Sam Newman: right? I mean, you know, this is, this is kind of no different. I remember the Google SREs. don't know if they still do it. They'd come around and use a thing called the wheel of misfortune because part of it is like, don't, know, as the people who are building and operating the system, almost by definition, you've tried to deal with the things that you think you need to deal with. It's very difficult to step outside yourself and and sort of think about those surprising scenarios. And so like the wheel of misfortune, what's happening today, it is a catastrophic fire in the data center. What's gonna happen to your system? I mean, I had a big network system problem, one of my first jobs, it was caused by rabbits moving into the network, ducking between buildings. You can't prepare for rabbits, right? So Anne: No. Sam Newman: it's having that kind of external element to those things can be really useful as well. because it takes you outside yourself a little bit. And you can have a bit of fun with that Anne: Yeah. Sam Newman: as well. Anne: Yeah, there's so much that can be done if people aren't too po faced about it. If you Sam Newman: Yes, you don't have an outage caused by regional conflict. You have an outage caused because Godzilla has woken up and he's angry. Right. Anne: Yeah. Sam Newman:

So it's the same circumstances, but a bit of levity is all right. Anne: Mm. Sam Newman: And I think, you know, it's, it's, it, sorts of things, think are really important. I think also coming off the back of that though, as well as, is an acknowledgement about what you're not going to try and do. And this is the point I kind of try and make in the book a lot. It's like the When you say is a system resilient or not, it's not like an absolute property on which everyone agrees. Resiliency is a quality of a system and you sort of have to decide how much of it you want. We can measure that, you know, in terms of SLOs and things like that, But that is a conversation that you have. And I think being explicit about what you aren't going to try and deal with is as important. I saw lots of You know, in the wake of the US East one outage, lots of companies being lambasted for only being in one region. Like they were idiots and didn't know this. And it's like, well, those big companies that were impacted, they absolutely knew that this was an issue. They knew this was a risk. The people like a place of like zoom that was severely impacted, for example, you're telling me they don't have people there that were aware that they were beholden to a single AWS region for a bunch of course. Of course they did, but they would have made the calculation that going multi-region. would not only have been much more expensive and added complexity, but there's a nasty paradox when it comes to resiliency. is something that David Woods kind of outlines in the Four Concepts paper. He kind of says that as we make systems, as we try and change our systems to deal with new sources of issue, new sources of errors, to make it more robust in some way, we actually can end up increasing the source. of new risks to our system. So basically, as we make our system more complex, that new complexity itself can open us up to new kinds of failure modes. So, you know, there are a bunch of challenges around, for example, doing, say, multi-region setups that beyond the cost can actually open you up to new kinds of outage models and new kinds of contagious bugs and things. And so more resiliency isn't always the answer. I think it should be a trade-off. So it comes back to seeing it as a socio-technical system, right? That includes the users of your software. What do they need? What's appropriate? And being very explicit about that. Anne: Yeah, it's interesting. I mean things like is U US East has gone down is such a good excuse that maybe actually you don't really need anything more than that. You know, it's it's Sam Newman: Yeah, I think it's sort of interesting is there is a kind of thing where if you're the only company impacted, you stand out when lots of companies are impacted. think, I think maybe five or 10 years ago, that would have given you more play. I don't think that happens now because I think we're too dependent on the software. And I think there was some quite egregious issues.

Like don't know why so many UK banks were impacted by us East one going down. Yeah. Perfectly cloud regions here, but they were right. So those are interesting. Discussions points for me. So I think there's I think you've got a little bit of that window But it kind of comes back to nobody ever got fired for buying IBM, right? When you bought IBM I'm old enough to him with that right you bought IBM software if it turned out to be terrible It can't be your fault because everyone buys IBM if you don't buy IBM and the software you bought goes wrong There's definitely your fault right? So it's safer. You sort of had had an air cover when something went wrong It was interesting to see the fallout from US East one where the blame publicly, there was a lot to go around and a lot of it went to those companies that were impacted. And I think the really interesting thing about that was seeing the contagion. know, seeing, for example, people not be able to unlock smart locks to get access to Airbnb's because Alexa is completely single home in US East one. People's alarms didn't go off. so they were late for work. I mean, I would definitely have used that excuse too. Sorry, I'm late. US East one went down again. you had Anne: Yeah. Sam Newman: a smart litter box that stopped working. I didn't know what smart litter box was. They're those, the eight sleep mattresses, which took connect to the cloud entirely. I don't know if it should have been changed by now. So if it was heating, it would keep heating the schedules were all off the cloud. It would get stuck in positions. Right. And I think that exposed a bunch of questionable decisions made by those vendors. But that, and, it's kind of going through a lot of the discourse around that. I think we've moved beyond that now where it's acceptable for people to, and people are just more generally angry with software. You know, I, there was a great tweet I saw the other day, which was trying to explain why the younger generations are so anti AI. And you and I techies we've been around a long time and they sort of pointed out that we've been around our generation and the older generation have been around to see some significant step changes in the tech industry. All right. So the web, right. was around at 95, I started doing software development, right? So you're just starting to the web being a thing. Public cloud agile with DevOps movement. We've had every five to 10 years, this interesting shift. that has made software delivery better, has made the software quality and what we can do with it better. And then the younger generation, what have they seen? I'm thinking about my son who's developer now. He's seeing web 3.0 and blockchain and AI, right? Which by and large are both giant steaming piles of doodoo in Anne: Mm. Sam Newman: terms of what they're able to do for us. mean, the benefits of AI is really only on software delivery. and isn't financially viable conversation on time. that kind of, think there, but yet software has never been more important and we're never more impacted when it goes wrong. Anne: Mm. Sam Newman: And so we're in this interesting point where people are really not happy with the continual and chitification of their services. They can't not use them. The appetite for these things failing is going down at the same time as companies are putting functionality into our systems that create new sources of instability. And I think that's, so I don't, I don't think there's, I think you're going to see a lot more societal pushback all over the world to these sorts of things. governments are catching up very slowly. You've got things like the Dora Act in the EU, for example, which for financial services institutions, which is really quite limp, but it's better than nothing. yeah, I mean, that's not why I wrote the book, but I'll definitely use that in the marketing push for it, Anne: So so when does the when do we c when can we get our hands on the book? When does the book come out? Or is it too soon to ask? Sam Newman: No, no, no, you can ask. So if you're desperate, you can get about half the book in early access form. If you have an O'Reilly subscription, can find out something for a free trial and you can read it, but that's all kind of the rough version. The files we going to the printer on the 18th of September. You can pre-order it now from all good bookshops and also Amazon, I guess, if that's your thing. If you're in the UK, use something like Hive where you can or books through local bookshops. You should be getting print copies probably arriving in your hands second week of October. E-books should be out by the end of September. For those of you weren't sure authors make more money from e-books than we do print books. They're also better for the environment. But yeah, that's kind of the plans we're looking at right now. There will be as normal like a local version going out in India, which is a... a better price point as well but I'm not sure when that's happening yet so yeah you can pre-order now if you want to but yeah really looking forward to getting this one out. Anne: Yeah, no, it's as it gets closer and closer, you're just desperate for the book to appear, aren't you really? Sam Newman: Yeah, I mean, I was really lucky. It took a year longer than it was supposed to. It took three years, which is supposed to be a two and a half. It was supposed to be a one 18 month book. It took me three years. And but I was still really enjoying the process of writing it. But I had to stop because I was I'd keep changing it. Right. It's a problem with books. So I've got to move on to the next thing. I'm working. We've already got another book planned. So I'm really lucky that the thing I enjoy most is actually writing. And so my goal is to make enough money to be able to write. So that's what, so for me, writing is the happy place. And so it's kind of, I'll be sad, I'm sad it's over in a way, but I hope it just opens me up to be able to have a lot more conversations around that topic as well. Cause that's one of the great ways, you there was a reason I wrote a second edition of my first book was cause I got to have loads of great conversations with people. And if I save just a few people from their miserable distributed systems that they've accidentally built without realizing what they were doing, then that'll be enough for me. Anne: That sounds fantastic. So basically, your community of people who read Building Microservices, if they if you haven't read it yet, read that first probably. And then don't build a distributed system. But if you find that you have accidentally built distributed systems, then definitely read Building Resilient Distributed Systems, because it will make your life much less painful. Sam Newman: I mean, and it's called resilient distributed systems because it is really for any distributed system. It's not specific to people build with microservice architecture. So I'm talking generically about the challenges of that. So whatever type of distributed architecture you might have, it's a good, I think, introduction to you. And if you're an expert, I build it into topics so you can jump right into the more detailed stuff. I'll start talking about pack, hell can harvest and yield, or you can start with easy stuff. So, I'm hoping it's a book that kind of tries to meet people where they are. Anne: It's a really fun read, as as was Building Microservices. it's it's enjoyable and it's informative at the same time. So Sam Newman: Thank you. Anne: thank you very much for being on the podcast. Sam Newman: You're very welcome. Thank you for inviting me. a nice opportunity to a chat. I really enjoyed it. Anne: Very nice opportunity to have a chat have a chat. And thank you very much to my viewers and listeners to for asynchronous and we didn't talk about how I called this podcast asynchronous and unreliable because you said don't call it asynchronous and unreliable. I thought, do you know actually, I think that might be a good name for a podcast. Sam Newman: Hahaha Anne: But we should have a separate conversation. Once your book comes out about asynchronous and unreliable, why that's a terrible name for a podcast. Sam Newman: don't think it's a terrible name for a podcast. It's more, we should talk about the word asynchronous. Because Anne: If we do need to talk about the word is it. Sam Newman: I spent a lot of time looking into that word and that's a discussion for another day. Anne: That is a whole that is a whole chapter. it's Sam Newman: yeah. Anne: like, what does asynchronous really mean? But in the meantime, thank you very much for to my listeners and viewers. I hope you enjoyed the episode and I will catch you again on next week's episode of Asynchronous and Unreliable. Thank you very much.