บทถอดความฉบับเต็มของ 323 - David Yanacek on 20 Years of Innovation at AWS

00:05:21.360 --> 00:05:23.519 I I moved down to Austin, Texas. 00:05:23.680 --> 00:05:28.639 And by the way, I don't know if we said uh I live in uh West Michigan, just outside of Grand Rapids. 00:05:29.040 --> 00:05:38.079 Um I moved down to Austin, Texas, partly to get away from the cold of Michigan, and I found that I missed the cold and like the overcast skies and stuff like that. 00:05:38.240 --> 00:05:40.000 I'm like, this kind of sucks. 00:05:40.079 --> 00:05:42.160 It's like beautiful every single day. 00:05:42.319 --> 00:05:44.560 I want it to be like crappy out sometimes. 00:05:45.040 --> 00:05:46.879 You can decide to have all four seasons. 00:05:47.199 --> 00:05:48.160 Yeah, yeah. 00:05:48.399 --> 00:05:53.600 And uh last question for you on on this topic: where exactly in Michigan did you live? 00:05:53.680 --> 00:05:54.399 Like what town? 00:05:54.720 --> 00:05:56.399 Oh, I grew up in Midland, Michigan. 00:05:56.480 --> 00:05:58.160 Uh and then I went to school in Ann Arbor. 00:05:58.399 --> 00:06:00.480 So U of M. 00:06:01.600 --> 00:06:03.600 Yeah, U of M, University of Michigan, yeah. 00:06:03.759 --> 00:06:11.920 Um but yeah, Midland, Michigan, yeah, it was a kind of a it's a large suburb. 00:06:12.000 --> 00:06:15.199 Uh I guess it calls itself a city, about 40,000 people or so. 00:06:15.519 --> 00:06:15.920 Mm-hmm. 00:06:16.399 --> 00:06:18.959 Yeah, I believe I've been through Midland. 00:06:19.199 --> 00:06:27.759 Um okay, so getting into the meat of it, I listened to a little bit of an episode that you recorded for a different podcast. 00:06:28.079 --> 00:06:34.480 Um, and you talked about AI and testing and spec driven development and stuff like that. 00:06:34.879 --> 00:06:37.920 Um this is right up my alley. 00:06:38.480 --> 00:06:54.399 Um and and I think it's kind of funny um because it's like the industry is rediscovering these things that a lot of people were doing all along, um, like test-driven development. 00:06:54.560 --> 00:07:10.800 And and it's so funny to watch because it's like, hey, if you just like decide what to do before you do it, and then write tests that represent what you intend to do, and then write code to fulfill those tests, everything turns out to work a lot better. 00:07:10.879 --> 00:07:17.360 And it's like, yeah, like uh we we knew that, but it's it's good that people are rediscovering this. 00:07:17.519 --> 00:07:20.079 Anyway, I'm I'm curious to hear your take on this. 00:07:20.399 --> 00:07:32.959 Yeah, I mean, I think uh I think that's that's actually quite a bit true around the transformation that teams I'm seeing software teams go through to be so to get so much more done uh than before. 00:07:33.040 --> 00:07:42.800 It's actually doing things that like we not necessarily new things, like just doing things that maybe we've been meaning to do and haven't gotten around to it yet. 00:07:42.959 --> 00:07:56.000 Um things where you can just increase the autonomy and and let the agentic coding loop just go and be super productive where um yeah, where before it was kind of on the backlog. 00:07:56.240 --> 00:08:09.199 Oh, yeah, that would make us a little bit faster if we had really good, you know, bet a little bit better unit tests that kind of would fail sooner if there was a problem versus an integration test, which is free further along in the pipeline or that kind of thing. 00:08:09.360 --> 00:08:19.439 So these things that teams like would always like to be able to do um just become that much more useful because you just get that much more done if you can let an agent loose with it. 00:08:19.519 --> 00:08:37.440 And and so spec-driven development, just as a starting point for all these other practices, if you want to talk about more beyond the spec, but um, you know, we it's something where we found um internally that when we were using vibe coding tools, agentic coding tools, um, we just found that they were, we could see the potential. 00:08:37.600 --> 00:08:42.320 They were so powerful and in how much it was so impressive that they could generate so much. 00:08:42.480 --> 00:08:45.039 Um, but we found that they would wander off. 00:08:45.279 --> 00:08:51.679 Maybe uh you'd come back and look at what it was doing, and it was doing some refactor that you didn't really want to do right now or ever. 00:08:52.080 --> 00:09:05.600 Um it would uh you know cheat on the test, it would kind of maybe comment out uh tests or you know, say, oh yeah, that seems about right, and and just not really do what we agreed on, or do, or maybe not even complete the total assignment. 00:09:05.679 --> 00:09:11.919 Um, and so we found that what we could do is is introduce specs, like you're talking about, the spec-driven development. 00:09:12.000 --> 00:09:31.519 And and we actually made a whole uh agentic IDE, a coding environment called Kiro, um, around spec-driven development to be able to uh to just do this, to get to be more useful for to have a more useful workflow for the agent so that it could do production grade coding, where it would actually get everything done. 00:09:31.679 --> 00:09:34.799 Um with all the resiliency and everything built in. 00:09:35.120 --> 00:09:46.000 Yeah, I want to talk a little bit more about the ways in which vanilla AI with no special guardrails will fail to do what you might want. 00:09:46.320 --> 00:09:52.639 Um, because there are a lot of different uh software development methodologies. 00:09:53.120 --> 00:10:03.279 Some are better than others, and the AI, you know, it it like knows everything, but it doesn't know how you want it to behave. 00:10:03.519 --> 00:10:06.799 It's like, what, you want me to behave smart or dumb? 00:10:06.960 --> 00:10:09.600 Like I'll I'll be dumb if that's what you want. 00:10:09.840 --> 00:10:18.720 Um and and out of the box, it's not gonna follow this like I don't know, Kent Beck style of development that I personally might want. 00:10:18.879 --> 00:10:21.279 But it can, it's

it's not like it doesn't know. 00:10:21.679 --> 00:10:32.000 It's just that it's not gonna do that by default, and then it's like it if if it does something that I think is stupid, and I'm like, oh, like don't do that. 00:10:32.159 --> 00:10:38.639 Use like the dependency inversion principle in instead of all these conditionals, and it's like, oh yeah, of course, I'll do that. 00:10:38.879 --> 00:10:41.519 But it's funny because it it won't do that on its own. 00:10:42.799 --> 00:10:43.120 Right. 00:10:43.279 --> 00:10:58.799 I think where people spend like the the the most productive time in in getting an agent to run um just better, is when they when they view everything that the agent did that they didn't want to do as a defect. 00:10:59.039 --> 00:11:01.440 Like as a defect that you can do something about. 00:11:01.679 --> 00:11:18.559 Like imagine like uh uh it's like uh if if you do like a code review, like you you actually have all the code ready and it's time to look at that code and see like, okay, if I have feedback on that code review, like obviously, okay, let's address that, but like let's figure out how to remember that for next time. 00:11:18.799 --> 00:11:45.759 And so if you kind of embrace this the this workflow of viewing every everything is okay, how do we make sure as I address this code review feedback that I add some steering, some skill, update my skills, update the steering with the project, or maybe so that the team uses so that it remembers next time when it's either generating a new spec and following it that way, or whether it's just going off and doing something that it would remember, remember how to address that thing that went wrong last time. 00:11:45.840 --> 00:11:49.279 Because you asked what uh what can go wrong. 00:11:49.440 --> 00:12:13.759 I mean, it's it's uh I guess I yeah, I just see it wander off and start and start uh you if it if it r actually, okay, so one class of things comes to when it um either hits a uh like a sometimes it'll hit a context window limit, like it'll it'll it'll just have been running for a long time and it kind of starts losing track of what it had done and what it had already learned in that session. 00:12:14.000 --> 00:12:23.039 And so that's why having having this like a spec that lays out this larger project that we want to do that might take a couple hours. 00:12:23.200 --> 00:12:25.120 It might take more than a couple hours. 00:12:25.279 --> 00:12:29.600 And that could potentially overflow context window, need to reset it, need to compress it. 00:12:29.679 --> 00:12:33.600 Um, and so having that all the whole plan written down helps it pick back up. 00:12:33.679 --> 00:12:39.360 If you say, okay, actually, let's just we've we've achieved a few tasks so far. 00:12:39.759 --> 00:12:42.080 Um, like we've done steps one through five. 00:12:42.159 --> 00:12:43.200 This is part of the spec. 00:12:43.279 --> 00:12:48.559 The spec is essentially a uh a requirements doc that you work

with the agent to develop. 00:12:48.720 --> 00:12:53.679 It's a design that you work with the agent to refine, and then it's a task list. 00:12:53.759 --> 00:13:04.240 And so these three things that make up the spec end up being a really useful thing to fall to deal with the fact that the agent, yeah, it can run out of essentially memory, context window memory. 00:13:04.399 --> 00:13:14.480 And if it does, it can just pick up which whether after it can look at what it tasks it was going to do and had already done and re-just kind of re-get its bearings and start over. 00:13:14.639 --> 00:13:18.000 So that's one thing, like the context window overflow. 00:13:18.159 --> 00:13:30.799 Um yeah, and if I if I may interrupt, um whenever I learn something new in programming, um I try I try to extract principles out of it. 00:13:30.960 --> 00:13:35.600 Um, because we're we're always presented with new tools and stuff like that. 00:13:35.840 --> 00:13:44.480 Um, but it if you can extract the principles out of it, the principles are much more portable and durable over time and stuff like that. 00:13:44.559 --> 00:13:49.200 And and it's there's a lot of profit there if you can extract the principles. 00:13:49.360 --> 00:13:51.440 I've been trying to do that with AI. 00:13:51.679 --> 00:13:59.840 Um, and I'm I I'm I'm kind of noticing multiple layers to the way the ways that people are harnessing AI. 00:14:00.159 --> 00:14:06.720 Um one is to just like tell it what to do, say like behave in this certain way. 00:14:07.039 --> 00:14:08.639 And that can work. 00:14:08.879 --> 00:14:13.200 It it seems that as the models improve, they're getting better at that. 00:14:13.360 --> 00:14:16.000 That's like my anecdotal perception on that. 00:14:16.960 --> 00:14:21.840 Act as like you are a professional senior developer, like uh that kind of thing. 00:14:22.399 --> 00:14:30.080 Yeah, yeah, just like in general, uh, don't mix refactorings with uh behavior changes, that stuff like that. 00:14:30.320 --> 00:14:39.600 Um it it seems to be getting better at that, but it's that is like a um it's like a what's how do I want to put it? 00:14:40.399 --> 00:14:42.639 It's not a very tight harness, you know? 00:14:42.960 --> 00:14:44.960 It it won't necessarily follow that. 00:14:45.200 --> 00:14:55.759 So you can do things like uh I now add pre-commit hooks that run the whole test suite before any commit can be committed. 00:14:56.000 --> 00:14:58.080 And that's like a uh physical barrier. 00:14:58.639 --> 00:14:58.960 Yeah. 00:14:59.759 --> 00:15:19.759 And I I I I think those things, whenever you can put in like a physical barrier, I think of it kind of like a woodworking jig or something like that, like a a jig that makes it impossible to make a mistake, like maybe a hole drilling jig where like you can't not drill the holes in the right place because you have this physical barrier there. 00:15:20.320 --> 00:15:21.519 So that's another thing. 00:15:21.840 --> 00:15:40.240 And

then the the kind of thing that you're describing, um, where you just save these files, and it's like, okay, I have this spec, which is kind of the authoritative source of truth for the project I'm working on, and that's durable, and you can always refer back to that. 00:15:40.559 --> 00:15:46.559 Do do you do you think about it in terms of like principles that you can apply to to harness the agent? 00:15:46.639 --> 00:15:47.759 How do you think about this? 00:15:48.399 --> 00:16:10.559 Yeah, I think though uh the the types of barriers that you can uh it's sort of actually partly about the agent harness that you build of like that it that it will the the program surrounding the agent won't let it succeed and proceed, excuse me, proceed until it uh until it uh does the thing and proves the proves the previous step. 00:16:10.720 --> 00:16:21.519 So it just enough workflow kind of along with the letting the uh LLM uh be the sort of Ouija board, uh if you will, that's like just kind of ghost that's in the machine. 00:16:21.679 --> 00:16:36.480 Um and so that one of those things uh by by breaking down the project into tasks, um those tasks would include uh actually some pretty interesting testing, uh testing barriers that you're talking about. 00:16:36.639 --> 00:16:44.320 Um one of them is uh a technique, again, another technique, like you say, that's been around a long time is is called property-based testing. 00:16:44.480 --> 00:16:47.200 It's just a type of testing, it's sort of a mindset. 00:16:47.519 --> 00:16:53.200 Um it there are frameworks out there and have been for some time around this testing technique. 00:16:53.440 --> 00:17:05.519 And this is one where, because you have a spec with really well-written um directions with like caps, like you shall, the program shall do this when in caps will this happens. 00:17:05.759 --> 00:17:11.279 And from those, it turns out those are pretty useful inputs to generate property-based testing. 00:17:11.440 --> 00:17:24.000 Um, property-based tests for just for the for everyone are things that um that test exhaustive um properties of of the program versus just boundary cases. 00:17:24.160 --> 00:17:30.000 And so you could let's take, let's say you're you're implementing a uh a traffic light uh system. 00:17:30.640 --> 00:17:37.200 You of one really important property invariant of it is that at most one direction has a green light at a time. 00:17:37.839 --> 00:17:42.720 Um that has to be held, that's its only job, really. 00:17:42.880 --> 00:17:44.960 And it's everything else is just goodness. 00:17:45.119 --> 00:17:51.279 Um and so you can just that would be described in the spec as a requirement. 00:17:51.359 --> 00:18:02.000 Um, and then a property-based test would uh generate input, they generate inputs like with a just a bunch of combination of the inputs to the program as sort of like a test driver. 00:18:02.160 --> 00:18:07.519 And they're testing along the way in every step of this that the program maintains those invariants. 00:18:07.839 --> 00:18:10.160 It's sort of just it's just a test framework, right? 00:18:10.240 --> 00:18:19.359 Uh but a good technique that is particularly useful when it comes to these requirements documents, that it can just be turned into all of these verifications. 00:18:19.440 --> 00:18:25.839 And so the coding agent won't continue until it has written the property-based tests and that they pass. 00:18:26.000 --> 00:18:28.720 And so it just it can't proceed until it has done that. 00:18:29.440 --> 00:18:40.480 I've never done property-based testing and I never I heard the name and that's it until I listened to that other podcast of yours where you where you talked about it. 00:18:40.799 --> 00:18:50.000 Um it the way it works, will the tool typically like generate some test cases? 00:18:50.079 --> 00:19:05.599 Like, if there are 17 different permutations, then it'll generate 17 tests and you do it that way, as opposed to like there is a chunk of of the test that like loops through and and does that kind of stuff, or maybe both. 00:19:05.680 --> 00:19:06.319 How does that work? 00:19:06.559 --> 00:19:11.279 Yeah, the input generate, there are a bunch of different input generator strategies that they that the frameworks supply. 00:19:11.519 --> 00:19:20.079 Some of them are include some amount of your randomness, um, looking at certainly making sure that the important combinations are all exercised. 00:19:20.160 --> 00:19:26.960 But yeah, input generator that just comes up with all the different there are different input generator strategies in the in the frameworks. 00:19:27.119 --> 00:19:34.720 Uh yeah, it's uh it tries to exhaust all possible uh inputs, uh as many as many as are practical. 00:19:34.960 --> 00:19:37.759 Um yeah, yeah, that's that's really great. 00:19:37.920 --> 00:19:48.079 Um, and that's you know, that seems to go along with this idea of like a jig that that makes it physically impossible to do the wrong thing. 00:19:48.319 --> 00:19:58.319 Um I have found sadly that LLMs tend to write fairly fairly poor quality tests, at least at this present moment. 00:19:58.480 --> 00:19:59.519 Maybe they'll get better. 00:19:59.680 --> 00:20:04.480 I think they have gotten better uh in the last couple years or whatever. 00:20:04.720 --> 00:20:07.200 Um, but at the moment, not great. 00:20:07.359 --> 00:20:25.440 I found that I if I give it a fairly thorough uh suite of examples, if I drop it into a program that already has a lot of well-written tests, it's good at extending that and writing more tests that resemble the existing ones, uh tests that I'll be pretty happy with. 00:20:25.599 --> 00:20:32.559 I found that when I do a greenfield project with Claude Code, for example, the tests are not always great. 00:20:32.720 --> 00:20:36.079 And it seems to depend somewhat on the language. 00:20:36.160 --> 00:20:45.759 There's kind of a testing culture base baked into each uh community, and and some

communities have better testing cultures than others. 00:20:46.160 --> 00:20:49.519 Anyway, I was writing a program in Rust. 00:20:49.680 --> 00:20:51.440 I I don't know Rust at all. 00:20:51.599 --> 00:20:57.279 I was just letting Claude Code write the Rust, and I was really unhappy with the tests that it was writing. 00:20:57.359 --> 00:21:01.920 They were very just like perfunctory, um tautological. 00:21:02.079 --> 00:21:05.680 Just if I put in five, assert that it's set to five. 00:21:05.839 --> 00:21:07.920 It's like, well, that's that's kind of pointless. 00:21:08.319 --> 00:21:22.079 So I hooked it up to mutation testing, and so I said, write the tests and then do the mutation testing so that um it's it's a much more rigorous, thorough way of testing it. 00:21:22.160 --> 00:21:24.799 And I found that to to work out a lot better. 00:21:24.880 --> 00:21:28.640 It kind of reminds me of of this property testing idea. 00:21:29.440 --> 00:21:33.759 Have you also applied mutation testing to this this at all also? 00:21:34.319 --> 00:21:36.880 I'm actually not familiar with the term mutation testing. 00:21:37.359 --> 00:21:39.039 Oh yeah, it I'm new to it. 00:21:39.279 --> 00:21:53.920 Um so the idea is that you write the test, you write the code to make the test pass, but then who's to say uh that that everything is as it should be? 00:21:54.079 --> 00:22:06.720 Like, could it be that your test gives some sort of false positive um And maybe you wrote too much application code beyond what you really needed to satisfy the test, or something like that. 00:22:07.039 --> 00:22:14.960 So the mutation testing library will mess with your code and it'll say, okay, this passes, but what if we tweak it like this? 00:22:15.599 --> 00:22:17.440 Do the tests still pass then? 00:22:17.599 --> 00:22:25.920 Oh, because if it does, then like, gotcha, these tests are invalid and we need more test cases and blah, blah, blah. 00:22:26.480 --> 00:22:30.559 Um I I I found that to be a really powerful technique. 00:22:30.960 --> 00:22:37.839 Yeah, I think like these verification and testing techniques that have been around, I think it's it's really their time to shine. 00:22:37.920 --> 00:23:10.799 Uh and so one thing we do uh at Amazon a bunch for these um um for we build a lot of distributed systems like in AWS, uh, you know, we have things like DynamoDB, things like S3, where we have to prove that the replication, like these distributed systems around failover during a network partition while bootstrapping another replica, like all these the this is the this is the stuff that personally I like I love so much is all of these uh distributed systems algorithms. 00:23:11.039 --> 00:23:17.359 And in order to prove these, um we do a lot of formal verification methods, like TLA plus. 00:23:17.599 --> 00:23:19.599 Uh, it's a really good verifier of a model. 00:23:19.680 --> 00:23:25.200 It's like I have this model, uh, and and this the TLA just it it beats it up. 00:23:25.440 --> 00:23:42.880 You have your your here's how my system will behave, and it beats it up with introducing like whatever latency at the critical moment, and to see if if you still maintain an invariant, like only one uh replica of a replicated database is the is the leader at any point in time. 00:23:42.960 --> 00:23:51.920 Like all the it just is very good at these kinds of synthetic uh tests around all the different cases that you would care about in a digit distributed system. 00:23:52.160 --> 00:23:57.039 And I think these this is they are kind of hard to write. 00:23:57.279 --> 00:23:59.039 Um they can be. 00:23:59.200 --> 00:24:03.359 Um, these models, and then you write your code, which isn't the model. 00:24:03.519 --> 00:24:06.880 It's actually your code is a rep is a is a manifestation of that model. 00:24:07.039 --> 00:24:24.799 And so I think it's these methods kind of time to shine, because you can now you can now write generate, write the model, and then generate the code and and have the LLM reason about and prove differences between the model, which already proved your algorithm, and your code to see does it match the model. 00:24:24.960 --> 00:24:32.960 There are a bunch of techniques around this, but I think it it my point is just that it is it's the time for these formal methods and and just testing methods to shine. 00:24:33.519 --> 00:24:35.200 Yeah, yeah, I totally agree. 00:24:35.359 --> 00:24:38.720 Um I I have to take us on a small digression. 00:24:38.880 --> 00:24:46.319 Um, my friend uh Stephen Baker mentioned the other day, uh he he made an analogy with oil. 00:24:46.720 --> 00:25:06.319 So uh back in the I don't know, mid-late 1800s, second half of the 1800s, um oil wasn't used for gasoline and automobiles, it was mainly used to make kerosene for lighting, and the market size was limited. 00:25:06.480 --> 00:25:13.839 Um Standard oil was was dominant at that time, but their their main market was um kerosene. 00:25:14.480 --> 00:25:38.559 Um but then uh industrialization came along, uh World War One happened, um battleships converted from coal to oil, um, people started buying cars, the the adoption of automobiles just totally exploded, and all of a sudden it it was oil's time to shine, you know. 00:25:38.720 --> 00:25:58.640 It it had been there this whole time under our feet for millions of years or whatever, uh and and then used for this limited application, just for for lighting and and stuff like that, but then the world changed and it got so much more useful, indispensable. 00:25:59.200 --> 00:26:07.519 Um and so these these testing techniques techniques and such uh could be compared to oil. 00:26:07.920 --> 00:26:13.200 We've had them for decades, and they've they've already been great. 00:26:13.359 --> 00:26:18.319 Uh but now it's like okay, there's there's I think two two things happening. 00:26:18.559 --> 00:26:25.039 One is a greater necessity for these techniques, because if you don't use them,

then AI will just go off the rails. 00:26:25.519 --> 00:26:30.240 Um and another is that the the bar has been lowered. 00:26:30.480 --> 00:26:37.759 Uh I I don't want to confuse my metaphors, but I've I've always talked about AI kind of like a mental lubricant. 00:26:37.920 --> 00:26:47.200 You know, it these these things that have friction, um, like writing tests, for example, there might be some mental friction there. 00:26:47.599 --> 00:26:55.279 Uh it's AI is like WD40, which is not a lubricant, but um it can unstick your mind. 00:26:55.599 --> 00:27:01.279 Um and it makes these things which were previously difficult much easier. 00:27:01.440 --> 00:27:06.799 So those two things, uh I think, you know, like you're saying, it's it's their time to shine. 00:27:07.440 --> 00:27:10.880 Those those are uh hurrying that along. 00:27:11.359 --> 00:27:11.920 For sure. 00:27:12.079 --> 00:27:23.519 I think uh the it's just you know, I've spent the last 20 years just with uh almost singular purpose in mind, and that's just to make developers' lives easier. 00:27:23.759 --> 00:27:35.359 That is the pattern that I have followed ever since I started at Amazon and had to operate a database in order to automate server operations because I needed to keep track of all the servers. 00:27:35.440 --> 00:27:36.640 I needed to operate a database. 00:27:36.799 --> 00:27:39.359 Operating a database is harder than operating the servers. 00:27:39.519 --> 00:27:45.119 And so then it's like, okay, how do I net how can I never have to do this difficult database ops again? 00:27:45.200 --> 00:27:50.160 Oh, I heard we're let's make this dynamo DB, let's make NoSQL, so then we don't have to do database ops from then. 00:27:50.240 --> 00:27:51.200 It'll just be managed. 00:27:51.359 --> 00:28:03.759 That sort of rinse and repeat with like I've worked on Lambda then for managing servers, the building serverless, so you don't have to manage servers, API gateway because API routing is tedious and and easy to get wrong. 00:28:03.839 --> 00:28:07.279 Um so just I've been chasing this this like constant. 00:28:07.519 --> 00:28:11.599 How do I build the next abstraction that's gonna make my life easier? 00:28:11.839 --> 00:28:19.039 And the really fantastic part about LLMs is that they can do that just they're so flexible. 00:28:19.119 --> 00:28:23.359 Like they can, they are that uh sort of this missing ingredient to be able to do that. 00:28:23.519 --> 00:28:26.400 Well, how do I how do I do the next part that I find tedious? 00:28:26.480 --> 00:28:28.880 Like server ups, okay, now I don't have to do that. 00:28:29.039 --> 00:28:31.440 But what about uh writing tests? 00:28:31.519 --> 00:28:32.160 You know, right? 00:28:32.240 --> 00:28:41.920 Like that is it is something that people we could have been writing property-based tests and and uh mutation tests this entire time. 00:28:42.079 --> 00:28:47.440 But it's just the it's been tedious uh to do all the time and spend all of our

time. 00:28:47.680 --> 00:28:48.400 It's not as fun. 00:28:48.559 --> 00:28:51.920 And so great, let's have let's have the agent do that. 00:28:52.079 --> 00:28:55.440 We have to now the work is to prove that it actually wrote good tests. 00:28:55.519 --> 00:29:08.000 You know, and and these frameworks can help help do that, provide that evidence or the the enough of that guarding guardrail, just like the spectrum development provides some guardrails and and and a path to success. 00:29:08.079 --> 00:29:10.640 Um, these tests can be a part of that too. 00:29:12.000 --> 00:29:14.559 Yeah, yeah, it's a really exciting time. 00:29:14.799 --> 00:29:20.880 Um and and uh uh uh what you say is exactly true. 00:29:21.119 --> 00:29:41.519 Um on the developer experience side, like I've invested a lot in my own developer experience because it's um you know you you pay a small price to get hopefully a large benefit, and then you can go faster, and it's kind of a positive feedback loop. 00:29:42.240 --> 00:30:07.119 Um so maybe if we if we can get into that a bit, like what kinds of stuff uh, and and it doesn't even have to be only AI, but like what kind of stuff have you done to make uh developer experience easier, especially things that people might not like most companies might not do, that kind of stuff, just anywhere you want to take that. 00:30:07.519 --> 00:30:07.839 Sure. 00:30:08.000 --> 00:30:18.400 Uh I'd say a lot of it stems from the that the first team I was on on Amazon.com, where I was on the team that ran Amazon.com's web server fleets. 00:30:18.559 --> 00:30:22.000 Um we had just a lot of servers that needed to get provisioned. 00:30:22.079 --> 00:30:24.799 We had to forecast how many to physic to actually buy. 00:30:24.880 --> 00:30:27.839 This was pre pre-EC2, pre-AWS cloud. 00:30:27.920 --> 00:30:29.359 We had to buy servers every year. 00:30:29.519 --> 00:30:40.079 So I had to manage keeping track of how many we have and how many we need to buy, what their utilization is, forecast the efficiency uh that we're going, like, is the website going to get less efficient or more efficient? 00:30:40.319 --> 00:30:43.920 Probably less efficient because that's the way that entropy works. 00:30:44.079 --> 00:30:45.759 Um, and so we had all these things. 00:30:45.839 --> 00:30:55.519 And so one of the things that might be a little bit uh should it seemed like it should be easy, but it wasn't, was uh calling other services. 00:30:55.680 --> 00:31:09.599 So I would make this a forecasting tool that would figure out how many servers to buy every year, but I would need to call and get the utilization data, the observability data about how like CPU usage, requests to the website, all that stuff. 00:31:09.759 --> 00:31:17.599 And that was in this other web service, um, the monitoring web services as it happened to be called, later became CloudWatch. 00:31:17.759 --> 00:31:31.680 Um but calling that service was actually kind of annoying because they it was

a sort of a new paradigm around REST, where it was like, okay, uh you don't have to generate a client library. 00:31:31.759 --> 00:31:33.440 It's like, but I'm writing a Java program. 00:31:33.519 --> 00:31:41.920 It's like, no, just like, you know, use what just make HTTP requests and fish around in the return to XML or JSON for the response. 00:31:42.400 --> 00:31:44.160 Okay, but I've I have a Java program here. 00:31:44.319 --> 00:31:46.799 Like I do need to generate types or write types. 00:31:46.880 --> 00:31:51.759 So I just found it sort of uh tedious to call some of these services. 00:31:51.920 --> 00:31:57.680 Whereas before we had some frameworks at Amazon that just generated strongly typed stuff, but it wasn't REST under the hood. 00:31:57.839 --> 00:32:03.519 And so it was there was this just kind of a new way that people were writing and new frameworks people were bringing in. 00:32:03.680 --> 00:32:15.440 And so I found it those made it maybe easier for the service author to write a service because they could do it in the newer, newer way and newer tools and newer frameworks, but those had left the clients behind. 00:32:15.680 --> 00:32:19.440 And so I found that tedious of like just calling services. 00:32:19.519 --> 00:32:20.640 It's supposed to be easy. 00:32:20.799 --> 00:32:25.039 And so I learned that there was a team making a new framework. 00:32:25.119 --> 00:32:26.960 Um, we call it Coral. 00:32:27.200 --> 00:32:41.920 It it its goal was to be a modern framework that that could speak REST, but also speak all the other protocols that we have, all the binary formats, all the everything to be back so that it would fit into whatever tooling that clients wanted to use. 00:32:42.000 --> 00:32:50.319 It would sort of actually just make everybody happy, gener be a nice modern like web service framework, but also support any kind of client that people want. 00:32:50.480 --> 00:32:53.039 Like also basically generate clients as well. 00:32:53.200 --> 00:32:56.799 And that kind of became like the AWS SDK generation in a way. 00:32:56.880 --> 00:33:01.839 Um so I found that this that's just an example of one of these that I've chased down over the years. 00:33:01.920 --> 00:33:03.279 That's it's like subtle. 00:33:03.359 --> 00:33:17.200 It's like, okay, a web services are are tricky and uh and and building the service is hard, but also making sure that you are paying attention to your customers so that calling the service is easy too. 00:33:17.279 --> 00:33:27.039 And so uh this coral sort of actually in a way became this API gateway uh thing to because frameworks are are also kind of hard. 00:33:28.240 --> 00:33:28.720 Right. 00:33:28.880 --> 00:33:37.440 Um okay, so this might be the the point in the show when we uh wildly wander off the trail as inevitably happens. 00:33:37.680 --> 00:33:40.480 Um but I I'm curious about something. 00:33:40.640 --> 00:33:51.759 So like if you have a large system with a bunch of different services and stuff like that, um there's a lot of different ways you can have them talk to each other. 00:33:52.000 --> 00:34:02.720 Um something I've seen that I haven't loved is when each service basically talks to every other service. 00:34:03.279 --> 00:34:13.039 Um and I I did some research because uh uh most of my career has been at very small startups that haven't had this situation. 00:34:13.519 --> 00:34:23.519 Um but I I I did some research once I started to work at a big company, and I'm like, okay, like what's what what are the different ways this could be architected? 00:34:23.760 --> 00:34:31.199 And it sounds like what I was looking at was point-to-point architecture, and I was seeking something more like hub and spoke. 00:34:31.360 --> 00:34:46.320 Um because the the thing that I the thing that makes me uncomfortable about the point-to-point architecture is every service had its own custom, unique way of talking to every other service. 00:34:46.480 --> 00:34:49.039 And then what happens when you add a new service? 00:34:49.199 --> 00:35:03.039 Now you have to like start from scratch and incur the cost of building ways to talk to all the other services, and you have to have all the credentials and all this stuff, and it's just like this this can't be the way to do this. 00:35:03.280 --> 00:35:11.039 With a Hub and Spoke, it seems like you just talk to the hub, everybody just talks to the hub, and it's like, hey, I want to send a message. 00:35:11.119 --> 00:35:21.519 I I'm not gonna integrate with Slack and tell Slack to send a message, I'm just gonna notify the hub that something happened, and the hub can say, Oh, okay, this thing happened. 00:35:21.599 --> 00:35:29.440 Uh I'll send a message to Slack and or send an email, whatever, whatever I, the hub, thinks need to happen. 00:35:29.599 --> 00:35:38.400 Um I'm just curious, any commentary you might have on that whole thing is somebody way more experienced than myself and those kind of things. 00:35:39.280 --> 00:35:44.960 I've seen so many patterns of this um over the years, like what work to different degrees. 00:35:45.119 --> 00:35:51.920 I think there are different problems that that are it's helpful to have more of a of this kind of hub and spoke. 00:35:52.000 --> 00:35:58.000 Uh there was one uh like looking at the um, so I worked on the Amazon.com web server fleet. 00:35:58.079 --> 00:36:01.679 That means the thing that ran that rendered HTML, ultimately. 00:36:01.840 --> 00:36:03.360 And it needs to make a lot of service. 00:36:03.519 --> 00:36:10.320 Get data about if you're looking at a product page, we need to get data about the product in order to show decide how to display it. 00:36:10.480 --> 00:36:19.920 Um turns out the information about what is a product with when you have when you can sell any kind of product, uh, whether it's digital or physical. 00:36:20.000 --> 00:36:25.119 And you know, when we started, we only had books, and then we had physical products, then we

had digital products, it just evolves a lot. 00:36:25.280 --> 00:36:30.159 And over time, yeah, we built uh one of a kind of a dedicated around products. 00:36:30.320 --> 00:36:52.800 We built some of what you describe as as a as one of these hub services that would kind of aggregate all of the and and deal with figuring out like, okay, given this request to look to display this product, it could it had logic in it to figure out where to get that from the different other services that had a little piece of this information here, a piece of information there. 00:36:53.039 --> 00:37:11.360 Similarly, with like the order processing pipeline that would because placing an order for something that is uh there's so many different types of products, and so that you need to be able to handle different products differently and the workflow changes on the fly as things get um depending on who's going to fulfill that order and everything. 00:37:11.519 --> 00:37:13.920 So it's uh we've seen a lot of it. 00:37:14.079 --> 00:37:35.760 I don't think there's any any one size fits all, but there is definitely like a lot of people will do what you described with uh with API gateway because it can you can just have that is the API layer and it handles all the different types of API requests, but um behind the scenes, some paths go to one service or another, and you don't have to think about point integrations like that. 00:37:35.840 --> 00:37:37.039 So that can be nice. 00:37:37.199 --> 00:37:52.400 Um, I do see think that one thing that worked particularly well that kind of works no matter which, whether it is point-to-point or hub and spoke or event bus or whatever is just common frameworks. 00:37:52.559 --> 00:38:04.400 If people are speaking the same, having the same identity, like the company uses the same identity system for what is a service, what identifies a service, and what authorizes one service to talk to another. 00:38:04.559 --> 00:38:08.400 Like in AWS, you you can that's one that's this identity access manager. 00:38:08.559 --> 00:38:18.559 IAM just is the policy that just and sort of a signing language that that describes you with with one set of credentials, you can scope those to be able to call any service. 00:38:18.639 --> 00:38:23.840 You don't have to figure out how to speak a different identity provider system or anything. 00:38:23.920 --> 00:38:26.239 Um internally, there are others. 00:38:26.400 --> 00:38:52.639 So by having some standardization, um, you can uh I guess uh having standardization is is really helpful because it because you don't have to uh it doesn't you can actually choose whether hub and spoke works for you or point-to-point or workflow or or anything because you uh but the standardization makes it so that you don't have to pay the price of of integration 10 times. 00:38:52.800 --> 00:38:56.079 You're just okay, that's the advantage of this framework that I looked on. 00:38:56.159 --> 00:38:57.599 It's like the and it's still around. 00:38:57.679 --> 00:39:06.960 Like uh pretty with a few exceptions, like every AWS API call is is run through this framework, even though there is no common API layer. 00:39:07.039 --> 00:39:10.000 Like you still talk to different endpoints, but it's all the same framework. 00:39:10.159 --> 00:39:16.559 And so the SDK is all generated off of the same service distribution, service definition library, uh their language. 00:39:16.639 --> 00:39:24.159 It's called Smithy, it's actually open source now, um, that defines services and what shapes and APIs and signing methods for identity. 00:39:24.320 --> 00:39:30.400 Um, even though they aren't, we don't, as all of AWS have one API layer, this like Hub and Spoke. 00:39:30.480 --> 00:39:37.679 It's still we don't have to reinvent the wheel every time we make a new service or make a new client for that service. 00:39:38.159 --> 00:39:39.440 Yeah, that's very interesting. 00:39:39.599 --> 00:39:41.679 I hadn't considered that possibility. 00:39:41.840 --> 00:39:45.280 Um I'm trying to think of an analogy to understand it better. 00:39:45.440 --> 00:39:54.320 Um I I like to buy all the same brand of power tools so that I can take a battery out of one and stick it in another tool. 00:39:54.480 --> 00:39:59.920 Um I know your podcast is probably isn't sponsored by any one of them, but what is the what is your go-to? 00:40:00.639 --> 00:40:01.280 Skill. 00:40:01.760 --> 00:40:02.159 Skill? 00:40:02.320 --> 00:40:02.880 Okay, okay. 00:40:03.039 --> 00:40:08.079 I've I I kind of the sorting hat put me into the uh Mikita line at one point. 00:40:08.239 --> 00:40:11.119 Uh so that's to mix to mix some metaphors there. 00:40:11.199 --> 00:40:15.679 But yeah, well, when my wife and I got together, she already had some skill tools. 00:40:16.400 --> 00:40:22.719 And and she has an insistence, which I agree with, on having everything uniform. 00:40:23.519 --> 00:40:29.679 So I was like, well, skill is what we have, and I think their tools are pretty good, so that's what we're gonna go with. 00:40:29.920 --> 00:40:40.880 Um yeah, okay, so the the the takeaway for me there, a takeaway, is Hub and Spoke isn't the only answer. 00:40:41.039 --> 00:40:47.679 You don't need to have the hub as the intermediary in order to get some efficiency gains. 00:40:48.000 --> 00:41:03.039 Um you can save the cost of those uh unique uh uh point-to-point integrations by having something in common between them that allows them to talk to each other. 00:41:03.360 --> 00:41:04.079 Yeah, exactly. 00:41:04.159 --> 00:41:10.400 I think identity and uh and uh especially framework for for protocol help a lot. 00:41:10.480 --> 00:41:28.000 Um another kind of cool aspect of this was uh basically like make enough the the key with the framework was to make it uh so um is to just solve a lot of the problems that people would have with point-to-point in a standard way, like metrics and observability, for example. 00:41:28.159 -->

00:41:36.639 Um, if you are going to call some other service and you have some other framework for that client, you'd have to add your own instrumentation. 00:41:36.719 --> 00:41:47.360 Um, where just to say, okay, when I call a service, I need to record whether or not it succeeded, what kind of error it gave back, specifics about the error it gave back, how long the latency was. 00:41:47.519 --> 00:41:49.840 I need to bucket the latency intelligently. 00:41:50.000 --> 00:41:57.119 Like if I just measure the latency, I could see my latency get really good, really low, but actually all those requests are failing. 00:41:57.199 --> 00:41:59.679 So that I shouldn't be happy that the latency is better. 00:41:59.840 --> 00:42:05.840 It's like it, so I need to actually have the right dimensionality of the observability to say successful request latency. 00:42:05.920 --> 00:42:07.039 And what does success mean? 00:42:07.199 --> 00:42:12.159 Okay, well, 201, 200, like what HTTP stuff is happening under the hood. 00:42:12.320 --> 00:42:15.519 So having the right observability is actually pretty tough. 00:42:15.760 --> 00:42:17.679 Now, open telemetry exists now. 00:42:17.920 --> 00:42:22.000 That is a nice framework that that instruments with tons of framework. 00:42:22.079 --> 00:42:23.519 It's an observability framework. 00:42:23.679 --> 00:42:39.440 I think that's fantastic, and it's changed a lot for the for the industry around being able to get this just correct automatic instrumentation for any client server, any database, any whatever anybody has come up with um to have just like great instrumentation that way. 00:42:39.599 --> 00:42:49.440 But that was this was before that, and it it just kind of out of the box gave everybody instrumentation that was compatible with our observability tool we had internally, which is Cloud Watch. 00:42:49.679 --> 00:43:00.960 Um so it just like that's just making that framework just do a lot for you, but also be flexible enough so that you don't feel like suffocated by like a framework that's overbearing. 00:43:01.039 --> 00:43:02.320 Uh, that's the dance. 00:43:02.880 --> 00:43:07.280 Yeah, well, a framework can give you um leverage. 00:43:07.760 --> 00:43:14.480 Um okay, so here's here's something I've been preaching lately to people I work with. 00:43:14.639 --> 00:43:20.079 Um and it's it's taken me a while, but I've gotten the message across, at least to some people, I think. 00:43:20.239 --> 00:43:30.400 Um I talk about the concept of universality, and some tools, some things possess universality and some don't. 00:43:30.719 --> 00:43:33.440 Um, for example, uh Grafana. 00:43:33.760 --> 00:43:39.280 Uh it it lets you plug in data sources and it gives you visual graphs. 00:43:39.840 --> 00:43:43.199 It absolutely does not have universality. 00:43:43.599 --> 00:43:53.360 Um you can customize dashboards and stuff like that, but like no matter how you slice it, you're gonna end up with a dashboard of graphs. 00:43:54.000 --> 00:43:59.679 Um whereas if you build, for example, a Ruby on Rails app. 00:44:00.239 --> 00:44:06.000 Application or a JavaScript application, just something using code, it does have universality. 00:44:06.159 --> 00:44:09.199 Anything that can be built, you can build. 00:44:09.599 --> 00:44:15.760 And there's a trade-off there where you might have to do some things completely from scratch. 00:44:16.480 --> 00:44:29.920 And then there are tools which give you like both at the same time, which is it it gives you the leverage, but it doesn't pin you down so that you can only do what the tool lets you do. 00:44:30.079 --> 00:44:39.360 It gives you a base to build on top of rather than something that uh that that squeezes you in and limits you. 00:44:40.000 --> 00:45:17.280 I think that's such a cool and and actually uh this universality is such an like a applicable to like just to full circle this to to to agent uh stuff is is actually very cool because universality is is exactly why I'm excited about and why I joined from like from being on on like uh you know infrastructure service teams to being in working on agents because I see this problem that I've always been trying to chase around improving operations, like improving DevOps, like every every annoying part about building and operating software that's like somewhat tedious, but so important. 00:45:17.440 --> 00:45:20.320 I've been I've been trying to just solve that. 00:45:20.400 --> 00:45:28.880 And so because the universality of agents, um, I think we we've figured out can be applied in some pretty amazing ways here around that. 00:45:29.039 --> 00:45:32.000 Like you mentioned Grafana, like as or like observability tool. 00:45:32.079 --> 00:45:34.079 Like you can, it's pluggable to so many things. 00:45:34.159 --> 00:45:38.719 That's one of the things that's so great about it is pluggable, pluggability to so many data sources. 00:45:38.880 --> 00:45:45.119 Um, but when I'm trying to solve, like observability is a is a means to operations. 00:45:45.199 --> 00:45:46.239 It's a means, not an end. 00:45:46.320 --> 00:45:48.880 Like you don't do observability for observability's sake. 00:45:48.960 --> 00:45:52.000 You you do it so that you can operate um well. 00:45:52.159 --> 00:46:05.360 And and so we tackled that when the thing that I'm working on currently that I'm the most I'm just so excited about, like over the last 20 years, is is we've made an agent that does DevOps for you, the AWS DevOps agent. 00:46:05.599 --> 00:46:10.320 And it is um DevOps means a lot of things to a lot of people. 00:46:10.400 --> 00:46:15.039 Um to us, it means just what the DevOps team does, right? 00:46:17.039 --> 00:46:22.000 No, no, to us, like I thought I've done DevOps this entire time, like because it developers do the ops. 00:46:22.079 --> 00:46:25.039 There is no, yeah, exactly, no DevOps team. 00:46:25.280 --> 00:46:29.840 Uh and so um, and that's what what I I've always liked wearing

all the hats. 00:46:29.920 --> 00:46:32.480 Like I also want to pay attention to security. 00:46:32.559 --> 00:46:38.320 I want to pay attention to what customers are saying, uh, and and where and so I can make the product better and support them. 00:46:38.400 --> 00:46:40.639 Like I want to wear all of the hats and I love that. 00:46:40.800 --> 00:46:44.960 And so with DevOps, like that's that my that lifestyle of wearing all the hats. 00:46:45.039 --> 00:46:59.199 And yeah, and I just want to say, um in in case not everybody caught that that joke of ours, um, you know, people put those two words together, DevOps team, which is like the antithesis of of DevOps. 00:46:59.440 --> 00:47:09.519 DevOps is where you put the dev uh the development and the operations together, that idea, as I think of it, of like you build it, you run it. 00:47:09.920 --> 00:47:13.039 Um so that's why the term DevOps team is so funny. 00:47:13.360 --> 00:47:16.000 Yeah, I the nothing against specialization and everything. 00:47:16.079 --> 00:47:20.960 Like for people who do uh who are like are DevOps engineers and everything like that. 00:47:21.119 --> 00:47:24.559 I think that that specialization has been has been really powerful. 00:47:24.880 --> 00:47:34.800 Even even at Amazon, where you know we're we we just we're all software developers, there are some who are just a lot better at at automating ops. 00:47:34.960 --> 00:47:37.360 And that's just their kind of flavor and focus. 00:47:37.440 --> 00:47:44.239 And and like, and I think I'm one of those, like compared to whereas I'm not as good at other like maybe abstractions and frameworks and stuff like that. 00:47:44.639 --> 00:47:52.960 To me, the significant thing, obviously, individuals vary in their strengths and inclinations and and backgrounds and stuff like that. 00:47:53.119 --> 00:48:20.960 Um, to me, one of the significant things has to do with incentives, because if you have all the developers over here and then all the SRE people over there, um the the developers might have needs that the SRE people don't have any natural reason to care about, uh other than like hopefully out of the goodness of their heart, but that's not like why people do things at work. 00:48:21.199 --> 00:48:25.599 Um and and so that creates really uncomfortable situations. 00:48:25.920 --> 00:48:36.079 Whereas if you slice that up differently and put some devs and some SRE people in the same team, then the the interests are are more aligned. 00:48:36.400 --> 00:48:36.880 That's right. 00:48:36.960 --> 00:48:43.440 And there's this a natural like back pressure and reacting to pain, uh, you know, which is which is I think so important. 00:48:43.599 --> 00:48:50.239 So instead of like negotiating a contract and everything like that, which is you know, and there are a lot of philosophy, there are a lot of these that work. 00:48:50.320 --> 00:48:54.719 I've talked to so many customers of AWS over the years, like back and forth about, hey, what do you do? 00:48:54.800 --> 00:48:55.440 Oh, what do you do? 00:48:55.519 --> 00:49:01.599 And like, and we just kind of there are a lot of different places and models and everything that work for different different companies. 00:49:01.760 --> 00:49:13.920 I of course have my I'm you know, I'm very biased and and opinionated about DevOps being this what we're describing as DevOps, like to be the true Scotsman fallacy of like you know, defining well, you're not doing real DevOps, yeah. 00:49:14.239 --> 00:49:17.039 Okay, so but uh I'm a fan of this model. 00:49:17.119 --> 00:49:30.719 I think I I think it works with how I like to to work and and I think it results in really good outcomes for customers because the developer teams are just you know, ultimately ops is a and observability are a lens with which to understand the customer experience. 00:49:30.880 --> 00:49:32.960 And I uh and I really like that. 00:49:33.039 --> 00:49:35.920 I like being so connected to it to how how are our customers doing? 00:49:36.000 --> 00:49:38.079 Well, let's let's look at how the service is operating. 00:49:38.159 --> 00:49:38.800 Oh, yeah, okay. 00:49:38.960 --> 00:49:42.480 Like, well, I don't know how if customers are happy based on my observability data. 00:49:42.559 --> 00:49:45.360 It's like, well, then we're missing some observability data. 00:49:45.519 --> 00:49:47.760 Let's let's measure customer experience better. 00:49:48.159 --> 00:49:53.519 Yeah, and that's a whole podcast episode in itself, at least. 00:49:53.679 --> 00:50:02.079 It's probably many, many podcast episodes, is like um, how much of the story uh do metrics tell? 00:50:02.480 --> 00:50:05.039 Um it's of course not the whole thing. 00:50:05.199 --> 00:50:20.559 Um the I I had an experience where uh numbers were being cited for months and months, and then finally uh we were like, hey, let's like go and actually talk to some people. 00:50:20.880 --> 00:50:26.400 Um and it turns out we had totally the wrong idea because we were just looking at the numbers. 00:50:26.800 --> 00:50:34.719 Yeah, measure it from the right place, got to talk to people, constantly check your assumptions about whether that's one of the kind of cultural things that's just so important. 00:50:34.800 --> 00:50:41.920 And I think what helps with DevOps helps a lot is that you know it at AWS teams uh get together every week. 00:50:42.000 --> 00:50:45.519 Like, and actually all of AWS gets together every week. 00:50:45.760 --> 00:50:47.440 Every week, like for two hours. 00:50:47.679 --> 00:50:49.519 And we talk about ops. 00:50:49.599 --> 00:50:56.960 We talked about what's going on with of things things that are size of formula, like every what what were the wins? 00:50:57.039 --> 00:51:00.000 What did somebody do that they want to just like tell the world about? 00:51:00.079 --> 00:51:05.840 Hey, we just used this new tool or we made this new tool, we had this big efficiency gain in the service, or or

whatever. 00:51:06.000 --> 00:51:07.440 And what do you mean? 00:51:07.679 --> 00:51:13.440 What do you mean everybody in AWS gets together like on a call in person? 00:51:13.760 --> 00:51:19.599 Uh yeah, every uh we used to be in person, but every every team is represented by at least one person. 00:51:19.760 --> 00:51:23.760 Um, a lot of people are on uh like a live stream uh watching. 00:51:23.920 --> 00:51:34.159 Anybody can participate on like we joined these days, kind of we were kind of forced this way, uh, but I think for the better, um, with COVID and everything, we went we went virtual for this, and we've stayed that way. 00:51:34.320 --> 00:51:43.119 We find that it's easier for everybody to participate uh without the kind of intimidation of being in a large room where you're trying to yell from the back and you don't necessarily want to. 00:51:43.360 --> 00:51:50.159 So um we have a channel, a Slack channel where we're all talking and uh about it during the meeting. 00:51:50.239 --> 00:51:53.840 Uh the kind of the side conversations are very interesting and ongoing. 00:51:53.920 --> 00:52:15.599 But yeah, there are people from from every team, um, every single like not just every service, but every team within every service um who attends uh and and watches and uh and and discusses like what what what went well, like what maybe we'll talk about retrospectives of things that that didn't go well that we wanted to share with everybody and and share how we think. 00:52:15.679 --> 00:52:18.400 Here's how I think about things that didn't go well and how to improve. 00:52:18.639 --> 00:52:40.400 Uh like uh and then uh we'll the agenda varies, but sometimes we'll actually what we used to do, especially more was we would pull up a random services dashboard, like in with a metric, a metric dashboard, and say, well, and then we would we would ask each other, like there would be uh a line that would show, say, uh no, here's uh oh, here's an error graph. 00:52:40.480 --> 00:52:45.119 Uh but these are just the 400s, these are the four XX HTTP status code responses. 00:52:45.280 --> 00:52:48.639 So those aren't those aren't server faults, those are clients calling us wrong. 00:52:48.880 --> 00:52:55.199 And people would ask, well, okay, like if that spikes though, like somebody isn't happy, they're not successful. 00:52:55.280 --> 00:52:58.239 Like, is there some some sharp edge in the service that's causing that? 00:52:58.320 --> 00:52:59.280 And we'll discuss that. 00:52:59.440 --> 00:53:01.840 Oh, no, those are just people exceeding their rate limit. 00:53:02.000 --> 00:53:05.199 Well, okay, like are they happy that they exceeded their rate limit? 00:53:05.280 --> 00:53:07.679 Maybe you could just increase their rate limit for them. 00:53:07.760 --> 00:53:18.480 And then we're we have good discussions about what like what limit, you know, about limits around input validation and everything about and that really do get to the question of, okay, is

is this the right customer experience? 00:53:18.559 --> 00:53:20.719 And we kind of challenge each other on that. 00:53:21.280 --> 00:53:22.800 Yeah, yeah. 00:53:23.039 --> 00:53:34.800 I so I I like can't help but think in analogies and something okay, so I've I've been like re-studying calculus lately, and I found this really nice video. 00:53:35.280 --> 00:53:38.639 Um there's this YouTube channel that's just called Math and Science. 00:53:38.800 --> 00:53:40.800 It seems to be this one guy. 00:53:40.960 --> 00:53:45.840 Um, and he did a really nice job of explaining what calculus was all about. 00:53:46.000 --> 00:53:56.639 And he used what I guess is a fairly common example of like here is a graph of um a particle's uh position over time. 00:53:56.960 --> 00:54:09.039 And then if you take the derivative of that, you get the um let's see, there's there's a position, velocity, and I don't remember what. 00:54:09.199 --> 00:54:09.440 Sorry? 00:54:11.920 --> 00:54:13.199 Exactly, yeah. 00:54:13.519 --> 00:54:19.599 And and so I like to take these things and take like the the derivative of each one. 00:54:19.760 --> 00:54:28.159 It's like okay, we have a culture of culture of firefighting and and we need to turn that around. 00:54:28.239 --> 00:54:37.440 We need to like be more preap proactive and have better monitoring and alerting so that we don't have that reactive culture of firefighting. 00:54:37.599 --> 00:54:42.480 It's like, hang on a second, why do we have a culture of firefighting? 00:54:43.119 --> 00:54:48.159 One of my favorite quotes is things are the way they are because they got that way. 00:54:48.559 --> 00:54:54.000 And so, like, if this is the way, like, why are we just looking at this now? 00:54:54.079 --> 00:54:57.760 Like, why didn't somebody look at this five years ago and straighten this out five years ago? 00:54:57.840 --> 00:55:00.320 Like, why now and not back then? 00:55:00.719 --> 00:55:04.079 Unfortunately, it's not very common to dig into that. 00:55:04.800 --> 00:55:04.960 Yeah. 00:55:05.360 --> 00:55:08.000 You're describing kind of one of these ops meetings pretty well. 00:55:08.159 --> 00:55:16.800 Like they this gets second and second derivative pretty fast of like, okay, it's like, and people say, oh, okay, why there's this, you know, something happened in a service. 00:55:16.880 --> 00:55:19.039 Oh, like well, why does that happen in services? 00:55:19.119 --> 00:55:20.960 Why do like why isn't this just out of the box? 00:55:21.119 --> 00:55:25.920 So oh yeah, we didn't have this uh alarm on on like file descriptors or something. 00:55:26.079 --> 00:55:29.280 Oh, okay, well, why doesn't why aren't those already always the case? 00:55:29.360 --> 00:55:40.239 And so we do talk about that whole that whole like shift left aspect of it of how do we just get everybody to be better, which which like back to that universality point you're talking about. 00:55:40.400 --> 00:55:49.280 Like this is like agents are really that you go, there's a lot of the culture you cannot bypass the culture around of Roundups about making things better. 00:55:49.440 --> 00:55:50.800 There is no magic wand. 00:55:50.880 --> 00:55:53.360 There's no compression algorithm for experience. 00:55:53.679 --> 00:56:02.079 But gosh, agents are pretty, pretty universally adaptable in both root-causing issues and plugging into and understanding any system. 00:56:02.239 --> 00:56:07.440 Doesn't matter what framework it has, doesn't matter what observability provider it has, there's an MCP server for it. 00:56:09.039 --> 00:56:11.840 It'll figure out how to get the right metric out that it's looking for. 00:56:12.000 --> 00:56:27.280 Um, for for the shift left aspect of it, scan through an application, look at all the observability, look at all the incidents, look at the code for it, match it against things that we know are are like not good, things that opportunities to improve, and just do those. 00:56:27.519 --> 00:56:37.920 So this like the both the React and also the improve, the proactive part are what we've built into DevOps Agent uh last sales pitch on DevOps Agent. 00:56:38.000 --> 00:56:51.920 But it it's like that universality and like being able to adapt to to to really any learn an architecture, learn an application just by looking at everything it can tell from it, uh the whatever tools you throw at it, whatever MCP servers. 00:56:52.000 --> 00:56:55.840 Um it's just it's working better than I expected almost. 00:56:56.000 --> 00:57:00.719 Like it's it's pretty incredible the universality of uh of agents. 00:57:01.119 --> 00:57:10.639 By the way, this is by far the fastest uptake I've gotten on on the term universality and having it make its way into the conversation. 00:57:10.800 --> 00:57:18.480 Um yeah, I got it from this book, um, The Beginning of Infinity by David Deutsch, or maybe it was The Fabric of Reality by David Deutsch. 00:57:18.559 --> 00:57:26.239 Both those books are are excellent and they've they've changed the way that I think about programming and just like the life in general. 00:57:26.400 --> 00:57:30.960 Um life-changing books, especially the beginning of infinity. 00:57:31.360 --> 00:57:41.920 Um where we're about at time, um there's there's one more thing that I have to ask, but let me know if you get to where where you need to have a hard stop. 00:57:42.239 --> 00:57:45.360 Um I want to ask about postmortems. 00:57:45.599 --> 00:58:07.440 Um something that really frustrates me in po postmortems is when a large number of people are brought together for a short amount of time, and the the incident is analyzed maybe superficially, and then out of the post mortem come uh several action items. 00:58:07.840 --> 00:58:14.079 It's like here here are the seven things we have to implement immediately to make sure this never happens again. 00:58:14.639 --> 00:58:18.400 And and I'm like, uh, is is this really the the way to do it? 00:58:18.559 -->

00:58:25.440 I think maybe we should have fewer people for a much longer time instead of like an a one-hour postmortem. 00:58:25.840 --> 00:58:38.000 You know, some postmortems maybe maybe they call for 30 minutes, some maybe they call for like three days of of picking it apart because different things call for different levels of thought and stuff like that. 00:58:38.159 --> 00:58:40.800 I don't know to you what what makes a good postmortem? 00:58:41.519 --> 00:58:46.480 I one of my favorite things to do is to is to write a postmortem doc. 00:58:46.719 --> 00:58:49.039 We call them COE correction of error. 00:58:49.199 --> 00:58:50.320 You can take any name. 00:58:50.480 --> 00:58:55.119 Um, but we have a pretty pretty standard formula for these, but I love writing them. 00:58:55.199 --> 00:58:58.239 They're a document um that a team will write. 00:58:58.400 --> 00:59:12.159 Like a generally like a maybe one or or two people will spend a lot of time with the pen, but then that they kind of just can keep riffing on it until with the like as a team and as then a larger team if it again, if it needs more scrutiny, uh talk about it. 00:59:12.239 --> 00:59:16.559 But first, what makes up a one of these COEs, we call it. 00:59:16.639 --> 00:59:21.679 Um, it's a description that we we write them for anybody in the company to be able to read. 00:59:21.760 --> 00:59:36.000 Like to to and and so we want to make sure we're not using too much jargon about oh, the the uh the BSF service called the you know, MVP service called that we do, you know, we don't want to use too many acronyms, or we certainly will put a like a description of it in it. 00:59:36.079 --> 00:59:44.000 But we say, here's what it is, here's what happened, here's the customer impact, like pre- uh here's the timeline of like what break it down really granular. 00:59:44.159 --> 00:59:52.559 The timeline includes uh when we learned about something, when an alarm went off, when the impact happens, when we took any action, every when everything. 00:59:52.719 --> 00:59:57.199 Um, and then the meat of it is the the whys, the five whys. 00:59:57.440 --> 01:00:03.360 My understanding is this is a Toyota method for for kind of looking back at when something happens. 01:00:03.679 --> 01:00:06.400 Say, well, uh service fails. 01:00:06.639 --> 01:00:07.440 Why did it fail? 01:00:07.679 --> 01:00:08.880 Uh it ran out of something. 01:00:08.960 --> 01:00:10.400 Well, why did it run out of something? 01:00:10.480 --> 01:00:13.039 Uh well, it ran out of something because we didn't have an alarm. 01:00:13.119 --> 01:00:13.920 Why didn't we have an alarm? 01:00:14.159 --> 01:00:15.360 So you just keep asking why. 01:00:15.519 --> 01:00:16.480 You don't ask five. 01:00:16.639 --> 01:00:20.320 And and these answers, it's not just a one-sentence response to the why. 01:00:20.480 --> 01:00:25.920 Like, that's where I put all the thinking of like, well, okay, gosh, well, how do I really keep this from happening

again? 01:00:26.000 --> 01:00:44.079 And then invariably you think, like, well, how would anybody else like I have people have that kind of, I think as engineers, you really maybe something just built in as being a programmer, but culturally, of you really want to help other people, like other teams, other people not have not face that same thing. 01:00:44.239 --> 01:00:49.039 That's why people really like writing frameworks, I think, is because you're like, oh, let's help other people do that thing. 01:00:49.199 --> 01:00:57.679 Uh, similarly with this, like, how do how would I make it so that uh that this like that everybody has an alarm on high CPU? 01:00:58.159 --> 01:00:58.800 Exactly this way. 01:00:58.960 --> 01:01:14.400 It's like I had an alarm, but I've alarmed on the median host CPU instead of really, it should always be the well, you should have that, but you should also have the the max, the P100, hundredth percentile host CPU to see like because maybe you're because it's really a distribution across your fleet. 01:01:14.480 --> 01:01:16.239 So all these little nuanced things. 01:01:16.320 --> 01:01:27.360 How do I make it so that and who do I talk to to who might have some some leverage, some tool that could be uh could apply this to everyone kind of out of the box? 01:01:27.519 --> 01:01:29.280 What is the abstraction that we could have? 01:01:29.440 --> 01:01:38.400 And just I love just unpacking and writing one of these documents that um where you're just digging into the data for a long time. 01:01:38.480 --> 01:01:40.719 Like this isn't a is a short thing. 01:01:40.880 --> 01:02:05.519 Um and uh it's a team, like I said, so in terms of review, then um some of these are like it they're always be reviewed as as a team, like uh whether that team's like 10 people or something, but then and also as a as a larger like a group of people, uh you know, maybe a larger team, maybe you bring in outside, help to be, oh, I wonder what this person thinks about how I got here. 01:02:05.679 --> 01:02:07.920 And so I'll just invite other people to review it. 01:02:08.000 --> 01:02:12.480 And and some of them, like this is part of that that Wednesday ops meeting I was talking about. 01:02:12.719 --> 01:02:17.280 Some of them uh we'll actually review every in a given week for for a while. 01:02:17.360 --> 01:02:23.679 Um, and and as a and talk about it and talk about our own the patterns that we've noticed over time. 01:02:23.760 --> 01:02:26.320 People will weigh in with, oh, I've seen this happen a bunch. 01:02:26.480 --> 01:02:31.760 And and uh and so like what do we do to what do we do to keep this from happening for any service? 01:02:31.920 --> 01:02:33.360 Um we also are analyzing. 01:02:33.760 --> 01:02:34.800 I don't know, David. 01:02:35.039 --> 01:02:37.440 This all sounds like it takes a lot of time. 01:02:37.599 --> 01:02:39.760 That doesn't sound very efficient. 01:02:40.239 --> 01:02:41.199 Oh, I don't know. 01:02:42.079 --> 01:02:43.760 I I'm I'm speaking as a character right now. 01:02:44.639 --> 01:02:45.199 Of course, yeah. 01:02:45.280 --> 01:02:47.360 Oh, there's no compression algorithm for experience. 01:02:47.440 --> 01:02:55.760 Like the thing is with like, and this isn't you know, maybe it there is just a a there are different businesses with different incentives. 01:02:55.840 --> 01:03:00.880 Like I like that ultimately that is totally totally the case, different different requirements. 01:03:01.039 --> 01:03:08.320 Um, I mean at AWS, like everybody is counting on us to be like we people to be perfect, right? 01:03:08.400 --> 01:03:13.039 So we we have to that that is the product, like is that we are doing ops so you don't have to. 01:03:13.360 --> 01:03:16.239 Like over the last 20 years, like that's like that's what we are doing. 01:03:16.400 --> 01:03:19.280 We are we are your ops team in a way. 01:03:19.440 --> 01:03:29.599 Uh not not not your application ops team, obviously, but like for the infrastructure, you're relying on so we spend an enormous amount of effort on this, obsessing over this. 01:03:29.760 --> 01:03:35.039 Um and uh yeah, I mean, we even have I didn't even describe everything. 01:03:35.199 --> 01:03:37.199 Like there are also teams. 01:03:37.360 --> 01:03:53.599 If you are a team, when I was on that framework team, for example, or when I was on the observability team, I would look at every one of these uh COEs, like we have it's a database of them, I can mine them and you know learn, like actually with LLMs, now it's even easier for me to mine them for for like patterns. 01:03:53.679 --> 01:04:03.679 And I'm like, well, if I own CloudWatch, that's the service I'm building, how could I make that just so much more out of the box and easier so that everybody has the alarm that they need all the time? 01:04:03.840 --> 01:04:06.800 Like, what do I need to build into my service? 01:04:06.960 --> 01:04:14.159 Like, how can I improve my service by seeing how how people maybe struggle with it or when some things don't always go exactly right? 01:04:14.320 --> 01:04:16.559 Um, every team I was on, it was like that. 01:04:16.719 --> 01:04:23.360 Okay, like, you know, when I was on on Dynamo DB, people would say, well, yeah, I had an outage because I like hit a throttling on my table. 01:04:23.440 --> 01:04:26.800 It's like, okay, how do I how do we add auto-scaling to DynamoDB? 01:04:26.880 --> 01:04:34.239 Or like make it so we we started DynamoDB with this notion of provisioned throughput because people wanted super predictable performance. 01:04:34.400 --> 01:04:46.400 So we would pre-provision capacity and like you could because it was elastic, you could call an API to add more or take away database capacity, and we would just do that behind the scenes, but um, we didn't have auto-scaling at first. 01:04:46.480 --> 01:04:53.760 And so when we saw people run into uh, oh yeah, like it's it's hard to actually monitor this all the time and

then dial it up. 01:04:53.840 --> 01:04:58.480 So let's just build that into the product, this like real-time elasticity into it. 01:04:58.719 --> 01:05:05.840 So, like by understanding where things go wrong for your customers, you you make your product better. 01:05:06.000 --> 01:05:13.519 Uh whether it's whether it's going wrong because the service has a rough edge, or whether you know we need to just be better ourselves. 01:05:13.679 --> 01:05:16.159 Uh, and both both cases apply. 01:05:16.320 --> 01:05:39.599 So these these these root uh retrospectives, these uh COEs, they we get so much out of this process uh because it is, it takes a lot of time, but we we we really get a lot out of it from different angles, from the team itself, from other teams not running into the same issue, to just making the underlying services that we build for all of you better by looking at how we use it ourselves. 01:05:40.000 --> 01:05:46.719 And I imagine that the patience and rigor pay themselves back many times over. 01:05:47.519 --> 01:05:48.079 I hope so. 01:05:48.239 --> 01:05:55.840 I mean, uh yeah, I hope I hope that yeah, I mean, I hope that people, you know, I hope that we are having the operational outcomes that our customers need. 01:05:56.000 --> 01:05:57.119 That's all that matters to me. 01:05:57.199 --> 01:06:04.079 So it I uh we're we're always looking And and wishing we could do more, spend more time on it, uh it'd be better. 01:06:04.239 --> 01:06:10.400 So uh always looking for that next next tool to give our leverage uh that will make things better. 01:06:10.559 --> 01:06:17.119 So uh always looking to to to figure out how to how to keep keep that bar going up. 01:06:17.360 --> 01:06:17.840 Mm-hmm. 01:06:18.239 --> 01:06:25.679 Well well David, I have about a million more questions for you, um, but unfortunately we don't have unlimited time. 01:06:26.159 --> 01:06:40.639 Um I have to say that this has been this this episode has had among the highest uh nuggets of wisdom per minute uh uh of of any episode I've done. 01:06:40.800 --> 01:06:46.000 So I really I personally have gotten a lot out of this. 01:06:46.559 --> 01:06:49.119 And that's partly because different directions. 01:06:49.199 --> 01:06:51.840 It's been it's been a lot of fun just visiting different things. 01:06:52.239 --> 01:06:53.039 Yeah, yeah. 01:06:53.280 --> 01:06:55.599 Um so yeah, I really enjoyed it. 01:06:55.760 --> 01:07:02.559 Uh before we sign off, anything that you want to share, links you want to send people to, that kind of stuff. 01:07:03.360 --> 01:07:04.320 Uh oh sure. 01:07:04.400 --> 01:07:05.440 I mean, certainly. 01:07:05.519 --> 01:07:08.079 Um I guess okay, two two things. 01:07:08.239 --> 01:07:27.360 Of course, AWS DevOps agent, uh, you know, that that's a uh I think it's I I think it's gonna change a lot in terms of where as you code more and ship more so much faster with agents, the the the the bottle the code just piles up unless you can get it shipped and run it. 01:07:27.440 --> 01:07:28.800 So I think that's very important. 01:07:29.039 --> 01:07:39.440 The second thing, um, when we talk about what we've learned as AWS over the last 20 years, um one thing that we did a few years ago kicked off. 01:07:39.519 --> 01:07:46.480 Um we we were trying to help like just capture this like to for ourselves of like, well, what have we learned? 01:07:46.559 --> 01:07:48.960 Like how what is the right way to build a service and operate it? 01:07:49.039 --> 01:07:51.440 So it was like a like we we were just doing that for ourselves. 01:07:51.519 --> 01:07:58.400 We had been a lot of um talks over the years that we give every week that that so how would how do we kind of distill that? 01:07:58.480 --> 01:08:02.000 So we wrote um we launched this thing called the Amazon Builders Library. 01:08:02.159 --> 01:08:05.760 We wrote these really long articles, honestly, like they got a little longer. 01:08:05.840 --> 01:08:10.320 It's like, oh, just write down write down what does it mean to do instrumentation of a service. 01:08:10.400 --> 01:08:14.480 It was like, oh, wait a minute, that's gonna take me like 12 pages to to touch on, right? 01:08:14.639 --> 01:08:25.760 And so we wrote we published this Amazon Builders Library, um, uh AWS.amazon.com slash it yeah, AWS.amazon, AWS.com or slash builders library, just just Google it, I guess. 01:08:25.840 --> 01:08:27.760 Builders Amazon Builders Library. 01:08:28.000 --> 01:08:35.920 And it uh it has a lot of articles in depth about everything from fairness in multi-tenant systems to how we do CI C D. 01:08:36.000 --> 01:08:37.520 Um, a lot of ways to go about that. 01:08:37.680 --> 01:08:50.720 I think we have three articles about CI C D to uh to shuffle sharding of like how to make multi-tenant systems at large scale be um uh seem like single tenant systems. 01:08:50.800 --> 01:08:53.680 Uh so yeah, I would I would recommend that folks check that out. 01:08:53.760 --> 01:09:02.640 It's uh uh something we've put a lot uh of in into, but uh you know there's all we're always learning something new, so we will add articles from time to time. 01:09:03.039 --> 01:09:03.439 Wow. 01:09:03.600 --> 01:09:07.359 Okay, that is gonna be a gold mine for me and my team. 01:09:07.600 --> 01:09:10.079 So I already Googled that on my phone just now. 01:09:10.159 --> 01:09:12.399 I'm gonna be digging into that later. 01:09:12.560 --> 01:09:15.920 Um Yeah, I I certainly will. 01:09:16.159 --> 01:09:19.520 Um again, this has been extremely educational. 01:09:19.680 --> 01:09:21.840 And David, thanks so much for coming on the show. 01:09:22.159 --> 01:09:22.720 Thanks a lot. 01:09:22.800 --> 01:09:33.039 Uh oh yeah, and I guess last link is uh yeah, I'm on Twitter, LinkedIn, etc., uh the Barrett Blue Sky, whatever social network is in the fragmented current world. 01:09:33.119 --> 01:09:34.960 Find me on there, David Janichek. 01:09:35.039 --> 01:09:37.680 Uh, and uh happy to chat with anybody anytime. 01:09:37.760 --> 01:09:39.840 So uh but yeah, thank you so much for having me on. 01:09:39.920 --> 01:09:42.000 It was this was a really fun conversation. 01:09:42.399 --> 01:09:43.359 Thank you.