บทถอดความฉบับเต็มของ Automating Application Lifecycles For Developer Happiness At Wayfair

repository that housed a cookie cutter 30 00:04:49.669 --> 00:04:52.810 template, and folks interested in creating their own decoupled application 31 00:04:53.430 --> 00:05:04.145 could clone that repo and run the basic cookie cutter command to template out some code. And that was all that it did. So it kind of gave you the code itself, but left it kind of at that point. 32 00:05:04.525 --> 00:05:09.990 When I spent about a year embedding with 4 different teams to kind of help knowledge transfer around 33 00:05:10.290 --> 00:05:17.270 setting up, you know, containers, the application, Python best practices, testing, linting, we got a lot of great feedback on our tooling. 34 00:05:17.574 --> 00:05:29.440 But 1 thing that stood out is it took even senior engineers with a lot of domain expertise weeks to get something set up. You know, you had to know exactly who to contact, how to click the right buttons, 35 00:05:29.740 --> 00:05:34.720 what ticket to file, and exactly how to fill it out. And we saw that as, like, a pretty big problem. 36 00:05:35.935 --> 00:05:55.645 So kind of the approach we took is we started by counting the manual steps that it actually took to get a very, very basic hello world rest API in our production environment. And to our surprise, it took about, like, 60 distinct steps, which was, like, pretty awful. Again, like, really experienced senior engineers would take weeks to get this set up. So 37 00:05:55.945 --> 00:05:59.164 without really having a focus on the product at first, 38 00:05:59.465 --> 00:06:08.760 we just had a focus on reducing the amount of manual steps. We knew that no matter what we did, we wanted to kind of drive that figure down. So I think I remember 39 00:06:09.074 --> 00:06:22.880 years ago, 1 of my kind of quarterly objectives was, can we take this number of 60 and reduce it to 40? So we kind of got started thinking about that. We ultimately released a product, which we call Mamba. It's on the theme of snakes, which is really cool. 40 00:06:23.180 --> 00:06:28.415 And, initially, the application life cycle team, which at the time was incubating within PyFlats, 41 00:06:28.875 --> 00:06:39.770 supported 1 main user story, which was, you know, as an engineer, I would like to create a production ready application in 10 minutes so that I can have a reliable and standardized application setup that follows best practices. 42 00:06:40.310 --> 00:06:47.944 That team ultimately grew out of Python platforms. We created our own team with our own scope. At this point, that was about 1 and a half years ago. So 43 00:06:48.245 --> 00:06:53.065 summer 2020 at that time. Given the time since the 44 00:06:53.639 --> 00:07:00.139 initial development of the application life cycle team to where we are now, how has the scope and responsibilities 45 00:07:01.080 --> 00:07:07.635 changed or evolved and some of the ways that the team is integrated into the rest of the organization 46 00:07:08.255 --> 00:07:10.835 started

to formulate and become more 47 00:07:11.295 --> 00:07:11.795 standardized? 48 00:07:12.530 --> 00:07:25.495 So when the team was created, we had the 1 solution to manage the application creation process, but we decided to carve out somewhat of an ambitious scope. And we decided that instead of just focusing on application creation, 49 00:07:25.875 --> 00:07:31.770 it made sense to deal with the entire life cycle of kind of abstractly managing a decoupled application. 50 00:07:32.470 --> 00:07:42.975 So not only did we want to support the creation of these apps, but we also recognized that even maintaining these apps and keeping them up to date and keeping them running in our production environment 51 00:07:43.330 --> 00:07:49.270 involved a lot of toil for our developers, so we wanted to support that as well. Also, we care about application decommissioning. 52 00:07:49.730 --> 00:08:02.620 1 thing we wanted to avoid was having a bunch of things running in our production environment that didn't really have anything referencing it and weren't actually providing production value. So we felt that it was necessary to also spin down applications at the end of their life. 53 00:08:03.080 --> 00:08:04.540 As far as the 54 00:08:05.240 --> 00:08:06.060 sort of formalization 55 00:08:06.520 --> 00:08:16.764 of that life cycle where a lot of times you say, at the outset, I have the need to be able to build the service or product, but there's usually not a lot of thought that goes into 56 00:08:17.240 --> 00:08:21.979 the sunsetting aspect of a project. It may just stay alive indefinitely 57 00:08:22.360 --> 00:08:27.884 and start to become kind of Frankensteined with unrelated functionality just because it's already there. 58 00:08:28.264 --> 00:08:30.444 And I'm wondering what your responsibility 59 00:08:30.745 --> 00:08:33.084 is as far as being able to help maybe 60 00:08:33.390 --> 00:08:36.210 combat that kind of organic growth and helping to 61 00:08:36.670 --> 00:08:38.130 create that overall 62 00:08:38.430 --> 00:08:40.290 life cycle plan at the outset 63 00:08:40.885 --> 00:08:44.585 and just some of the ways that that is becoming 64 00:08:45.285 --> 00:08:47.785 standardized in the Wayfair organizational 65 00:08:48.485 --> 00:08:48.985 culture? 66 00:08:49.580 --> 00:08:57.760 I think of the 3 parts of our scope, decommissioning is where we have invested the least amount of effort. So right now, we have a self-service workflow 67 00:08:58.214 --> 00:09:00.795 where you can opt in to have your application decommissioned. 68 00:09:01.255 --> 00:09:07.035 We'll do some very, very basic stuff, you know, renaming your GitHub repo, spinning down your build pipelines, etcetera. 69 00:09:07.380 --> 00:09:10.440 The idea is you should be able to generate and decommission 70 00:09:10.899 --> 00:09:11.800 apps indefinitely 71 00:09:12.180 --> 00:09:14.120 using the same name and the same terminology. 72 00:09:14.885 --> 00:09:17.305 Though, we aren't

doing anything super intelligent 73 00:09:17.685 --> 00:09:23.305 on the side of, you know, proactively identifying applications that would be candidates for decommissioning. 74 00:09:23.960 --> 00:09:39.715 Though 1 project that we're working on now is kind of getting that insight into the various state of the applications that we have at Wayfair. So we think that insight will help us, you know, focus attention to things that require it. Given the fact that your 75 00:09:40.415 --> 00:09:41.875 mission is to 76 00:09:42.470 --> 00:09:48.089 be kind of an enabling team to the rest of the engineering group, I'm curious how you think about 77 00:09:48.709 --> 00:09:49.209 identifying 78 00:09:49.510 --> 00:09:53.675 what are the areas of greatest leverage that you can focus your efforts in 79 00:09:54.055 --> 00:09:54.555 identifying 80 00:09:55.175 --> 00:09:58.960 and scoping the types of projects that you want to work on, whether that's 81 00:09:59.440 --> 00:10:02.580 from external inputs from other engineering teams or visibility 82 00:10:02.960 --> 00:10:10.454 of the work that's in flight and maybe internal insights that you're building from working with the various stakeholders across the organization? 83 00:10:11.154 --> 00:10:35.945 Cool. Yeah. So I think 1 of the fundamental approaches we take is we're willing to make assumptions, but we're also willing to be wrong. So anytime we want to build a new solution or kind of experiment, we like to validate our assumptions with our customers. So we'll kind of take baby steps initially and make sure that what we set out to build is indeed being used and provides value. Part of this is knowing customer pain points. So 84 00:10:36.570 --> 00:10:49.834 the approach I described earlier where I went and embedded with different teams, that's something that I think has really helped us historically and currently for sure. Another thing that is interesting about the way we work is we have a concept called free day Fridays, 85 00:10:50.295 --> 00:10:52.954 which is just about what it sounds like, where 86 00:10:53.400 --> 00:10:56.300 on Fridays, engineers are encouraged to 87 00:10:56.680 --> 00:11:00.060 spend time solving problems that they really wouldn't get to solve 88 00:11:00.360 --> 00:11:21.435 in their, you know, otherwise day to day work, the work that's, you know, fulfilling objectives and key results. That's where our innovation happens. You know, we're comfortable taking risks and using that Friday time to do some things that are a little bit out of the box even if they're thrown away. These solutions that I wanna talk to you about today all came from that 3 day Friday time where we got to experiment. 89 00:11:21.735 --> 00:11:35.100 People ended up liking kind of the the stuff that we were building, and we just took it from there. So we never once said, we're gonna build and release this new product, and it's gonna be great. We were always, you know, solving a problem or kind of, like,

building off of innovation 90 00:11:35.400 --> 00:11:41.285 that we had kind of landed on. Before we dig into the specific projects, I'm also interested in understanding 91 00:11:41.905 --> 00:11:43.685 how you approach the 92 00:11:44.225 --> 00:11:51.960 kind of marketing of the capabilities that you're providing to the rest of the organization because Wayfair is a fairly large company. 93 00:11:52.339 --> 00:11:59.405 My understanding is that there are a number of different physical locations, and I'm sure that over the past couple of years, a lot of your work has been done remotely. And so 94 00:11:59.785 --> 00:12:15.870 just the overall challenge of raising awareness of the work that you're doing so that other teams can take advantage of it and some of the ways that you measure the overall impact that you're having on the kind of developer velocity of the engineering group at Wayfair? 95 00:12:16.425 --> 00:12:32.870 Definitely. Yeah. Wayfair is a pretty big place. There's about 3, 000 plus engineers. So we do some level of broad marketing. You know, we'll send out what we call release notes when something is ready to be announced publicly, but release notes certainly aren't the main contributor to adoption. 96 00:12:33.235 --> 00:12:34.775 1 of the things we focus on 97 00:12:35.155 --> 00:13:04.000 is selling our products and our solutions to our, early adopters. So we really love working with people that love working with us. So the people that are giving us positive feedback and helping make our product better, we really enjoy working with them. And those people end up being our promoters. So we try to find, you know, the interesting problems that need to be solved and kinda do our homework to vet that these are indeed problems that need to be solved. But we'll we just try to build a strong product and close collaboration with these early adopters. 98 00:13:04.380 --> 00:13:15.855 And, you know, by the time we have a mature product, it's already being marketed for us. So 1 thing about our work is that we don't force people to use our products. 1 of the ways we measure success 99 00:13:16.450 --> 00:14:08.279 is by adoption of our tooling. So like I mentioned, we'll market things to the early adopter category and from there, see kind of where things land. So if we see that traffic to a product or net usage is increasing organically, that's usually a good sign. Early on, we cared about looking at, you know, just the net number of microservices being spun up because we are heavily involved in the application creation space. We kind of viewed more is better, which is kind of narrow minded. But, like, I think when we're spinning up that new product and decoupling, that was the way to go. A lot of what we do also has to do with automated pull requests. Like, a lot of our work has to do with that. So we will just measure the wrong number of automated pull requests being created and merged. Again, you know, more automated pull requests being

merged isn't necessarily a good thing. So all of our work, we do try to tie to what are called the DORA metrics. 100 00:14:08.740 --> 00:14:19.215 Those are 4 distinct measurements that, Wayfair instruments and other, you know, enterprises do as well, and they have to do a throughput. So deployment frequency and lead time for changes 101 00:14:19.675 --> 00:14:23.615 as well as stability, which is change failure rate and mean time to recovery. 102 00:14:24.140 --> 00:14:33.305 So the idea is, like, if virtual link can help deploy frequency increase and lead time for changes go down, I think we're doing our job well. Given the fact that the 103 00:14:34.005 --> 00:14:34.505 initial 104 00:14:35.045 --> 00:14:35.545 core 105 00:14:36.085 --> 00:14:47.579 software capability of Wayfair is built around this PHP monolith. I'm curious if you can talk to the role that Python has in the work that you're doing and some of the ways that you think about the 106 00:14:48.425 --> 00:14:55.965 selection of language and ecosystem and tooling for being able to provide to other members of the engineering org? 107 00:14:56.330 --> 00:15:07.084 So within our team, we prefer to use Python for a couple of reasons. 1 of the main reasons was just based on history. We do have that existing skill set from our incubation phase within Python platforms. 108 00:15:07.464 --> 00:15:14.365 But, also, we do think it's the right tool for the job that we're doing. We run a lot of lightweight automation. We're hitting a lot of APIs. 109 00:15:14.890 --> 00:15:17.390 We define a lot of user facing specifications 110 00:15:17.850 --> 00:15:27.795 in order to integrate with our tools. So we just find Python to be really, really easy to use for what we're trying to do. An example is something like Pydantic has been fantastic 111 00:15:28.255 --> 00:15:31.714 for creating descriptive human and machine readable specifications. 112 00:15:32.095 --> 00:15:35.790 So a lot of the way we interface from a technical standpoint, 113 00:15:36.810 --> 00:15:39.470 with our customers is through YAML specifications, 114 00:15:40.175 --> 00:15:45.635 and I can talk more about that later. But Pydantic has made it really, really easy to version these specifications 115 00:15:46.175 --> 00:15:48.035 and enforce, you know, different validations. 116 00:15:48.560 --> 00:15:54.260 That's stuff that, sure, would be possible in other programming languages, but we just find it really, really easy to do with Python. 117 00:15:54.800 --> 00:15:56.260 In terms of the actual 118 00:15:56.745 --> 00:16:00.685 specific tools and work that you're doing, 1 of the things that you mentioned is 119 00:16:01.225 --> 00:16:11.190 simplifying the process of being able to spin up a new application and get it into production. I'm wondering if you can talk to the way that you have built that system and 120 00:16:11.595 --> 00:16:12.335 some of 121 00:16:12.715 --> 00:16:19.375 the approach that you've taken to be able to

reduce those manual steps and automate as much as possible while still being 122 00:16:19.740 --> 00:16:22.400 understandable and maintainable for the consumers? 123 00:16:23.180 --> 00:16:30.695 So 1 thing that's really neat about this is although the system itself is written in Python, it can template out code for any language. So 124 00:16:31.075 --> 00:16:33.335 Wayfair supports Python, PHP, 125 00:16:33.795 --> 00:16:46.209 Java, dot net, JavaScript, some Go. So there's just a lot of variety out there, and we don't see ourselves as being opinionated at all as far as, like, what the right tooling choices, and we want to enable 126 00:16:46.575 --> 00:16:49.715 all sorts of solutions to be built. So our system 127 00:16:50.095 --> 00:16:50.595 generally 128 00:16:50.975 --> 00:16:54.675 is 2 phases. It will both template out code according to templates 129 00:16:55.190 --> 00:16:58.330 and run automation to onboard an application to production. 130 00:16:58.870 --> 00:17:01.690 So that's things like creating a GitHub repository, 131 00:17:02.470 --> 00:17:03.690 creating a build pipeline, 132 00:17:04.015 --> 00:17:15.620 calling out to various APIs to signal that a new app is present, stuff like that. There's a lot of, like, some proprietary stuff, but just things that developers would have to go through in order to, you know, stand their thing up. So 133 00:17:15.920 --> 00:17:20.500 any team interested in integrating with our solution will create a GitHub repository containing 134 00:17:20.880 --> 00:17:22.660 1 to many cookie cutter templates. 135 00:17:22.975 --> 00:17:31.715 So it is using cookie cutter under the hood. 1 of the cool things is on top of that, they will provide a specification for what questions to ask users. 136 00:17:32.140 --> 00:17:34.320 So with cookie cutter, generally, you'll provide 137 00:17:34.700 --> 00:17:36.240 a JSON payload describing 138 00:17:36.700 --> 00:17:41.434 what questions you want asked and what their default values are. For us, there was a little bit of a limitation 139 00:17:41.815 --> 00:17:42.315 where 140 00:17:42.615 --> 00:17:48.795 we want our question asking approach to be dynamic. An example is a Python fast API application 141 00:17:49.289 --> 00:17:55.710 will have a completely different set of questions that we wanna ask our users than a Java library, for example. 142 00:17:56.090 --> 00:18:11.669 Additionally, if you are building a new application and you want to use a database or have an integration with a messaging application such as Kafka or Google Pub Sub, and you say, yes. I want that thing. We may have an entirely different set of questions we wanna ask in addition. So 143 00:18:11.970 --> 00:18:16.465 1 of my teammates, Pat Lanigan, wrote an open source library called Colombo. 144 00:18:17.085 --> 00:18:21.505 And what that offers our template designers is basically a Python DSL 145 00:18:22.000 --> 00:18:25.620 to describe the set of questions to ask users. It allows them to 146 00:18:26.000 --> 00:18:31.675 determine whether or not a question should be asked in the 1st place based on previous answers. So there's the dynamism. 147 00:18:32.295 --> 00:18:44.490 It allows for a rich validation, default values, and a lot of different types of questions that can be asked as well. So you might just wanna echo some text. You might want a Boolean value. You might want a multi choice, those types of things. 148 00:18:45.190 --> 00:18:47.130 And as far as the 149 00:18:47.590 --> 00:18:50.170 templating aspect and being able to 150 00:18:50.855 --> 00:18:56.155 multipath the questions that are being asked. I'm wondering if you can talk to some of the 151 00:18:56.615 --> 00:18:57.595 ways that 152 00:18:57.950 --> 00:19:02.050 you and your team have had to educate other members of the org 153 00:19:02.590 --> 00:19:10.885 about how to think about either building out their own templates or some of the edge cases that they need to be aware of or, you know, maybe integrating 154 00:19:11.345 --> 00:19:33.855 language ecosystem best practices into these templates to be able to make sure that they're as useful as possible and that you don't, you know, generate a bunch of boilerplate, and then the end user then has to go and change, you know, 20% of what's there? Yeah. I think that's a great question. It puts our team in a very, very interesting position where we aren't super opinionated about the content of the templates themselves 155 00:19:34.490 --> 00:19:35.630 and want to enable 156 00:19:36.010 --> 00:19:39.150 teams that are interested in creating project templates to do so. 157 00:19:39.529 --> 00:19:41.870 We don't have too much insight 158 00:19:42.325 --> 00:19:47.305 or standards in terms of what the actual content of the generated application is, though 159 00:19:48.085 --> 00:19:52.480 we do have ways of verifying that templates are doing what they're supposed to do. 160 00:19:52.860 --> 00:19:54.160 So the template repositories 161 00:19:54.460 --> 00:19:54.960 themselves, 162 00:19:55.420 --> 00:19:59.120 we know where these template designers will create their logic for their templates. 163 00:19:59.575 --> 00:20:02.955 We have enabled different build steps to basically verify, 164 00:20:03.335 --> 00:20:09.519 hey. For all of these templates that you have and all of these questions that you're going to ask, we wanna make sure that at a minimum, 165 00:20:09.980 --> 00:20:13.919 we are capable of successfully running the cookie cutter command to actually 166 00:20:14.220 --> 00:20:15.760 generate out a new application. 167 00:20:16.125 --> 00:20:20.785 And there are some various things that we might wanna check and assert on your application. 168 00:20:21.165 --> 00:20:32.110 If these base set of expectations aren't true, we will fail your build. A really, really high level example is Wafer is undergoing a migration from on premise data centers to Google Cloud Platform. 169 00:20:32.490 --> 00:20:38.575 And

in 1 of our manifest files where we describe the data center to deploy containers to, you can imagine, 170 00:20:39.115 --> 00:20:42.815 because we're at such a high leverage point in the application creation process, 171 00:20:43.500 --> 00:21:00.220 we probably want to assert that all templates now are, at a minimum, deploying new applications to GCP and not our old on premise data centers. So those are the types of opinions we will hold, and we will hold them strongly. But aside from that, it's very much on the implementers 172 00:21:00.520 --> 00:21:01.980 to maintain standards. 173 00:21:02.360 --> 00:21:10.195 We've found it to be, like, fairly successful so far where the teams that own these templates are incentivized internally 174 00:21:11.054 --> 00:21:21.200 to have these best practices in mind. So that hasn't really been a problem for us. Though if something out in the wild is found as an improvement to be made, we generally fold it right back into the templates. 175 00:21:21.740 --> 00:21:22.559 1 of the 176 00:21:22.955 --> 00:21:25.215 always interesting and challenging aspects 177 00:21:25.755 --> 00:21:26.975 of templating applications 178 00:21:27.275 --> 00:21:28.575 from a cookie cutter 179 00:21:29.115 --> 00:21:29.855 is that 180 00:21:30.440 --> 00:21:33.740 at the time that they're created, they represent the best practice 181 00:21:34.360 --> 00:21:37.500 and, you know, maybe they have the updated dependencies 182 00:21:37.960 --> 00:21:39.260 in the generated 183 00:21:39.595 --> 00:21:41.054 lock file, for instance. 184 00:21:41.755 --> 00:21:44.414 But as time goes on and bit rod sets in 185 00:21:44.715 --> 00:21:47.135 and the language ecosystem evolves, 186 00:21:47.590 --> 00:21:54.330 you might go back to that template and say, okay. You know, we used to use unit test and knows. Now we wanna use Pytest, 187 00:21:54.950 --> 00:21:55.450 and, 188 00:21:55.825 --> 00:22:05.125 you know, now we need to update the version of Django or Flask or what have you in the cookie cutter so that when you start a new project, you're starting at the latest version. 189 00:22:05.730 --> 00:22:06.870 And then being able 190 00:22:07.490 --> 00:22:09.350 to bring that forward into 191 00:22:09.890 --> 00:22:29.680 applications that were previously generated from that template. I know, for instance, there's the copier project that is designed to be able to handle some of that capability. There are some add ons to the cookie cutter ecosystem to be able to handle that aspect. And I'm just wondering how you think about the ongoing maintenance of those templates for being able to bring in those 192 00:22:30.175 --> 00:22:33.475 upgrades to dependencies or upgrades to best practices 193 00:22:33.855 --> 00:22:34.675 or the 194 00:22:34.975 --> 00:22:44.420 organizational standards, and then being able to alert teams who previously used that template to those changes to determine when and how to incorporate those updates? 195 00:22:45.040

--> 00:22:52.385 Yeah. It's a really, really hard problem. Not just hard, but, like, borderline impossible when you're dealing with the skill that Wayfair does because 196 00:22:53.165 --> 00:23:00.530 the diff that you have between a template and applications that exist out in the wild can be anything. We don't know really 197 00:23:00.830 --> 00:23:10.815 what the state of all the different applications are. Presents a really big challenge. So that actually kind of folds into the second portion of our team's scope, which is application maintenance. 198 00:23:11.195 --> 00:23:27.615 I think our stance is we acknowledge that over time, applications are going to be out of date with the latest and greatest standards. And an app generated today might look great, but 3 years from now might not be reflective of, you know, the desired tooling or standards or even dependencies. 199 00:23:27.995 --> 00:23:30.895 So I would say, like, instead of trying to solve 200 00:23:31.230 --> 00:23:32.529 the technical problem 201 00:23:32.830 --> 00:23:37.649 from a template standpoint, you know, hey. This template's evolving, and let's fold these changes indirectly. 202 00:23:37.950 --> 00:23:39.970 We've taken a little bit of a different approach 203 00:23:40.384 --> 00:23:44.245 where through different pieces of tooling that create automated pull requests, 204 00:23:44.865 --> 00:23:55.320 we use that to try to keep applications up to date. I can tell you a little bit more about that. Yeah. Definitely interested in exploring some of that kind of automated maintenance aspect and some of the ways that you're able to 205 00:23:55.620 --> 00:23:56.840 introspect the current 206 00:23:57.245 --> 00:23:59.345 structure of the project and the dependencies 207 00:23:59.645 --> 00:24:00.465 and understand 208 00:24:00.925 --> 00:24:02.385 what types of modifications 209 00:24:03.405 --> 00:24:07.639 are useful to make, which ones are safe to make, and just being able to 210 00:24:08.340 --> 00:24:13.240 hook that into the overall ongoing life cycle and maintenance of the systems that you're supporting? 211 00:24:13.625 --> 00:24:22.044 Totally. I'll break up the application maintenance space into 2 broad categories. 1 of them is very easy to describe and another is less easy to describe. 212 00:24:22.410 --> 00:24:27.390 First 1, I'll just call versioned dependencies. So if you have an application, Python application, 213 00:24:28.010 --> 00:24:33.375 you might rely on Python packages. You might rely on some base docker image. 214 00:24:33.675 --> 00:24:39.535 If you're using Kubernetes and a technology such as Helm, you are relying on Helm charts potentially. 215 00:24:40.070 --> 00:24:45.370 And if you have a build system like we use, you know, that may have versioned plugins as well. 216 00:24:45.670 --> 00:24:48.090 So managing those versioned dependencies 217 00:24:48.445 --> 00:24:51.505 is 1 of the problems that we've addressed over the past year and a half. 218 00:24:51.965 --> 00:24:54.865 We started off using a variety of in house solutions, 219 00:24:55.165 --> 00:24:58.790 but are now favoring an open source solution called Renovate, 220 00:24:59.170 --> 00:25:03.190 which is able to handle all of the things I described above. It's highly configurable, 221 00:25:03.705 --> 00:25:06.205 and it allows application engineers to 222 00:25:06.585 --> 00:25:10.445 really, really describe the way that they want to manage their versioned dependencies. 223 00:25:11.030 --> 00:25:20.355 So our team takes direct responsibility for helping application owners keep their dependencies up to date, and that's kind of 1 half of the application maintenance 224 00:25:20.835 --> 00:25:30.990 problem. And so the dependency management aspect of it, you know, definitely seen a few different tools that approach that. So there's the Dependabot project that was acquired by GitHub. 225 00:25:31.530 --> 00:25:43.054 There are language specific aspects. I have come across Renovate, but haven't used it specifically. So I'm maybe interested in understanding some of the benefits that that provides of its stance as being a multilanguage 226 00:25:43.434 --> 00:25:43.934 ecosystem 227 00:25:44.235 --> 00:25:44.735 tool 228 00:25:45.430 --> 00:25:47.850 and some of the ways that it's able to 229 00:25:48.470 --> 00:25:51.770 work across languages and maybe some of the capabilities 230 00:25:52.285 --> 00:26:06.030 that would be nice to have from some of these more focused solutions such as the PyUp bot or something like that? The way View Renovate and its advantages over other tooling that we've used before is renovate is very, very, very configurable. 231 00:26:06.490 --> 00:26:21.230 So we don't have to build our own configuration and kind of reinvent the wheel. So we just get to ask people like, hey. Listen. You know, RenovA open source has really great docs on how to set up auto merging and how to, you know, write your own plug ins for managing 232 00:26:21.690 --> 00:26:24.190 various different types of Nuance to dependencies 233 00:26:24.795 --> 00:26:30.655 that maybe renovate doesn't handle out of the box. Certainly, that's an advantage there where it's, like, very, very flexible. 234 00:26:31.195 --> 00:26:46.255 Another interesting aspect of it is the way that renovate actually ships is through a docker image. So I believe it's a docker image and or, like, an NPM package. So the way that we actually because Renovate is open source and not a vendored solution, 235 00:26:46.635 --> 00:27:11.970 we have to kind of stand it up and run it in house. From a technical standpoint, the way that we actually do that is very, very straightforward. Whereas other tools, you know, instead of shipping a Docker image or an NPM package, it's more of an SDK where we have to stitch everything together and kind of reinvent the wheel config wise and figure out how to expose different things to users. We just don't have to solve a lot of those

problems with renovate. So for those reasons, it's certainly preferred. 236 00:27:12.270 --> 00:27:19.315 Digression. I don't wanna flame dependabot. We use dependabot at Wayfair. We don't use it in GitHub, but there's open source Dependabot core. 237 00:27:19.695 --> 00:27:21.315 And it's basically a Ruby SDK. 238 00:27:21.615 --> 00:27:39.735 So we've had to, like, stitch together a bunch of, like, weird janky Ruby stuff and write our own Ruby scripts to figure out how to do this. We've also had to literally reinvent the wheel config wise to, like, figure out how do we take Dependabot's config that they've talked about and then, like, expose that to our users in a way that is compatible 239 00:27:40.170 --> 00:27:48.965 with the way that we've implemented the, like, Ruby SDK. So I just didn't wanna speak ill of dependent about, but that's, like, what we're going through. Fair enough. K. 240 00:27:49.425 --> 00:27:49.925 And 241 00:27:50.305 --> 00:27:55.205 the other aspect of the kind of automated maintenance that you mentioned is beyond just 242 00:27:55.580 --> 00:27:59.120 keeping the dependencies up to date. There's also the question of 243 00:27:59.580 --> 00:28:00.400 maybe doing 244 00:28:00.700 --> 00:28:02.560 vulnerability and security scanning, 245 00:28:02.985 --> 00:28:05.965 linting, maybe adding in some fitness functions for 246 00:28:06.345 --> 00:28:12.044 enforcing different architectural aspects of the application. I'm wondering if you can talk to some of the ways that you have 247 00:28:12.370 --> 00:28:13.350 worked with teams 248 00:28:14.049 --> 00:28:19.110 to automate some of that and some of the tools or internal capabilities that you've developed as a result. 249 00:28:19.544 --> 00:28:23.485 Absolutely. Going back to the question that you're asking before about, you know, 250 00:28:23.865 --> 00:28:27.245 diff when an app is created from a template and time passes, 251 00:28:27.610 --> 00:28:32.030 We realized pretty early on that that was a big problem for us, and we wanted to address it. 252 00:28:32.330 --> 00:28:37.230 So the way we view that is we just wanted a kind of general solution for 253 00:28:37.675 --> 00:28:42.335 creating pull requests at scale and basically creating a platform that allowed 254 00:28:42.795 --> 00:28:44.975 various engineers at Wayfair to 255 00:28:45.299 --> 00:28:50.280 describe changes that they wanted to make at scale. So we have an in house solution, 256 00:28:50.660 --> 00:29:02.475 called Gator that we've built, which is really, really neat. An example of something you might wanna do with Gator is imagine you're the Python platform team at Wayfair and Python apps are using the black tool to format code. 257 00:29:02.820 --> 00:29:13.745 And our Python platform team, for some reason, this wouldn't actually happen, but they're opinionated about the line length that they would like all Python apps to use. Our tooling enables them to make declarations 258 00:29:14.205 --> 00:29:14.705 over, 259 00:29:15.005 --> 00:29:21.059 hey. I want to find all Python apps. I want to search in a specific file such as like a pyproject.toml, 260 00:29:22.080 --> 00:29:24.419 and I might wanna do something like a modification, 261 00:29:24.720 --> 00:29:26.419 like a regex replace 262 00:29:26.960 --> 00:29:31.015 to look for, you know, the line length of 88 and replace it with 120. 263 00:29:31.315 --> 00:29:33.015 That's fairly basic. But 264 00:29:33.315 --> 00:29:38.480 with a couple lines of YAML, we enable the Python platform team to make that type of change at scale. 265 00:29:38.780 --> 00:29:42.240 Another more complex example would be, imagine 266 00:29:42.540 --> 00:29:43.040 you 267 00:29:43.395 --> 00:29:46.775 own a shared library that many, many applications are using. 268 00:29:47.315 --> 00:29:51.575 A new version of it is released, but it includes a breaking interface change. 269 00:29:52.050 --> 00:29:58.630 Now, historically, you know, that would happen, and we'd ask people to, hey. Please use this new version of this new library, 270 00:29:58.985 --> 00:30:05.485 and you have a month to do it. And, obviously, people get upset about that. And when they inevitably don't do it, things break. 271 00:30:05.865 --> 00:30:14.410 So to support the shift from we need you to do this to, hey. I'm proactively letting you know that something needs to change, and I made the change for you. We, 272 00:30:14.870 --> 00:30:16.970 through the system, allow for 273 00:30:17.305 --> 00:30:19.965 very, very complex code manipulation to happen. 274 00:30:20.345 --> 00:30:24.845 We basically allow you to run your own container in your own environment that describes 275 00:30:25.440 --> 00:30:32.260 exactly the type of code modification you wanna do. So our Python platform team has gone as far as performing AST manipulation 276 00:30:32.865 --> 00:30:35.125 to resolve the breaking change for people. 277 00:30:35.745 --> 00:30:52.644 So, you know, the YAML config will reference a Docker container where they've written these instructions up. That's how it integrates with our system. Those are the types of things that are being done with our system. Been very, very helpful on a variety of use cases, things that we wouldn't have expected are being done with it, and it's really neat. 278 00:30:52.945 --> 00:30:54.485 On the AST manipulation 279 00:30:55.825 --> 00:30:56.725 and code restructuring 280 00:30:57.220 --> 00:31:01.400 aspect of it. I know that in Python, there are a lot of capabilities built into the AST 281 00:31:01.780 --> 00:31:05.880 and pretty similarly with some of the other dynamic languages. I'm curious if there are any 282 00:31:06.345 --> 00:31:09.085 language environments where you have run up against 283 00:31:09.544 --> 00:31:10.924 complexities or limitations 284 00:31:11.544 --> 00:31:13.245 in the kind of introspectability 285 00:31:13.865 --> 00:31:18.779 of the software to be able to automate some of these

changes and you've had to fall back on other approaches? 286 00:31:19.320 --> 00:31:21.899 You know, I think 1 of the benefits of 287 00:31:22.255 --> 00:31:38.990 being where we are in terms of running this platform is we don't need to know about those details. So we haven't really dug into those. It's kind of like, hey. We'll give you the tools to do whatever you need to do. And if you can't, that's kind of on you to figure out. So, you know, we're not necessarily privy to all of those details. I think, like, 288 00:31:39.295 --> 00:31:48.730 within different teams, they're trying to figure out how to do some complex stuff. But I'm kind of unaware of all of that, which I think is kind of a good thing in this case. So we've taken the approach of being as unopinionated 289 00:31:49.110 --> 00:31:49.769 as possible 290 00:31:50.230 --> 00:31:56.065 with our tooling and and letting people do what they wanna do. To the point that you're making about being able to 291 00:31:56.605 --> 00:31:57.105 automatically 292 00:31:57.725 --> 00:32:00.945 make pull requests and suggest changes to 293 00:32:01.450 --> 00:32:03.630 either update dependencies or 294 00:32:04.010 --> 00:32:05.870 improve code structures or 295 00:32:06.250 --> 00:32:07.470 introduce bug fixes. 296 00:32:08.250 --> 00:32:12.245 What is your approach for being able to actually automate the creation 297 00:32:12.705 --> 00:32:16.405 and management of those pull requests across such a large number of repositories? 298 00:32:17.190 --> 00:32:41.220 Yeah. It's an interesting problem. We use the Git Python library, which is an extremely powerful tool. Basically, sits on top of Git and allows you to do anything that Git does. Its API is fairly involved and, by design, doesn't provide the type of high level abstractions we need. Like, if you think about how a human will create a pull request, they'll likely clone a repo, check out a future branch, apply some changes, 299 00:32:41.679 --> 00:32:44.500 stage commit, push up the branch, and create the PR. 300 00:32:44.895 --> 00:32:48.915 So we thought it made sense to have kind of high level abstractions to 301 00:32:49.215 --> 00:32:50.195 mirror that workflow. 302 00:32:50.575 --> 00:33:05.325 We have a couple systems at Wayfair that do create automated PRs. So it made sense for us to first, you know, create those within 1 system and then abstract it out into a library that we could reuse. We ended up creating an open source library called Pygitops. 303 00:33:05.865 --> 00:33:15.770 Again, it's built completely on top of Git Python, which we think is an amazing library. It just kind of adds some stuff on top of it. And, again, it's a high level abstraction for the workflow I described. 304 00:33:16.309 --> 00:33:21.304 1 of the coolest parts about it that I think is, like, the most Pythonic part that I'm really in love with 305 00:33:21.605 --> 00:33:23.784 is the feature branch context manager. 306 00:33:24.164 --> 00:33:35.000 So, again, as a human, if you are making code changes that ultimately get pushed up, you're likely going to be making those changes in the context of a future branch. So, you know, when you 307 00:33:35.434 --> 00:33:36.815 intentionally or accidentally 308 00:33:37.115 --> 00:33:48.559 leave the context of a feature branch, we always want certain things to be true. We always want you to be on the default or, you know, main branch of your repo. We want a clean work dir. We want no unstaged changes, etcetera. 309 00:33:48.940 --> 00:33:56.304 This is important because when you're creating many, many feature branches and PRs and against many repos kind of all at once in batch, 310 00:33:56.605 --> 00:34:01.024 we wanna make sure that the git state of that repository 311 00:34:01.485 --> 00:34:12.250 is clean always. We've had some interesting edge cases where changes from 1 PR get pulled into another PR, or we have Git just completely barf and break things because, 312 00:34:12.575 --> 00:34:13.235 you know, 313 00:34:13.535 --> 00:34:21.155 we had an unstaged change when we check out a future branch or something crazy like that. And we built a lot of those learnings and edge cases into Py GitOps. 314 00:34:21.480 --> 00:34:30.135 You know, it's become more stable over time. When you write PR automation, you generally don't have to worry about those edge cases anymore as a result. PygidOps has 100% 315 00:34:30.515 --> 00:34:34.454 unit test coverage. And it's not just a lot of, like, mock unit tests. 316 00:34:34.755 --> 00:34:37.815 Because of the way Git works and because a remote 317 00:34:38.230 --> 00:34:40.010 repo can just be another directory, 318 00:34:40.310 --> 00:34:41.930 we get to use through pytest 319 00:34:42.550 --> 00:35:01.190 the temp path fixture in our test cases. So we'll start our test cases by, you know, creating a remote repo on disk. We'll clone it. We'll do some stuff to it, and we'll make a lot of assertions on really a live repo. So we're not really faking anything. We're making sure that things work in a sensible way, which I think is a really, really powerful part about 320 00:35:01.490 --> 00:35:04.069 Git, Git Python, Python itself, Pytest, 321 00:35:04.369 --> 00:35:05.155 all the above. 322 00:35:05.555 --> 00:35:07.975 1 of the things that's always interesting to explore 323 00:35:08.275 --> 00:35:08.775 is 324 00:35:09.475 --> 00:35:09.975 the 325 00:35:10.515 --> 00:35:11.735 philosophy around 326 00:35:12.195 --> 00:35:14.135 when and how to open source projects 327 00:35:14.560 --> 00:35:18.260 when building specifically for internal consumption. 328 00:35:18.880 --> 00:35:21.140 And I'm curious if you can talk to 329 00:35:21.785 --> 00:35:24.445 how you think about which tools are worth open sourcing, 330 00:35:25.145 --> 00:35:32.680 whether you bias towards building the tools that you're supporting the rest of the organization with in a way that it can be released publicly. 331 00:35:33.380 -->

00:35:33.880 And 332 00:35:34.260 --> 00:35:35.240 if you generally 333 00:35:35.700 --> 00:35:43.105 approach the sort of open sourcing of the tools up front or if you build the tool first and then decide, okay. This can be extracted. 334 00:35:43.405 --> 00:35:47.665 And then just some of the process that's involved in actually sanitizing it and 335 00:35:48.030 --> 00:35:53.410 updating it so that it is more broadly applicable and not tied to your organizational assumptions? 336 00:35:54.190 --> 00:35:56.050 Yeah. Great question. I think 337 00:35:56.464 --> 00:36:00.885 in both of the open source libraries I referenced, there's a little bit of a different story. 338 00:36:01.185 --> 00:36:04.484 So I will say with Colombo, which again is the 339 00:36:04.880 --> 00:36:09.940 library that allows for specification of questions being asked in the the kind of entire workflow, 340 00:36:10.559 --> 00:36:13.140 the teammate of mine that wrote that, Patrick Lanigan, 341 00:36:13.475 --> 00:36:23.589 I think had a more intentional approach from the beginning where he had a good vision for the project itself, knew that it would be a good candidate for open source. And that came after, 342 00:36:23.970 --> 00:36:27.270 you know, doing a lot of research on solutions that were available, 343 00:36:27.650 --> 00:36:33.335 didn't do exactly what we needed it to do. And so because it led to us having to create a custom solution, 344 00:36:33.635 --> 00:36:39.734 I think part of his idea was giving back to the community from the very beginning and and having it be open sourced. 345 00:36:40.110 --> 00:36:42.610 The story with PygadOps is a little bit different. 346 00:36:42.990 --> 00:36:43.490 Initially, 347 00:36:43.870 --> 00:36:51.605 we have Git automation that touches just about every project that we run in a variety of ways. So it definitely started off, you know, 2 years ago as 348 00:36:51.905 --> 00:37:08.425 how do we even do this in the first place? Like, how do you actually technically make this possible? So So it was very scrappy, very, very iterative. We didn't have kind of design in mind, and so it was just a whole mess of various things that we needed to do. And it wasn't until 349 00:37:08.885 --> 00:37:09.385 we 350 00:37:09.765 --> 00:37:22.860 had system number 2 that wanted to do a very similar thing where I I had even considered abstracting things out. So we then took the step of of kind of, like, making a little bit of a cleaner API, creating a library that was reusable. 351 00:37:23.245 --> 00:37:32.785 And by the time we had the 3rd or 4th system actually using Py GitOps, we thought that it would be appropriate to open source just because, you know, we really like Git Python, 352 00:37:33.309 --> 00:37:43.585 and we wanted to, like, give it credit and call out the accomplishments of Git Python where possible, but also provide a nice abstraction for other folks looking to create TRs. 353 00:37:44.205 --> 00:37:46.385 And in terms of your 354 00:37:46.685 --> 00:37:50.225 work that you've done with the engineering group at Wayfair 355 00:37:50.605 --> 00:37:54.369 and some of the tools that you have built to enable their 356 00:37:54.829 --> 00:37:57.010 application delivery and life cycle management 357 00:37:57.470 --> 00:38:04.745 and improve the overall developer experience and workflow? What are some of the most interesting or innovative or unexpected ways that you have seen 358 00:38:05.285 --> 00:38:08.825 the different tools that you've built used and maybe some of the 359 00:38:09.160 --> 00:38:11.260 interesting approaches that you have 360 00:38:11.640 --> 00:38:13.980 developed internally to be able to 361 00:38:14.760 --> 00:38:15.900 speed up the 362 00:38:16.200 --> 00:38:18.299 time to delivery for these different applications? 363 00:38:19.085 --> 00:38:21.505 I think this answers your question in a way. 364 00:38:21.805 --> 00:38:30.510 1 of the really, really interesting things that happened in recent history, if you're familiar with the Log 4 j issue that happened back in December, 365 00:38:30.970 --> 00:38:34.029 the change propagation system I told you about earlier, Gator, 366 00:38:34.410 --> 00:38:41.985 has a flexible resource model, and it kind of supports the notion of, like, we don't know what we don't know. So it allows people to very abstractly, 367 00:38:42.365 --> 00:38:54.145 you know, target repos and then run some outputs against them. So 1 of the really, really neat things is that Gator was actually used in the remediation of Log 4J where, as 368 00:38:54.445 --> 00:39:21.015 I mentioned, we allow for, you know, implementers of these Gator change sets to call out to a container. So we actually had a Java platform engineers writing some bash scripts to detect whether or not our repo had a log4j vulnerability, and that was able to be plugged into our system. The output of all that was we were able to create GitHub issues on these repos to kind of flag them to our internal ops teams. And, that was a pretty interesting way that our tooling was used. 369 00:39:21.339 --> 00:39:22.240 In your own experience 370 00:39:22.540 --> 00:39:31.735 of working on this team and helping to figure out how to reduce friction in the process from going from idea to delivery? 371 00:39:32.195 --> 00:39:36.455 What are some of the most interesting or unexpected or challenging lessons that you've learned in the process? 372 00:39:36.950 --> 00:39:46.935 1 of the lessons that I've learned along the way that I think is applicable to, you know, most of software engineering, especially when you're doing a platform work for a large enterprise, it goes back to the 373 00:39:47.395 --> 00:39:48.375 fusion of innovation 374 00:39:48.915 --> 00:39:59.000 or the adoption curve that I talked about earlier where, you know, now we know very well to target the early majority, and we know that that's a formula that works for us because 375 00:39:59.540 --> 00:41:04.775

of all the reasons I described earlier. 1 hard lesson along the way was kind of this expectation that I had that I'd built a thing and then everyone would immediately wanna use it. Even in recent history, you know, maybe, like, a year and a half ago, I built a solution that I just kind of assume that people would wanna use. Some people were really excited about it, and that was great and reassuring. But it was kind of a letdown when not everyone wanted to adopt it right away. You know, when you think about it and kind of think about things in the context of that adoption curve, it makes total sense. You know, you're marketing to a huge set of people with vastly different concerns and sets of priorities and tolerances for risk. That was definitely a challenge kind of, like, identifying that as, like, a tough pill to swallow. But I like us thinking about our work in kind of this new format where we market to people that wanna work with us. And then from there, you know, things kind of spread out organically. So it's okay if, you know, it takes a year for a solution to reach a different engineer that that wouldn't otherwise adopted immediately. Like, we're totally okay with that. As you continue to work with the various teams at Wayfair and 376 00:41:05.075 --> 00:41:13.050 manage your own team's concerns, what are some of the things you have planned for the near to medium term that you're excited to dig into? 377 00:41:13.590 --> 00:41:21.775 Yeah. I think the focus for us for the rest of the year is continuing to do what we're doing and supporting the same products we've been supporting, 378 00:41:22.075 --> 00:41:25.934 but with an eye for how it fits into our larger platform. 379 00:41:26.234 --> 00:41:35.930 So in other words, we've built some useful stuff that has, you know, smoothed out developer experience and helped reduce toil. And now it's a matter of folding these into 380 00:41:36.525 --> 00:41:43.105 other products in a way that's very, very seamless. So if you're a wafer engineer, you don't have to understand, 381 00:41:43.579 --> 00:42:00.145 you know, product a, b, and c and tool x, y, z. You're just using, you know, Wafer's platform, and the stuff that we've talked about today is just a part of it. So that's pretty vague and abstract, but a lot of it is, you know, getting our products and our tooling in the direction where it fits more cohesively with with surrounding tools. 382 00:42:00.600 --> 00:42:26.325 Are there any other aspects of the work that you're doing on the application life cycle team at Wayfair that we didn't discuss yet that you'd like to cover before we close out the show? Yeah. 1 thing we didn't talk about yet is the change propagation system, which we call Gator. I think it's an interesting thing to consider for open sourcing. The tool itself in the entire product is quite complex, and there's a lot of, you know, wafer specific stuff involved there. 383 00:42:26.645 --> 00:42:28.105 Though the core technology,

384 00:42:28.724 --> 00:42:33.305 which, again, very similar to Kubernetes resource model, allows you to specify abstractly 385 00:42:33.970 --> 00:42:36.150 ways to grab repos that we care about 386 00:42:36.450 --> 00:42:43.915 and what you actually want to, you know, run against them in terms of code changes that automate PRs. I think that that kind of central 387 00:42:44.295 --> 00:42:52.280 piece is an interesting part of, you know, the innovation that we've done with Gator that I think would be a good candidate for open source. We've had feedback from others 388 00:42:52.660 --> 00:42:54.360 in the Wafer Engineering Org 389 00:42:54.740 --> 00:42:58.520 that, you know, maintain open source projects of their own on public GitHub 390 00:42:58.855 --> 00:43:01.195 and would be interested in running Gator 391 00:43:01.575 --> 00:43:16.440 against those repos on GitHub. And so we think that, you know, if we were to open source, even a very, very minimalistic way to replicate kind of part of what we're doing here, then individuals may run it against their own repos, against their pet projects, 392 00:43:16.785 --> 00:43:36.785 and enterprises may even adopt it. We have to be thoughtful about, like, the approach we take there because, again, I think there's a lot of technical challenges with ripping out this central piece. But I think it could be done, and I think that would be a really, really interesting thing for us to consider. So I hope to be able to give you an update in the future, but that may be something to look forward to. 393 00:43:37.244 --> 00:43:49.725 Alright. Well, for anybody who wants to get in touch with you, I'll have you add your preferred contact information to the show notes. And so with that, we'll move us into the picks. This week, I'm going to choose something called nuchiolata, 394 00:43:50.025 --> 00:44:02.359 which is hazelnut spread. It's very similar to the Nutella product that more people will be familiar with. Just a very tasty treat. Just spread it on a piece of toast for a snack. Just add some of that before this show. So definitely 395 00:44:02.740 --> 00:44:15.320 always great to add to whatever you're eating. So with that, I'll pass it to you, Josh. What do you have for picks this week? Sure. So I love simulation video games. 1 game that I really like playing on my PC is called Cities Skylines, so I do want to 396 00:44:15.620 --> 00:44:16.520 recommend that. 397 00:44:16.820 --> 00:44:24.684 More specifically, though, I've been watching a YouTuber called City Planner Plays. It's a guy who actually is a city planner by profession, 398 00:44:25.065 --> 00:44:37.259 but he also loves simulation games. He has a series called Verde Beach, and it's, like, I think, 75 parts by now. He's been doing it for the past 2 years, but I've been watching him iteratively build a city. 399 00:44:37.615 --> 00:44:40.035 He, of course, pays a ton of attention to detail 400 00:44:40.575 --> 00:45:08.480 and has an eye for all of the things that he considers in his job as a city planner. So it's really, really interesting to watch, and so I recommend that. That's a fun thing to follow along. Yeah. It's definitely interesting to think about how accurate the simulation responds to the real world city planning considerations that he's adding to it. For sure. Alright. Well, thank you very much for taking the time today to join me and share the work that you're doing at Wayfair to help your application teams 401 00:45:08.940 --> 00:45:27.205 build and deliver faster. So it's definitely always interesting to think about the platform aspects of being able to help developers do their jobs. So I appreciate all of the time and energy you put into that and the open source projects that you've released out of that effort, and I hope you enjoy the rest of your day. Cool. Well, thanks for chatting with me, Tobias. Really appreciate it. 402 00:45:29.685 --> 00:45:34.880 Thank you for listening. Don't forget to check out our other show, the Data Engineering Podcast at dataengineeringpodcast.com 403 00:45:36.300 --> 00:45:38.240 for the latest on modern data management. 404 00:45:39.020 --> 00:45:40.480 And visit the site at pythonpodcast.com 405 00:45:41.420 --> 00:45:44.895 to subscribe to the show, sign up for the mailing list, and read the show notes. 406 00:45:45.275 --> 00:45:49.680 And if you've learned something or tried out a project from the show, then tell us about it. Email host@podcastinit.com 407 00:45:51.099 --> 00:45:51.920 with your story. 408 00:45:52.700 --> 00:45:56.800 To help other people find the show, please leave a review on Itunes and tell your friends and coworkers.