The Future of Dev Experience: Spotify’s Playbook for Organization‑Scale AI의 전체 녹취록

00:04:31.014 --> 00:04:36.695 our entire fleet of software. So it used to be that every team would own their own components. 50 00:04:36.695 --> 00:04:38.055 And when we, 51 00:04:38.215 --> 00:04:40.455 let's say, needed to upgrade some 52 00:04:40.879 --> 00:05:07.735 language version or library or whatever, we would tell each team that, hey. You need to move to the latest version of Python or Java or whatever it would be. That just didn't scale very well. So we made all of these investments into instead of being able to manage our entire fleet of software as one thing that we could sort of mutate. And I'm mentioning this because this will come back in terms of where we where we're also focusing when it comes to AI at the moment. So so that's roughly the backstory for us, like a few 53 00:05:08.134 --> 00:05:11.495 or number of thousands of production components, 800 teams, 54 00:05:12.240 --> 00:05:26.800 very high scale for what we do, at least compared to my previous experiences. Like, we have millions and millions of users connected at any given point of time. We have lots and lots of traffic. So it's it's a lot of fun and always been very fast paced over here. 55 00:05:28.344 --> 00:05:29.065 And 56 00:05:29.224 --> 00:05:34.025 one of the key aspects of developer experience is that you're trying to 57 00:05:34.185 --> 00:06:00.045 keep engineers in that flow state, keep them focused on the problem at hand, and not get bogged down by all of the various roadblocks and things that are technically necessary but don't really contribute to the actual product goals. So things like making sure that your linter checks passed, make sure that your CI runs in a timely fashion, make sure that you have good test coverage. And I'm just wondering as you were investing in that developer experience, 58 00:06:00.045 --> 00:06:01.965 what are some of the main 59 00:06:02.125 --> 00:06:10.285 pain points or struggles that you were dealing with prior to the introduction of AI? And then we can get into some of the things that you're trying to use AI to solve for. 60 00:06:10.970 --> 00:06:24.570 Yeah. So I think one of the main challenges that we saw was just a fragmentation of tooling for our developers. So it used to be as as a developer at Spotify, you have to jump in between many, many different tools that 61 00:06:25.115 --> 00:06:31.915 either we had, you know, some open source tool we were using or we had built internal tooling, but in a very fragmented. 62 00:06:31.915 --> 00:06:47.600 So that was how we came about investing into what we now call Backstage that we've also made available outside of Spotify, and it's it's it's this, like, single pane of glass that developers interact with, and it's very, very present as a developer at Spotify. You do a lot of your 63 00:06:47.760 --> 00:06:53.360 tasks every day within within Backstage. And that's been I think if I would point out, like, a single 64 00:06:53.974 --> 00:07:00.855 productivity unblock that

we've done maybe prior to AI, that's gonna be the one. Like, that that was the one that really 65 00:07:00.935 --> 00:07:16.070 simplified the life of a developer. Now I can go in within in Backstage. I can look at any of our thousands of software components, and I can see all the actions that I can take on them and all the data and information that is available around them. And that was, for us, very, very different from 66 00:07:16.470 --> 00:07:23.895 the fragmented state we were in before that. And I think this is true for many other companies as well, but it's been a particularly particular focus for us. 67 00:07:25.015 --> 00:07:27.015 And now digging into 68 00:07:27.095 --> 00:07:31.335 the introduction of AI in the engineering workflow, 69 00:07:31.335 --> 00:07:32.535 there are various 70 00:07:32.870 --> 00:07:39.030 shapes that that can take. There is a never ending set of tools with new ones being introduced every day. 71 00:07:39.270 --> 00:07:44.150 And I'm wondering as you were starting to see some of the potential 72 00:07:44.310 --> 00:07:45.030 for 73 00:07:45.110 --> 00:07:51.405 using AI to accelerate software development, what was the overall process for determining 74 00:07:51.645 --> 00:07:56.285 how and where and why to apply some of those technologies? 75 00:07:56.285 --> 00:08:06.930 Was it more of a bottom up and organic growth situation? Was it more of a top down mandate or, you know, maybe some hybrid structure? I'm just wondering kind of some of the ways that you thought through that overall 76 00:08:06.930 --> 00:08:08.370 process and investment. 77 00:08:09.250 --> 00:08:19.625 It has been, I'm gonna say, largely a bottoms up driver within Fortify. We started out with a very strong internal demand for Copilot 78 00:08:19.625 --> 00:08:40.100 back in the days, and we went through the journey of being able to clear that so we could use it. It was a bit of a early days, so there was a lot of legal questions and security questions and things like that that we had to figure out, but we were pretty early on on jumping on Copilot, and then the journey has just continued from there. Like, people we had a incredibly 79 00:08:40.100 --> 00:08:46.935 fast adoption of of Copilot within the company, and then Cursor came along and then CloudCode 80 00:08:47.334 --> 00:08:56.134 came along, and we've continued to seeing these, like, extremely rapid adoption, not like, nothing I've seen for any other tool that we've introduced. So there's been a incredible 81 00:08:56.134 --> 00:09:07.870 bottoms up demand. We are, of course, trying to balance that with our being somewhat intentional and strategic about this. But I'm gonna say we have gone very heavy on 82 00:09:08.030 --> 00:09:09.070 experimentation 83 00:09:09.070 --> 00:09:30.805 and sharing learnings and being pretty humble about that. We don't know exactly what's gonna happen tomorrow to your point of new tools being released all the time. So we've been pretty open and allowing for a pretty broad set of tools and then encouraging our engineers to experiment and share as much as they can. And we have some very, very active internal 84 00:09:30.900 --> 00:09:32.660 Slack channels and 85 00:09:33.140 --> 00:09:44.500 repositories with Claude skills and things like that where people are experimenting every day. So it's it is amazing to see at the moment, and I imagine that this will continue. Like, if the pace has only been increasing. 86 00:09:46.225 --> 00:09:56.225 And to that point of the rapid pace of change, the rapid introduction of new tools, the evolution of existing tools, particularly from the perspective of 87 00:09:56.625 --> 00:10:02.589 developer experience and investing in the aggregate velocity of the organization, 88 00:10:03.149 --> 00:10:04.510 that variance 89 00:10:04.589 --> 00:10:12.350 can allow some people to move faster, but maybe at the expense of the broader organizational velocity, 90 00:10:13.065 --> 00:10:14.585 whereas standardization, 91 00:10:14.585 --> 00:10:22.425 such as what you were mentioning with the backstage investment of this is the way that you get to the thing, can make sure that everybody is moving 92 00:10:22.585 --> 00:10:33.630 in the same direction and at, you know, roughly the same multiplier of pace. And I'm wondering how you're thinking about some of that tension of wanting to standardize 93 00:10:33.710 --> 00:10:38.030 the tool chain builds around some of those capabilities 94 00:10:38.030 --> 00:10:52.095 and share those best practices while at the same time enabling people to do that pioneering exploration of figuring out what are the capabilities, what are the boundaries, and how do we determine what best practices to bring to the rest of the organization? 95 00:10:52.735 --> 00:11:03.340 Yeah. It's it's a great question. I think that's that's putting the the point very clearly on some of the trade offs that we were having to make. So much like you described, like, we are betting on standardizing 96 00:11:03.340 --> 00:11:17.675 a lot of our underlying foundational pieces. So that will be the the infrastructure that we have in place, like the tools that we make available to our coding agents and so on and on the code level. So as I mentioned before, we've been on a multiyear 97 00:11:17.755 --> 00:11:26.160 journey to drive that standardization, and we're continuing that now and accelerating that. And we're doing that through various means, like, where you one of the 98 00:11:26.720 --> 00:11:45.975 investments that we we currently have ongoing is to move all our code into mono repos, which allows us to have a much, much higher degree of standardization than we have before, like, where our code lived in thousands of small repositories owned by each and every team. So we were kind of lucky to make these, like, standardization, 99 00:11:46.135 --> 00:11:57.120 the fleet management, and the monorepo investments as AI happened to us. But so we're, in that

sense, like, feel that we're relatively well set up, but it is also moving 100 00:11:57.360 --> 00:12:01.120 incredibly fast. So we'll we're we're trying to adjust as we go along. 101 00:12:02.160 --> 00:12:12.165 And to your point of some of the tools that are available to the agents, some of the coding standards that can be enforced using things like linters, CICD, 102 00:12:12.165 --> 00:12:16.645 etcetera, that is definitely one of those core engineering disciplines. 103 00:12:16.965 --> 00:12:20.720 And I'm wondering what are some of the other foundational 104 00:12:20.800 --> 00:12:27.760 kind of first principles that are effective regardless of the specific harness that you're using for a given LLM 105 00:12:27.920 --> 00:12:31.520 and some of the ways that you are investing in 106 00:12:31.745 --> 00:12:44.384 that kind of platform layer of enabling the agents to be able to get the context, understand the understand the tasks, and maybe just some of the ways that the introduction of those agentic coding capabilities 107 00:12:44.384 --> 00:12:50.520 even shifts the focus of the people who are guiding and, responsible for that work. 108 00:12:51.240 --> 00:13:16.230 Yeah. It it comes in many shapes and forms. So one that you mentioned in there was we need to guide those agents. So we're spending quite a bit of time at the moment to make sure that we have the right skills and and and rules set up for the agents so they are able to to follow those and that we have the right where we needed the right sandboxing as well. So we're doing it in a secure way. We're also investing 109 00:13:16.230 --> 00:13:18.390 into improving the verification 110 00:13:18.390 --> 00:13:43.565 loop. So one thing that we've seen is that the better testing we have in place, the better both agents and, in particular, these, like, fleet wide agents that we haven't really talked about, but I'll come back to work. So let let me dive down that rabbit hole a little bit. So I mentioned before that we made this investment a few years ago on on what we call fleet management, which is all about, like, managing our entire fleet of software as one thing that we can make changes to. 111 00:13:43.964 --> 00:14:25.855 The challenge with that was that the type of changes that we could make were fairly simplistic because as it turns out, code is very complex to change. Like, it has a very wide API surface in in a way. And so we could make code changes, but they were fairly simplistic and shallow in a way. So one thing that has fundamentally changed now is that we can use AI agents to make those changes for us and then orchestrate and roll those out across our fleet. And that means that we can significantly raise the bar for how complex changes we can make to our fleet. But that means that we really need to have a good test automation, a way to verify that those changes are correct and safe. 112 00:14:26.015 --> 00:14:51.790 Again, luckily, we did some of those improvements as we started rolling out a fleet management stuff a bunch of years ago, but we think that we need to improve that further now. So that is one of the investments we're doing. We're both revisiting our practices around how we do tests and also our test coverage in the fleet. Luckily, agents are actually an excellent tool to also write some of those tests for us, so we think we can get some benefits from that. So those are some examples of these, like, more found foundational 113 00:14:52.515 --> 00:15:04.195 capabilities that we're building out, but there's lots more of those. Like, we built out an MCP infrastructure internally as well, for example, to make sure that we can provide many different MCP tools and do that in a in a secure way. 114 00:15:05.570 --> 00:15:09.810 Your comment on fleet management also brings up an interesting 115 00:15:09.970 --> 00:15:25.155 nuance to the overall engineering challenges that there are different styles of engineering within an organization where you have your application engineers who are focused on writing the features, making sure that the end user experience is fulfilling the overall organizational 116 00:15:25.155 --> 00:15:26.275 product goals. 117 00:15:26.515 --> 00:15:29.315 And then you also have data engineers, ML 118 00:15:30.360 --> 00:15:31.320 operations engineers, 119 00:15:31.320 --> 00:15:40.440 platform engineers, and all of those roles are going to have some set of code that they are interacting with that they're responsible for that they're going to be managing. 120 00:15:40.760 --> 00:15:46.440 But all of that also has to be deployed and maintained and monitored in the production 121 00:15:46.305 --> 00:15:54.145 environments, which is its own set of engineering challenges that has its own set of agentic capabilities that are being developed across the ecosystem. 122 00:15:54.385 --> 00:16:00.464 And I'm wondering how you're seeing the adoption and application of these agentic 123 00:16:00.650 --> 00:16:01.850 capabilities 124 00:16:02.170 --> 00:16:03.050 differ 125 00:16:03.050 --> 00:16:08.410 of across those different styles of engineering or domains of responsibility. 126 00:16:09.370 --> 00:16:39.260 So I agree with the sort of classification of different disciplines that you outlined, but I think they've started to meld together a little bit both as we've been, again, like, doing these investments in in thinking about our entire fleet. So it used to be that as a developer, you were at Spotify, you were very close to the code that you owned or your team owned. But now we're more in a mode of, like, yeah, you still own some piece of or some number of components within your team, but many other teams, in particular, are more platform teams. 127 00:16:39.420 --> 00:16:45.100 We'll also make changes to that code. So code has gone from being this thing that, like, I own to 128 00:16:45.255 --> 00:16:59.890 Spotify owns and in in a sense, like, everyone can make changes to them. There are

still access rules and and ownership around them, but but the principle has changed quite a bit here. So so I feel that some of those like, the the distinction between a 129 00:17:00.529 --> 00:17:15.275 what we would call a feature engineer, like the app engineer and a platform engineer used to be very distinct, but it's much less so today. That being said, I I still think that the way that we apply AI tooling into this, there's sorry. Lots of similarities, 130 00:17:15.275 --> 00:17:24.315 but also a few differences. So, again, the typical feature engineer, like, working on one of our mobile apps or, let's say, on our playlist systems, 131 00:17:24.395 --> 00:17:33.030 they will be fairly focused on those systems and, like, building new features and rolling those out to our users and and then operating those serve those 132 00:17:33.350 --> 00:17:34.230 components. 133 00:17:34.950 --> 00:17:38.790 And that tends to be that then you're focused on, like, a smallish 134 00:17:38.790 --> 00:17:55.895 set of code and you're very, like, deep expertise on that particular domain. And that me means that you will use AI agents typically, like, I would say in the sort of mode that they were primarily developed towards, like what most engineers would do. And then if you look at our platform engineers, 135 00:17:56.210 --> 00:18:01.090 they are very focused around, again, like, managing our entire fleet of software. 136 00:18:01.250 --> 00:18:04.050 So they will then use more scheduled 137 00:18:04.050 --> 00:19:14.215 jobs and and things like that. So one of the things that we built out as part of our AI agents is something we call home. It's just kind of silly name, but it's it's really way for us to and, again, using this management infrastructure that I I mentioned before, really way to take, like, a AI agent in currently clawed code under the hood to apply that to our entire, like, millions and millions of lines of of software. And that's been incredibly powerful in terms of being able to drive changes to our software and, you know, do code mononize modernization and things like that. So and that's really something that, like, I'm gonna say mostly our platform engineer mostly, but not exclusively our platform engineers are are focused on. So under the hood, they're using the same tools and the same many of the same practices. It's the same way of prompting. It's the same way of working with agentic tools and whatnot, but the focus for their efforts tend to be a little bit different. So that's the main distinction that we have. We don't have you mentioned, like, operations engineers. We don't have separate operations engineers. Like, it's the same teams that build and own our software that also operates it. So we don't have some of those categories that you mentioned. But I would say that's the big distinction that we have. 138 00:19:15.335 --> 00:19:17.335 And then in terms of the 139 00:19:17.735 --> 00:19:21.809 shape of the engineering work, I'm wondering how you're seeing 140 00:19:22.050 --> 00:19:22.770 the 141 00:19:22.930 --> 00:19:23.890 distribution 142 00:19:23.890 --> 00:19:24.610 of 143 00:19:24.930 --> 00:19:34.284 work change as far as what stage of the software life cycle more of the time goes into now where for most of the history of writing software, 144 00:19:34.365 --> 00:19:38.044 you had maybe a product manager or 145 00:19:38.125 --> 00:19:56.830 an architect who would be responsible for saying, this is how everything is going to fit together. These are the features that the end users are going to be delivered, and then I'm going to write all of the tickets, and then I'm gonna hand that off to an engineer who's going to write all the code and test it and make sure that it gets deployed effectively and monitored. 146 00:19:56.990 --> 00:19:59.710 And with agentic coding 147 00:19:59.790 --> 00:20:02.830 in the mix, it changes that equation a bit where 148 00:20:03.225 --> 00:20:17.465 the writing of the code part can sometimes move from taking weeks down to minutes depending on exactly what you're doing. But making sure that it's generating the right code is really the challenge now, and I'm just wondering how you're seeing that change in dynamics 149 00:20:17.465 --> 00:20:30.740 shift the time investments where maybe rather than it being that middle point that takes up most of the work of actually writing the different lines of code, you spend more time instead making sure that you understand the problem and the specification. 150 00:20:31.385 --> 00:20:47.720 And then it also puts a higher burden on the review and deployment stage because you need to make sure that the code that was generated is actually accurate and correct in doing what it was designed to do and just how that's changing the dynamics of the engineering org at Spotify? 151 00:20:48.360 --> 00:21:25.945 Yeah. It's it's a great question, and many of the things you mentioned there really rings through. So so let let me talk through some of them. So first of all, I I I think our product development process always been a little bit more collaborative than what you described. So it hasn't been those, like, clear handovers of, like, the product manager writes down a a spec and hands it over to a developer. That's always been very iterative within Spotify, but we do have those roles. We have, like, product managers that will think deeply about our product strategy and then engineers to your point who will then design and build and operate the the software to to manage that. And, yeah, and and like you said, like, that part of turning 152 00:21:26.024 --> 00:21:48.620 those ideas into code, is obviously dramatically changing now. Like, it it is just like you said going from potentially weeks to to hours, and and the work for the work for for us and me as a developer has changed dramatically as well. Like, I used to be someone that loved the, you know, hands on keyboard, solving problems through code type of

challenge. 153 00:21:48.780 --> 00:22:28.445 And I haven't literally haven't written a line of code now in, like, two months because all of that is being done for me with the agents that I use. And that is a very, very different way of thinking about it. And I me, personally, I really enjoyed that transition, actually. It it turned out that it wasn't that handled keyboard thing that I enjoyed. I enjoyed being able to solve problems and shipping those to users more than the the hands on keyboard. This is obviously gonna be personal and different for diff different people. But for me, it's been a a pretty fun fun experience in in so many ways. And it is shifting some of the dynamics that you mentioned. Like, it is shifting the work for us as developers more towards how do we effectively translate 154 00:22:28.445 --> 00:22:32.604 these ideas of things we want to do into input for 155 00:22:32.684 --> 00:23:07.554 the agents that we use, and then to your point, then reviewing the output from those agents. And we are leaning heavily on applying human judgment in in both of those cases, like being a making sure that we provide the right set of inputs for the agents so they do a good job together with all the stuff that we've talked about before with making sure that we have the right foundations in place. Like, the agents, just like humans, knows what code standards we expect and what levels of conventions and security and so on that we expect. And then we put a lot of accountability on the developer to then review the output of the agent, 156 00:23:07.715 --> 00:23:16.510 both in terms of me sitting in front of the agent, but also as part of our peer code review. So that is remains super important to us. But it is also 157 00:23:16.670 --> 00:23:24.255 challenging. Right? Because now we're able to produce much more code changes, and we and a lot of that happens purely, 158 00:23:24.255 --> 00:23:34.415 like, in this agent loop and much, much more quickly than before. But we still need humans to review all of those code changes. So it's putting a lot of pressure on the code review stage 159 00:23:34.655 --> 00:23:46.269 before things goes into production. And that is something that we're working on figuring out right now. So putting more we already have, but we're we need to improve the AI assistance that we provide for code reviewing, 160 00:23:46.350 --> 00:23:51.710 and we're gonna change a little bit the policies that we have around how we do code reviews so 161 00:23:52.235 --> 00:24:04.955 more people can contribute to those reviews than is the case today, for example. And then the last reflection I'll I'll I'll add in is that you mentioned the again, like, going back to those different roles that we have with, like, product managers, designers, engineers, 162 00:24:05.290 --> 00:24:24.414 the similar to what I said before in in between the different disciplines within in within engineering, we're also seeing that those boundaries between what it means to be a product manager and an engineer are loosening up as well. Like, we see a lot of PMs spending time in cursor and Claude code and, you know, building prototypes, 163 00:24:24.495 --> 00:24:29.375 contributing to our code in a completely different way that was the that was the case before. 164 00:24:29.535 --> 00:25:16.760 So I don't necessarily expect that these different disciplines will completely disappear because there are different expertise there. But these, like, handover boundaries that that you started with, like, those, I think, gonna look very, very different in the future. And I fully expect that where we have congestion points today within that process, like you mentioned, like, the code part having taken a lot of time historically and that now moving away, that's just gonna create new congestion points. So one of the things that we're preparing for is being able to detect those quickly and then being able to make decisions so we can iterate our way out of that. I think this is gonna be a big part for us in the next few years just to transition through this, and we don't have a perfect, you know, view of the future of what this is gonna look like. But yeah. I mean, everyone are using these tools today. 165 00:25:18.255 --> 00:25:21.934 We have a ton of adoption outside of the engineering 166 00:25:22.415 --> 00:25:23.375 cohort 167 00:25:23.375 --> 00:25:24.174 for 168 00:25:24.255 --> 00:25:28.414 both cursor and claw to code, and, like, everyone's trying to use them to 169 00:25:28.735 --> 00:25:46.965 for many different things that you would be surprised about. That, like, wasn't necessarily what the tools were intended to from from the beginning, but people are having a lot of good luck in in, like, using them for for other things as well. And the the world of just like the world of an engineer has has been completely 170 00:25:46.965 --> 00:25:53.205 revamped in the last few months, I expect that to happen with all disciplines within Spotify in the next year or two. 171 00:25:54.165 --> 00:26:09.360 There are a couple of different things I wanna touch on here, and I'll just note them now before we get too far down the path. But I think that one of the things I really wanna dig into is some of those other ways that you're seeing some of these agentic utilities being applied. But before we explore that, 172 00:26:09.680 --> 00:26:14.320 in terms of the validation and review stage, I'm wondering how 173 00:26:14.640 --> 00:26:17.095 this speed of code generation 174 00:26:17.175 --> 00:26:31.219 and the need for context queues and rapid feedback for the agents is changing some of the ways that you think about what languages to use, how to invest in things like type annotations, 175 00:26:31.380 --> 00:26:35.620 and just some of the overall tooling that you're investing in for linting, 176 00:26:35.620 --> 00:26:37.059 testing, validation, 177 00:26:37.220 --> 00:26:43.859 canary deploys, etcetera, just some of the

ways that it's shifting your thinking about which languages and which tool chains are 178 00:26:44.019 --> 00:26:56.674 best now that you don't necessarily have to spend all of those extra keystrokes on putting in those type annotations and just because you don't have the benefit of engineers being lazy, but but also the the 179 00:26:57.075 --> 00:26:57.794 potential, 180 00:26:58.115 --> 00:27:09.039 downsides that that can have as far as not wanting to put in the extra work of, oh, well, I don't need that type annotation right now. I know what I'm trying to do, but maybe the LLM doesn't, just some of those trade offs. 181 00:27:10.080 --> 00:27:39.100 Yeah. I'm I'm gonna say that as as as a company with lots and lots and lots of code, as I mentioned before, and many, many different teams, and we have a fair I didn't touch on this before, but we also have a fairly it's very often that that, like, ownership of components will move around within the company. Like, I built something, but it's now closer to the business problem that you are trying to solve, so I will hand over those back end services or whatever it might be to you. So given this, and this has been the the case for many, many years at Spotify, and given this, we have 182 00:27:39.634 --> 00:27:41.154 always been pretty, 183 00:27:41.794 --> 00:28:05.360 pretty strict around, like, what languages do we use, what conventions do we have in place. And so a lot of these things we had to solve many years ago prior to AI taking over, and it I think it has turned out that a lot of those things that were good things at scale when you have thousands of engineers also turns out to be good for AI agents. At least that's with the current generation of those agents. 184 00:28:06.195 --> 00:28:23.930 So I'm not seeing us making it ask me again in six months and maybe maybe my view has changed. But at the moment, I'm not seeing us making a ton of changes in those, like, very foundational decisions on what languages we use or or whatnot. What we are doing is that one of the big shifts that we're going through at the moment is 185 00:28:24.090 --> 00:28:35.635 a lot of the standards that we've used like, the the way that we've used to promote the standards within Spotify has largely been guidance towards engineers. So we've had TechRadar 186 00:28:35.635 --> 00:28:36.355 and 187 00:28:36.675 --> 00:28:49.875 various documents that tells you how to build a ML pipeline within Spotify, and that's been reasonably effective, but we don't think it works very well for for agents. So one of the things we're doing now is really to 188 00:28:50.115 --> 00:28:56.249 put more of those, like, closer into our code bases rather than being guide guidelines, they're gonna be links 189 00:28:56.410 --> 00:29:02.570 instead. So that is one of the changes that we're doing at the moment, and we're shifting a lot of those things into 190 00:29:02.705 --> 00:29:10.784 into our actual code base. And this also goes hand in hand with the investments that that I mentioned before in of of, like, moving to more of a mono repo structure because 191 00:29:10.945 --> 00:29:46.914 one of the things that have been really hard for us historically has been I mentioned having thousands and thousands of production components. They lived in thousands and thousands of git repos and trying to manage that and, like, reasonably enforce our standards in in all of those tiny repositories with hundreds of teams owning them has been very hard. So one of the things one of the there's many challenges for us to move to mono repos, but one of the benefits we're getting is, like, we have this one place where we can put all of these standards in place and all of the linking rules in place. We've done this for our clients for many, many years. They've always been in in mono repos, 192 00:29:46.915 --> 00:29:54.700 but not the rest of our software. So we're sort of catching up with that now. So that gives us much better tooling for doing some of these things as well. 193 00:29:55.419 --> 00:29:59.740 And, also, one of the things that you mentioned is identifying 194 00:29:59.740 --> 00:30:03.465 what are the points where you are hitting bottlenecks in the path to delivery. 195 00:30:03.705 --> 00:30:14.665 And that's definitely an area where the overall industry has been investing time recently as in figuring out how do you identify those bottlenecks using things like DoraMetrics, 196 00:30:14.665 --> 00:30:21.040 I think is the most widely adopted one. But just wondering what are some of the ways that you are quantifying 197 00:30:21.040 --> 00:30:28.240 some of those aspects of the engineering life cycle and I in order to be able to identify those bottlenecks and hotspots. 198 00:30:29.455 --> 00:30:31.214 Yeah. So we have very, 199 00:30:31.534 --> 00:30:33.695 I'm gonna say, very good instrumentation 200 00:30:33.695 --> 00:30:35.695 of our developer workflows. 201 00:30:35.695 --> 00:30:40.975 So we capture data from our production systems and whatnot, then that goes into 202 00:30:41.450 --> 00:30:50.009 BigQuery in our case, and and you can go in there and and make queries around pretty much everything that goes into our developer workflow. 203 00:30:50.250 --> 00:31:05.605 So in that space, we're in a pretty good state to and we have lots and lots of metrics that we track, like Dora metrics, but also much more fine grained metrics that we that we catch and track all the time. So we're in a pretty good state there. Unfortunately, that is not true for the broader production 204 00:31:06.165 --> 00:31:14.710 sorry, product development process that we have. So we do have pretty significant gaps around being able to measure and quantify, 205 00:31:14.790 --> 00:31:21.750 let's say, how our PMs are working or how our designers are working. So as we're seeing these congestion points 206 00:31:22.165 --> 00:31:54.764 shifting around, I'm reasonably confident that we'll be able to see them

within what our engineers do. So I mentioned before the code review. We're starting to see some of that popping up in code review, and we could clearly see that in our metrics. It's not as obvious how we're gonna see that in a quantifiable way when it comes to the broader product development process. I think we're gonna have to rely much more on, you know, user research and surveying and those types of things to to capture some of those. I I I honestly think that's gonna be a bit of a challenge for us over the next few years to to really 207 00:31:54.925 --> 00:32:09.670 detect those early enough before you know, as they become a real problem for people, then it's obviously gonna bubble up and we'll be very aware of them, but I would not love to be able to identify them earlier than that. And but that, I think, is gonna be a challenge for us in some cases. 208 00:32:10.870 --> 00:32:14.230 And so digging now into some of the 209 00:32:14.550 --> 00:32:15.430 broader, 210 00:32:15.830 --> 00:32:18.470 I guess, side effects of introducing 211 00:32:18.550 --> 00:32:26.364 agentic capabilities into the engineering life cycle. I'm curious some of the ways that you're seeing those capabilities applied 212 00:32:26.525 --> 00:32:40.389 beyond just the stage of I can generate a boatload of code as fast as I want and just some of the ways that it's leaking into other aspects of engineering work, product work, and just some of the maybe 213 00:32:40.710 --> 00:32:49.830 new code that it enables because of the fact that it is so fast and easy and also maybe some of the ways that it changes some of your patterns around documentation 214 00:32:49.830 --> 00:32:50.950 and diagramming. 215 00:32:52.085 --> 00:33:24.755 Yeah. So so let let me actually go a little bit broader than that to to start out. So one of the very eager set of users that user group that we've seen has been our our product managers that we've been talking about a little bit before here. And they are real like, many of them are really picking up these tools. And, again, like, they are using the same tools that our engineers are doing. So they're trying to figure out their ways around, like, how to how to bend the cursor and clawed clawed code, and it's, you know, many different ways of improving how we write PRDs or how we evaluate experiments 216 00:33:24.755 --> 00:33:25.475 and 217 00:33:25.715 --> 00:33:26.754 doing 218 00:33:27.794 --> 00:33:33.570 insights analysis and, like, many, many different things. And we've had a a number of internal 219 00:33:33.810 --> 00:33:35.250 tools and websites 220 00:33:35.250 --> 00:33:40.210 pop up for for that target audience of, you know, again, managing, 221 00:33:40.210 --> 00:34:05.789 like, the life of being a product manager at Spotify in in in a better way. So so that is one one way that I've been particularly impressed with some of the innovation that's come out of come out of that group. And I think we haven't talked about vibe coding yet,

but I think that's been one of the things that they've really leaned into. And compared to before where they would need to go to like, they might have had some of these ideas around, like, these tools before, 222 00:34:06.029 --> 00:34:18.805 but they weren't really able to build them because they would have to go to an engineer and convince them to spend time on building out some web tool for that idea they had. And now that's, you know, a few prompts away. So we have a lot of these experimental 223 00:34:18.805 --> 00:34:24.005 tools popping up and people having ideas around how to, again, like, improve the nonengineering 224 00:34:24.005 --> 00:34:27.525 parts of of how we build products. Within engineering, 225 00:34:27.605 --> 00:34:47.815 what we've seen is, like, we see this sort of coding part of this, which we've been talking about. One of the interesting parts with coding is that it's actually a fairly small part of what engineers spend their time on. So it's roughly out of your eight days sorry. Eight hours a day, it is roughly one hour that engineers spend, like, hands on keyboard 226 00:34:47.895 --> 00:35:00.260 writing code. This seems to be fairly consistent across companies, so it's not unique for us. But, like, there's been a bunch of companies sharing that research, and it's true for us as well. Like, we're able to measure that. And then there's a broader set of development 227 00:35:00.260 --> 00:35:02.900 tasks. So this will be, you know, debugging, 228 00:35:03.220 --> 00:35:04.740 operational duties, 229 00:35:04.820 --> 00:35:12.215 code review, those types of things. That builds up to roughly three or four hours of the day, and then the rest is spent on planning, 230 00:35:12.375 --> 00:35:14.295 coordinating with other teams, 231 00:35:14.535 --> 00:35:16.135 reading, writing documentation, 232 00:35:16.135 --> 00:35:36.390 those types of things. So it is easy to, like, I think, overly focus on this, like, one hour a day and trying to make that as quickly as possible, which, like, the tools that we've been talking about today has been amazing at. But we're really even if we're 10 x ing that share of the work that engineers do to the previous points of just, like, finding new congestion points, 233 00:35:36.470 --> 00:35:57.530 we're we won't actually make Spotify a ton faster by just optimizing that. So we've been focusing on, like, what are other use cases outside of that one hour where we can apply AI early on. And it used to be that, like, if you would have if we would have talked a few months ago, you would have heard me talk about, like, the various strategies that we had for that of, like, how are we bringing AI into 234 00:35:57.770 --> 00:36:08.010 being on call or managing incidents or whatever it might be. And that is still very much something that we think about and have strategies for, but also where we've been out innovated by our employees. 235 00:36:08.170 --> 00:36:11.130 So there are now all of those tools available internally. 236 00:36:11.130 --> 00:36:56.045 People have, like, multiple tools that has that's been built for how to, for example, help people managing incidents. And we've taken one of those and made them, like, the sort of canonical tool that we now provide to the company, but other people are, like, still experimenting with other ways of doing that, which is amazing. So I think, like, we're seeing this spread out. So, like, what I imagine, again, if you would've we would've talked a few months ago, what I would imagine was, like, we had this, like, the coding cost at one hour a day, and then we would gradually spread the word the use of agents out from that because that was really what they were optimized for. I think what we're seeing in practice is that they're now being used for everything people do during their full workdays. Like, everyone will use them for I mean, I'm I use quote code to write documents today or, like, prepare for 237 00:36:56.160 --> 00:37:07.440 hiring interviews or whatever it might be. Like, it is it is something where it's become these tools have become ever present in every task that we do as as not just engineers, but more broadly as well. 238 00:37:08.575 --> 00:37:11.455 Yeah. And speaking as a technical manager, 239 00:37:11.535 --> 00:37:29.349 I was very pleased the first time that I realized that I could use things like Copilot or Cloud or whatever to take the fuzzy sketch of these are the things that I need to do and then stick them into the issue template so I could actually write comprehensive and useful issues for being able to assign to team members. 240 00:37:29.670 --> 00:37:46.635 Yeah. Yeah. It is it is amazing. And I think, again, like, the the amount of experimentation innovation that we're seeing from people just being able to take these tools, which, again, like, were built for a kind of different purpose, but being able to reuse them for every problem they're faced with has been amazing to see and continues being amazing to see. 241 00:37:47.675 --> 00:37:56.360 And one of the aspects of these agents being a double edged sword is that while they are immensely powerful and can 242 00:37:56.520 --> 00:38:00.040 allow people to move much faster, they can also amplify 243 00:38:00.040 --> 00:38:09.985 problems where if you have somebody who doesn't properly understand the problem domain or doesn't properly understand the code base, they can guide the agent in a direction that is counterproductive. 244 00:38:10.225 --> 00:38:15.505 And then the other aspect of challenge in terms of the deployment of agentic capabilities 245 00:38:15.505 --> 00:38:28.530 is the cost factor of how do you make sure that you don't accidentally end up spending the entire company's budget on a question and answer session that ultimately does not provide any actual return on investment? 246 00:38:29.490 --> 00:39:23.130 Yeah. So the the first part, I think honestly, I think the first part of that is is is

the most important, and that is really to create a culture and and practices where you have that human judgment in the loop so you don't blindly trust whatever is coming out of of the agent. One good and recent example on that that we've that we've done is we have an internal tool for for essentially essentially being able to prompt our data. So I mentioned before that we have, for example, data on the the engineer the developer process, and there's a bunch of datasets for that. That can be pretty daunting if you're not super familiar with SQL or our internal datasets and whatnot. So one of the ways that we're trying to bring that capability to more folks within Spotify is is to allow you to prompt for that instead of prompt the type of problem you want to solve, and that tool will generate the SQL for you to try to answer that question. That is great, but it's also pretty prone to 247 00:39:23.450 --> 00:39:24.650 introducing errors 248 00:39:24.890 --> 00:39:31.050 into those SQL queries that will be very hard for a less experienced user to detect. 249 00:39:31.050 --> 00:40:07.575 So that's a tool where we have spent a ton of work to make sure that we have the right validation in place of those queries before we show them to the user. One of the this could otherwise lead to, you know, us taking decisions on incorrect data and and things like that. So we really want to be careful about that. So that's one of the cases where we've spent a lot of effort to make sure that we do the validation before we, again, like, show it back to the user. And we're trying to do the same in all of these tools. And like I said, we're trying to make sure that we have places within the processes where we make sure where we still have that human in the loop remaining. 250 00:40:07.895 --> 00:40:22.980 I'm sure that will change long term, and there will be more of these cases where quality of the output of the agents are of high quality enough that we can remove some of those. But at the moment, like, we're still very aware of that and and and trying to be very careful about. On the cost 251 00:40:23.300 --> 00:40:27.380 side of things, we've certainly seen cases where things goes a little bit haywire. 252 00:40:27.380 --> 00:40:28.420 But generally speaking, 253 00:40:29.355 --> 00:40:40.155 in most cases, the ROI of using these tools have been very strong for us. So we're not trying to constrain people too much by putting cost guardrails 254 00:40:40.235 --> 00:40:51.990 in their face. We do have those guardrails in place behind the scene to make sure that, yeah, you know, what you said, like, in terms of some single use case going completely haywire, but we've seen very few cases of that in practice. 255 00:40:53.285 --> 00:40:54.645 Another interesting 256 00:40:54.725 --> 00:40:55.605 aspect 257 00:40:55.605 --> 00:40:56.405 of 258 00:40:56.485 --> 00:40:58.565 Spotify as an organization 259 00:40:58.645 --> 00:41:00.565 and the incorporation 260 00:41:00.565 --> 00:41:01.525 of these 261 00:41:01.685 --> 00:41:07.300 AI and ML capabilities in the engineering life cycle is that as an organization, 262 00:41:07.300 --> 00:41:13.060 you've been investing very heavily in various ML features and capabilities 263 00:41:13.060 --> 00:41:16.740 in terms of your core product. And I'm wondering how 264 00:41:16.900 --> 00:41:17.940 some of the 265 00:41:18.180 --> 00:41:23.685 expertise and lessons learned in the process process of developing that operational capacity 266 00:41:23.765 --> 00:41:25.285 has fed into 267 00:41:25.285 --> 00:41:30.325 some of your overall adoption process for these AI utilities 268 00:41:30.645 --> 00:41:32.725 broadly in the organization, 269 00:41:32.805 --> 00:41:34.645 in engineering and otherwise, 270 00:41:34.645 --> 00:41:39.520 and also some of the ways that the lessons learned through that experimentation 271 00:41:39.520 --> 00:41:43.280 within the engineering team and across the different organizational 272 00:41:43.360 --> 00:41:47.120 areas has fed back into some of either the 273 00:41:47.280 --> 00:41:53.795 operational excellence of the ML features that you provide as well as some of the maybe more generative capabilities 274 00:41:53.795 --> 00:41:56.035 that are being introduced in the product? 275 00:41:56.675 --> 00:42:09.690 Yeah. I it's gone both ways just like you say. So like like I said early on, like, we have more than a decade of experience of building machine learning products at scale, and there's been quite a few of those things that we've been able to 276 00:42:10.010 --> 00:42:23.415 bring over to to our to the productivity side of things. So one such case is, for example, how we do evals of LLM based products. So to to take a very concrete example, we have this internal chat agent 277 00:42:23.415 --> 00:42:26.855 called Aika, and it's it's all about providing 278 00:42:26.855 --> 00:42:30.214 our internal Spotify knowledge to our employees. 279 00:42:30.214 --> 00:42:45.240 And very much based on how we had experience working with our traditional machine learning consumer facing products, we reused a lot of that eval practices and infrastructure for how to build that chat agent. And the same is true for many other things like how we run experiments 280 00:42:45.640 --> 00:42:46.599 and a b testing 281 00:42:47.175 --> 00:42:56.455 for internal tools is very we're reusing the same tools that we use for our consumer facing products, for example. I think it also, to be honest, allowed us to 282 00:42:56.855 --> 00:43:45.155 not do some things. So one decision that we were faced with, I don't know, two years ago maybe was, do we need to train our own model on our code or will the rest of the world sort of catch up and be able to do that well enough without us having to make that investment? And I think that was one of the case where we had a fair amount of experience from training both our own traditional machine learning model models and transformer

based models where we could better predict the future, and we actually avoided making that investment. And I know some of our friends in the industry went down that journey of trying to do that and then had to back out of it because the tools we've been talking about today just became better than what they had internally. And then thinking about it in the other direction. So you mentioned this, but one of the one of the, I think, interesting examples there has been 283 00:43:45.795 --> 00:43:50.835 we've now started talking externally about how we're moving towards the future with more 284 00:43:51.234 --> 00:43:53.075 control for the user over 285 00:43:53.660 --> 00:44:02.780 the recommendations that you get. And that has been very much informed by, like, how we're able to prompt these types of tools that we're using every day now. 286 00:44:03.020 --> 00:44:04.140 And then imagining, 287 00:44:04.140 --> 00:44:07.335 like, what would that look like in the Spotify 288 00:44:07.335 --> 00:44:08.935 consumer experience. 289 00:44:08.935 --> 00:44:17.655 And you can imagine that many of those things that you're able to do with, let's say, Claude Coat today would be as applicable to be able to do within 290 00:44:17.735 --> 00:44:25.380 the Spotify experience. So we already like I said, we already started shipping some of those products, and you'll see us talk more about this in the future. 291 00:44:26.180 --> 00:44:28.740 Yeah. I know that as a user of Spotify, 292 00:44:28.740 --> 00:44:48.335 I was pretty happy when I saw the natural language prompt to generate a playlist and being able to experiment with that. And I think your point of being able to inform the type of recommendations that you want versus it just being a black box that I as a user get whatever comes out of it is an interesting 293 00:44:48.780 --> 00:45:04.780 way of exposing more of these AI capabilities beyond the typical you know, the the the trope that was first introduced when ChatGPT came on the scenes of, hey. Here's a chat box where you can talk to whatever, but it's not actually useful as an end user. It's just a gimmick. 294 00:45:05.795 --> 00:45:13.315 Yeah. We think there's lots and lots of opportunity in that space, so we're very excited about what that will look like for for Spotify as well. 295 00:45:14.194 --> 00:45:18.434 And as you have been introducing these capabilities, 296 00:45:18.434 --> 00:45:21.260 investing in them, building some core 297 00:45:21.420 --> 00:45:31.180 tooling around that overall life cycle? What are some of the most interesting or innovative or unexpected ways that you've seen your team bringing these agentic and AI capabilities to bear? 298 00:45:32.220 --> 00:45:41.635 Yeah. There's been many, many, many ways. We talked about a few before, like how people are using these tools now to build strategies. Like, one of the things that I 299 00:45:41.875 --> 00:45:54.170 if you would ask me a year ago that I would imagine being, like, one of the things where it would be hardest to apply these types of tools, but turns out that they're incredibly good at, like, you know, spitballing and working with you and brainstorming 300 00:45:54.170 --> 00:46:09.005 back and forth. And then in the end, you have a a product strategy that is as strong or stronger than we could produce before, for example. We talked about incident management and how we're able to apply it there. And in our our designers has been incredibly 301 00:46:09.565 --> 00:46:13.165 what's the word I'm looking for? Creative in terms of, like, how they're able to use 302 00:46:13.484 --> 00:46:19.565 we have a bunch of designers who spent time to, like, figure out how to use cursor and ended up building these 303 00:46:20.390 --> 00:46:27.590 tools for other designers for, like, how to prototype things or how to one very concrete example. If you go into 304 00:46:27.670 --> 00:46:35.444 a Spotify app and you go into our DJ feature, which is another one of these, like, more agentic features. That is like this visualization, 305 00:46:35.444 --> 00:46:58.760 and it's actually, like, driven by a bunch of parameters on on how that behaves. And we actually had a designer again, like, vibe coding something with cursor where you can go in and, like, play with those parameters and make new visualizations based on that. So we've seen ton of those examples, including, like, people using these tools for making their taxes or whatever problems they're faced with. So people are innovative around how to use these tools. 306 00:47:00.120 --> 00:47:03.355 And in your own journey of understanding 307 00:47:03.355 --> 00:47:09.915 the use cases and capabilities and best practices around how to integrate these agentic 308 00:47:09.915 --> 00:47:16.900 workflows into your engineering organization, what are some of the most interesting or unexpected or challenging lessons that you learned in the process? 309 00:47:18.020 --> 00:47:24.100 Yeah. There's been a lot of lot around what we've been covering before in terms of having better verification 310 00:47:24.180 --> 00:47:31.275 both in our in our code basis in terms of testing and so on, but also in that agentic loop. So we 311 00:47:31.275 --> 00:47:36.955 mentioned before this tool that we call honk, which is this agent that we use to manage our entire 312 00:47:37.195 --> 00:48:08.744 manage changes to our entire code base. And one of the real breakthroughs we had in developing honk was really when we added in LLM as a judge loop into that. So you can imagine, like, if you would be using Cloud Code to make some changes, but but but before those changes are shown to you as users, there's there's another sort of independent LLM that takes a look at those and takes a look at the prompt you were using and saying, like, is this a good change or not? So you're like, you have that review loop always being built in, and that dramatically 313 00:48:08.744 --> 00:48:10.585 improved the

performance 314 00:48:10.585 --> 00:48:11.145 of 315 00:48:11.385 --> 00:48:27.310 of that tool. When you're using Claude, like, interactively, that's not a big problem because if Claude does does something wonky, you will just tell it, like, change this thing. Like, I do that a 100 times every day. But when you're authoring changes that is gonna be sent out, so that's pull request to, like, a thousand different repositories, 316 00:48:27.714 --> 00:48:34.835 You don't want people to then have to iterate on those in 90% of those repositories. You want to catch those most much earlier. 317 00:48:34.915 --> 00:48:36.195 So having 318 00:48:36.275 --> 00:48:37.795 80 or 90% 319 00:48:37.954 --> 00:48:39.234 good output 320 00:48:39.234 --> 00:48:43.875 is just not enough in that scenario. You really need to get to that, like, 99% 321 00:48:43.530 --> 00:48:50.810 plus. And that was one of those cases where we were able to do that and really, like, make that jump in in quality that was transformational 322 00:48:50.810 --> 00:48:52.650 for that particular tool for us. 323 00:48:53.690 --> 00:48:56.810 And I think to one of the key things 324 00:48:57.385 --> 00:49:00.505 in the adoption of these agentic capabilities 325 00:49:00.505 --> 00:49:02.585 that we've touched on throughout 326 00:49:02.585 --> 00:49:07.385 is the challenge of a lot of these tools being very much designed around 327 00:49:07.465 --> 00:49:08.105 a 328 00:49:08.345 --> 00:49:11.320 single user interacting with it directly. 329 00:49:11.320 --> 00:49:13.400 So it's more of the single player mode, 330 00:49:13.640 --> 00:49:16.200 but it has broader ramifications 331 00:49:16.200 --> 00:49:31.455 at the team and organization level. And I'm just wondering what are some of the key ways that you have found effective in terms of being able to break out of that siloed mode of interacting with these utilities and turn it into more of a collaborative experience and 332 00:49:31.535 --> 00:49:35.695 making sure that some of those useful patterns get 333 00:49:35.775 --> 00:49:36.735 captured 334 00:49:36.895 --> 00:49:49.180 without necessarily the engineer having to click the button to say, yes. This is the right thing or just maybe figuring out what are some of those implicit signals around, oh, this is a useful prompting style. This one let us down a dead end, things like that. 335 00:49:49.740 --> 00:49:53.420 Yeah. I'm gonna say that I think we're still very early in that journey, 336 00:49:53.994 --> 00:49:56.795 both as an industry and and and at Spotify. 337 00:49:57.435 --> 00:50:00.795 Today, like, the patterns that we're able to use for that are 338 00:50:00.875 --> 00:50:03.195 I I find, like, still pretty manual 339 00:50:03.195 --> 00:50:15.980 in many ways, so making sure that we update the skills or, like, what MD calls files or whatever that we I would love to be in a in a place in the in the future where just like you said, like, these tools are much more collaborative. 340 00:50:15.980 --> 00:50:31.345 So if you and I are are working in in one of these agents, like, the things that you do and, like, the the agent can learn from that and then reuse that when I work on the same code base, for example, so that we build out that build up that collective 341 00:50:31.505 --> 00:50:35.025 memory in the agent without that having to be in this very 342 00:50:35.570 --> 00:50:44.530 explicit way that we'd what that we do today. I very much imagine that this is something that we'll see coming in the tools in the in the in the near term future. 343 00:50:44.850 --> 00:50:52.145 I'm very excited about that because I think that's gonna make often often, we talk about these agents as being like another 344 00:50:52.225 --> 00:51:00.625 human in the team, and that's not what they feel like today to me. Like, they're incredibly powerful. Don't get me wrong, but they don't get that collective learning 345 00:51:00.785 --> 00:51:04.940 unless we encode that into into skills or whatever. So 346 00:51:05.339 --> 00:51:19.345 I that seems to me like one of the things that will be coming. I know that there's projects out there where people where people are experimenting with with exactly those things today, and and I imagine a lot of that will be coming in the near term future. I think it will be pretty powerful. 347 00:51:20.705 --> 00:51:21.425 And 348 00:51:21.585 --> 00:51:29.745 when you're working with your engineering team and you are tackling a given problem, what are the situations where you decide 349 00:51:29.825 --> 00:51:35.650 that a human is the better operator and you issue agentic utilities entirely? 350 00:51:36.690 --> 00:51:41.250 I don't know if there's gonna be a lot of cases where we completely not use 351 00:51:41.490 --> 00:51:59.615 agents. There's definitely gonna continue being cases where the human will be in the loop and pretty closely so. Again, like I mentioned, code reviews before, which we think is are those one of those important things. Other things will be, like, working with working with our production data and and things like that, but that is not something that, like 352 00:52:00.300 --> 00:52:06.300 much of that has been automated since many years ago prior to to AI, so it's not a very frequent 353 00:52:06.540 --> 00:52:23.625 task that we do. So I I don't think there's gonna be a ton of cases where we don't use agents at all. I think there's gonna be cases where we still want the human to be in the loop. And And then there's gonna be cases that I think increasingly over time, we feel more and more comfortable, more or less fully outsourcing to to agents. 354 00:52:25.545 --> 00:52:27.785 And as you continue to 355 00:52:28.025 --> 00:52:38.190 build the Spotify product, work with your engineers, grow your capabilities around these agentic utilities, and make investments in leveling up the broad 356 00:52:38.270 --> 00:52:42.270 capacity for bringing AI to bear on your

engineering challenges, 357 00:52:42.675 --> 00:52:48.115 what are some of the things you have planned for the near to medium term or any particular projects that you're excited to explore? 358 00:52:49.395 --> 00:53:01.090 So I I really want as I'm as we we were talking about before, I want to broaden the set of use cases where we get that, like, deep genetic impact that we have for coding today. I would love to for that to be deeply 359 00:53:01.170 --> 00:53:04.290 integrated into how we manage incidents because, 360 00:53:04.450 --> 00:53:27.000 in particular, I think that's gonna lower the time to recovery for for those incidents, which is gonna be important from user experience point of view. I think it's like I said before, it's gonna integrate a lot of the work that we do within our product development process. So, like, bring together more of our disciplines, working more closely together. So as a PM, I can visualize more clearly through prototyping 361 00:53:27.000 --> 00:53:45.735 what I what I want us to build, and then as developer, I can pick up from there. Like, those types of things, I'm I'm pretty excited about. And really to zoom out and look at that full product development workflow rather than, like, only be looking at the coding tasks where we've been primarily focused historically here. So I'm 362 00:53:45.815 --> 00:53:54.780 super excited about that world. And, of course, there are a million things we we need to get right on the foundational layer that we talked about before, and a lot of that we're working on at the moment. 363 00:53:55.579 --> 00:54:03.660 Are there any other aspects of your work that you're doing in the investment for your engineering teams and 364 00:54:03.660 --> 00:54:11.555 sharing best practices and investing in platform capabilities for agentic coding that we didn't discuss yet that you'd like to cover before we close out the show? 365 00:54:12.515 --> 00:54:17.395 I don't think so. This was a broad and and fun discussion. We went into many, many different areas. 366 00:54:17.955 --> 00:54:24.119 There are surely other things that we're doing, but I think we covered most of the most of the ground on the interesting stuff we've been focused on. 367 00:54:24.680 --> 00:54:41.155 Well, anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest gaps in the tooling or technology or human training that's available for AI systems today? 368 00:54:42.115 --> 00:54:52.470 That's a big question. But so so but continuing within the scope that we've been talking about here today, like, I would love to see these tools getting broader into 369 00:54:52.550 --> 00:54:57.670 more of the that full, like, development life cycle and product life cycle. 370 00:54:57.910 --> 00:55:09.515 Even though we're able to use them for many of those purposes today, I think there are still a fair amount of gaps that would make them much more impact within companies like Spotify going forward. So and I a 100% think 371 00:55:09.675 --> 00:55:14.795 100% believe that that's gonna be happening. I would just love to have that today rather than tomorrow. 372 00:55:15.595 --> 00:55:28.210 Alright. Well, thank you very much for taking the time today to join me and sharing the work that you're doing at Spotify to make sure that everybody is able to take advantage of these new and exciting capabilities 373 00:55:28.210 --> 00:55:42.175 and all of the investment that you're putting into that and some of the lessons learned. It's definitely a very interesting problem space that all of us are trying to come to grips with, so I appreciate the time and effort you're putting into that. And I hope you enjoy the rest of your day. Thanks. Thanks for having me. This was great fun. 374 00:55:47.820 --> 00:55:59.900 Thank you for listening. Don't forget to check out our other shows. The data engineering podcast covers the latest on modern data management, and podcast dot in it covers the Python language, its community, and the innovative ways it is being 375 00:56:00.155 --> 00:56:10.828 Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email hosts@aiengineeringpodcast.com 376 00:56:10.828 --> 00:56:11.548 with your story.