The Future of Dev Experience: Spotify’s Playbook for Organization‑Scale AI

WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 01/20/2026 00:19:38
Duration: 3377.294
Channels: 1
1
00:00:05.120 --> 00:00:08.000
Hello, and welcome to the AI Engineering Podcast,
2
00:00:08.000 --> 00:00:12.160
your guide to the fast moving world of building scalable and maintainable AI systems.
3
00:00:19.685 --> 00:00:24.485
Unlock the full potential of your AI workloads with a seamless and composable data infrastructure.
4
00:00:25.060 --> 00:00:32.739
Bruin is an open source framework that streamlines integration from the command line, allowing you to focus on what matters most, building intelligent systems.
5
00:00:33.300 --> 00:00:41.255
Write Python code for your business logic and let Bruin handle the heavy lifting of data movement, lineage tracking, data quality monitoring, and governance enforcement.
6
00:00:41.735 --> 00:00:48.935
With native support for ML and AI workloads, Bruin empowers data teams to deliver faster, more reliable, and scalable AI solutions.
7
00:00:49.495 --> 00:00:55.255
Harness Bruin's connectors for hundreds of platforms, including popular machine learning frameworks like TensorFlow and PyTorch.
8
00:00:55.910 --> 00:01:00.230
Build end to end AI workflows that integrate seamlessly with your existing tech stack.
9
00:01:00.550 --> 00:01:05.510
Join the ranks of forward thinking organizations that are revolutionizing their data engineering with Bruin.
10
00:01:05.750 --> 00:01:09.575
Get started today at aiengineeringpodcast.com/bruin.
11
00:01:09.575 --> 00:01:14.135
And for dbt cloud customers, enjoy a $1,000 credit to migrate to Bruin Cloud.
12
00:01:14.455 --> 00:01:23.415
Your host is Tobias Maci, and today I'm interviewing Niklas Gustavsson about how Spotify is scaling AI usage and engineering and product work. So, Niklas, can you start by introducing yourself?
13
00:01:24.579 --> 00:01:33.539
Yes. Thanks for having me. Yes. I'm Niklas. I'm a chief architect at Spotify. I've been here for, I guess, coming up on fifteen years in in the summer, so quite a while. And
14
00:01:33.780 --> 00:01:42.994
one of the areas that I've been focused on for the last few years is very much around our developer experience and how we leverage AI in our development workflows.
15
00:01:43.875 --> 00:01:47.314
And do you remember how you first got started working in the ML and AI space?
16
00:01:48.290 --> 00:01:51.490
Yeah. My background is actually not in in ML. I
17
00:01:52.050 --> 00:02:12.415
my primary background before Spotify was in back end and the distributor systems and and things like that. But, of course, as I joined Spotify and we started heavily investing into machine learning, it's more than a decade now. It's always been close to what we do. It's deeply integrated into all our product development and so on. So it's always always been very close to me.
18
00:02:12.735 --> 00:02:28.720
But then, of course, as transformers and LLMs came about a few years ago, that got much closer to everything we do. So that's where I where I really got deep into our AI strategy and and has ever since been spending a lot of my time on that.
19
00:02:29.840 --> 00:02:31.600
And digging into
20
00:02:31.760 --> 00:02:38.375
the main topic here before we get too deep in the weeds, can you just start by giving a bit of an overview about the
21
00:02:38.535 --> 00:02:39.335
general
22
00:02:39.335 --> 00:02:40.295
engineering
23
00:02:40.295 --> 00:02:45.095
structure and some of the developer experience investments that you've made independent
24
00:02:45.095 --> 00:02:47.015
of any AI tooling,
25
00:02:47.015 --> 00:02:50.440
you know, maybe in the years before GPT came onto the scenes?
26
00:02:51.079 --> 00:02:57.400
Yeah. Absolutely. So it it might be good to know as I think we'll probably come back to some of this as well. Spotify,
27
00:02:57.480 --> 00:02:59.960
we run a highly distributed
28
00:03:00.120 --> 00:03:01.079
architecture.
29
00:03:01.079 --> 00:03:01.560
So we have
30
00:03:02.345 --> 00:03:08.584
many thousands of production components that might be, you know, back end services, mobile components,
31
00:03:08.584 --> 00:03:09.545
machine learning,
32
00:03:09.944 --> 00:03:12.265
pipelines, and models, and so on. And
33
00:03:12.504 --> 00:03:19.740
these are designed and owned and operated by a fairly large number of development teams,
34
00:03:20.140 --> 00:03:22.300
something like 800 development teams.
35
00:03:22.620 --> 00:03:25.340
So in a sense, like, it's this, like, very
36
00:03:25.580 --> 00:03:37.105
there's both both a lot of intention about our our architecture and the way that we work, but it's also highly distributed. So there's some level of, you know, chaos to it, which is both fun and challenging.
37
00:03:37.185 --> 00:03:39.505
It is a very collaborative
38
00:03:39.985 --> 00:03:40.785
environment.
39
00:03:40.785 --> 00:03:45.470
We do a lot of back and forth between teams. Something like 40%
40
00:03:45.470 --> 00:04:00.575
of our pull requests are from a nonowning team. So if I own a back end service, a lot of the changes to that back end service is actually gonna come from some some other team. And this is always going on in between teams. So, like, there's, you know, lots and lots and lots of collaboration.
41
00:04:00.655 --> 00:04:02.095
And a few years ago,
42
00:04:02.415 --> 00:04:08.495
we shifted a little bit. So it used to be that we were a super autonomous company. Like, every team
43
00:04:08.894 --> 00:04:09.694
had a very
44
00:04:11.040 --> 00:04:12.080
high flexibility
45
00:04:12.080 --> 00:04:15.440
around how what they built and how they built the things,
46
00:04:15.920 --> 00:04:24.640
and we just didn't see that this worked very well with the scale that we were at. So a few years ago, we started efforts around an increasing standardization
47
00:04:24.640 --> 00:04:26.400
of our code and practices
48
00:04:26.775 --> 00:04:30.855
and also made an investment into better ways of managing
49
00:04:31.014 --> 00:04:36.695
our entire fleet of software. So it used to be that every team would own their own components.
50
00:04:36.695 --> 00:04:38.055
And when we,
51
00:04:38.215 --> 00:04:40.455
let's say, needed to upgrade some
52
00:04:40.879 --> 00:05:07.735
language version or library or whatever, we would tell each team that, hey. You need to move to the latest version of Python or Java or whatever it would be. That just didn't scale very well. So we made all of these investments into instead of being able to manage our entire fleet of software as one thing that we could sort of mutate. And I'm mentioning this because this will come back in terms of where we where we're also focusing when it comes to AI at the moment. So so that's roughly the backstory for us, like a few
53
00:05:08.134 --> 00:05:11.495
or number of thousands of production components, 800 teams,
54
00:05:12.240 --> 00:05:26.800
very high scale for what we do, at least compared to my previous experiences. Like, we have millions and millions of users connected at any given point of time. We have lots and lots of traffic. So it's it's a lot of fun and always been very fast paced over here.
55
00:05:28.344 --> 00:05:29.065
And
56
00:05:29.224 --> 00:05:34.025
one of the key aspects of developer experience is that you're trying to
57
00:05:34.185 --> 00:06:00.045
keep engineers in that flow state, keep them focused on the problem at hand, and not get bogged down by all of the various roadblocks and things that are technically necessary but don't really contribute to the actual product goals. So things like making sure that your linter checks passed, make sure that your CI runs in a timely fashion, make sure that you have good test coverage. And I'm just wondering as you were investing in that developer experience,
58
00:06:00.045 --> 00:06:01.965
what are some of the main
59
00:06:02.125 --> 00:06:10.285
pain points or struggles that you were dealing with prior to the introduction of AI? And then we can get into some of the things that you're trying to use AI to solve for.
60
00:06:10.970 --> 00:06:24.570
Yeah. So I think one of the main challenges that we saw was just a fragmentation of tooling for our developers. So it used to be as as a developer at Spotify, you have to jump in between many, many different tools that
61
00:06:25.115 --> 00:06:31.915
either we had, you know, some open source tool we were using or we had built internal tooling, but in a very fragmented.
62
00:06:31.915 --> 00:06:47.600
So that was how we came about investing into what we now call Backstage that we've also made available outside of Spotify, and it's it's it's this, like, single pane of glass that developers interact with, and it's very, very present as a developer at Spotify. You do a lot of your
63
00:06:47.760 --> 00:06:53.360
tasks every day within within Backstage. And that's been I think if I would point out, like, a single
64
00:06:53.974 --> 00:07:00.855
productivity unblock that we've done maybe prior to AI, that's gonna be the one. Like, that that was the one that really
65
00:07:00.935 --> 00:07:16.070
simplified the life of a developer. Now I can go in within in Backstage. I can look at any of our thousands of software components, and I can see all the actions that I can take on them and all the data and information that is available around them. And that was, for us, very, very different from
66
00:07:16.470 --> 00:07:23.895
the fragmented state we were in before that. And I think this is true for many other companies as well, but it's been a particularly particular focus for us.
67
00:07:25.015 --> 00:07:27.015
And now digging into
68
00:07:27.095 --> 00:07:31.335
the introduction of AI in the engineering workflow,
69
00:07:31.335 --> 00:07:32.535
there are various
70
00:07:32.870 --> 00:07:39.030
shapes that that can take. There is a never ending set of tools with new ones being introduced every day.
71
00:07:39.270 --> 00:07:44.150
And I'm wondering as you were starting to see some of the potential
72
00:07:44.310 --> 00:07:45.030
for
73
00:07:45.110 --> 00:07:51.405
using AI to accelerate software development, what was the overall process for determining
74
00:07:51.645 --> 00:07:56.285
how and where and why to apply some of those technologies?
75
00:07:56.285 --> 00:08:06.930
Was it more of a bottom up and organic growth situation? Was it more of a top down mandate or, you know, maybe some hybrid structure? I'm just wondering kind of some of the ways that you thought through that overall
76
00:08:06.930 --> 00:08:08.370
process and investment.
77
00:08:09.250 --> 00:08:19.625
It has been, I'm gonna say, largely a bottoms up driver within Fortify. We started out with a very strong internal demand for Copilot
78
00:08:19.625 --> 00:08:40.100
back in the days, and we went through the journey of being able to clear that so we could use it. It was a bit of a early days, so there was a lot of legal questions and security questions and things like that that we had to figure out, but we were pretty early on on jumping on Copilot, and then the journey has just continued from there. Like, people we had a incredibly
79
00:08:40.100 --> 00:08:46.935
fast adoption of of Copilot within the company, and then Cursor came along and then CloudCode
80
00:08:47.334 --> 00:08:56.134
came along, and we've continued to seeing these, like, extremely rapid adoption, not like, nothing I've seen for any other tool that we've introduced. So there's been a incredible
81
00:08:56.134 --> 00:09:07.870
bottoms up demand. We are, of course, trying to balance that with our being somewhat intentional and strategic about this. But I'm gonna say we have gone very heavy on
82
00:09:08.030 --> 00:09:09.070
experimentation
83
00:09:09.070 --> 00:09:30.805
and sharing learnings and being pretty humble about that. We don't know exactly what's gonna happen tomorrow to your point of new tools being released all the time. So we've been pretty open and allowing for a pretty broad set of tools and then encouraging our engineers to experiment and share as much as they can. And we have some very, very active internal
84
00:09:30.900 --> 00:09:32.660
Slack channels and
85
00:09:33.140 --> 00:09:44.500
repositories with Claude skills and things like that where people are experimenting every day. So it's it is amazing to see at the moment, and I imagine that this will continue. Like, if the pace has only been increasing.
86
00:09:46.225 --> 00:09:56.225
And to that point of the rapid pace of change, the rapid introduction of new tools, the evolution of existing tools, particularly from the perspective of
87
00:09:56.625 --> 00:10:02.589
developer experience and investing in the aggregate velocity of the organization,
88
00:10:03.149 --> 00:10:04.510
that variance
89
00:10:04.589 --> 00:10:12.350
can allow some people to move faster, but maybe at the expense of the broader organizational velocity,
90
00:10:13.065 --> 00:10:14.585
whereas standardization,
91
00:10:14.585 --> 00:10:22.425
such as what you were mentioning with the backstage investment of this is the way that you get to the thing, can make sure that everybody is moving
92
00:10:22.585 --> 00:10:33.630
in the same direction and at, you know, roughly the same multiplier of pace. And I'm wondering how you're thinking about some of that tension of wanting to standardize
93
00:10:33.710 --> 00:10:38.030
the tool chain builds around some of those capabilities
94
00:10:38.030 --> 00:10:52.095
and share those best practices while at the same time enabling people to do that pioneering exploration of figuring out what are the capabilities, what are the boundaries, and how do we determine what best practices to bring to the rest of the organization?
95
00:10:52.735 --> 00:11:03.340
Yeah. It's it's a great question. I think that's that's putting the the point very clearly on some of the trade offs that we were having to make. So much like you described, like, we are betting on standardizing
96
00:11:03.340 --> 00:11:17.675
a lot of our underlying foundational pieces. So that will be the the infrastructure that we have in place, like the tools that we make available to our coding agents and so on and on the code level. So as I mentioned before, we've been on a multiyear
97
00:11:17.755 --> 00:11:26.160
journey to drive that standardization, and we're continuing that now and accelerating that. And we're doing that through various means, like, where you one of the
98
00:11:26.720 --> 00:11:45.975
investments that we we currently have ongoing is to move all our code into mono repos, which allows us to have a much, much higher degree of standardization than we have before, like, where our code lived in thousands of small repositories owned by each and every team. So we were kind of lucky to make these, like, standardization,
99
00:11:46.135 --> 00:11:57.120
the fleet management, and the monorepo investments as AI happened to us. But so we're, in that sense, like, feel that we're relatively well set up, but it is also moving
100
00:11:57.360 --> 00:12:01.120
incredibly fast. So we'll we're we're trying to adjust as we go along.
101
00:12:02.160 --> 00:12:12.165
And to your point of some of the tools that are available to the agents, some of the coding standards that can be enforced using things like linters, CICD,
102
00:12:12.165 --> 00:12:16.645
etcetera, that is definitely one of those core engineering disciplines.
103
00:12:16.965 --> 00:12:20.720
And I'm wondering what are some of the other foundational
104
00:12:20.800 --> 00:12:27.760
kind of first principles that are effective regardless of the specific harness that you're using for a given LLM
105
00:12:27.920 --> 00:12:31.520
and some of the ways that you are investing in
106
00:12:31.745 --> 00:12:44.384
that kind of platform layer of enabling the agents to be able to get the context, understand the understand the tasks, and maybe just some of the ways that the introduction of those agentic coding capabilities
107
00:12:44.384 --> 00:12:50.520
even shifts the focus of the people who are guiding and, responsible for that work.
108
00:12:51.240 --> 00:13:16.230
Yeah. It it comes in many shapes and forms. So one that you mentioned in there was we need to guide those agents. So we're spending quite a bit of time at the moment to make sure that we have the right skills and and and rules set up for the agents so they are able to to follow those and that we have the right where we needed the right sandboxing as well. So we're doing it in a secure way. We're also investing
109
00:13:16.230 --> 00:13:18.390
into improving the verification
110
00:13:18.390 --> 00:13:43.565
loop. So one thing that we've seen is that the better testing we have in place, the better both agents and, in particular, these, like, fleet wide agents that we haven't really talked about, but I'll come back to work. So let let me dive down that rabbit hole a little bit. So I mentioned before that we made this investment a few years ago on on what we call fleet management, which is all about, like, managing our entire fleet of software as one thing that we can make changes to.
111
00:13:43.964 --> 00:14:25.855
The challenge with that was that the type of changes that we could make were fairly simplistic because as it turns out, code is very complex to change. Like, it has a very wide API surface in in a way. And so we could make code changes, but they were fairly simplistic and shallow in a way. So one thing that has fundamentally changed now is that we can use AI agents to make those changes for us and then orchestrate and roll those out across our fleet. And that means that we can significantly raise the bar for how complex changes we can make to our fleet. But that means that we really need to have a good test automation, a way to verify that those changes are correct and safe.
112
00:14:26.015 --> 00:14:51.790
Again, luckily, we did some of those improvements as we started rolling out a fleet management stuff a bunch of years ago, but we think that we need to improve that further now. So that is one of the investments we're doing. We're both revisiting our practices around how we do tests and also our test coverage in the fleet. Luckily, agents are actually an excellent tool to also write some of those tests for us, so we think we can get some benefits from that. So those are some examples of these, like, more found foundational
113
00:14:52.515 --> 00:15:04.195
capabilities that we're building out, but there's lots more of those. Like, we built out an MCP infrastructure internally as well, for example, to make sure that we can provide many different MCP tools and do that in a in a secure way.
114
00:15:05.570 --> 00:15:09.810
Your comment on fleet management also brings up an interesting
115
00:15:09.970 --> 00:15:25.155
nuance to the overall engineering challenges that there are different styles of engineering within an organization where you have your application engineers who are focused on writing the features, making sure that the end user experience is fulfilling the overall organizational
116
00:15:25.155 --> 00:15:26.275
product goals.
117
00:15:26.515 --> 00:15:29.315
And then you also have data engineers, ML
118
00:15:30.360 --> 00:15:31.320
operations engineers,
119
00:15:31.320 --> 00:15:40.440
platform engineers, and all of those roles are going to have some set of code that they are interacting with that they're responsible for that they're going to be managing.
120
00:15:40.760 --> 00:15:46.440
But all of that also has to be deployed and maintained and monitored in the production
121
00:15:46.305 --> 00:15:54.145
environments, which is its own set of engineering challenges that has its own set of agentic capabilities that are being developed across the ecosystem.
122
00:15:54.385 --> 00:16:00.464
And I'm wondering how you're seeing the adoption and application of these agentic
123
00:16:00.650 --> 00:16:01.850
capabilities
124
00:16:02.170 --> 00:16:03.050
differ
125
00:16:03.050 --> 00:16:08.410
of across those different styles of engineering or domains of responsibility.
126
00:16:09.370 --> 00:16:39.260
So I agree with the sort of classification of different disciplines that you outlined, but I think they've started to meld together a little bit both as we've been, again, like, doing these investments in in thinking about our entire fleet. So it used to be that as a developer, you were at Spotify, you were very close to the code that you owned or your team owned. But now we're more in a mode of, like, yeah, you still own some piece of or some number of components within your team, but many other teams, in particular, are more platform teams.
127
00:16:39.420 --> 00:16:45.100
We'll also make changes to that code. So code has gone from being this thing that, like, I own to
128
00:16:45.255 --> 00:16:59.890
Spotify owns and in in a sense, like, everyone can make changes to them. There are still access rules and and ownership around them, but but the principle has changed quite a bit here. So so I feel that some of those like, the the distinction between a
129
00:17:00.529 --> 00:17:15.275
what we would call a feature engineer, like the app engineer and a platform engineer used to be very distinct, but it's much less so today. That being said, I I still think that the way that we apply AI tooling into this, there's sorry. Lots of similarities,
130
00:17:15.275 --> 00:17:24.315
but also a few differences. So, again, the typical feature engineer, like, working on one of our mobile apps or, let's say, on our playlist systems,
131
00:17:24.395 --> 00:17:33.030
they will be fairly focused on those systems and, like, building new features and rolling those out to our users and and then operating those serve those
132
00:17:33.350 --> 00:17:34.230
components.
133
00:17:34.950 --> 00:17:38.790
And that tends to be that then you're focused on, like, a smallish
134
00:17:38.790 --> 00:17:55.895
set of code and you're very, like, deep expertise on that particular domain. And that me means that you will use AI agents typically, like, I would say in the sort of mode that they were primarily developed towards, like what most engineers would do. And then if you look at our platform engineers,
135
00:17:56.210 --> 00:18:01.090
they are very focused around, again, like, managing our entire fleet of software.
136
00:18:01.250 --> 00:18:04.050
So they will then use more scheduled
137
00:18:04.050 --> 00:19:14.215
jobs and and things like that. So one of the things that we built out as part of our AI agents is something we call home. It's just kind of silly name, but it's it's really way for us to and, again, using this management infrastructure that I I mentioned before, really way to take, like, a AI agent in currently clawed code under the hood to apply that to our entire, like, millions and millions of lines of of software. And that's been incredibly powerful in terms of being able to drive changes to our software and, you know, do code mononize modernization and things like that. So and that's really something that, like, I'm gonna say mostly our platform engineer mostly, but not exclusively our platform engineers are are focused on. So under the hood, they're using the same tools and the same many of the same practices. It's the same way of prompting. It's the same way of working with agentic tools and whatnot, but the focus for their efforts tend to be a little bit different. So that's the main distinction that we have. We don't have you mentioned, like, operations engineers. We don't have separate operations engineers. Like, it's the same teams that build and own our software that also operates it. So we don't have some of those categories that you mentioned. But I would say that's the big distinction that we have.
138
00:19:15.335 --> 00:19:17.335
And then in terms of the
139
00:19:17.735 --> 00:19:21.809
shape of the engineering work, I'm wondering how you're seeing
140
00:19:22.050 --> 00:19:22.770
the
141
00:19:22.930 --> 00:19:23.890
distribution
142
00:19:23.890 --> 00:19:24.610
of
143
00:19:24.930 --> 00:19:34.284
work change as far as what stage of the software life cycle more of the time goes into now where for most of the history of writing software,
144
00:19:34.365 --> 00:19:38.044
you had maybe a product manager or
145
00:19:38.125 --> 00:19:56.830
an architect who would be responsible for saying, this is how everything is going to fit together. These are the features that the end users are going to be delivered, and then I'm going to write all of the tickets, and then I'm gonna hand that off to an engineer who's going to write all the code and test it and make sure that it gets deployed effectively and monitored.
146
00:19:56.990 --> 00:19:59.710
And with agentic coding
147
00:19:59.790 --> 00:20:02.830
in the mix, it changes that equation a bit where
148
00:20:03.225 --> 00:20:17.465
the writing of the code part can sometimes move from taking weeks down to minutes depending on exactly what you're doing. But making sure that it's generating the right code is really the challenge now, and I'm just wondering how you're seeing that change in dynamics
149
00:20:17.465 --> 00:20:30.740
shift the time investments where maybe rather than it being that middle point that takes up most of the work of actually writing the different lines of code, you spend more time instead making sure that you understand the problem and the specification.
150
00:20:31.385 --> 00:20:47.720
And then it also puts a higher burden on the review and deployment stage because you need to make sure that the code that was generated is actually accurate and correct in doing what it was designed to do and just how that's changing the dynamics of the engineering org at Spotify?
151
00:20:48.360 --> 00:21:25.945
Yeah. It's it's a great question, and many of the things you mentioned there really rings through. So so let let me talk through some of them. So first of all, I I I think our product development process always been a little bit more collaborative than what you described. So it hasn't been those, like, clear handovers of, like, the product manager writes down a a spec and hands it over to a developer. That's always been very iterative within Spotify, but we do have those roles. We have, like, product managers that will think deeply about our product strategy and then engineers to your point who will then design and build and operate the the software to to manage that. And, yeah, and and like you said, like, that part of turning
152
00:21:26.024 --> 00:21:48.620
those ideas into code, is obviously dramatically changing now. Like, it it is just like you said going from potentially weeks to to hours, and and the work for the work for for us and me as a developer has changed dramatically as well. Like, I used to be someone that loved the, you know, hands on keyboard, solving problems through code type of challenge.
153
00:21:48.780 --> 00:22:28.445
And I haven't literally haven't written a line of code now in, like, two months because all of that is being done for me with the agents that I use. And that is a very, very different way of thinking about it. And I me, personally, I really enjoyed that transition, actually. It it turned out that it wasn't that handled keyboard thing that I enjoyed. I enjoyed being able to solve problems and shipping those to users more than the the hands on keyboard. This is obviously gonna be personal and different for diff different people. But for me, it's been a a pretty fun fun experience in in so many ways. And it is shifting some of the dynamics that you mentioned. Like, it is shifting the work for us as developers more towards how do we effectively translate
154
00:22:28.445 --> 00:22:32.604
these ideas of things we want to do into input for
155
00:22:32.684 --> 00:23:07.554
the agents that we use, and then to your point, then reviewing the output from those agents. And we are leaning heavily on applying human judgment in in both of those cases, like being a making sure that we provide the right set of inputs for the agents so they do a good job together with all the stuff that we've talked about before with making sure that we have the right foundations in place. Like, the agents, just like humans, knows what code standards we expect and what levels of conventions and security and so on that we expect. And then we put a lot of accountability on the developer to then review the output of the agent,
156
00:23:07.715 --> 00:23:16.510
both in terms of me sitting in front of the agent, but also as part of our peer code review. So that is remains super important to us. But it is also
157
00:23:16.670 --> 00:23:24.255
challenging. Right? Because now we're able to produce much more code changes, and we and a lot of that happens purely,
158
00:23:24.255 --> 00:23:34.415
like, in this agent loop and much, much more quickly than before. But we still need humans to review all of those code changes. So it's putting a lot of pressure on the code review stage
159
00:23:34.655 --> 00:23:46.269
before things goes into production. And that is something that we're working on figuring out right now. So putting more we already have, but we're we need to improve the AI assistance that we provide for code reviewing,
160
00:23:46.350 --> 00:23:51.710
and we're gonna change a little bit the policies that we have around how we do code reviews so
161
00:23:52.235 --> 00:24:04.955
more people can contribute to those reviews than is the case today, for example. And then the last reflection I'll I'll I'll add in is that you mentioned the again, like, going back to those different roles that we have with, like, product managers, designers, engineers,
162
00:24:05.290 --> 00:24:24.414
the similar to what I said before in in between the different disciplines within in within engineering, we're also seeing that those boundaries between what it means to be a product manager and an engineer are loosening up as well. Like, we see a lot of PMs spending time in cursor and Claude code and, you know, building prototypes,
163
00:24:24.495 --> 00:24:29.375
contributing to our code in a completely different way that was the that was the case before.
164
00:24:29.535 --> 00:25:16.760
So I don't necessarily expect that these different disciplines will completely disappear because there are different expertise there. But these, like, handover boundaries that that you started with, like, those, I think, gonna look very, very different in the future. And I fully expect that where we have congestion points today within that process, like you mentioned, like, the code part having taken a lot of time historically and that now moving away, that's just gonna create new congestion points. So one of the things that we're preparing for is being able to detect those quickly and then being able to make decisions so we can iterate our way out of that. I think this is gonna be a big part for us in the next few years just to transition through this, and we don't have a perfect, you know, view of the future of what this is gonna look like. But yeah. I mean, everyone are using these tools today.
165
00:25:18.255 --> 00:25:21.934
We have a ton of adoption outside of the engineering
166
00:25:22.415 --> 00:25:23.375
cohort
167
00:25:23.375 --> 00:25:24.174
for
168
00:25:24.255 --> 00:25:28.414
both cursor and claw to code, and, like, everyone's trying to use them to
169
00:25:28.735 --> 00:25:46.965
for many different things that you would be surprised about. That, like, wasn't necessarily what the tools were intended to from from the beginning, but people are having a lot of good luck in in, like, using them for for other things as well. And the the world of just like the world of an engineer has has been completely
170
00:25:46.965 --> 00:25:53.205
revamped in the last few months, I expect that to happen with all disciplines within Spotify in the next year or two.
171
00:25:54.165 --> 00:26:09.360
There are a couple of different things I wanna touch on here, and I'll just note them now before we get too far down the path. But I think that one of the things I really wanna dig into is some of those other ways that you're seeing some of these agentic utilities being applied. But before we explore that,
172
00:26:09.680 --> 00:26:14.320
in terms of the validation and review stage, I'm wondering how
173
00:26:14.640 --> 00:26:17.095
this speed of code generation
174
00:26:17.175 --> 00:26:31.219
and the need for context queues and rapid feedback for the agents is changing some of the ways that you think about what languages to use, how to invest in things like type annotations,
175
00:26:31.380 --> 00:26:35.620
and just some of the overall tooling that you're investing in for linting,
176
00:26:35.620 --> 00:26:37.059
testing, validation,
177
00:26:37.220 --> 00:26:43.859
canary deploys, etcetera, just some of the ways that it's shifting your thinking about which languages and which tool chains are
178
00:26:44.019 --> 00:26:56.674
best now that you don't necessarily have to spend all of those extra keystrokes on putting in those type annotations and just because you don't have the benefit of engineers being lazy, but but also the the
179
00:26:57.075 --> 00:26:57.794
potential,
180
00:26:58.115 --> 00:27:09.039
downsides that that can have as far as not wanting to put in the extra work of, oh, well, I don't need that type annotation right now. I know what I'm trying to do, but maybe the LLM doesn't, just some of those trade offs.
181
00:27:10.080 --> 00:27:39.100
Yeah. I'm I'm gonna say that as as as a company with lots and lots and lots of code, as I mentioned before, and many, many different teams, and we have a fair I didn't touch on this before, but we also have a fairly it's very often that that, like, ownership of components will move around within the company. Like, I built something, but it's now closer to the business problem that you are trying to solve, so I will hand over those back end services or whatever it might be to you. So given this, and this has been the the case for many, many years at Spotify, and given this, we have
182
00:27:39.634 --> 00:27:41.154
always been pretty,
183
00:27:41.794 --> 00:28:05.360
pretty strict around, like, what languages do we use, what conventions do we have in place. And so a lot of these things we had to solve many years ago prior to AI taking over, and it I think it has turned out that a lot of those things that were good things at scale when you have thousands of engineers also turns out to be good for AI agents. At least that's with the current generation of those agents.
184
00:28:06.195 --> 00:28:23.930
So I'm not seeing us making it ask me again in six months and maybe maybe my view has changed. But at the moment, I'm not seeing us making a ton of changes in those, like, very foundational decisions on what languages we use or or whatnot. What we are doing is that one of the big shifts that we're going through at the moment is
185
00:28:24.090 --> 00:28:35.635
a lot of the standards that we've used like, the the way that we've used to promote the standards within Spotify has largely been guidance towards engineers. So we've had TechRadar
186
00:28:35.635 --> 00:28:36.355
and
187
00:28:36.675 --> 00:28:49.875
various documents that tells you how to build a ML pipeline within Spotify, and that's been reasonably effective, but we don't think it works very well for for agents. So one of the things we're doing now is really to
188
00:28:50.115 --> 00:28:56.249
put more of those, like, closer into our code bases rather than being guide guidelines, they're gonna be links
189
00:28:56.410 --> 00:29:02.570
instead. So that is one of the changes that we're doing at the moment, and we're shifting a lot of those things into
190
00:29:02.705 --> 00:29:10.784
into our actual code base. And this also goes hand in hand with the investments that that I mentioned before in of of, like, moving to more of a mono repo structure because
191
00:29:10.945 --> 00:29:46.914
one of the things that have been really hard for us historically has been I mentioned having thousands and thousands of production components. They lived in thousands and thousands of git repos and trying to manage that and, like, reasonably enforce our standards in in all of those tiny repositories with hundreds of teams owning them has been very hard. So one of the things one of the there's many challenges for us to move to mono repos, but one of the benefits we're getting is, like, we have this one place where we can put all of these standards in place and all of the linking rules in place. We've done this for our clients for many, many years. They've always been in in mono repos,
192
00:29:46.915 --> 00:29:54.700
but not the rest of our software. So we're sort of catching up with that now. So that gives us much better tooling for doing some of these things as well.
193
00:29:55.419 --> 00:29:59.740
And, also, one of the things that you mentioned is identifying
194
00:29:59.740 --> 00:30:03.465
what are the points where you are hitting bottlenecks in the path to delivery.
195
00:30:03.705 --> 00:30:14.665
And that's definitely an area where the overall industry has been investing time recently as in figuring out how do you identify those bottlenecks using things like DoraMetrics,
196
00:30:14.665 --> 00:30:21.040
I think is the most widely adopted one. But just wondering what are some of the ways that you are quantifying
197
00:30:21.040 --> 00:30:28.240
some of those aspects of the engineering life cycle and I in order to be able to identify those bottlenecks and hotspots.
198
00:30:29.455 --> 00:30:31.214
Yeah. So we have very,
199
00:30:31.534 --> 00:30:33.695
I'm gonna say, very good instrumentation
200
00:30:33.695 --> 00:30:35.695
of our developer workflows.
201
00:30:35.695 --> 00:30:40.975
So we capture data from our production systems and whatnot, then that goes into
202
00:30:41.450 --> 00:30:50.009
BigQuery in our case, and and you can go in there and and make queries around pretty much everything that goes into our developer workflow.
203
00:30:50.250 --> 00:31:05.605
So in that space, we're in a pretty good state to and we have lots and lots of metrics that we track, like Dora metrics, but also much more fine grained metrics that we that we catch and track all the time. So we're in a pretty good state there. Unfortunately, that is not true for the broader production
204
00:31:06.165 --> 00:31:14.710
sorry, product development process that we have. So we do have pretty significant gaps around being able to measure and quantify,
205
00:31:14.790 --> 00:31:21.750
let's say, how our PMs are working or how our designers are working. So as we're seeing these congestion points
206
00:31:22.165 --> 00:31:54.764
shifting around, I'm reasonably confident that we'll be able to see them within what our engineers do. So I mentioned before the code review. We're starting to see some of that popping up in code review, and we could clearly see that in our metrics. It's not as obvious how we're gonna see that in a quantifiable way when it comes to the broader product development process. I think we're gonna have to rely much more on, you know, user research and surveying and those types of things to to capture some of those. I I I honestly think that's gonna be a bit of a challenge for us over the next few years to to really
207
00:31:54.925 --> 00:32:09.670
detect those early enough before you know, as they become a real problem for people, then it's obviously gonna bubble up and we'll be very aware of them, but I would not love to be able to identify them earlier than that. And but that, I think, is gonna be a challenge for us in some cases.
208
00:32:10.870 --> 00:32:14.230
And so digging now into some of the
209
00:32:14.550 --> 00:32:15.430
broader,
210
00:32:15.830 --> 00:32:18.470
I guess, side effects of introducing
211
00:32:18.550 --> 00:32:26.364
agentic capabilities into the engineering life cycle. I'm curious some of the ways that you're seeing those capabilities applied
212
00:32:26.525 --> 00:32:40.389
beyond just the stage of I can generate a boatload of code as fast as I want and just some of the ways that it's leaking into other aspects of engineering work, product work, and just some of the maybe
213
00:32:40.710 --> 00:32:49.830
new code that it enables because of the fact that it is so fast and easy and also maybe some of the ways that it changes some of your patterns around documentation
214
00:32:49.830 --> 00:32:50.950
and diagramming.
215
00:32:52.085 --> 00:33:24.755
Yeah. So so let let me actually go a little bit broader than that to to start out. So one of the very eager set of users that user group that we've seen has been our our product managers that we've been talking about a little bit before here. And they are real like, many of them are really picking up these tools. And, again, like, they are using the same tools that our engineers are doing. So they're trying to figure out their ways around, like, how to how to bend the cursor and clawed clawed code, and it's, you know, many different ways of improving how we write PRDs or how we evaluate experiments
216
00:33:24.755 --> 00:33:25.475
and
217
00:33:25.715 --> 00:33:26.754
doing
218
00:33:27.794 --> 00:33:33.570
insights analysis and, like, many, many different things. And we've had a a number of internal
219
00:33:33.810 --> 00:33:35.250
tools and websites
220
00:33:35.250 --> 00:33:40.210
pop up for for that target audience of, you know, again, managing,
221
00:33:40.210 --> 00:34:05.789
like, the life of being a product manager at Spotify in in in a better way. So so that is one one way that I've been particularly impressed with some of the innovation that's come out of come out of that group. And I think we haven't talked about vibe coding yet, but I think that's been one of the things that they've really leaned into. And compared to before where they would need to go to like, they might have had some of these ideas around, like, these tools before,
222
00:34:06.029 --> 00:34:18.805
but they weren't really able to build them because they would have to go to an engineer and convince them to spend time on building out some web tool for that idea they had. And now that's, you know, a few prompts away. So we have a lot of these experimental
223
00:34:18.805 --> 00:34:24.005
tools popping up and people having ideas around how to, again, like, improve the nonengineering
224
00:34:24.005 --> 00:34:27.525
parts of of how we build products. Within engineering,
225
00:34:27.605 --> 00:34:47.815
what we've seen is, like, we see this sort of coding part of this, which we've been talking about. One of the interesting parts with coding is that it's actually a fairly small part of what engineers spend their time on. So it's roughly out of your eight days sorry. Eight hours a day, it is roughly one hour that engineers spend, like, hands on keyboard
226
00:34:47.895 --> 00:35:00.260
writing code. This seems to be fairly consistent across companies, so it's not unique for us. But, like, there's been a bunch of companies sharing that research, and it's true for us as well. Like, we're able to measure that. And then there's a broader set of development
227
00:35:00.260 --> 00:35:02.900
tasks. So this will be, you know, debugging,
228
00:35:03.220 --> 00:35:04.740
operational duties,
229
00:35:04.820 --> 00:35:12.215
code review, those types of things. That builds up to roughly three or four hours of the day, and then the rest is spent on planning,
230
00:35:12.375 --> 00:35:14.295
coordinating with other teams,
231
00:35:14.535 --> 00:35:16.135
reading, writing documentation,
232
00:35:16.135 --> 00:35:36.390
those types of things. So it is easy to, like, I think, overly focus on this, like, one hour a day and trying to make that as quickly as possible, which, like, the tools that we've been talking about today has been amazing at. But we're really even if we're 10 x ing that share of the work that engineers do to the previous points of just, like, finding new congestion points,
233
00:35:36.470 --> 00:35:57.530
we're we won't actually make Spotify a ton faster by just optimizing that. So we've been focusing on, like, what are other use cases outside of that one hour where we can apply AI early on. And it used to be that, like, if you would have if we would have talked a few months ago, you would have heard me talk about, like, the various strategies that we had for that of, like, how are we bringing AI into
234
00:35:57.770 --> 00:36:08.010
being on call or managing incidents or whatever it might be. And that is still very much something that we think about and have strategies for, but also where we've been out innovated by our employees.
235
00:36:08.170 --> 00:36:11.130
So there are now all of those tools available internally.
236
00:36:11.130 --> 00:36:56.045
People have, like, multiple tools that has that's been built for how to, for example, help people managing incidents. And we've taken one of those and made them, like, the sort of canonical tool that we now provide to the company, but other people are, like, still experimenting with other ways of doing that, which is amazing. So I think, like, we're seeing this spread out. So, like, what I imagine, again, if you would've we would've talked a few months ago, what I would imagine was, like, we had this, like, the coding cost at one hour a day, and then we would gradually spread the word the use of agents out from that because that was really what they were optimized for. I think what we're seeing in practice is that they're now being used for everything people do during their full workdays. Like, everyone will use them for I mean, I'm I use quote code to write documents today or, like, prepare for
237
00:36:56.160 --> 00:37:07.440
hiring interviews or whatever it might be. Like, it is it is something where it's become these tools have become ever present in every task that we do as as not just engineers, but more broadly as well.
238
00:37:08.575 --> 00:37:11.455
Yeah. And speaking as a technical manager,
239
00:37:11.535 --> 00:37:29.349
I was very pleased the first time that I realized that I could use things like Copilot or Cloud or whatever to take the fuzzy sketch of these are the things that I need to do and then stick them into the issue template so I could actually write comprehensive and useful issues for being able to assign to team members.
240
00:37:29.670 --> 00:37:46.635
Yeah. Yeah. It is it is amazing. And I think, again, like, the the amount of experimentation innovation that we're seeing from people just being able to take these tools, which, again, like, were built for a kind of different purpose, but being able to reuse them for every problem they're faced with has been amazing to see and continues being amazing to see.
241
00:37:47.675 --> 00:37:56.360
And one of the aspects of these agents being a double edged sword is that while they are immensely powerful and can
242
00:37:56.520 --> 00:38:00.040
allow people to move much faster, they can also amplify
243
00:38:00.040 --> 00:38:09.985
problems where if you have somebody who doesn't properly understand the problem domain or doesn't properly understand the code base, they can guide the agent in a direction that is counterproductive.
244
00:38:10.225 --> 00:38:15.505
And then the other aspect of challenge in terms of the deployment of agentic capabilities
245
00:38:15.505 --> 00:38:28.530
is the cost factor of how do you make sure that you don't accidentally end up spending the entire company's budget on a question and answer session that ultimately does not provide any actual return on investment?
246
00:38:29.490 --> 00:39:23.130
Yeah. So the the first part, I think honestly, I think the first part of that is is is the most important, and that is really to create a culture and and practices where you have that human judgment in the loop so you don't blindly trust whatever is coming out of of the agent. One good and recent example on that that we've that we've done is we have an internal tool for for essentially essentially being able to prompt our data. So I mentioned before that we have, for example, data on the the engineer the developer process, and there's a bunch of datasets for that. That can be pretty daunting if you're not super familiar with SQL or our internal datasets and whatnot. So one of the ways that we're trying to bring that capability to more folks within Spotify is is to allow you to prompt for that instead of prompt the type of problem you want to solve, and that tool will generate the SQL for you to try to answer that question. That is great, but it's also pretty prone to
247
00:39:23.450 --> 00:39:24.650
introducing errors
248
00:39:24.890 --> 00:39:31.050
into those SQL queries that will be very hard for a less experienced user to detect.
249
00:39:31.050 --> 00:40:07.575
So that's a tool where we have spent a ton of work to make sure that we have the right validation in place of those queries before we show them to the user. One of the this could otherwise lead to, you know, us taking decisions on incorrect data and and things like that. So we really want to be careful about that. So that's one of the cases where we've spent a lot of effort to make sure that we do the validation before we, again, like, show it back to the user. And we're trying to do the same in all of these tools. And like I said, we're trying to make sure that we have places within the processes where we make sure where we still have that human in the loop remaining.
250
00:40:07.895 --> 00:40:22.980
I'm sure that will change long term, and there will be more of these cases where quality of the output of the agents are of high quality enough that we can remove some of those. But at the moment, like, we're still very aware of that and and and trying to be very careful about. On the cost
251
00:40:23.300 --> 00:40:27.380
side of things, we've certainly seen cases where things goes a little bit haywire.
252
00:40:27.380 --> 00:40:28.420
But generally speaking,
253
00:40:29.355 --> 00:40:40.155
in most cases, the ROI of using these tools have been very strong for us. So we're not trying to constrain people too much by putting cost guardrails
254
00:40:40.235 --> 00:40:51.990
in their face. We do have those guardrails in place behind the scene to make sure that, yeah, you know, what you said, like, in terms of some single use case going completely haywire, but we've seen very few cases of that in practice.
255
00:40:53.285 --> 00:40:54.645
Another interesting
256
00:40:54.725 --> 00:40:55.605
aspect
257
00:40:55.605 --> 00:40:56.405
of
258
00:40:56.485 --> 00:40:58.565
Spotify as an organization
259
00:40:58.645 --> 00:41:00.565
and the incorporation
260
00:41:00.565 --> 00:41:01.525
of these
261
00:41:01.685 --> 00:41:07.300
AI and ML capabilities in the engineering life cycle is that as an organization,
262
00:41:07.300 --> 00:41:13.060
you've been investing very heavily in various ML features and capabilities
263
00:41:13.060 --> 00:41:16.740
in terms of your core product. And I'm wondering how
264
00:41:16.900 --> 00:41:17.940
some of the
265
00:41:18.180 --> 00:41:23.685
expertise and lessons learned in the process process of developing that operational capacity
266
00:41:23.765 --> 00:41:25.285
has fed into
267
00:41:25.285 --> 00:41:30.325
some of your overall adoption process for these AI utilities
268
00:41:30.645 --> 00:41:32.725
broadly in the organization,
269
00:41:32.805 --> 00:41:34.645
in engineering and otherwise,
270
00:41:34.645 --> 00:41:39.520
and also some of the ways that the lessons learned through that experimentation
271
00:41:39.520 --> 00:41:43.280
within the engineering team and across the different organizational
272
00:41:43.360 --> 00:41:47.120
areas has fed back into some of either the
273
00:41:47.280 --> 00:41:53.795
operational excellence of the ML features that you provide as well as some of the maybe more generative capabilities
274
00:41:53.795 --> 00:41:56.035
that are being introduced in the product?
275
00:41:56.675 --> 00:42:09.690
Yeah. I it's gone both ways just like you say. So like like I said early on, like, we have more than a decade of experience of building machine learning products at scale, and there's been quite a few of those things that we've been able to
276
00:42:10.010 --> 00:42:23.415
bring over to to our to the productivity side of things. So one such case is, for example, how we do evals of LLM based products. So to to take a very concrete example, we have this internal chat agent
277
00:42:23.415 --> 00:42:26.855
called Aika, and it's it's all about providing
278
00:42:26.855 --> 00:42:30.214
our internal Spotify knowledge to our employees.
279
00:42:30.214 --> 00:42:45.240
And very much based on how we had experience working with our traditional machine learning consumer facing products, we reused a lot of that eval practices and infrastructure for how to build that chat agent. And the same is true for many other things like how we run experiments
280
00:42:45.640 --> 00:42:46.599
and a b testing
281
00:42:47.175 --> 00:42:56.455
for internal tools is very we're reusing the same tools that we use for our consumer facing products, for example. I think it also, to be honest, allowed us to
282
00:42:56.855 --> 00:43:45.155
not do some things. So one decision that we were faced with, I don't know, two years ago maybe was, do we need to train our own model on our code or will the rest of the world sort of catch up and be able to do that well enough without us having to make that investment? And I think that was one of the case where we had a fair amount of experience from training both our own traditional machine learning model models and transformer based models where we could better predict the future, and we actually avoided making that investment. And I know some of our friends in the industry went down that journey of trying to do that and then had to back out of it because the tools we've been talking about today just became better than what they had internally. And then thinking about it in the other direction. So you mentioned this, but one of the one of the, I think, interesting examples there has been
283
00:43:45.795 --> 00:43:50.835
we've now started talking externally about how we're moving towards the future with more
284
00:43:51.234 --> 00:43:53.075
control for the user over
285
00:43:53.660 --> 00:44:02.780
the recommendations that you get. And that has been very much informed by, like, how we're able to prompt these types of tools that we're using every day now.
286
00:44:03.020 --> 00:44:04.140
And then imagining,
287
00:44:04.140 --> 00:44:07.335
like, what would that look like in the Spotify
288
00:44:07.335 --> 00:44:08.935
consumer experience.
289
00:44:08.935 --> 00:44:17.655
And you can imagine that many of those things that you're able to do with, let's say, Claude Coat today would be as applicable to be able to do within
290
00:44:17.735 --> 00:44:25.380
the Spotify experience. So we already like I said, we already started shipping some of those products, and you'll see us talk more about this in the future.
291
00:44:26.180 --> 00:44:28.740
Yeah. I know that as a user of Spotify,
292
00:44:28.740 --> 00:44:48.335
I was pretty happy when I saw the natural language prompt to generate a playlist and being able to experiment with that. And I think your point of being able to inform the type of recommendations that you want versus it just being a black box that I as a user get whatever comes out of it is an interesting
293
00:44:48.780 --> 00:45:04.780
way of exposing more of these AI capabilities beyond the typical you know, the the the trope that was first introduced when ChatGPT came on the scenes of, hey. Here's a chat box where you can talk to whatever, but it's not actually useful as an end user. It's just a gimmick.
294
00:45:05.795 --> 00:45:13.315
Yeah. We think there's lots and lots of opportunity in that space, so we're very excited about what that will look like for for Spotify as well.
295
00:45:14.194 --> 00:45:18.434
And as you have been introducing these capabilities,
296
00:45:18.434 --> 00:45:21.260
investing in them, building some core
297
00:45:21.420 --> 00:45:31.180
tooling around that overall life cycle? What are some of the most interesting or innovative or unexpected ways that you've seen your team bringing these agentic and AI capabilities to bear?
298
00:45:32.220 --> 00:45:41.635
Yeah. There's been many, many, many ways. We talked about a few before, like how people are using these tools now to build strategies. Like, one of the things that I
299
00:45:41.875 --> 00:45:54.170
if you would ask me a year ago that I would imagine being, like, one of the things where it would be hardest to apply these types of tools, but turns out that they're incredibly good at, like, you know, spitballing and working with you and brainstorming
300
00:45:54.170 --> 00:46:09.005
back and forth. And then in the end, you have a a product strategy that is as strong or stronger than we could produce before, for example. We talked about incident management and how we're able to apply it there. And in our our designers has been incredibly
301
00:46:09.565 --> 00:46:13.165
what's the word I'm looking for? Creative in terms of, like, how they're able to use
302
00:46:13.484 --> 00:46:19.565
we have a bunch of designers who spent time to, like, figure out how to use cursor and ended up building these
303
00:46:20.390 --> 00:46:27.590
tools for other designers for, like, how to prototype things or how to one very concrete example. If you go into
304
00:46:27.670 --> 00:46:35.444
a Spotify app and you go into our DJ feature, which is another one of these, like, more agentic features. That is like this visualization,
305
00:46:35.444 --> 00:46:58.760
and it's actually, like, driven by a bunch of parameters on on how that behaves. And we actually had a designer again, like, vibe coding something with cursor where you can go in and, like, play with those parameters and make new visualizations based on that. So we've seen ton of those examples, including, like, people using these tools for making their taxes or whatever problems they're faced with. So people are innovative around how to use these tools.
306
00:47:00.120 --> 00:47:03.355
And in your own journey of understanding
307
00:47:03.355 --> 00:47:09.915
the use cases and capabilities and best practices around how to integrate these agentic
308
00:47:09.915 --> 00:47:16.900
workflows into your engineering organization, what are some of the most interesting or unexpected or challenging lessons that you learned in the process?
309
00:47:18.020 --> 00:47:24.100
Yeah. There's been a lot of lot around what we've been covering before in terms of having better verification
310
00:47:24.180 --> 00:47:31.275
both in our in our code basis in terms of testing and so on, but also in that agentic loop. So we
311
00:47:31.275 --> 00:47:36.955
mentioned before this tool that we call honk, which is this agent that we use to manage our entire
312
00:47:37.195 --> 00:48:08.744
manage changes to our entire code base. And one of the real breakthroughs we had in developing honk was really when we added in LLM as a judge loop into that. So you can imagine, like, if you would be using Cloud Code to make some changes, but but but before those changes are shown to you as users, there's there's another sort of independent LLM that takes a look at those and takes a look at the prompt you were using and saying, like, is this a good change or not? So you're like, you have that review loop always being built in, and that dramatically
313
00:48:08.744 --> 00:48:10.585
improved the performance
314
00:48:10.585 --> 00:48:11.145
of
315
00:48:11.385 --> 00:48:27.310
of that tool. When you're using Claude, like, interactively, that's not a big problem because if Claude does does something wonky, you will just tell it, like, change this thing. Like, I do that a 100 times every day. But when you're authoring changes that is gonna be sent out, so that's pull request to, like, a thousand different repositories,
316
00:48:27.714 --> 00:48:34.835
You don't want people to then have to iterate on those in 90% of those repositories. You want to catch those most much earlier.
317
00:48:34.915 --> 00:48:36.195
So having
318
00:48:36.275 --> 00:48:37.795
80 or 90%
319
00:48:37.954 --> 00:48:39.234
good output
320
00:48:39.234 --> 00:48:43.875
is just not enough in that scenario. You really need to get to that, like, 99%
321
00:48:43.530 --> 00:48:50.810
plus. And that was one of those cases where we were able to do that and really, like, make that jump in in quality that was transformational
322
00:48:50.810 --> 00:48:52.650
for that particular tool for us.
323
00:48:53.690 --> 00:48:56.810
And I think to one of the key things
324
00:48:57.385 --> 00:49:00.505
in the adoption of these agentic capabilities
325
00:49:00.505 --> 00:49:02.585
that we've touched on throughout
326
00:49:02.585 --> 00:49:07.385
is the challenge of a lot of these tools being very much designed around
327
00:49:07.465 --> 00:49:08.105
a
328
00:49:08.345 --> 00:49:11.320
single user interacting with it directly.
329
00:49:11.320 --> 00:49:13.400
So it's more of the single player mode,
330
00:49:13.640 --> 00:49:16.200
but it has broader ramifications
331
00:49:16.200 --> 00:49:31.455
at the team and organization level. And I'm just wondering what are some of the key ways that you have found effective in terms of being able to break out of that siloed mode of interacting with these utilities and turn it into more of a collaborative experience and
332
00:49:31.535 --> 00:49:35.695
making sure that some of those useful patterns get
333
00:49:35.775 --> 00:49:36.735
captured
334
00:49:36.895 --> 00:49:49.180
without necessarily the engineer having to click the button to say, yes. This is the right thing or just maybe figuring out what are some of those implicit signals around, oh, this is a useful prompting style. This one let us down a dead end, things like that.
335
00:49:49.740 --> 00:49:53.420
Yeah. I'm gonna say that I think we're still very early in that journey,
336
00:49:53.994 --> 00:49:56.795
both as an industry and and and at Spotify.
337
00:49:57.435 --> 00:50:00.795
Today, like, the patterns that we're able to use for that are
338
00:50:00.875 --> 00:50:03.195
I I find, like, still pretty manual
339
00:50:03.195 --> 00:50:15.980
in many ways, so making sure that we update the skills or, like, what MD calls files or whatever that we I would love to be in a in a place in the in the future where just like you said, like, these tools are much more collaborative.
340
00:50:15.980 --> 00:50:31.345
So if you and I are are working in in one of these agents, like, the things that you do and, like, the the agent can learn from that and then reuse that when I work on the same code base, for example, so that we build out that build up that collective
341
00:50:31.505 --> 00:50:35.025
memory in the agent without that having to be in this very
342
00:50:35.570 --> 00:50:44.530
explicit way that we'd what that we do today. I very much imagine that this is something that we'll see coming in the tools in the in the in the near term future.
343
00:50:44.850 --> 00:50:52.145
I'm very excited about that because I think that's gonna make often often, we talk about these agents as being like another
344
00:50:52.225 --> 00:51:00.625
human in the team, and that's not what they feel like today to me. Like, they're incredibly powerful. Don't get me wrong, but they don't get that collective learning
345
00:51:00.785 --> 00:51:04.940
unless we encode that into into skills or whatever. So
346
00:51:05.339 --> 00:51:19.345
I that seems to me like one of the things that will be coming. I know that there's projects out there where people where people are experimenting with with exactly those things today, and and I imagine a lot of that will be coming in the near term future. I think it will be pretty powerful.
347
00:51:20.705 --> 00:51:21.425
And
348
00:51:21.585 --> 00:51:29.745
when you're working with your engineering team and you are tackling a given problem, what are the situations where you decide
349
00:51:29.825 --> 00:51:35.650
that a human is the better operator and you issue agentic utilities entirely?
350
00:51:36.690 --> 00:51:41.250
I don't know if there's gonna be a lot of cases where we completely not use
351
00:51:41.490 --> 00:51:59.615
agents. There's definitely gonna continue being cases where the human will be in the loop and pretty closely so. Again, like I mentioned, code reviews before, which we think is are those one of those important things. Other things will be, like, working with working with our production data and and things like that, but that is not something that, like
352
00:52:00.300 --> 00:52:06.300
much of that has been automated since many years ago prior to to AI, so it's not a very frequent
353
00:52:06.540 --> 00:52:23.625
task that we do. So I I don't think there's gonna be a ton of cases where we don't use agents at all. I think there's gonna be cases where we still want the human to be in the loop. And And then there's gonna be cases that I think increasingly over time, we feel more and more comfortable, more or less fully outsourcing to to agents.
354
00:52:25.545 --> 00:52:27.785
And as you continue to
355
00:52:28.025 --> 00:52:38.190
build the Spotify product, work with your engineers, grow your capabilities around these agentic utilities, and make investments in leveling up the broad
356
00:52:38.270 --> 00:52:42.270
capacity for bringing AI to bear on your engineering challenges,
357
00:52:42.675 --> 00:52:48.115
what are some of the things you have planned for the near to medium term or any particular projects that you're excited to explore?
358
00:52:49.395 --> 00:53:01.090
So I I really want as I'm as we we were talking about before, I want to broaden the set of use cases where we get that, like, deep genetic impact that we have for coding today. I would love to for that to be deeply
359
00:53:01.170 --> 00:53:04.290
integrated into how we manage incidents because,
360
00:53:04.450 --> 00:53:27.000
in particular, I think that's gonna lower the time to recovery for for those incidents, which is gonna be important from user experience point of view. I think it's like I said before, it's gonna integrate a lot of the work that we do within our product development process. So, like, bring together more of our disciplines, working more closely together. So as a PM, I can visualize more clearly through prototyping
361
00:53:27.000 --> 00:53:45.735
what I what I want us to build, and then as developer, I can pick up from there. Like, those types of things, I'm I'm pretty excited about. And really to zoom out and look at that full product development workflow rather than, like, only be looking at the coding tasks where we've been primarily focused historically here. So I'm
362
00:53:45.815 --> 00:53:54.780
super excited about that world. And, of course, there are a million things we we need to get right on the foundational layer that we talked about before, and a lot of that we're working on at the moment.
363
00:53:55.579 --> 00:54:03.660
Are there any other aspects of your work that you're doing in the investment for your engineering teams and
364
00:54:03.660 --> 00:54:11.555
sharing best practices and investing in platform capabilities for agentic coding that we didn't discuss yet that you'd like to cover before we close out the show?
365
00:54:12.515 --> 00:54:17.395
I don't think so. This was a broad and and fun discussion. We went into many, many different areas.
366
00:54:17.955 --> 00:54:24.119
There are surely other things that we're doing, but I think we covered most of the most of the ground on the interesting stuff we've been focused on.
367
00:54:24.680 --> 00:54:41.155
Well, anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest gaps in the tooling or technology or human training that's available for AI systems today?
368
00:54:42.115 --> 00:54:52.470
That's a big question. But so so but continuing within the scope that we've been talking about here today, like, I would love to see these tools getting broader into
369
00:54:52.550 --> 00:54:57.670
more of the that full, like, development life cycle and product life cycle.
370
00:54:57.910 --> 00:55:09.515
Even though we're able to use them for many of those purposes today, I think there are still a fair amount of gaps that would make them much more impact within companies like Spotify going forward. So and I a 100% think
371
00:55:09.675 --> 00:55:14.795
100% believe that that's gonna be happening. I would just love to have that today rather than tomorrow.
372
00:55:15.595 --> 00:55:28.210
Alright. Well, thank you very much for taking the time today to join me and sharing the work that you're doing at Spotify to make sure that everybody is able to take advantage of these new and exciting capabilities
373
00:55:28.210 --> 00:55:42.175
and all of the investment that you're putting into that and some of the lessons learned. It's definitely a very interesting problem space that all of us are trying to come to grips with, so I appreciate the time and effort you're putting into that. And I hope you enjoy the rest of your day. Thanks. Thanks for having me. This was great fun.
374
00:55:47.820 --> 00:55:59.900
Thank you for listening. Don't forget to check out our other shows. The data engineering podcast covers the latest on modern data management, and podcast dot in it covers the Python language, its community, and the innovative ways it is being
375
00:56:00.155 --> 00:56:10.828
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email hosts@aiengineeringpodcast.com
376
00:56:10.828 --> 00:56:11.548
with your story.