逐字稿 🔗
WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 7/6/2024 4:49:00 PM
Duration: 3216.801
Channels: 1
1
00:00:13.315 --> 00:00:18.055
Hello, and welcome to Podcast Dot in It, the podcast about Python and the people who make it great.
2
00:00:18.699 --> 00:00:25.439
When you're ready to launch your next app or want to try a project you hear about on the show, you'll need somewhere to deploy it. So take a look at our friends over at Linode.
3
00:00:26.345 --> 00:00:34.285
With the launch of their managed Kubernetes platform, it's easy to get started with the next generation of deployment and scaling powered by the battle tested Linode platform,
4
00:00:34.700 --> 00:00:36.800
including simple pricing, node balancers,
5
00:00:37.100 --> 00:00:38.240
40 gigabit networking,
6
00:00:38.780 --> 00:00:42.560
dedicated CPU and GPU instances, and worldwide data centers.
7
00:00:43.275 --> 00:00:44.575
Go to python podcast.com/linode,
8
00:00:46.155 --> 00:00:56.160
that's l I n o d e, today and get a $100 credit to try out a Kubernetes cluster of your own. And don't forget to thank them for their continued support of this show.
9
00:00:56.699 --> 00:00:58.559
And do you want to get better at Python?
10
00:00:59.019 --> 00:01:01.600
Now is an excellent time to take an online course.
11
00:01:02.255 --> 00:01:12.820
Whether you're just learning Python or you're looking for deep dives on topics like APIs, memory management, async and await, and more, our friends at Talk Python Training have a top notch course for you.
12
00:01:13.200 --> 00:01:23.875
If you're just getting started, be sure to check out the Python for absolute beginners course. It's like the 1st year of computer science that you never took compressed into 10 fun hours of Python coding and problem solving.
13
00:01:24.895 --> 00:01:26.195
Go to python podcast.com/talkpython
14
00:01:27.935 --> 00:01:31.460
today and get 10% off the course that will help you find your next level.
15
00:01:31.760 --> 00:01:32.500
That's pythonpodcast.com/talkpython.
16
00:01:34.800 --> 00:01:46.765
And don't forget to thank them for supporting the show. Your host as usual is Tobias Macy. And today, I'm interviewing Lynn Root about Clio, an open source pipeline for processing audio and binary data. So, Lynn, can you start by introducing yourself?
17
00:01:47.180 --> 00:01:50.320
Yeah, sure. Thanks for having me. My name is Lynn Root.
18
00:01:50.700 --> 00:01:53.040
I am a staff engineer at Spotify.
19
00:01:53.740 --> 00:01:57.305
I've been at Spotify for about a little over 7 years now.
20
00:01:57.765 --> 00:02:01.545
Past 2 years, I've been working on audio processing infrastructure.
21
00:02:02.165 --> 00:02:04.585
And do you remember how you first got introduced to Python?
22
00:02:05.710 --> 00:02:07.490
Like it was yesterday. Yes.
23
00:02:07.869 --> 00:02:12.450
So I myself have like a bachelor's degree in finance and economics,
24
00:02:12.910 --> 00:02:13.810
but I wanted
25
00:02:14.235 -->
Load full transcript
00:02:17.135
to, you know, pursue a master's degree in financial engineering.
26
00:02:18.155 --> 00:02:20.895
So in order to apply, I had to like learn how to code.
27
00:02:21.275 --> 00:02:22.895
And so I got exposed
28
00:02:23.530 --> 00:02:30.110
to Python and among other languages. And I basically said, screw finance programming is so much more fun.
29
00:02:30.410 --> 00:02:31.150
So then
30
00:02:31.795 --> 00:02:34.135
I started, you know, teaching myself Python.
31
00:02:34.754 --> 00:02:39.015
Doing so started the PyLadies chapter in San Francisco.
32
00:02:39.555 --> 00:02:41.829
This is just basically me finding
33
00:02:42.530 --> 00:02:44.790
friends to learn Python with me.
34
00:02:45.409 --> 00:02:46.709
And soon after
35
00:02:47.329 --> 00:02:48.864
I joined Spotify
36
00:02:49.165 --> 00:02:52.705
and then became known as our internal Python person,
37
00:02:53.245 --> 00:02:53.745
maintaining
38
00:02:54.685 --> 00:02:58.780
a lot of infrastructure tooling for developers to use for the language.
39
00:02:59.640 --> 00:03:04.700
Somehow got lucky enough to get stuck with a Python 2 to 3 migration road management.
40
00:03:05.000 --> 00:03:08.165
It got quickly introduced to Python, and I'm kind of stuck with it since.
41
00:03:09.105 --> 00:03:27.575
And so you mentioned that you've been responsible for a lot of the internal tooling. And I know that Spotify uses Python in a number of areas, most recently with this Clio project. But I'm wondering if you've also been involved with things like the Luigi project or some of the other interesting things that you've been tasked with aside from just the Python 2 to 3 migration.
42
00:03:28.195 --> 00:03:31.095
For a hot second, I was a maintainer for Luigi.
43
00:03:31.500 --> 00:03:37.280
But I was on a team that didn't own Luigi. So it didn't really make sense for me to maintain beyond, like, a few months.
44
00:03:37.580 --> 00:03:39.600
In terms of other, like, open source
45
00:03:40.015 --> 00:03:40.674
Python related
46
00:03:41.215 --> 00:03:42.275
things from Spotify,
47
00:03:42.974 --> 00:03:44.674
I, maybe in 2015, released
48
00:03:45.135 --> 00:03:47.715
a tool to help parse what's called RAML,
49
00:03:48.040 --> 00:03:49.820
REST API modeling language.
50
00:03:50.120 --> 00:03:52.220
Basically like YAML, but specific to
51
00:03:52.520 --> 00:03:53.820
defining a REST API.
52
00:03:54.360 --> 00:03:55.020
And then
53
00:03:55.400 --> 00:03:56.220
more recently,
54
00:03:56.815 --> 00:04:04.595
not under the Spotify name, but as a part of my job, I wrote a tool and released it called interrogate, which helps you identify
55
00:04:05.570 --> 00:04:07.510
what code is missing its documentation,
56
00:04:08.290 --> 00:04:11.430
kinda like test coverage to make sure you have documentation coverage.
57
00:04:12.290 --> 00:04:18.965
So in terms of the Clio project, can you give a bit of an overview about what it is and some of the origin story behind it?
58
00:04:19.345
--> 00:04:24.165
So so basically, Clio is Python based framework for for data processing.
59
00:04:24.610 --> 00:04:28.150
It's actually built on top of and wraps Apache Beam.
60
00:04:28.610 --> 00:04:33.270
And Apache Beam is an open source project. It actually originated in Google
61
00:04:33.575 --> 00:04:35.755
and provides a way to, I guess, define
62
00:04:36.455 --> 00:04:38.155
batch and streaming pipelines
63
00:04:38.535 --> 00:04:42.440
while being quote unquote embarrassingly parallel. That's their quote.
64
00:04:42.920 --> 00:04:48.780
So Clio is built on top of Beam, and it could be used for general data processing,
65
00:04:49.320 --> 00:04:51.515
but Clio is a bit more opinionated
66
00:04:51.895 --> 00:04:52.875
and it's actually
67
00:04:53.335 --> 00:04:53.835
particularly
68
00:04:54.215 --> 00:04:59.660
tuned to process media or I guess any sort of binary file. And so
69
00:05:00.120 --> 00:05:03.820
the generic use case is processing large files,
70
00:05:04.360 --> 00:05:05.180
but it's also
71
00:05:05.560 --> 00:05:06.380
meant to
72
00:05:06.755 --> 00:05:07.255
handle
73
00:05:07.635 --> 00:05:08.135
particularly
74
00:05:08.675 --> 00:05:10.215
resource intensive processing.
75
00:05:10.755 --> 00:05:11.575
So for instance,
76
00:05:12.035 --> 00:05:21.430
digital signal processing, where you look at the actual waveform of a media file. So you might be using packages like Librosa or Py Audio or PyDeb
77
00:05:21.810 --> 00:05:24.150
or, like, image stuff like Pill or
78
00:05:24.565 --> 00:05:29.625
pillow, or maybe some system level tools like FFmpeg or ImageMagick,
79
00:05:30.485 --> 00:05:38.720
or maybe you need to run a bunch of media files through an ML model that might be trained to do some sort of signal processing for you.
80
00:05:39.260 --> 00:05:40.000
And so
81
00:05:40.380 --> 00:05:42.720
the reason that Clio got started was
82
00:05:43.085 --> 00:05:49.745
researchers and ML engineers at Spotify were getting deeper and deeper into this audio processing
83
00:05:50.285 --> 00:05:54.169
workspace and wanted slash needed infrastructure to support
84
00:05:54.470 --> 00:05:56.090
that processing at scale.
85
00:05:56.630 --> 00:05:57.449
You can imagine
86
00:05:57.990 --> 00:06:00.250
we have a lot of analytics
87
00:06:00.745 --> 00:06:18.120
and our models that are based on the metadata of audio, like artists, genre, that kind of thing. But we also have the actual audio file itself to look at. And so I kind of want to step back a bit just to give context for those who are not familiar, because I wasn't until 2 years ago.
88
00:06:18.495 --> 00:06:30.250
Audio can be represented as like a wave. I guess if you were to go back into algebra, you could draw a little graph where the x axis is time, the y axis is amplitude.
89
00:06:30.790 --> 00:06:38.935
The waveform can be just drawn going up and down with y equals 0 at its center. And so the higher the amplitude
90
00:06:39.315 --> 00:06:41.175
or the wave, the louder it sounds.
91
00:06:41.555 --> 00:06:43.175
And then these waves can be sinusoidal,
92
00:06:43.555 --> 00:06:44.455
like smooth
93
00:06:44.819 --> 00:06:47.720
up and down or triangular or square or sawtooth.
94
00:06:48.180 --> 00:06:49.400
Then there's also frequency,
95
00:06:49.780 --> 00:06:55.645
which is how often that wave repeats in a given time. And so with audio frequency
96
00:06:56.105 --> 00:06:57.245
translates to pitch
97
00:06:57.625 --> 00:06:58.764
and where a waveform
98
00:06:59.385 --> 00:07:00.445
at a certain frequency
99
00:07:00.824 --> 00:07:05.449
translates to like a specific note, like a 4 or something like that. And so
100
00:07:05.830 --> 00:07:09.849
looking at this information can reveal some really cool things,
101
00:07:10.150 --> 00:07:12.169
something as simple as estimating
102
00:07:12.475 --> 00:07:15.215
a song's tempo, since you can see
103
00:07:15.515 --> 00:07:18.095
all the beats of a song in its wave.
104
00:07:18.555 --> 00:07:21.215
You can track its pitch from a waveform,
105
00:07:21.949 --> 00:07:34.125
providing the ability to estimate what key the song is in. You can start looking into patterns as well, like an audio that's just spoken word, like a podcast, will look different than a pop song.
106
00:07:34.585 --> 00:07:40.340
We can detect, does an audio have speech in it? And then maybe from there, detect its language.
107
00:07:40.720 --> 00:08:03.720
And this isn't necessarily anything new that Spotify is doing. So it's called, like, music informational information retrieval. I mean, it's an actual proper, like, academic subject matter for for at least a couple of decades now. But the idea of machine hearing is kind of new still, maybe a decade old, where we try to teach machines to listen to audio as humans do.
108
00:08:04.345 --> 00:08:05.005
So far,
109
00:08:05.384 --> 00:08:15.190
a couple of years ago, our researchers and engineers started to do some real novel things with this signal processing and machine learning or machine hearing. For instance, it's always been difficult
110
00:08:16.370 --> 00:08:22.355
to take a single song and separate out what's called stems, like vocals and instrumentals.
111
00:08:22.975 --> 00:08:24.595
But about 3 years ago, researchers
112
00:08:24.975 --> 00:08:29.020
essentially found a way to do it, and there's actually even a white paper about it.
113
00:08:29.419 --> 00:08:41.855
And from there, they wanted to essentially do that for all of our Spotify catalog, which as you can imagine is huge with like tens of millions of song or music files. And that's not even including podcasts.
114
00:08:42.475 --> 00:08:43.375
So these folks,
115
00:08:43.940 --> 00:08:48.519
you know, they started to slap together their own duct tape system, but quickly
116
00:08:48.899 --> 00:08:50.839
realized that they're not infrastructure
117
00:08:51.140 --> 00:08:51.640
engineers.
118
00:08:52.135 --> 00:08:54.555
They had a bit difficulty scaling
and
119
00:08:54.855 --> 00:09:01.980
the fact that they had a lot of required maintenance kind of took away a lot of time from their actual research and development.
120
00:09:02.440 --> 00:09:07.335
And so, that's essentially how Cleo was born. It's, you know, built
121
00:09:07.955 --> 00:09:09.095
by infrastructure engineers
122
00:09:09.395 --> 00:09:11.815
who should know a thing or 2 about scaling, hopefully
123
00:09:12.195 --> 00:09:14.135
about observability and reliability,
124
00:09:14.820 --> 00:09:23.399
and then just empowers researchers and other engineers or anyone interested in, I guess, in audio processing to only care about the processing logic itself.
125
00:09:24.195 --> 00:09:34.220
And you mentioned a lot of interesting problems that you're working to solve by being able to process these audio files and particularly at the scale of Spotify, where you have so many different
126
00:09:34.680 --> 00:09:38.380
assets that you need to be able to process because of music and recently podcasts.
127
00:09:39.080 --> 00:09:40.540
I'm interested in understanding
128
00:09:40.920 --> 00:09:42.300
some of the specifics
129
00:09:42.865 --> 00:09:52.880
of the problems that are faced by engineers and researchers who are trying to work with this audio and binary data as compared to some of the other types of large scale
130
00:09:53.340 --> 00:09:58.240
stream processing or data processing that is focused primarily on text or structured data?
131
00:09:58.755 --> 00:10:28.310
It's quite different, the problem space for binary versus like textual based data. It is, you know, inherently a lot quicker to crunch some analytics on audio metadata, like, you know, trying to figure out what songs to recommend to a user that the songs that are labeled in a similar genre as some other songs, and that computation, I guess, should be relatively quick. Dealing with like binary data brings a lot of resource constraints that you wouldn't, I guess, otherwise be concerned with. For instance,
132
00:10:28.690 --> 00:10:36.705
like the average song, like 3 to 4 minutes in duration might be about 8 megabytes in size, and that's the compressed size.
133
00:10:37.165 --> 00:10:50.930
Downloading that takes up network bandwidth and potentially disk space if you're sending it to disk. And compressing that audio to actually load it into memory and to be able to analyze it will then take up 40 megabytes of Ram
134
00:10:51.365 --> 00:10:59.065
processing. 40 megabytes of data itself can take maybe 1 to 2 minutes of CPU time, depending on what you're doing. Maybe even longer,
135
00:10:59.500 --> 00:11:00.240
Maybe you
136
00:11:00.540 --> 00:11:02.960
actually need GPU's to do all this processing.
137
00:11:03.580 --> 00:11:04.720
And then if you're using
138
00:11:05.100 --> 00:11:06.400
frameworks like TensorFlow,
139
00:11:06.780 --> 00:11:07.520
yet another
140
00:11:08.095 --> 00:11:09.635
heavy memory footprint,
141
00:11:10.335 --> 00:11:10.995
some libraries
142
00:11:11.695 --> 00:11:14.275
might not be well engineered for concurrency
143
00:11:14.815 --> 00:11:22.529
or not thread safe in terms of processing. So that might actually force you to write a pipeline that can only be 1 thread per process.
144
00:11:23.070 --> 00:11:31.065
A lot of these audio frameworks in general are just actually wrappers around tools implemented in other languages or have C bindings,
145
00:11:31.685 --> 00:11:33.200
maybe even need to like
146
00:11:33.680 --> 00:11:40.420
shell out to call like F of MPEG or something like that. So you need to make sure that you have your runtime
147
00:11:40.800 --> 00:11:44.605
environment set up properly with with all those system level dependencies. So
148
00:11:45.065 --> 00:11:47.805
I guess overall, it does get a bit hairy,
149
00:11:48.345 --> 00:11:59.770
but it's like a super interesting space to be in because I don't remember the last time I really had to deeply care about lower level constraints like this, especially with the advent of the cloud computing kind of thing.
150
00:12:00.305 --> 00:12:01.365
Digging more into
151
00:12:01.825 --> 00:12:14.590
the Clio architecture itself, I know that you mentioned that it's built on top of Apache Beam, which is a framework for being able to handle both batch and stream processing over large volumes of data and on arbitrary compute substrates.
152
00:12:15.370 --> 00:12:15.870
And
153
00:12:16.170 --> 00:12:18.110
particularly because of these performance
154
00:12:18.675 --> 00:12:21.495
sensitive characteristics, I'm curious what the
155
00:12:21.875 --> 00:12:29.000
decision process looked like as far as choosing the underlying framework to work with, but also the decision to use Python
156
00:12:29.300 --> 00:12:33.880
versus something that might be more close to the metal as far as the performance optimizations?
157
00:12:34.820 --> 00:12:37.880
So that's a good question. We definitely looked elsewhere first,
158
00:12:38.214 --> 00:12:40.875
like just kind of tried to understand the lay of the land.
159
00:12:41.255 --> 00:12:44.875
We didn't want to necessarily write something that already existed.
160
00:12:45.440 --> 00:12:48.100
There's certainly like a lot of audio processing
161
00:12:48.560 --> 00:12:50.660
tools out there, but that's not necessarily
162
00:12:51.040 --> 00:12:57.315
the problem. The problem here is scaling with resources, right, as you said. And so we actually kind of looked at
163
00:12:57.935 --> 00:12:59.555
data processing frameworks
164
00:13:00.095 --> 00:13:01.395
in hopes that they could
165
00:13:01.750 --> 00:13:09.050
help with, with our requirements. So we looked at, you know, beyond Beam, we looked at stuff like Argo or Airflow, even Luigi,
166
00:13:09.510 --> 00:13:19.890
some others. And, I mean, there's so many, I guess there's a new data processing framework popping up every week. But some frameworks, you know, don't support streaming or event based processing,
167
00:13:20.350 --> 00:13:21.650
only batched processing.
168
00:13:22.350 --> 00:13:23.570
Some didn't support
169
00:13:24.030 --> 00:13:31.485
being able to configure the types of hardware that you needed, like, which is very job specific since 1 might need more memory ability
170
00:13:31.865 --> 00:13:35.085
to provide these like a dockerized environment or
171
00:13:35.490 --> 00:13:46.565
specify the exact runtime requirements needed. And so like some of the problems that we're needing to solve is we did need that streaming slash event based so folks can
172
00:13:46.945 --> 00:13:50.965
process audio as soon as it's available to our catalog.
173
00:13:51.399 --> 00:13:54.620
We had users with very different like hardware requirements.
174
00:13:55.000 --> 00:13:57.420
So it had to be flexible in that regard.
175
00:13:57.800 --> 00:14:00.060
We have, you know, so many
176
00:14:00.505 --> 00:14:02.445
different frameworks that folks can use
177
00:14:02.985 --> 00:14:03.885
needed to ensure
178
00:14:04.665 --> 00:14:11.459
that like the runtime environment can, can be kind of essentially hard pin, like versions and stuff like that, because it is research.
179
00:14:11.760 --> 00:14:13.060
It needs to be reproducible.
180
00:14:13.760 --> 00:14:17.060
So again, dockerization of environment was ideal.
181
00:14:17.519 --> 00:14:18.899
We do have Shio,
182
00:14:19.375 --> 00:14:22.435
which is a Scala implementation of Beam,
183
00:14:22.815 --> 00:14:25.795
but we could have looked more deeper into that. But
184
00:14:26.255 --> 00:14:26.995
the actual
185
00:14:27.455 --> 00:14:31.860
lingua franca, I guess, of audio processing and ML is Python based.
186
00:14:32.160 --> 00:14:34.100
So researchers and ML engineers
187
00:14:34.800 --> 00:14:41.195
are far more common, I guess, are comfortable in Python than Scala or another JVM based language.
188
00:14:41.735 --> 00:14:44.875
And then we have actual tools
189
00:14:45.430 --> 00:14:46.250
developed in
190
00:14:46.630 --> 00:14:48.730
Python. There are some JVM based tooling
191
00:14:49.110 --> 00:15:04.630
for folks for like ML processing or audio processing, but they definitely do not have the reach or the development as a lot of these Python libraries do. And so that kind of sort of enforced a decision for us is that we want to provide tools that are actually,
192
00:15:05.250 --> 00:15:10.550
you know, helpful to engineers rather than forcing them to learn another language and adapt,
193
00:15:11.075 --> 00:15:12.375
you know, kind of lesser
194
00:15:12.835 --> 00:15:17.495
libraries to their needs. And I guess translate from like Python into
195
00:15:17.875 --> 00:15:20.949
a JVM by these things. So I went with Python.
196
00:15:21.490 --> 00:15:23.029
We chose Beam specifically
197
00:15:23.810 --> 00:15:28.709
because Beam is or it was already widely accepted in our infrastructure.
198
00:15:29.745 --> 00:15:30.245
Spotify's
199
00:15:31.024 --> 00:15:33.285
motto to kind of keep things,
200
00:15:34.305 --> 00:15:34.964
I guess
201
00:15:35.504 --> 00:15:36.964
not too many competing
202
00:15:37.504 --> 00:15:40.519
products to do the same thing. We want folks to
203
00:15:40.899 --> 00:15:52.584
have like less cognitive overload when, when writing their pipeline. So if they are familiar with Beam for a simple data pipeline, then they should be familiar with Beam and writing a media processing.
204
00:15:53.125 --> 00:15:54.745
So, yeah, that's how we eventually
205
00:15:55.524 --> 00:15:59.360
arrive at building on top of the Beam framework with with Python.
206
00:15:59.980 --> 00:16:13.865
You mentioned briefly some of the constraints that you were working with and the design goals of the project. And I'm wondering if you can dig a bit more into some of the alternative options that you looked at for being able to work with this volume of binary data
207
00:16:14.240 --> 00:16:17.140
and be able to fulfill the use cases that you were targeting
208
00:16:17.760 --> 00:16:25.194
and highlight some of the capabilities that were lacking in those options or some of the edge cases that they weren't able to cover that
209
00:16:25.495 --> 00:16:26.555
made it worthwhile
210
00:16:27.014 --> 00:16:30.954
to invest the time and effort into building this new system from scratch?
211
00:16:31.480 --> 00:16:42.154
It's kind of difficult to answer this question because there wasn't really competing full fledged libraries that handles that handle this, like, binary data. About a year into our development,
212
00:16:42.535 --> 00:16:49.649
I think Microsoft released a framework that actually does, like video processing, and I think it's event based.
213
00:16:50.110 --> 00:16:54.610
It supports like ML, but it's particularly targeted for like analytics
214
00:16:55.525 --> 00:16:56.025
generation,
215
00:16:56.565 --> 00:17:07.230
not necessarily, like, processing, like, heavy actual media processing itself. Also, it's closely tied to Azure, which we are a Google Cloud Platform based company.
216
00:17:07.610 --> 00:17:14.424
But as well as like, we were already a year into development with folks starting to use our product. A lot of
217
00:17:14.965 --> 00:17:23.384
the existing processing frameworks had bits and pieces of what we need, but you can't exactly cobble together a processing framework
218
00:17:23.910 --> 00:17:26.010
from other processing frameworks. Right? So
219
00:17:26.310 --> 00:17:28.410
it was better for us to essentially
220
00:17:28.790 --> 00:17:29.530
build upon
221
00:17:30.230 --> 00:17:30.730
something
222
00:17:31.355 --> 00:17:37.215
that gives us the biggest head start, I guess, which, ended up being Beam because they do support
223
00:17:37.515 --> 00:17:38.975
the ability to configure
224
00:17:39.630 --> 00:17:44.370
your resource requirements, like, you know, more memory or CPUs, etcetera.
225
00:17:45.070 --> 00:17:46.450
They do support
226
00:17:47.455 --> 00:17:52.275
running on data
flow, which is in Google Cloud. So we don't have to actually
227
00:17:52.735 --> 00:17:55.155
manage the running of the pipelines ourselves.
228
00:17:55.775 --> 00:17:56.515
It already
229
00:17:57.040 --> 00:17:57.540
has
230
00:17:58.240 --> 00:17:58.740
Python
231
00:17:59.040 --> 00:18:00.980
SDK, so you can actually
232
00:18:01.440 --> 00:18:06.825
write, you know, a vanilla Beam job that kinda looks similar to Clio, but, you know, you'd
233
00:18:07.285 --> 00:18:09.705
have to do a lot of the boiler plate that Cleo does.
234
00:18:10.085 --> 00:18:13.225
So it's, it's actually quite easy to go between the 2
235
00:18:13.685 --> 00:18:14.825
if, if need be.
236
00:18:15.320 --> 00:18:17.580
So, yeah, they had a lot of
237
00:18:18.120 --> 00:18:19.180
the basic needs.
238
00:18:19.560 --> 00:18:20.460
Yeah. They also
239
00:18:21.000 --> 00:18:30.565
ability to customize our runtime environment with, you know, whatever dependencies we need. And so the framework, there's a lot of architecture around it and tries to be a bit more opinionated.
240
00:18:31.750 --> 00:18:33.450
In terms of the actual
241
00:18:33.750 --> 00:18:53.710
design of the framework, you mentioned that Clio provides a decent amount of boilerplate for people who are looking to build these jobs on top of Beam. I'm wondering if you can talk a bit more about the design of Clio itself and some of the capabilities that you've baked into it to simplify this work of processing these large binary files and being able to
242
00:18:54.169 --> 00:18:55.870
integrate with the broader ecosystem
243
00:18:59.895 --> 00:19:00.395
stupid
244
00:19:01.015 --> 00:19:01.515
easy
245
00:19:02.135 --> 00:19:03.675
to write an audio processing pipeline. Clio, I guess, tries to do a lot of things and essentially
246
00:19:04.535 --> 00:19:05.310
make it
247
00:19:05.870 --> 00:19:08.930
stupid easy to write an audio processing pipeline.
248
00:19:09.630 --> 00:19:15.725
So when you're writing like a Clio job, you're essentially, you know, writing a VM pipeline, which should look familiar.
249
00:19:16.585 --> 00:19:18.924
But all you need to do is, is just write
250
00:19:19.304 --> 00:19:21.085
what the processing logic is.
251
00:19:21.530 --> 00:19:24.430
A user's perspective, you have like a CLI tool
252
00:19:24.890 --> 00:19:26.350
that maybe creates
253
00:19:26.890 --> 00:19:29.310
like some template files for you for a new job.
254
00:19:29.665 --> 00:19:36.005
And you just add some job specific boiler plate code that defines your processing logic in the pipeline.
255
00:19:36.385 --> 00:19:37.205
That pipeline
256
00:19:37.550 --> 00:19:43.410
itself might be a few steps. Like first download this file from this like cloud storage bucket,
257
00:19:44.030 --> 00:19:51.475
pipe that into logic that actually handles processing, like stem separation, like separate vocals from instrumentals,
258
00:19:52.000 --> 00:19:55.940
and then take that output. And now you have like 2 outputs
and, and upload
259
00:19:56.400 --> 00:19:58.500
to their respective cloud storage buckets.
260
00:19:59.039 --> 00:20:00.659
So like the CLI helps
261
00:20:00.995 --> 00:20:02.534
the management, including
262
00:20:02.914 --> 00:20:03.414
building
263
00:20:03.794 --> 00:20:17.630
a Docker image for the job, deploying a job, stopping it. What What you actually don't see is when, when you run the job locally, everything is actually running in the Docker container, not within, not on like locally the host, and this is to help with
264
00:20:18.205 --> 00:20:18.705
consistency
265
00:20:19.325 --> 00:20:19.825
environments.
266
00:20:20.445 --> 00:20:29.730
But if you step back a bit and take a look at the job as more generically, it has maybe typically 2 types of input, like an event input, which is like a trigger,
267
00:20:30.110 --> 00:20:33.250
and then another input, the actual binary file to process.
268
00:20:33.605 --> 00:20:35.785
Similarly, it has, you know, 2 outputs,
269
00:20:36.325 --> 00:20:38.184
event output and the data
270
00:20:38.565 --> 00:20:46.750
output. And so a job receives an event where the event contains a unique reference that points to a binary file.
271
00:20:47.210 --> 00:20:50.590
This is basically a trigger saying some work needs to be done,
272
00:20:50.924 --> 00:20:53.745
and then that gets mapped to
273
00:20:54.125 --> 00:21:07.160
the actual binary data lookup, some like cloud storage bucket. The job then downloads the file, does the processing, uploads the result, as I said before, to another bucket and event to its event output signaling that the
274
00:21:07.525 --> 00:21:14.105
processing has been done for that particular file. And this is kind of akin to, like, a microservice,
275
00:21:15.030 --> 00:21:18.890
back end, but instead of, like, the request response paradigm,
276
00:21:19.510 --> 00:21:21.289
it's actually events and queues.
277
00:21:21.774 --> 00:21:26.674
And so that's what, like, a single Clio job looks like from a user's perspective.
278
00:21:27.375 --> 00:21:30.595
And if you step back a little bit further, you can also
279
00:21:30.990 --> 00:21:35.250
connect jobs together based on these event inputs and outputs.
280
00:21:35.710 --> 00:21:38.450
Say for instance, I may have a colleague
281
00:21:39.195 --> 00:21:44.255
that wants to look at the vocals of a song, maybe to detect its language.
282
00:21:44.635 --> 00:21:49.490
So they can write a job that subscribes to like my job that separates vocals from instruments.
283
00:21:49.950 --> 00:21:52.850
And then you essentially have like a graph forming.
284
00:21:53.495 --> 00:22:01.675
This kind of provides a way to build upon each other's work, you know, a bit easier rather than kind of being siloed or sharing code and
285
00:22:02.190 --> 00:22:03.730
doing duplicate work essentially,
286
00:22:04.350 --> 00:22:06.610
save on compute time. And so
287
00:22:06.990 --> 00:22:08.370
say that you want to
288
00:22:08.670 --> 00:22:09.170
process
289
00:22:09.550 --> 00:22:14.785
songs as soon as a label delivers them, like a music label delivers them.
290
00:22:15.325 --> 00:22:17.585
We actually have this really cool feature
291
00:22:17.965 --> 00:22:24.240
built upon backed by Clio that allows users in Japan to, like, sing along to songs
292
00:22:24.620 --> 00:22:31.045
so that you can, you know, essentially turn off the vocals of a song and just hear the instrumentals and then see its lyrics
293
00:22:31.425 --> 00:22:35.445
timed along with the song. So, you know, have fun fun singing along.
294
00:22:35.825 --> 00:22:37.530
But it would be a bit frustrating
295
00:22:37.990 --> 00:22:43.850
to not be able to sing along to a song that was just released, just added to our catalog.
296
00:22:44.465 --> 00:22:48.085
So perhaps you want to, you know, process on demand.
297
00:22:48.545 --> 00:22:49.365
So Clio
298
00:22:49.905 --> 00:22:59.330
allows you to hook into a queue, like, pops up to listen to an event for that new song. But that event producing system doesn't necessarily
299
00:22:59.870 --> 00:23:01.890
have to be a Clio job itself,
300
00:23:02.265 --> 00:23:07.405
just so long as it has data that you need to process, like a reference to an audio file.
301
00:23:07.865 --> 00:23:09.804
And so this leads me to
302
00:23:10.270 --> 00:23:12.370
kind of like a bigger part of Clio's
303
00:23:12.670 --> 00:23:15.330
architecture is what's in that event data
304
00:23:16.270 --> 00:23:21.435
and refer to it as like a Clio message. And it's just basically a protobuf message that contains
305
00:23:22.135 --> 00:23:23.515
Clio related metadata
306
00:23:23.815 --> 00:23:30.830
as well. The pipeline that transforms actually see that the process, like a reference to a file.
307
00:23:31.450 --> 00:23:38.765
In reality, you can set up your Clio job to process any sort of data. Like, it can be just like JSON data or something like that.
308
00:23:39.065 --> 00:23:45.760
But Clio essentially uses this pointer, whatever that data is, probably an audio file, plus the job's configuration
309
00:23:46.460 --> 00:23:55.645
to find that audio file. And then it makes it easy for the user to say, okay, just go ahead and download this file. You already know where to find it. But then under the hood,
310
00:23:56.185 --> 00:23:59.405
Clio first actually checks to see if that output
311
00:23:59.705 --> 00:24:01.565
for that particular file exists.
312
00:24:01.930 --> 00:24:03.230
So say that your
313
00:24:03.530 --> 00:24:06.670
output is, you know, 2 audio files of vocals and instrumentals.
314
00:24:07.290 --> 00:24:10.590
They live in separate buckets in some cloud storage.
315
00:24:11.085 --> 00:24:25.820
Google in our case, Clio takes the file name, looks up the file name in both those bucket buckets. And if it discovers that both already exist, it doesn't actually process it unless you, you know, explicitly want it to, but it doesn't spend unnecessary resources.
316
00:24:26.440 --> 00:24:37.840
But then so maybe the output does not yet exist, so, you know, we should try and process it. But then we first need to check to see if our input data exists, if that original audio file is available.
317
00:24:38.300 --> 00:24:48.024
If it doesn't, the job can be set up to trigger its parent job to generate that input for it, its dependency input. And so these little checks
318
00:24:48.565 --> 00:24:54.105
allows us to do some interesting things with how the overall graph of jobs gets executed.
319
00:24:54.490 --> 00:24:57.309
We have, essentially 2 modes of execution
320
00:24:57.769 --> 00:25:04.645
support in what we call top down and bottom up. So with top down, every job in a graph gets triggered.
321
00:25:05.184 --> 00:25:11.044
So this is useful when you want to process audio, as soon as it's available to catalog or to dataset
322
00:25:11.570 --> 00:25:12.070
event
323
00:25:12.450 --> 00:25:16.070
triggers the Apex job or like to process a new file.
324
00:25:16.370 --> 00:25:21.110
And then all the child jobs then get triggered once that Apex job finishes
325
00:25:21.554 --> 00:25:25.894
and then their children get triggered and so on and so on. But it's not necessarily
326
00:25:26.434 --> 00:25:27.894
efficient or necessary
327
00:25:28.274 --> 00:25:32.040
for every job in a graph to process a file. Maybe
328
00:25:32.340 --> 00:25:34.040
you only want to trigger
329
00:25:34.340 --> 00:25:37.240
your job, but not any downstream jobs.
330
00:25:37.780 --> 00:25:44.585
So you trigger your own job for it to do work for a file. Maybe the input data dependencies are missing,
331
00:25:45.045 --> 00:25:47.865
meaning that its parent job hasn't produced
332
00:25:48.165 --> 00:25:50.345
the output for that particular file.
333
00:25:50.890 --> 00:25:53.470
So the parent job then gets triggered.
334
00:25:53.770 --> 00:25:57.150
It recurses all the way up as far as it needs to in the graph.
335
00:25:57.610 --> 00:26:01.335
And so basically all the data dependencies get generated,
336
00:26:02.675 --> 00:26:05.255
but not all child jobs are subsequently
337
00:26:05.635 --> 00:26:06.135
executed.
338
00:26:06.755 --> 00:26:14.440
Only the jobs that are in the direct path of the original job. So like no selling jobs and no downstream jobs are doing unnecessary
339
00:26:14.980 --> 00:26:17.960
processing. So this is essentially describing
340
00:26:18.340 --> 00:26:19.635
bottom up execution.
341
00:26:20.174 --> 00:26:21.475
It's to help optimize
342
00:26:22.414 --> 00:26:22.914
for,
343
00:26:23.215 --> 00:26:27.090
you know, costs of resource processing time because it can be expensive
344
00:26:27.390 --> 00:26:28.130
to unnecessarily
345
00:26:28.510 --> 00:26:30.770
process, you know, audio files.
346
00:26:31.390 --> 00:26:35.330
1 of the interesting pieces that I am interested in digging a bit more into
347
00:26:35.745 --> 00:26:44.325
is the ability to compose these different pipelines together.
Because I know that with a number of different frameworks for being able to build out some of these
348
00:26:44.779 --> 00:26:56.304
DAGs of tasks, it can be difficult to actually wire them together where, you know, 1 pipeline is largely self contained. But it sounds like from what you're saying that if somebody builds a particular
349
00:26:56.684 --> 00:26:57.184
workflow
350
00:26:57.485 --> 00:27:13.385
for processing some of this audio data and extracting a certain set of information from it, Somebody else can then discover that pipeline and then be able to hook in their additional processing based on what was already completed upstream or maybe even at some node in the midst of that task graph. Is that accurate?
351
00:27:15.525 --> 00:27:18.345
And then in terms of the actual
352
00:27:18.885 --> 00:27:30.915
integration with Beam, I know that particularly in things like Spark and PySpark, people complain about some of the impedance mismatch between the ways that the Spark framework thinks about objects
353
00:27:31.535 --> 00:27:38.950
and memory allocation and the way that Python is trying to do it. I'm curious if you've seen a similar impedance mismatch between the Python SDK
354
00:27:39.330 --> 00:27:41.590
and the actual Beam execution layer.
355
00:27:42.130 --> 00:27:53.165
That's an interesting question because we have seen, I guess, a few, but Python, in my opinion, is quite flexible enough to help work around that. There's still some, like, headaches,
356
00:27:53.700 --> 00:27:55.960
fairly difficult to control concurrency
357
00:27:56.660 --> 00:27:57.160
from
358
00:27:57.540 --> 00:28:07.545
the Python SDK part down to, like, the Java part. Sometimes you do need to like limit processing to only 1 thread or 2 threads per like process
359
00:28:07.925 --> 00:28:10.345
or like maybe 1 thread per core,
360
00:28:10.740 --> 00:28:13.400
because you know, you have a really memory intensive
361
00:28:14.100 --> 00:28:20.044
TensorFlow model or something that's not thread safe. So the ability to do that doesn't necessarily
362
00:28:20.585 --> 00:28:21.085
exist
363
00:28:21.705 --> 00:28:22.205
within
364
00:28:22.505 --> 00:28:25.404
at least the Apache Beam part of it,
365
00:28:25.820 --> 00:28:26.320
and
366
00:28:27.260 --> 00:28:30.480
the underlying Java framework for that, doesn't
367
00:28:30.780 --> 00:28:44.010
expose any kind of knobs that you can turn to help limit it on that end. So a lot of times we need to kind of force something like this feature of concurrency management into into Clio itself. What Clio does is try to like
368
00:28:44.630 --> 00:28:52.804
not make that a concern for its users. So you don't get, you know, that kind of complaint from users that kind of abstracts that away,
369
00:28:53.184 --> 00:28:54.325
all those kind of difficulties.
370
00:28:54.945 --> 00:29:12.195
But I do have to say that there hasn't been much of that issue between like the Java and Python SDK or the Java under implementation within the Python SDK that talks to it. It's been very kind of honestly easygoing, but no other sort of real big, insurmountable challenges
371
00:29:12.975 --> 00:29:14.435
come up due to
372
00:29:14.735 --> 00:29:16.355
differences of the 2 languages.
373
00:29:16.960 --> 00:29:34.365
And then digging more into some of the optimizations that you've built into Clio, particularly for things like the bottom up processing that you mentioned of being able to determine whether an upstream job needs to be re executed or not based on your downstream requirements. So I'm wondering if there are any other
374
00:29:34.850 --> 00:29:35.350
useful
375
00:29:35.809 --> 00:29:48.245
or interesting capabilities that you've built into Clio for being able to handle some of these particular edge cases or some of the optimizations that are necessary for working with these large files and the volume of IO that's necessary as a result?
376
00:29:48.785 --> 00:30:03.765
Yeah. There there's a handful, and we have so many ideas in development or on our roadmap. But right now, there are tools that allow users to download files, but download them to memory so you don't, you know, write to disk and take up disk space.
377
00:30:04.145 --> 00:30:10.010
And then in doing that, you then have, like, a choice on how it can be pickled. For instance,
378
00:30:10.390 --> 00:30:14.809
NumPy, a lot of like, the waveforms are represented in NumPy arrays.
379
00:30:15.315 --> 00:30:23.015
NumPy's pickling optimizes for memory where as a standard lib pickle optimizes for speed. So you do have that choice.
380
00:30:23.395 --> 00:30:24.295
It's quite nice.
381
00:30:24.759 --> 00:30:26.779
Sometimes you need to like debug
382
00:30:27.080 --> 00:30:28.539
a dag of jobs,
383
00:30:29.000 --> 00:30:32.059
so you don't necessarily want to make it do work.
384
00:30:32.385 --> 00:30:34.965
You're able to kind of debug a a day of jobs by
385
00:30:35.265 --> 00:30:38.725
sending, like, a message in ping mode. So you can kind of
386
00:30:39.185 --> 00:30:58.610
track what happened if this audio file were to trigger, you know, a certain job and you can see it go through the graph and what actually would happen if it would, you know, find output or input or if it, you know, raises an error at a certain place that you can easily find. It's a rudimentary approach to tracing, like, a back end request kind of thing.
387
00:30:59.150 --> 00:31:09.865
And then there are some kind of cool, I guess, usability features that Clio has in in my opinion. I mean, it's stuff like, you know, running unit tests, which is just handy to have and to verify
388
00:31:10.245 --> 00:31:19.210
that your configuration is is set properly and that your declared resources exist, like whatever storage bucket you have declared, but then also allows you to
389
00:31:19.510 --> 00:31:20.570
profile locally,
390
00:31:20.870 --> 00:31:22.730
your jobs like memory and CPU
391
00:31:23.030 --> 00:31:23.530
footprint.
392
00:31:24.335 --> 00:31:25.955
This is particularly helpful to
393
00:31:26.335 --> 00:31:27.875
understand the kind of resources
394
00:31:28.335 --> 00:31:30.435
your job will require once deployed.
395
00:31:30.960 --> 00:31:34.100
And then there's even like a command, an underappreciated
396
00:31:34.400 --> 00:31:34.900
command.
397
00:31:35.360 --> 00:31:43.495
We have a lot of ideas for it. We just need to develop them. It's this idea of auditing your code so you can audit your job. Right now you can audit for
398
00:31:43.955 --> 00:31:45.095
things like not
399
00:31:45.460 --> 00:31:53.400
cleaning up temporary files, and using up disk space or not being thread safe when you use certain libraries.
400
00:31:54.020 --> 00:31:54.920
It's quite handy.
401
00:31:55.514 --> 00:32:02.414
Those are all, like, the features and cool tools that we're trying to provide that to sort of help that resource intensive
402
00:32:02.875 --> 00:32:03.375
processing.
403
00:32:07.059 --> 00:32:11.400
Python has become the default language for working with data, whether as a data scientist,
404
00:32:11.735 --> 00:32:14.795
data engineer, data analyst, or machine learning engineer.
405
00:32:15.735 --> 00:32:24.540
Springboard has launched their school of data to help you get a career in the field through a comprehensive set of programs that are 100% online and tailored to fit your busy schedule.
406
00:32:25.480 --> 00:32:38.005
With a network of expert mentors who are available to coach you during weekly 1 on 1 video calls, a tuition back guarantee that means you don't pay until you get a job, resume preparation, and interview assistance, there's no reason to wait.
407
00:32:38.680 --> 00:32:45.180
Springboard is offering up to 20 scholarships of $500 towards the tuition cost exclusively to listeners of this show.
408
00:32:45.695 --> 00:32:47.075
Go to python podcast.com/springboard
409
00:32:48.735 --> 00:32:52.035
today to learn more and give your career a boost to the next level.
410
00:32:55.200 --> 00:32:59.700
In terms of the actual project itself, I'm curious what the
411
00:33:00.240 --> 00:33:00.740
motivation
412
00:33:01.120 --> 00:33:21.975
was for releasing it as open source and if there were there's any particular process involved in cleaning up the code for it to be publicly consumable or if it was the intent from the outset to be open source and if that helps to direct the overall design and architecture of the project to avoid tight coupling with the internal Spotify systems?
413
00:33:22.755 --> 00:33:23.575
From the beginning,
414
00:33:23.955 --> 00:33:26.054
Clio was certainly developed
415
00:33:26.434 --> 00:33:37.410
with the intention to open source. I mean, I've been heavily involved in Spotify's open source strategy for, like, 5 years now. So it wasn't really necessarily a question, it was kind of like the default approach.
416
00:33:37.945 --> 00:33:41.725
And we did get some hints early on that open sourcing
417
00:33:42.585 --> 00:33:44.765
framework like this would be well received,
418
00:33:45.220 --> 00:33:47.800
The research community or even some of our competitors,
419
00:33:48.340 --> 00:33:54.520
yeah, we first wanted to make a viable product and to have our internal folks kind of kick the tires
420
00:33:54.865 --> 00:33:57.044
and make sure it's actually it has some traction.
421
00:33:57.585 --> 00:34:01.284
And then with that, we did go through a couple of API redesigns
422
00:34:01.585 --> 00:34:04.360
and iterations and then add some more optimizations
423
00:34:05.220 --> 00:34:10.760
adding like highly requested basic features like, like batch and stuff like that. Because we were first streaming
424
00:34:11.215 --> 00:34:12.755
and still in a lot of development
425
00:34:13.215 --> 00:34:13.715
and
426
00:34:14.175 --> 00:34:22.790
we're now comfortable releasing it. You are right to point out that there was a lot of time spent kind of cleaning up code first. I am particularly a perfectionist,
427
00:34:23.250 --> 00:34:23.750
so
428
00:34:24.130 --> 00:34:36.715
I tried to have things documented and tested. And so like, I'm not embarrassed to have my code out there. And so I did do, like, maybe a couple of months kind of focused effort on actually cleaning the code base
429
00:34:37.150 --> 00:34:38.850
and as well as adding
430
00:34:39.470 --> 00:34:47.825
really good basic features like, yep, like batch or handling different input and outputs like, you know, storage or, like, local files,
431
00:34:48.365 --> 00:34:48.865
etcetera.
432
00:34:49.405 --> 00:34:53.345
So, yeah, some time was spent just focusing on that, but we
433
00:34:54.220 --> 00:34:54.720
conveniently
434
00:34:55.100 --> 00:34:58.400
timed the open source open sourcing of Clio with
435
00:34:58.780 --> 00:35:00.400
conference that we presented,
436
00:35:00.780 --> 00:35:02.640
Clio at. It's called, ISMIR,
437
00:35:03.105 --> 00:35:05.365
International Society of Music Information Retrieval.
438
00:35:05.905 --> 00:35:06.645
So we basically
439
00:35:07.025 --> 00:35:13.020
hosted a tutorial to show off Clio to see, like, what their reaction was would be. And so that
440
00:35:13.320 --> 00:35:16.860
was kind of the fire under our butts to, to get it out there. But also,
441
00:35:17.160 --> 00:35:19.740
I think we finally decided that, yes, we were comfortable.
442
00:35:20.434 --> 00:35:25.575
You know, we might be a little bit lacking in full documentation coverage or full test coverage, but
443
00:35:26.434 --> 00:35:37.934
we are also just as eager to, you know, learn from a broader range of users, not just, like, internal Spotify users and maybe use cases we haven't thought of and hopefully
444
00:35:38.234 --> 00:35:39.535
invite, you know, contributions
445
00:35:39.994 --> 00:35:44.335
and essentially invite others to help improve and and build upon Clio.
446
00:35:45.230 --> 00:36:02.225
For people who are actually using Clio for doing their own
work, I'm wondering if you can talk through some of the workflow that's involved with getting a Clio project set up and being able to handle third party dependencies that you might want and managing the deployment and life cycle of the overall project.
447
00:36:02.990 --> 00:36:03.490
Yeah.
448
00:36:04.349 --> 00:36:16.265
So building a pipeline, I guess, starts off with a very, like, simple command, like Clio job create. It, like creates, like, I guess, a cookie cutter template of a job. You just declare your dependencies in
449
00:36:16.965 --> 00:36:18.025
1 or 2 places,
450
00:36:18.619 --> 00:36:23.039
The standard like requirements dot TXT for your 3rd party Python dependencies.
451
00:36:23.500 --> 00:36:25.359
And then maybe you need
452
00:36:25.885 --> 00:36:30.705
like, system level dependencies, like, actual, like, C libraries that maybe your Python dependencies
453
00:36:31.245 --> 00:36:32.465
plug into, or
454
00:36:32.930 --> 00:36:35.590
you might actually want to shell out to something like FFmpeg.
455
00:36:36.210 --> 00:36:39.030
There, you would declare it in like the Docker file.
456
00:36:39.570 --> 00:36:41.110
And so you, you add your dependencies
457
00:36:41.490 --> 00:36:44.715
and you kind of go about your regular Python development.
458
00:36:45.255 --> 00:36:46.715
You kind of declare
459
00:36:47.015 --> 00:36:48.155
the actual processing
460
00:36:48.535 --> 00:36:49.835
that you want handled.
461
00:36:50.369 --> 00:36:53.029
Behind the scenes, Clio actually takes care of
462
00:36:53.410 --> 00:36:58.549
reading those event trigger inputs and writing, and it can handle,
463
00:36:59.009 --> 00:37:12.090
like, the data reading input and writing output for you. But essentially, all you have to worry about is, okay, you've received a trigger that has this reference ID. What should the job do now? And so maybe
464
00:37:12.630 --> 00:37:14.890
you take that file and
465
00:37:15.350 --> 00:37:17.930
run it through your model and then just return the result.
466
00:37:18.295 --> 00:37:23.595
Or maybe you tell Clio to upload the result to a bucket. And so you really only have to focus
467
00:37:24.055 --> 00:37:27.275
on the actual business logic of the pipeline.
468
00:37:27.710 --> 00:37:28.530
And then
469
00:37:28.910 --> 00:37:30.370
once that's all defined,
470
00:37:30.830 --> 00:37:32.690
you probably want to test it out locally.
471
00:37:33.390 --> 00:37:37.865
The CLI tool allows you to easily kind of get it running locally,
472
00:37:38.405 --> 00:37:43.145
see if it works. You're able to kind of publish a message to its queue
473
00:37:43.460 --> 00:37:44.839
from the CLI as well.
474
00:37:45.299 --> 00:37:48.279
And then there from there, you should be able to see some really pretty logs
475
00:37:48.819 --> 00:38:01.400
and maybe good enough to write some unit tests As I've kind of hinted before, Clio kind of will run those unit tests in the Docker image for you. So it should be the same environment
as it would be when it's deployed.
476
00:38:01.940 --> 00:38:07.934
Then maybe you're totally comfort comfortable and confident and wanna, you know, deploy it. Clio job deploy
477
00:38:08.234 --> 00:38:16.335
command will just, you know, put it out there, put it on your configured runner, which right now is either direct runner, the local runner or data flow.
478
00:38:16.720 --> 00:38:18.260
So on the Google Cloud dataflow,
479
00:38:19.040 --> 00:38:23.619
it should, like, then take up all of the heavy lifting of orchestration.
480
00:38:24.405 --> 00:38:25.785
You won't have to worry about
481
00:38:26.085 --> 00:38:32.505
spinning up all the resources needed. If you want auto scaling, you can configure that so that maybe
482
00:38:32.920 --> 00:38:38.060
all of a sudden, you have a huge load of audio files processed. It'll take care of scaling for you.
483
00:38:38.360 --> 00:38:43.155
With that, kind of comes in, like, built in metrics and observability,
484
00:38:43.695 --> 00:38:47.875
which Clio can help you kind of discover, like help you find
485
00:38:48.175 --> 00:38:48.710
those dashboards
486
00:38:57.270 --> 00:38:58.724
a show up onto
487
00:38:59.185 --> 00:38:59.765
the dashboards.
488
00:39:00.785 --> 00:39:02.565
Maybe you have like a new
489
00:39:03.105 --> 00:39:05.845
model, like a retrained model, and you want to
490
00:39:06.145 --> 00:39:06.645
redeploy
491
00:39:07.590 --> 00:39:10.010
your job with this new, like ML model,
492
00:39:10.310 --> 00:39:14.810
you then kind of like go through that process again of just like updating your code,
493
00:39:15.164 --> 00:39:19.184
kind of rebuilding the image and then Cleo job deploy will
494
00:39:19.644 --> 00:39:20.785
take care of
495
00:39:21.164 --> 00:39:23.184
taking down that old job
496
00:39:23.670 --> 00:39:24.170
without
497
00:39:24.630 --> 00:39:28.250
losing the in process work for you. Like things
498
00:39:28.790 --> 00:39:36.215
work in progress and the actual pipeline will kind of get drained, will continue, but any work still in the queue will just kind of be paused
499
00:39:36.675 --> 00:39:39.815
until the new Dataflow job is run.
500
00:39:40.150 --> 00:39:41.930
And now you have like this new
501
00:39:42.550 --> 00:39:45.050
model deployed and you kind of, you need to reprocess
502
00:39:45.430 --> 00:39:50.245
the old data that had been processed by the old model. You can then re trigger,
503
00:39:50.785 --> 00:39:54.645
like all those audio files that need reprocessing by just like a simple
504
00:39:54.945 --> 00:40:04.829
force through trigger. So Clio will say, oh, I do find this output already existing, but you told me to force it. So we're gonna go ahead and process it for you. So, yeah, that's sort of like an overview
505
00:40:05.130 --> 00:40:08.655
work through or walk through of sort of the general workflow
506
00:40:09.035 --> 00:40:10.095
of the Clio job.
507
00:40:10.395 --> 00:40:20.579
Yeah. Or integrate with it more directly. What are some of the extension or integration points that you have exposed for being able to expand its utility beyond just the core feature set?
508
00:40:20.960 --> 00:40:24.259
There are actually quite a few points of, like, extension.
509
00:40:25.105 --> 00:40:28.805
For instance, like, you would just have to write, like, a small little class
510
00:40:29.345 --> 00:40:32.005
to, like, read from or write to, like, an AWS
511
00:40:32.500 --> 00:40:38.359
service or Azure service for, like, SNS or s 3 or q storage or whatever.
512
00:40:38.900 --> 00:40:40.440
Similarly, we have, like,
513
00:40:41.155 --> 00:40:42.135
support for metrics,
514
00:40:42.835 --> 00:40:43.655
only for,
515
00:40:43.955 --> 00:40:46.935
like, log based metrics and and Google Stackdriver,
516
00:40:47.235 --> 00:40:48.215
but it's extremely
517
00:40:48.515 --> 00:40:49.335
easy to
518
00:40:49.690 --> 00:40:53.150
extend the metrics client beyond that, maybe StatsD or Grafana.
519
00:40:53.609 --> 00:40:58.030
And it would take a bit more code, but it should be able to extend Clio
520
00:40:58.765 --> 00:41:03.185
to actually work on other runners supported by Beam. So Beam supports
521
00:41:03.565 --> 00:41:05.105
Dataflow, but also
522
00:41:05.565 --> 00:41:06.945
Flink or Spark,
523
00:41:07.410 --> 00:41:10.069
And it's just a little bit more legwork, but
524
00:41:10.530 --> 00:41:14.069
defining support for that kind of runner is definitely
525
00:41:14.530 --> 00:41:16.069
possible and relatively
526
00:41:16.369 --> 00:41:16.869
simple.
527
00:41:17.475 --> 00:41:19.095
And right now, Clio
528
00:41:19.395 --> 00:41:26.055
has an optional library for, like, specific audio handling based off of, like, the Librosa package,
529
00:41:26.515 --> 00:41:32.680
but folks could easily make tools for easier pipelining of like image and video handling.
530
00:41:33.460 --> 00:41:36.519
So there's, there's many, many areas, many touch points
531
00:41:37.015 --> 00:41:39.915
that should be relatively easy to kind of extend.
532
00:41:40.775 --> 00:41:45.115
And for people who are building on top of Clio, what are some of the challenges
533
00:41:45.460 --> 00:41:50.280
or misunderstandings or edge cases that they should be aware of or that you've seen people run into?
534
00:41:50.820 --> 00:41:52.599
There's in general, like, a common misunderstanding
535
00:41:52.900 --> 00:42:00.265
when you're first learning about Clio, particularly if you're not in, like, the research space of, like, audio processing. Is that, like, Clio
536
00:42:00.640 --> 00:42:05.619
does the audio or AML processing for you, or it's a framework for that when really
537
00:42:06.000 --> 00:42:12.035
Clio is meant to help scale that processing distributed way. Sometimes that can be a misunderstanding
538
00:42:12.654 --> 00:42:15.395
when going into the project, you know, fresh.
539
00:42:15.775 --> 00:42:19.075
But then in general, the the space itself is very complex
540
00:42:19.910
--> 00:42:20.890
and has a natural,
541
00:42:21.190 --> 00:42:22.650
like, steep learning curve.
542
00:42:22.950 --> 00:42:29.530
So you might have to do a bit of self education in order to get beyond, like, a simple hello world, like example.
543
00:42:30.135 --> 00:42:33.755
And it's difficult for the user because it's hard to tell where
544
00:42:34.215 --> 00:42:38.795
Clio ends and where Apache begins or where Dataflow begins.
545
00:42:39.190 --> 00:42:44.490
You're interacting with Clio mostly, but Beam is doing, like, all of the pipelining work and Dataflow
546
00:42:44.790 --> 00:42:53.755
is doing all the orchestration work or another runner, the orchestration work. But hopefully, that doesn't necessarily discourage folks. What Clio has done is essentially
547
00:42:54.055 --> 00:42:55.595
make the nearly impossible
548
00:42:56.375 --> 00:42:56.875
possible.
549
00:42:57.420 --> 00:42:59.040
It is still complex.
550
00:42:59.740 --> 00:43:05.280
And what we're working on next is to make that complex a bit more simple.
551
00:43:05.740 --> 00:43:06.240
So
552
00:43:06.585 --> 00:43:07.964
in our roadmap, we have
553
00:43:08.265 --> 00:43:12.204
ideas for tools like better audio chunking or just file chunking
554
00:43:12.664 --> 00:43:15.484
right now, processing extremely large
555
00:43:15.800 --> 00:43:18.620
files like hours, hours long podcasts
556
00:43:19.240 --> 00:43:24.780
is, like, less reliable, more a little bit more shaky than, you know, the
557
00:43:25.265 --> 00:43:28.325
3 or 10 minute, like, song. We're hoping
558
00:43:28.945 --> 00:43:29.605
to build
559
00:43:29.905 --> 00:43:31.205
more useful utilities
560
00:43:31.505 --> 00:43:35.730
in the near future to adjust that sort of use case of extremely large files.
561
00:43:36.510 --> 00:43:40.450
Or for the odd music track that actually happens to run for a solid hour.
562
00:43:40.830 --> 00:43:41.330
Yeah.
563
00:43:42.164 --> 00:43:44.424
And so in terms of your own experience
564
00:43:44.724 --> 00:43:56.069
of working on the Clio project and being involved in this space of audio data processing, what are some of the most interesting or unexpected or challenging lessons that you've learned in the process?
565
00:43:56.745 --> 00:44:06.640
Of course, I had to learn a lot about audio processing. I still couldn't write like a PhD based off of it or a dissertation based off it, but sort of, yeah, I had to do a crash course
566
00:44:06.940 --> 00:44:08.880
in, I guess, digital signal processing
567
00:44:09.339 --> 00:44:11.680
kind of thing, reading a lot of textbooks,
568
00:44:12.220 --> 00:44:14.875
but actually the challenges that were more in
569
00:44:15.175 --> 00:44:16.795
our faces in my face was
570
00:44:17.335 --> 00:44:18.555
understanding pickling.
571
00:44:18.935 --> 00:44:23.390
I haven't actually, I mean, I, I know pickling, but I hadn't actually been exposed to it before
572
00:44:23.849 --> 00:44:26.750
this and, and specifically with how Beam pickles.
573
00:44:27.115 --> 00:44:31.934
And so when launching a pipeline, particularly for like a remote runner like Dataflow,
574
00:44:32.395 --> 00:44:37.630
user code gets pickled locally and then unpickled on worker machines.
575
00:44:38.170 --> 00:44:38.670
So
576
00:44:38.970 --> 00:44:54.540
anytime pickling is used, it kind of can limit what you can do. And so this is actually 1 of the reasons why we dockerize everything to to make sure that the environment in which you launch your pipeline in, the environment that gets pickled is the same environment that the pipeline
577
00:44:55.000 --> 00:45:01.875
itself gets unpickled on the remote worker. And so that might be obvious to some, but I had yet been exposed to that problem space.
578
00:45:02.175 --> 00:45:06.630
But with that, like deeper understanding of pickling can come
579
00:45:06.930 --> 00:45:09.269
ideas of how to abuse it, I guess.
580
00:45:09.890 --> 00:45:10.630
So during
581
00:45:10.930 --> 00:45:11.430
unpickling,
582
00:45:12.049 --> 00:45:14.630
there's like this gender set state method
583
00:45:15.045 --> 00:45:18.025
defined in the class that gets called when unpickling an instance.
584
00:45:18.565 --> 00:45:22.585
And this provides us with, I guess, ways to manipulate
585
00:45:23.200 --> 00:45:25.060
worker environment as it starts up.
586
00:45:25.600 --> 00:45:29.220
Now, Cleo doesn't do this yet at all, but we could,
587
00:45:29.520 --> 00:45:31.060
but we've seen it used
588
00:45:31.605 --> 00:45:36.105
within Clio to maybe start demons on workers, like profilers,
589
00:45:36.405 --> 00:45:38.025
like, the Stackdriver profiler,
590
00:45:38.565 --> 00:45:39.785
which can be really useful.
591
00:45:40.340 --> 00:45:46.280
In generic kind of problem that I have regularly is kind of designing a framework for other developers.
592
00:45:46.820 --> 00:45:52.985
For instance, how best to design an API that makes sense and feels ergonomic to use.
593
00:45:53.285 --> 00:45:58.890
And then like how much magic should we provide the user? How much of that magic will actually
594
00:45:59.349 --> 00:46:01.609
limit the user maybe too much?
595
00:46:01.910 --> 00:46:03.930
How can we provide flexibility
596
00:46:04.390 --> 00:46:06.695
for the user while still being opinionated?
597
00:46:07.255 --> 00:46:13.115
And then the maintenance and development cost for us when it comes to like making those decisions.
598
00:46:13.575 --> 00:46:14.715
I'm like regularly
599
00:46:15.015 --> 00:46:15.515
paralyzed
600
00:46:15.895 --> 00:46:16.395
by
601
00:46:17.079 --> 00:46:22.460
like, a framework design, making sure that it's good and well thought out. And so I find personally
602
00:46:22.920 --> 00:46:24.540
that to be a big challenge.
603
00:46:25.305 --> 00:46:30.525
And as far as the ways that Clio is being used, what are some of the most interesting or unexpected
604
00:46:30.905 --> 00:46:31.965
or innovative
605
00:46:32.265 --> 00:46:33.800
projects that you've seen built with it?
606
00:46:34.280 --> 00:46:49.565
Yeah. I think I mentioned this before, but I still think it's super cool is is the sing along feature available in Japan and I think other markets as well. That's basically based off of Clio generated data of separating vocals from instruments and just taking the instrumentals
607
00:46:50.025 --> 00:46:51.005
and and the lyrics.
608
00:46:51.540 --> 00:46:56.359
I mean, we've also seen internally Clio used for like audio based ad generation,
609
00:46:56.900 --> 00:46:58.359
where you can like dynamically
610
00:46:58.885 --> 00:47:00.565
piece together someone's voice,
611
00:47:01.285 --> 00:47:04.345
like reading an ad with various background music.
612
00:47:04.645 --> 00:47:09.140
You can then create like multiple versions of the same ad. So this would help with
613
00:47:09.520 --> 00:47:17.540
like, maybe you're listening to a Chopin album and then it wouldn't be a great experience if that adds background music was heavy metal.
614
00:47:17.985 --> 00:47:22.485
So the fact that you're able to generate, you know, different types of ads with different backgrounds
615
00:47:22.945 --> 00:47:25.765
and kind of target based off that it provides a better experience.
616
00:47:26.260 --> 00:47:33.079
Some things that might be boring, to those like in this research space or in general, but I find super cool is
617
00:47:33.385 --> 00:47:44.270
we fingerprint audio, which means generate a unique identification or a signature for each audio or even segments of audio. And then that, can be used for like deduplication,
618
00:47:45.130 --> 00:47:46.430
cover song identification,
619
00:47:46.970 --> 00:47:47.710
many things.
620
00:47:48.250 --> 00:47:54.355
And there are some other research based things that we're doing, including like song and speech transcriptions,
621
00:47:54.975 --> 00:47:59.395
instrument separation, like separating the actual guitar from bass from drums.
622
00:47:59.810 --> 00:48:02.470
And then we can build on top of that. Personally
623
00:48:02.770 --> 00:48:05.430
ideating here, but like maybe we could build
624
00:48:05.970 --> 00:48:09.590
automatic generated, like sheet music with that. So that'd be really cool.
625
00:48:09.955 --> 00:48:21.069
Some external users, I haven't seen any actual, like, product of it, but I know external users are, like, really interested in in trying out Clio for an auto DJing feature.
626
00:48:21.369 --> 00:48:24.109
Basically, being able to create a playlist
627
00:48:24.650 --> 00:48:28.775
where, like, the tracks all fade in and out perfectly.
628
00:48:29.555 --> 00:48:30.375
A while ago,
629
00:48:30.915 --> 00:48:31.815
New York Times
630
00:48:32.115 --> 00:48:33.335
Sunday review
631
00:48:33.954 --> 00:48:34.454
feature,
632
00:48:35.290 --> 00:48:37.150
It's actually like a little web app.
633
00:48:37.530 --> 00:48:40.990
It's titled like why songs in summer sound the same.
634
00:48:41.450 --> 00:48:46.995
And it's a cool website that like highlights how summer songs have certain attributes or qualities like danceability
635
00:48:47.535 --> 00:48:49.235
or energy and loudness.
636
00:48:49.775 --> 00:48:50.275
Then
637
00:48:50.575 --> 00:49:03.985
the review itself actually goes to show, like, over the past 40 or 50 years, songs released in the summer or for the summer have become less diverse and have been converging on the same levels of danceability and loudness and other attributes.
638
00:49:04.365 --> 00:49:16.440
I don't wanna spoil it any further, and I'll, add the link to this cool app in the show notes. But this research kind of research is built upon our public audio features API
639
00:49:17.154 --> 00:49:26.570
And the the data for that public API is backed by clear jobs where the audio processing that takes place estimates how danceable a song is or the acoustickness
640
00:49:27.270 --> 00:49:30.090
or the tempo or what's called speechiness.
641
00:49:30.790 --> 00:49:37.355
So, like, it's making research a lot easier and allowing us to also build some really cool features.
642
00:49:38.295 --> 00:49:47.230
And you mentioned a few things that you've got planned for the future of the project. Are there any other things that you'd like to call out for plans that you have or
643
00:49:47.690 --> 00:49:50.510
contributions that would be helpful to help drive it forward?
644
00:49:51.375 --> 00:49:53.954
Yeah. So there's definitely a lot of development
645
00:49:54.255 --> 00:49:55.795
ahead. I say that while
646
00:49:56.255 --> 00:50:00.690
Clio is production ready, it can do a lot more to help users
647
00:50:01.069 --> 00:50:03.490
focus on what they care about, make things more simpler.
648
00:50:03.950 --> 00:50:05.329
And so, yeah, this includes,
649
00:50:05.675 --> 00:50:06.734
you know, more optimization
650
00:50:07.035 --> 00:50:08.335
tools like file chunking
651
00:50:08.875 --> 00:50:10.815
and improved concurrency management.
652
00:50:11.435 --> 00:50:19.000
And this is coming with, you know, more and more interest of processing podcasts and just longer audio in general. But we're also hoping to
653
00:50:19.460 --> 00:50:19.960
improve
654
00:50:20.295 --> 00:50:23.755
the current ability to observe more observability features,
655
00:50:24.295 --> 00:50:25.195
allow you to
656
00:50:25.655 --> 00:50:26.555
be able to
657
00:50:27.015 --> 00:50:29.270
diagnose any sort of, like,
658
00:50:29.570 --> 00:50:37.990
resource constraints that your job might be having, kind of uncover any sort of really ugly, like, deep down into the system issues,
659
00:50:38.345 --> 00:50:40.045
basically help diagnose, essentially.
660
00:50:40.825 --> 00:50:43.085
We also have plans for
661
00:50:43.705 --> 00:50:53.650
better testing. Like, right now, you can certainly easily unit test, but what about integration tests, particularly if you're a job that's a part of a bigger
662
00:50:53.950 --> 00:50:54.930
graph of
jobs?
663
00:50:55.405 --> 00:51:00.465
In general, like, I'm curious, I guess, to see what the open source community will bring.
664
00:51:00.765 --> 00:51:03.905
I know that folks do wish to use this
665
00:51:04.210 --> 00:51:04.710
Amazon's,
666
00:51:05.490 --> 00:51:06.390
cloud architecture
667
00:51:06.690 --> 00:51:11.349
and as well as, like, kind of homegrown architecture, like maybe a self maintained, Kubernetes
668
00:51:11.970 --> 00:51:12.470
cluster.
669
00:51:12.865 --> 00:51:16.484
And so that's what I hope to anticipate in terms of contributions.
670
00:51:17.025 --> 00:51:22.140
But in general, we're like a very small team and so we can only focus on what's in front of us.
671
00:51:23.019 --> 00:51:24.559
So I would be wholeheartedly
672
00:51:25.260 --> 00:51:30.160
excited for any contributions that like, you know, different use cases or prove
673
00:51:30.460 --> 00:51:30.960
usability,
674
00:51:31.855 --> 00:51:33.075
anything of that sort.
675
00:51:33.615 --> 00:51:51.964
Well, for anybody who wants to follow along with the work that you're doing and get in touch, I'll have you add your preferred contact information to the show notes. And so with that, I'll move us into the picks. And this week, I'm going to choose the PSF fundraiser that's going on right now. They're looking to close out q 4 with some, hopefully, additional funds to help
676
00:51:52.265 --> 00:52:05.020
support the overall Python community. So if you have some time and are able to, definitely recommend contributing back to the PSF for all the work that they do for the people who use Python. And so with that, I'll pass it to you, Lynn. Do you have any picks this week?
677
00:52:05.640 --> 00:52:13.595
Yeah. Just 1 pick. I hope this doesn't make me sound like a fan girl, but about a month ago, I started using, this note taking tool called Roam.
678
00:52:14.055 --> 00:52:25.995
It's super easy to take notes and create a graph of your notes, catalog everything that you might think. It's just better aligned with how I approach my own note taking. So I highly recommend taking a look at it.
679
00:52:26.375 --> 00:52:39.700
Well, thank you very much for taking the time today to join me and discuss the work that you've been doing on Clio. It's definitely a very interesting project and interesting problem space. So I'm excited to see where it goes and the new capabilities that you're able to
680
00:52:40.135 --> 00:52:52.060
add to Spotify as a result of that and some of the ways that the overall developer and research community can take advantage of it. So thank you for all of your time and effort on that, and I hope you enjoy the rest of your day. Thank you very much. I appreciate you having
681
00:52:52.599 --> 00:52:53.099
me.
682
00:52:54.760 --> 00:52:59.994
Thank you for listening. Don't forget to check out our other show, the Data Engineering Podcast at dataengineeringpodcast.com
683
00:53:01.414 --> 00:53:03.355
for the latest on modern data management.
684
00:53:04.134 --> 00:53:05.595
And visit the site of pythonpodcast.com
685
00:53:06.615 --> 00:53:09.869
to subscribe to the show, sign up for the mailing list, and read the show
686
00:53:10.170 --> 00:53:14.910
notes. And if you've learned something or tried out a project from the show, then tell us about it. Email host@podcastthenit.com
687
00:53:16.170 --> 00:53:17.070
with your story.
688
00:53:17.795 --> 00:53:21.975
To help other people find the show, please leave a review on Itunes and tell your friends and coworkers.