TRANSCRIPCIÓN 🔗
WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 7/6/2024 12:17:47 PM
Duration: 3600.330
Channels: 1
1
00:00:11.264 --> 00:00:15.445
Hello, and welcome to the data engineering podcast, the show about modern data management.
2
00:00:16.545 --> 00:00:18.564
This episode is supported by CodeCommence,
3
00:00:19.000 --> 00:00:28.140
an original podcast from Red Hat. As someone who listens to the data engineering podcast, you know that the road from tool selection to production readiness is anything but smooth or straight.
4
00:00:28.675 --> 00:00:37.895
In code comments, host Jamie Parker, Red Hatter and experienced engineer, shares the journey of technologists from across the industry and their hard won lessons in implementing new technologies.
5
00:00:38.820 --> 00:00:48.645
I listened to the recent episode, transforming your database, and appreciated the valuable advice on how to approach the selection and integration of new databases and applications into the impact on team dynamics.
6
00:00:49.344 --> 00:00:58.390
There are 3 seasons of great episodes and new ones landing everywhere you listen to podcasts. Search for Code Comments in your podcast player or go to data engineering podcast dot com/codecomments
7
00:00:59.010 --> 00:01:00.070
today to subscribe.
8
00:01:00.370 --> 00:01:04.790
My thanks to the team at CodeCommons for their support. Data lakes are notoriously complex.
9
00:01:05.245 --> 00:01:15.730
For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst is an end to end data lake has platform built on Trino, the query engine Apache Iceberg Iceberg was designed for.
10
00:01:16.510 --> 00:01:21.730
Starburst has complete support for all table formats, including Apache Iceberg, Hive, and Delta Lake.
11
00:01:22.095 --> 00:01:26.275
And Starburst is trusted by teams of all sizes, including Comcast and DoorDash.
12
00:01:26.815 --> 00:01:30.035
Want to see Starburst in action? Go to data engineering podcast.com/starburst
13
00:01:31.560 --> 00:01:37.660
today and get $500 in credits to try Starburst Galaxy, the easiest and fastest way to get started using Trino.
14
00:01:38.945 --> 00:01:50.270
Your host is Tobias Macy, and today I'm interviewing Sriram Panayam about his experiences conducting large scale data migrations and the useful strategies that he's learned in the process. So, Shrir, can you start by introducing yourself?
15
00:01:51.049 --> 00:01:55.149
Yeah. I'm Shriram. I'm currently the CTO at Diagnose.
16
00:01:55.615 --> 00:01:57.375
We we build the tooling for,
17
00:01:57.855 --> 00:02:00.435
automating runbooks for SRS and dev teams,
18
00:02:00.735 --> 00:02:06.300
to kind of get out of all the manual repetitive work. Right now, I probably focused on, I focus on
19
00:02:06.680 --> 00:02:15.395
satisfying our platform and building its control plane. But my background has been in data for a while, especially at, at, in large scale systems like, you know, Google
Load full transcript
and LinkedIn.
20
00:02:15.775 --> 00:02:22.310
And and, you know, 1 of the concepts that, you know, make me passionate passionate about is data migrations and data movement at scale. Right?
21
00:02:23.010 --> 00:02:24.470
Yeah. So hey. How are you doing?
22
00:02:24.850 --> 00:02:28.790
I'm doing well. Thanks. And do you remember how you first got started working in data?
23
00:02:30.194 --> 00:02:30.694
Yeah.
24
00:02:30.995 --> 00:02:31.815
Purely randomly.
25
00:02:32.275 --> 00:02:34.694
My career began, you know, in,
26
00:02:35.235 --> 00:02:41.030
as a network designer long, long, long time ago. I guess what's more, data than packets sitting on, I think.
27
00:02:41.890 --> 00:02:47.605
But to me, I think, you know, almost any useful or FX for computing is about moving and processing
28
00:02:48.145 --> 00:02:53.445
data. And I guess my real my journey was about 15 years ago when I started doing my first start up,
29
00:02:54.065 --> 00:02:54.950
back in Sydney.
30
00:02:55.430 --> 00:02:59.610
It was called Street Hawk. It was a location based marketing platform for matching
31
00:02:59.989 --> 00:03:00.489
retailers
32
00:03:00.790 --> 00:03:03.930
and shoppers, you know, as they as they are walking past based on location.
33
00:03:04.665 --> 00:03:06.045
And it was a
34
00:03:06.665 --> 00:03:07.245
a really
35
00:03:07.864 --> 00:03:08.364
interesting,
36
00:03:08.825 --> 00:03:11.885
you know, evolution for me because prior to that, I was a embedded
37
00:03:12.185 --> 00:03:16.860
systems kind of engineer, you know, working on device drivers and games. Right?
38
00:03:17.239 --> 00:03:21.875
So the best thing or the close thing to data was how do you load really large asset files for games.
39
00:03:22.575 --> 00:03:29.250
When I started getting into web, you know, even though it looks like, hey, you know, user comes in, gets a page, downloads something, goes away,
40
00:03:29.810 --> 00:03:36.310
There's so much behind the scenes in terms of how data is managed, loaded, moved, you know, kept constant, kept clean
41
00:03:36.685 --> 00:03:38.385
despite bugs happening all around.
42
00:03:38.925 --> 00:03:45.050
And and that was kind of the start of my world of startups, into data, into large scale systems, and moving into the US
43
00:03:45.430 --> 00:03:46.970
around 12 years ago
44
00:03:47.349 --> 00:03:50.010
just made that more fun and accelerated for me.
45
00:03:50.390 --> 00:03:53.450
Later on, I was at LinkedIn working on the profile and feed platforms.
46
00:03:54.215 --> 00:03:59.435
Again, a lot of data problems, a lot of fun stuff there. And then, you know, going from
47
00:03:59.815 --> 00:04:04.100
a user of these platforms and building the right platforms, I moved on to Google where I was working on Dataflow,
48
00:04:05.200 --> 00:04:08.180
where I was, I guess, selling a platform. Yeah.
49
00:04:08.640 --> 00:04:11.140
That's kind of my journey into data and, continual,
50
00:04:11.825 --> 00:04:13.925
you know,
current learning as I go through.
51
00:04:15.105 --> 00:04:19.685
Yeah. It's interesting when people start from that very low level,
52
00:04:20.139 --> 00:04:23.440
in your case, the network dealing with packets, how that
53
00:04:24.139 --> 00:04:25.759
colors their overall experience
54
00:04:26.139 --> 00:04:27.360
with other technologies.
55
00:04:27.660 --> 00:04:44.400
Myself, I started as a systems administrator. It was my first job in tech, and so that is definitely very much colored the way that I think about software and programming because I wanna make sure that things are reliable and aren't gonna break and wake me up in the middle of the night because that's where I started my career.
56
00:04:44.860 --> 00:04:51.175
I I mean, I saw what you you did some really, you know, interesting things. And I can see that, you know, those experiences,
57
00:04:51.555 --> 00:04:58.490
even though they may sound very different and offbeat, they really bring a different perspective for you when you solve these problems. Right?
58
00:04:58.950 --> 00:05:01.290
For example, in the networking world, you know,
59
00:05:02.070 --> 00:05:06.890
back then, 20 something years ago, it was all about network and switch design. Right?
60
00:05:07.555 --> 00:05:10.695
Everything was about how do I relieve load on some
61
00:05:11.635 --> 00:05:32.270
processing cluster on some rack on some network. Right? And then if you think about data, it's about, you know, okay, I have this giant, giant, giant blob of data, like, petabytes of data. I can't process them, you know, in a single core. Right? So how do I route it? How do I switch it? How do I break it down? How do I shot it? Like, all that comes in as part of this. Right? And and I think this is certain bit of,
62
00:05:33.710 --> 00:05:40.605
gleeful joy, you know, when you when you bring in those completely different unrelated topics into something else you're working on. Right?
63
00:05:41.565 --> 00:05:42.065
Absolutely.
64
00:05:42.445 --> 00:05:42.945
Yeah.
65
00:05:43.725 --> 00:05:44.225
And
66
00:05:45.405 --> 00:05:47.620
in terms of the topic that we're covering
67
00:05:50.100 --> 00:05:51.240
migration projects,
68
00:05:51.860 --> 00:05:52.360
obviously,
69
00:05:52.740 --> 00:05:57.575
data gets moved around all the time, but when we say migration, that usually means that there's
70
00:05:58.055 --> 00:06:06.955
some specific body of work that hopefully has an end, but definitely has a beginning. And I'm warn I'm wondering if you can just give some of your
71
00:06:07.440 --> 00:06:07.940
perspective
72
00:06:08.240 --> 00:06:11.060
on what you see as the
73
00:06:11.680 --> 00:06:12.900
kind of broad categories
74
00:06:13.280 --> 00:06:14.500
of data migration
75
00:06:14.960 --> 00:06:22.544
as a project and maybe some of the ways that people might think about it that we are explicitly excluding from this conversation.
76
00:06:23.005 --> 00:06:24.285
Yeah. Yeah. Now,
77
00:06:25.450 -->
00:06:27.310
if you are a Seinfeld fan,
78
00:06:27.690 --> 00:06:32.590
or if you haven't seen any episodes, there's a really good episode where Josh Costanza talks about risk management,
79
00:06:33.475 --> 00:06:35.175
and how there's a really boring
80
00:06:35.875 --> 00:06:41.895
introduction, you know. To understand risk management, you have to understand risk. Alright. Nothing can be drier and more boring than that.
81
00:06:42.275 --> 00:06:47.110
And if you look at data migrations or database migrations, it's very much like risk management.
82
00:06:47.490 --> 00:06:47.990
Right?
83
00:06:48.290 --> 00:06:58.005
If you look at it purely as, you know, migrating a database to something else, it can get very dry and very boring as as you as you called out. Right? There's a start, there's an end, and voila, off you go.
84
00:06:58.465 --> 00:07:01.685
But so much happens when you bring in other constraints.
85
00:07:02.099 --> 00:07:08.599
Right? If you think of migration or or data migration as, look, I have this big system. I need to move it somewhere else.
86
00:07:08.900 --> 00:07:17.305
That's it. Right? The easiest, dumbest thing you can do is, you know, take a backup or or or sorry. 1st, cut off your customers. Take a backup,
87
00:07:17.605 --> 00:07:18.665
load it on the database,
88
00:07:19.120 --> 00:07:22.979
have them downtime at 3 AM in the morning, move your customers, and restart it.
89
00:07:23.440 --> 00:07:25.379
That's kind of not how
90
00:07:25.680 --> 00:07:26.659
today's systems
91
00:07:27.285 --> 00:07:29.384
or or more importantly, it's not how
92
00:07:29.685 --> 00:07:37.840
customers would be happy with you if you did that today. Right? They want, you know, near a 100% uptime or 0 to 0 downtime.
93
00:07:38.460 --> 00:07:52.060
They want all kinds of other functionality in between. They want, you know, they have there's there's so much diversity in the kinds of data in terms of cardinality, the the structures and schemes and so on. And, you know, it and you're talking about, you know, global scale,
94
00:07:53.320 --> 00:07:55.180
and with global distributions. Right?
95
00:07:55.560 --> 00:07:58.140
You might have data for customers in America
96
00:07:58.645 --> 00:07:59.865
being served out of India.
97
00:08:00.405 --> 00:08:06.025
Right? So as you start looking at migrations as a data systems problem, right, things get very interesting.
98
00:08:06.460 --> 00:08:18.455
Yes. There is a start and there's an end, but it's a very, very interesting middle. Right? We just made data migration tools and strategies and systems really shine and, you know, take it away from being anything of a boring topic. Right?
99
00:08:18.835 --> 00:08:22.135
For example, is migration not just on the form of replication?
100
00:08:23.350 --> 00:08:24.730
At the nth degree. Right?
101
00:08:25.190 --> 00:08:31.850
Yes. You might have 1 database to another database or 1 MySQL to under MySQL or SQL to SQL or then you can go SQL to
NoSQL.
102
00:08:32.595 --> 00:08:36.135
Talking about building indexes, you know, as as of all migration.
103
00:08:36.515 --> 00:08:40.135
You might not wanna migrate the whole database. You can just look at partial migrations.
104
00:08:40.760 --> 00:08:43.980
A good example is when you when you have start ups
105
00:08:44.680 --> 00:08:47.500
or any company, right, building a product that's a v 0 over 0.1,
106
00:08:48.175 --> 00:08:49.635
everything gets stored in 1 database,
107
00:08:50.015 --> 00:08:51.475
served through some kind of ORM,
108
00:08:51.775 --> 00:09:00.430
off you go. But as you grow, you know, 1,000,000,000, 10,000,000,000 kind of record a 1,000,000,000,000, 10,000,000,000 kind of records, 1 database is not gonna do. Right? Or you might you will have to shot it. Suddenly, it becomes a set of partial migrations. Right?
109
00:09:06.235 --> 00:09:08.334
We talked about, you know, different kinds of databases
110
00:09:08.795 --> 00:09:09.615
to and from.
111
00:09:09.995 --> 00:09:14.894
And then, you know, there's things like, cardinality where you can be, you know, are you going from
112
00:09:15.380 --> 00:09:22.360
I I I do want to many relationships or many to many or even redistribution of things. Right? So there's a lot of these interesting cases that come in in the middle,
113
00:09:23.005 --> 00:09:26.065
when you look at migrations and transformations and moves.
114
00:09:27.165 --> 00:09:31.585
In terms of migrations, usually, there's some sort of impetus or some
115
00:09:32.080 --> 00:09:32.580
hopeful
116
00:09:32.960 --> 00:09:38.660
improvement in your overall state at the end of it. I'm curious if you can talk to some of the
117
00:09:39.435 --> 00:09:40.654
typical motivations
118
00:09:41.035 --> 00:09:45.855
for starting on a migration project, and then we can dig into some of the
119
00:09:46.530 --> 00:09:51.990
strategic elements of how to think about it, how to plan for it, how to execute it. Yeah. Yeah. So
120
00:09:52.530 --> 00:09:53.750
they can be different
121
00:09:54.825 --> 00:09:56.205
triggers for migration.
122
00:09:56.665 --> 00:10:02.500
Right? You you you might just be at a point where, you know, you put everything in a single data store,
123
00:10:03.460 --> 00:10:07.240
and you kinda reach limits. Now storage limits, QPS limits,
124
00:10:07.700 --> 00:10:13.485
like like, how many customers are accessing it, you know, like, what what kind of patterns you're doing, they can change.
125
00:10:14.025 --> 00:10:16.125
You might have you know, as your
126
00:10:16.505 --> 00:10:24.790
application or platform or API or the client facing thing morphs from day 1 to day 100, your schemas change. Your use cases change. Right?
127
00:10:26.130 --> 00:10:28.550
As pure as we wanna be,
128
00:10:29.315 --> 00:10:29.975
you know,
129
00:10:30.435 --> 00:10:46.515
your front end or your application is a reflection of your data, and your data organization is a reflection of your use case. Right?
So when either of them so so when the use case changes, your data organization is gonna change. So a good example is, you know, you might suddenly need a whole bunch of indexes just to make certain queries
130
00:10:46.835 --> 00:10:47.575
faster. Right?
131
00:10:48.275 --> 00:11:04.680
So you that that might trigger, you know, how often those use cases are exercise exercised. You know, what kind of SLOs do you want on that access? Do you want the same access for viewing my feeds or same SLOs for viewing my my my, you know, LinkedIn feed versus viewing my LinkedIn profile?
132
00:11:05.045 --> 00:11:16.089
Right? So those might mean that you're okay with certain sacrifices and certain trade offs you might need, which might mean that, hey. Look. Why am I keeping everything in the same database? Move it around, break it up, break it to multiple microservices
133
00:11:16.709 --> 00:11:19.449
that other teams can manage, you know, change ownership,
134
00:11:20.070 --> 00:11:23.850
you know, and that results in more reasons for migrating material.
135
00:11:24.535 --> 00:11:29.435
You might even have things like compliance and privacy requirements that might
136
00:11:29.815 --> 00:11:37.180
change how your data is stored and, you know, retrieve and processed and what kind of ACLs you might wanna have on your data. Right?
137
00:11:37.720 --> 00:11:52.225
And applying those ACLs may have different sorry. Applying those, axe controls might have different, you know, different costs. Right? That may not be homogeneous to who's paying for it. Right? So again, you might wanna break that up to serve different needs and different SLOs.
138
00:11:52.720 --> 00:11:53.220
And
139
00:11:53.680 --> 00:12:01.220
also, you might have things like, you know, team expertise and team limitations where certain data stores could be useful for certain certain
140
00:12:01.625 --> 00:12:06.925
kinds of features and teams that that team would be good at managing, whereas some teams might might be more,
141
00:12:07.545 --> 00:12:09.780
oriented towards a different cause of problems. Right?
142
00:12:10.260 --> 00:12:16.360
Then you have your whole offline, online, nearline use cases where, you know, you go dump data to some data lake for analytics purposes
143
00:12:16.900 --> 00:12:17.785
and a lot more.
144
00:12:18.264 --> 00:12:19.644
In the intro, I also
145
00:12:20.105 --> 00:12:28.060
had that keyword of large scale because if you're just dealing with the data migration of a 50 megabyte Postgres database to MySQL,
146
00:12:28.760 --> 00:12:35.820
there's not a lot of planning that has to happen. You just need to make sure both databases are running. You do, SQL dump, SQL load, and then you cut over.
147
00:12:36.365 --> 00:12:41.264
And I'm wondering what are some of the ways that gradations of scale
148
00:12:41.565 --> 00:12:44.464
impact the overall time and effort needed
149
00:12:45.029 --> 00:12:49.209
to be able to actually plan for and implement 1 of these migration projects?
150
00:12:50.389 --> 00:13:02.335
Right. So, you know, looking at the grades and sorry, gradations and scales. Right? I would look at it as, look, you know, what kind of consistency, what kind of freshness, what kind of, you know, latency requirements,
151
00:13:02.920 --> 00:13:04.300
you know, your access
152
00:13:04.680 --> 00:13:07.020
needs. Right? Now take something like,
153
00:13:07.640 --> 00:13:17.705
a 50 megawatt, you know, database that you wanna move from 1 to another. It could be SQL to SQL or SQL to SQL, you know, the size here is a big factor here. Right? In this case, a big factor.
154
00:13:18.380 --> 00:13:28.134
So, you know, it is fun. Even if you were to kind of do a manual physical, you know, 1 time cutover, the downtime itself is gonna be pretty minimal. Right? You might have more,
155
00:13:28.834 --> 00:13:35.800
effort in testing the migration and so on, but the oral downtime is kinda cheap. It's almost a lift and shift or a lift and transform and shift.
156
00:13:36.279 --> 00:13:36.779
And
157
00:13:37.640 --> 00:13:38.459
fairly manual,
158
00:13:39.080 --> 00:13:43.019
downtime if you for users probably in the order of minutes. Right?
159
00:13:43.615 --> 00:13:49.635
As you as as the data sizes kinda grow, right, you might say, look, I have a petabyte,
160
00:13:50.095 --> 00:13:55.579
you know, like, few terabytes of data in some data store that I do break and move and and so on.
161
00:13:56.040 --> 00:13:59.259
I can either do it from a streaming perspective,
162
00:13:59.560 --> 00:14:17.385
but we'll come to that later. But I I can do a full full, you know, dump and reload. Right? We're talking about a few hours, if not days of downtime for the customers. Right? Again, if all you care about is the ease of lift and shift, that's fine. You might be okay with your customers having downtime for a few hours in a very
163
00:14:18.084 --> 00:14:23.865
bespoke use case, and it's perfectly fine. In today's system, right, the it's not just 1
164
00:14:25.080 --> 00:14:51.910
access of data that a customer demands. Right? It's not just that, hey. I have a bunch of profiles in this database. I wanna move to database and start using it. What customers wanna do is I have my profiles that I'm using. I also wanna be able to post feeds. I wanna create content. I wanna like and, you know, like and follow other people's content. How can I do all all that all the time without a downtime? And now I need to start thinking about how can I do how can I reduce my downtime as much as possible
165
00:14:52.885 --> 00:15:09.274
while make making this process less complex? The other aspect is we talked about index. Right? And if you think about all these secondary, you know, 3rd degree, 4th degree views that you can get from your data. For instance, you might have, you know, a Twitter timeline, which is a collection of the of the
166
00:15:09.654 --> 00:15:29.485
of the of the post that you can see of all
your friends. Right? I mean, this is a classic system design problem. Right? You know, I'm posting. I wanna see all my all my tweets, but somebody else wants to see mine as quickly as possible. I could be an influencer. What are the consequences of that data getting around here at what point in time? So when you're migrating so if you look at that as a replication problem, how quickly
167
00:15:30.105 --> 00:15:38.530
I can see it or or or or or hapically I'm I'm okay to tolerate in terms of the wait time can affect, you know, what hurdles I make in my,
168
00:15:39.150 --> 00:15:46.665
migration process. So 1 example is you can use something like a a change capture process where any changes to a database, right, are,
169
00:15:47.365 --> 00:15:47.865
extracted,
170
00:15:48.725 --> 00:15:51.350
processed, transformed, and written to a target database.
171
00:15:51.829 --> 00:15:57.769
This is the proverbial middle that we're talking about. Right? You're reading from source database. It goes to a whole pipeline of changes right somewhere.
172
00:15:58.470 --> 00:16:03.445
And there are many tools for this, right, that are out there that can do this too. The but if you say that I want my
173
00:16:03.904 --> 00:16:06.325
freshness to be as fresh as possible,
174
00:16:06.785 --> 00:16:10.005
if I write to 1 database, I want that to be visible in the second system immediately,
175
00:16:10.530 --> 00:16:24.765
Then you can use something like a dual right approach, which has its own kind of caveats and trade offs. So the planning is important. They got a plan, what kind of what kind of experience the user is okay to tolerate, what that means for our business metrics, and how it will impact our own engagement and other KPIs.
176
00:16:25.350 --> 00:16:29.130
And then look at how expensive and costly it is to pick which way you go.
177
00:16:29.910 --> 00:16:31.690
In terms of the
178
00:16:32.755 --> 00:16:34.215
end goals of a migration,
179
00:16:34.915 --> 00:16:35.415
there
180
00:16:35.875 --> 00:16:47.449
are obviously a large variety. Sometimes it's you want to move from 1 database technology to another because it has some new shiny feature you wanna take advantage of. Maybe you need to move from
181
00:16:48.144 --> 00:17:01.279
maybe it's for reasons of scale, as we mentioned, where your single node database can't handle the amount of data that you're dealing with anymore, so you need to move to something that has sharding and replication built in, or maybe you need to move to a completely different
182
00:17:01.820 --> 00:17:06.735
style of database, maybe moving from a relational database to something like Cassandra
183
00:17:07.115 --> 00:17:17.580
for those reasons of scale. Or it might be for cost purposes where you start off with a cloud data warehouse, but then you start spending $1,000,000 a year on it, and you,
184
00:17:17.880 --> 00:17:24.060
aren't getting all the return on investment that you hope for, and so you decide to move
to more of a data lake or a lakehouse style architecture.
185
00:17:25.355 --> 00:17:32.495
Or maybe it's a continual migration process where you need to keep a subset of your data in a preproduction
186
00:17:32.875 --> 00:17:33.375
environment
187
00:17:33.780 --> 00:17:38.520
for development and testing purposes. I'm wondering if you can just talk to some of the ways that those
188
00:17:38.900 --> 00:17:39.400
motivations
189
00:17:39.940 --> 00:17:51.524
and end goals can color the overall appetite and the organizational processes necessary to be able to get, buy in and support for that migration process.
190
00:17:52.200 --> 00:17:57.020
Yeah. So, you know, your costs and I mean, your your cost cost play a huge factor in this. Right?
191
00:17:57.560 --> 00:18:01.020
And cost can come in a bunch of ways. Like, there's obviously the
192
00:18:01.345 --> 00:18:14.029
raw cost. As you call that, putting something in data warehouse, you know, is gonna be expensive from a compute perspective and not just storage. Right? You're topping everything. You need a lot of, you know, transformational compute there to make sense for the consumers of that,
193
00:18:14.570 --> 00:18:16.669
of all the data stored in your database. Right?
194
00:18:17.529 --> 00:18:27.050
You know, and if depending on who it's available for, that could be an acceptable cost. Now what these warehouses and data list give you is well, I mean, the promise there is that,
195
00:18:28.410 --> 00:18:42.135
you know, you you can do all that with very little effort, and there's a whole bunch of ingestion pipelines that give you things out of the box. There's a lot of connectors from all kinds of data sources that you can just dump in and then people can take advantage of it. If the ROI of that
196
00:18:42.675 --> 00:18:43.075
exceeds
197
00:18:43.740 --> 00:18:51.600
I mean, if it's high enough ROI and if the return exceeds the cost, it's okay for a certain point of time. But if you find that you're doing all that
198
00:18:52.025 --> 00:18:57.165
ingestion of data just for 3 or 4 bespoke use cases and you're unnecessarily spending the
199
00:18:57.465 --> 00:18:58.765
spending all all the resources
200
00:18:59.225 --> 00:18:59.940
just for that,
201
00:19:00.820 --> 00:19:03.159
It doesn't quite need that kind of heavy
202
00:19:03.620 --> 00:19:07.720
heavy ingestion. Right? And the other thing is, you know, you have you also have technical
203
00:19:08.075 --> 00:19:11.275
aspects like who's gonna be managing this, who's gonna be, you know,
204
00:19:11.835 --> 00:19:15.515
who's gonna manage the start of it, you know, the the the actual process of,
205
00:19:16.160 --> 00:19:17.940
continue continuous or 1 off migration.
206
00:19:18.560 --> 00:19:24.020
And post migration, who's gonna manage the system? Those costs would also have to be looked at,
207
00:19:24.434 --> 00:19:34.440
in the overall cost of doing this migration from a to b. We also I mean, 1 thing we our our kind
of glossed over was security and compliance. Right? Like, you know, when you're doing this, what is the security,
208
00:19:34.980 --> 00:19:42.425
you know, what is current state of the how secure and compliant your data is? What will be the final state in the target
209
00:19:42.825 --> 00:19:49.725
and during the migration itself. So I think these play a decent role in how you choose, you know, when and why and how you do the migration.
210
00:19:50.260 --> 00:19:58.995
Now so it's very easy to talk about a lot of these aspects of data migrations in the abstract where you maybe have various motivations,
211
00:19:59.535 --> 00:20:01.155
different organizational structures.
212
00:20:01.455 --> 00:20:10.910
But bringing it to a more concrete reference point, I'm wondering if you can pick 1 or 2 experiences that you've had of actually going through this process of
213
00:20:11.290 --> 00:20:12.430
understanding the motivations
214
00:20:12.730 --> 00:20:13.230
for
215
00:20:13.695 --> 00:20:18.034
why and how fast and where you need to migrate data from and to
216
00:20:18.414 --> 00:20:23.610
and the overall technical and project management process of that and then the
217
00:20:24.070 --> 00:20:24.570
validation
218
00:20:25.110 --> 00:20:32.965
and final sign off to say, yes. This is done. I have achieved my overall objective, and now I'm gonna go and get a big bonus because of it.
219
00:20:33.665 --> 00:20:35.265
Well, I don't think I got that big bonus, but,
220
00:20:36.465 --> 00:20:40.015
Yeah. So at at LinkedIn, what if I mean, I joined LinkedIn in 2015
221
00:20:40.580 --> 00:20:41.720
15 and right?
222
00:20:42.180 --> 00:20:44.120
And initially, I was part of the,
223
00:20:44.500 --> 00:20:45.640
LinkedIn feed platform.
224
00:20:46.100 --> 00:20:48.440
And part of the feed was, you know, your user posts,
225
00:20:49.185 --> 00:20:57.740
your use users' social actions around it, other kind of shares and comments and likes and, you know, social activity. I mean, LinkedIn itself has
226
00:20:58.040 --> 00:21:09.495
has gone through several stages of data evolution. They started off with this you know, I believe it's our goal first for a long, long, long time. And then LinkedIn built its own, like, extremely high performing, you know, espresso storage,
227
00:21:09.875 --> 00:21:13.895
which was a very, very, you know, highly shot shottable, you know, or shot at MySQL.
228
00:21:14.760 --> 00:21:20.059
And based on based on based on MySQL, they were sharded back. Right? It was essentially a Key Valley store.
229
00:21:20.600 --> 00:21:23.735
And the profile also sorry. And the feed also had many aspects
230
00:21:24.455 --> 00:21:26.235
in, many components in
231
00:21:26.934 --> 00:21:42.284
1 of my platforms, the social actions, you know, how to use a Slack comment. We had to move that from 1 espresso store to a different espresso store, right, for a lot of performance reasons. You know? And 2 aspects there were really important. Right? Like, how do you ensure that these actions were generalizable?
232
00:21:43.225 --> 00:21:45.325
Previously, they were very much tied to,
233
00:21:45.945 --> 00:22:03.684
the feed themselves. So we want to build a platform where we could take these actions and generalize it across any other entity in LinkedIn for the user. Effectively, what we called it was a hedge store or follow store. Now we called it unified follows for UFO. So the simplest thing was I mean, we couldn't do a lifting shift at the time because
234
00:22:04.065 --> 00:22:06.725
doing so would have impacted about 100,000,000,000
235
00:22:07.585 --> 00:22:08.560
edges at the time,
236
00:22:09.040 --> 00:22:21.105
and that was just not gonna happen in any reasonable amount of time. We, you know, saw a few options. You know, obviously, 1 of them was Lifecn shift, and the other 1 was see if we can somehow do a bulk dump initially
237
00:22:21.565 --> 00:22:24.065
followed by some form of CDC or change,
238
00:22:24.445 --> 00:22:25.140
data capture.
239
00:22:25.620 --> 00:22:30.200
LinkedIn already has what's called DataBus. It's a it's a very, you know, high again, high performance,
240
00:22:30.660 --> 00:22:32.760
CDC system. We yeah.
241
00:22:33.140 --> 00:22:38.995
We're coupled with espresso as well. Right? Essentially, espresso, the way it works I mean, CDC for espresso, the way it worked was
242
00:22:39.295 --> 00:22:39.955
it would,
243
00:22:40.335 --> 00:22:45.860
ride both to the MySQL table as well as the change log in a transactional manner, and that gets picked up by, you know, Dataverse,
244
00:22:46.559 --> 00:22:48.320
and it gets listened to by a bunch of other
245
00:22:48.960 --> 00:22:49.940
so our,
246
00:22:50.480 --> 00:22:58.075
goals for this were quite quite straightforward. Right? We wanna make sure that when this data these edges in the source database
247
00:22:58.455 --> 00:23:02.230
was migrated, it was all migrated. We had to make sure that there weren't missing
248
00:23:02.610 --> 00:23:10.495
edges in this. Last thing we wanted were last thing we wanted were customers, you know, liking something and finding finding that those lives were missing.
249
00:23:11.115 --> 00:23:14.975
We wanted to make sure that the migration was repeatable,
250
00:23:15.595 --> 00:23:18.575
just in case we did something, we wanted to kinda go back and,
251
00:23:18.919 --> 00:23:27.100
you know, fix pots and paths of it as opposed to as opposed to go through a full migration process all over again. Our original target was to make sure the client
252
00:23:27.425 --> 00:23:30.165
did not have to change as in the clients the LinkedIn
253
00:23:30.545 --> 00:23:41.280
user facing clients don't have to change. And I'll get to how we worked around that or worked through parts of it. We also wanted some testability on it and see how it's affecting any impact for the customer.
254
00:23:41.820 --> 00:23:46.145
Load times are critical for, you know,
engagement on LinkedIn or the impacts
255
00:23:46.525 --> 00:23:47.345
or to,
256
00:23:48.125 --> 00:23:58.170
improve engagement on LinkedIn. So we had to make sure that those are monitored. And, and, obviously, we wanna make sure that they were consistent. We didn't wanna have out of order out of ordering of rights affect any of
257
00:23:58.550 --> 00:24:05.514
the constancy and correctness of the edges before and after. By the way, the schemas were different. So that's that's a key part of it. It wasn't a homogeneous,
258
00:24:06.215 --> 00:24:07.995
transfer from the old to new. Right?
259
00:24:09.014 --> 00:24:18.870
So the the series so so we took a 2 part approach to this. 1 was at any point in time, you would take a dump off these databases or or the source database and load it into
260
00:24:19.184 --> 00:24:25.205
the target DB. Now the taking a snapshot wasn't your typical snapshot process. We would have hourly
261
00:24:25.505 --> 00:24:27.285
and or daily snapshots
262
00:24:27.760 --> 00:24:49.580
saved in HDFS from our our espresso stores. Right? So we would use that and the LSI or sequence number in those snapshots as the initial load at any point in time. The CDC would store these changes from the database into I believe it was Kafka at the time. Right? So you can just read those Kafka topics and write back to these systems. Now because we were we were shouting sorry, because our
263
00:24:50.095 --> 00:24:57.315
follow source was user. Right? It could be any entity, but we we chose the source as a sharding pattern. So we can
264
00:24:57.720 --> 00:24:58.460
pick for
265
00:24:59.000 --> 00:24:59.660
any source,
266
00:25:00.040 --> 00:25:21.195
only look at changes for that source in the last 1 day, 7 days, 30 days, so on, and replay that when we need it. Now once we set up this this pipeline, we wanted to see if we can play along with more of the freshness of rights because we wanted users to have their own rights available to them on both systems. So we did we did take on a dual right approach in this.
267
00:25:21.655 --> 00:25:34.909
Now dual rights can be I mean, they were simple. You know, you have a single client, the client writes to both the source and target database, and you also do a backfill of data from the source to data as a source to target through some kind of either CDC or some kind of replication
268
00:25:36.275 --> 00:25:46.810
method. Now it looks simple, but there are so many things that can go wrong here. Right? If you look at the traditional traditional dual ride, you know, you you you do this right to both, reach from both, validate, do transformations,
269
00:25:47.270 --> 00:25:51.610
and off you go. So when you do this, if you had distributed transactions,
270
00:25:52.065 --> 00:25:53.445
that can be a problem. Thankfully,
271
00:25:53.745 --> 00:25:59.365
you know, on the client or at least on the API facing layer, we knew that rights to a single
272
00:25:59.905 --> 00:26:08.460
user's follow, we could send it to the right shot. We are routing at that level. We didn't worry too much about backups to the point, because we we depending on the hourly
273
00:26:08.840 --> 00:26:09.340
daily,
274
00:26:09.800 --> 00:26:24.220
HDFS dumps to, you know, know, take care of those backups for us. We also didn't focus too much on disaster recovery because those mechanisms existed on individual databases. We could always fall back on these snapshots to do a recovery rather than falling back on the Doctor aspect. Yeah. So this process
275
00:26:24.760 --> 00:26:25.580
took about,
276
00:26:26.200 --> 00:26:30.505
you know, a month and a half at a time. The initial load was
277
00:26:30.805 --> 00:26:34.505
the longest at about a week and then bunch of incremental
278
00:26:34.885 --> 00:26:36.505
testing of these CDC,
279
00:26:36.885 --> 00:26:43.400
CDC and the other combinations to make sure that our constancy was in place. Now 1 challenge was how do you handle deletes,
280
00:26:43.860 --> 00:26:45.000
because in, sometimes,
281
00:26:45.940 --> 00:26:49.544
CDCs may not give you the right kind of delete events,
282
00:26:50.005 --> 00:26:54.024
and this thing, I'd be careful of. But, actually, Dataverse had a mechanism to
283
00:26:54.485 --> 00:27:00.240
store the right delta on deletes as well. So we're able to use that to ensure that deletes were probably taken care of and there weren't any
284
00:27:00.620 --> 00:27:21.590
new follows or new edges on this target system that we deleted from the old 1. On the client side, there were a change of schema spread. So we ensured that clients are migrated in a way so so the schema themselves were compatible in a way that both old clients and new clients will continue working. So the old clients would come to the old system and any new fields that are added to the new the new schema
285
00:27:21.965 --> 00:27:48.370
would have both the new and the old values in it. So even if the old files reading this 1, they can have the old fields in there until the until the old database will was fully, fully deprecated and removed. This took a I mean, this whole process I mean, the actual migration step took about 6 to 8 weeks roughly with a whole bunch of testing following for a few months. I mean, it was a reasonable success with, I don't think we had any major downtime and similar use. Now the bonus, I don't know. I I should,
286
00:27:48.770 --> 00:27:50.390
talk to somebody about that.
287
00:27:51.250 --> 00:27:54.455
I wonder if there's a statute of limitations there. I hope not.
288
00:27:56.275 --> 00:28:04.600
And in terms of these style of projects, there's the inherent risk that it starts, you get underway,
289
00:28:04.900 --> 00:28:11.480
you're in the thick of it, you get to what you think is the end, and then the goalposts keep moving or
290
00:28:11.815 --> 00:28:15.195
interruptions come along, and so it becomes a perpetual migration.
291
00:28:15.735 --> 00:28:23.980
I'm wondering if
you could just talk to some of the other risk factors that are involved in embarking down the path of executing a migration
292
00:28:24.520 --> 00:28:27.100
and some of the other failure modes that you've experienced.
293
00:28:27.400 --> 00:28:29.775
I don't know if I can talk about the politics of this, but,
294
00:28:31.215 --> 00:28:41.390
so like any project, right, I mean, when we started off doing this, the initial goal was, hey. User has a bunch of things they acted on. And I use that very vaguely. A user can like, share,
295
00:28:41.770 --> 00:28:48.235
do x on another entity. And initially, this was user can just do like, comment, share on feed posts. Now
296
00:28:48.615 --> 00:28:51.115
being engineers, our you know, both from
297
00:28:51.575 --> 00:28:53.355
our own kind of, ambition
298
00:28:53.815 --> 00:28:59.260
as well as, you know, trying to get buy in on this from other, you know, parts of the organization,
299
00:28:59.640 --> 00:29:00.940
it ended up becoming,
300
00:29:01.480 --> 00:29:05.945
you know, any any entity to any entity to any any other entity,
301
00:29:06.245 --> 00:29:17.940
you know, action. Effectively an edge store. Like, a very, very general type of edge store. I mean, you can do this, but, you know, it becomes a very it becomes a very very large scale project. Pretty much migrating every entity in LinkedIn,
302
00:29:18.565 --> 00:29:27.945
and each and and every 1 of its relationships to every other entity in LinkedIn. That would have been a multiyear project disrupting many, many, many fields, like, like, many, many, projects sorry,
303
00:29:28.309 --> 00:29:37.049
features and products with LinkedIn. And we obviously wanted to start with a smaller scope. So we settled on a user to a select select half a dozen entities,
304
00:29:37.945 --> 00:29:39.404
a few other actions,
305
00:29:39.784 --> 00:29:54.835
along the way. Because there is a temptation to kind of do everything because it is obviously glorious, but you have to kind of, you know, wrap you know, put a put a put a line around that v 0 or v 1 before you start taking more things on. You might find that the other things are nice to have and may never actually materialize
306
00:29:55.375 --> 00:29:55.875
for
307
00:29:56.335 --> 00:29:57.794
resourcing and prioritization
308
00:29:58.495 --> 00:29:58.995
constraints.
309
00:29:59.429 --> 00:30:06.169
And I know I'm being intentionally kind of vague there because it'll be giving you too much No. That's I I totally understand.
310
00:30:07.350 --> 00:30:10.465
No worries. And through your work of
311
00:30:11.005 --> 00:30:12.465
interacting with these systems,
312
00:30:13.245 --> 00:30:19.770
planning for and implementing some of these data migration projects, I'm wondering how that has influenced
313
00:30:20.150 --> 00:30:21.050
your overall
314
00:30:21.670 --> 00:30:25.370
thought process and approach to the design and implementation
315
00:30:25.830 --> 00:30:28.565
of data systems,
data infrastructure,
316
00:30:29.025 --> 00:30:31.525
and also the maybe modeling architecture
317
00:30:32.065 --> 00:30:42.435
that's involved and some of the overall initial planning process that you have factored into the early stages of building these types of systems because of that? You know,
318
00:30:43.075 --> 00:31:08.525
when you do a lot of these, you learn you you learn a few best practices and you learn along the way. And I feel like despite that, we will still make these mistakes and repeat them every now and then just for a sanity hood, just for a reality check. But regardless. Right? I mean, like anything else, you know, there are a few things that can help reduce the risk of any migration. And I think, you know, having a really, really, really good plan and seeing what stakeholders are and knowing what the current
319
00:31:08.870 --> 00:31:13.690
state of all your metrics and all these terms is, are is very important. Right?
320
00:31:14.150 --> 00:31:16.170
In the case of, you know, the follows
321
00:31:16.630 --> 00:31:17.130
migration
322
00:31:17.430 --> 00:31:27.080
or or or the edge migration, we kinda had an idea of who the customers were. It was it was a leading fit. Yes. Later on, we we found a few good use cases for how how can we apply that for companies
323
00:31:27.460 --> 00:31:41.145
and pages and so on. But having that initial plan and knowing what a rough endpoint looks like was kinda key. I mean, even with that, we had to analyze what the source like, what the starting point was, what the starting data store and the the type of data and the cardinality.
324
00:31:41.525 --> 00:31:48.250
All that was key. And, I mean, we we we give it the right project management rigor like anything else. We came up with a migration plan.
325
00:31:48.550 --> 00:32:13.845
I'd like to think it was detailed, you know. And looking back now, clearly, we missed a few things and it wasn't detailed enough. Right? But starting off with what you thought or what we thought was a detailed plan was a good point. Right? Getting that review vetted by most senior folks, more experienced folks, or even talking to other teams that have done something like this was helpful. Again, having the timelines and how much headcount this would need, all that helped getting buy in from which 2. Right? And, again, you know, understanding,
326
00:32:14.559 --> 00:32:18.179
changing this data I mean, going to this target data store,
327
00:32:18.480 --> 00:32:39.020
what that would mean for the product. Previously, I I mean, from memory, the data access on the previous data store, I think, was, I believe is low single digit number of queries for a certain use case. This changed, I mean our analysis showed that this would double in the new use case. So we had talked to our product folks who, hey, is that, you know, latency hit,
328
00:32:39.400 --> 00:32:39.900
acceptable?
329
00:32:40.495 --> 00:32:41.155
How can we
330
00:32:41.695 --> 00:32:44.595
parallelize slash, you know, make it manageable?
331
00:32:45.135 --> 00:32:48.995
And are there other ways to handle the same user experience
332
00:32:49.640 --> 00:33:02.605
that, you know, can improve, can be improved in other case? But this this these impacts can can be mitigated. Right? The other 1 was how do we do this, you know, how do we do this in multiple stages? We didn't want this to be a, you know, a
333
00:33:02.905 --> 00:33:17.995
a full blown 1 giant migration where we start today, there's silence for, like there's the silence from the team for, like, 9 months, and then suddenly, hey, here's an instant and goes to Stewart. Right? We didn't want that to be the case. We want to be something more iterative so we can bring on the front end teams to also
334
00:33:18.535 --> 00:33:19.674
AB test the new
335
00:33:19.975 --> 00:33:34.804
target system at different points of time and use that feedback and and the impact of those changes on metrics. Right? The other 1 was kind of data quality and testing that the migration actually is kind of going the right way. When we got up to x percent, we
336
00:33:35.105 --> 00:33:36.164
marked a few
337
00:33:36.705 --> 00:33:40.325
aspects of the original data store that we wanted to be reflected
338
00:33:40.625 --> 00:33:43.030
in the target data store or the target data,
339
00:33:43.410 --> 00:33:58.794
you know, database. Right? Things like how many users have, how many follows on this kind of entity type. Is it is it the does the same x percent hold in this smaller sample as well? Things like that. Same thing from data quality. Right? Like, you know, how many entries are missing? How many entries have been wrongly
340
00:33:59.410 --> 00:33:59.910
transformed.
341
00:34:00.450 --> 00:34:05.270
This should this should ideally be known before you started, but knowing what that criteria
342
00:34:05.650 --> 00:34:23.640
is helps you better test, right, as part of transformation. And and really, you know, having a pros around again, there's the usual pros around any other service that you do, like monitoring and, you know, getting feedback and getting all logs and the the the operational observability side of things applies to data migrations too. You find that data migration is also
343
00:34:24.295 --> 00:34:25.035
an opportunity
344
00:34:25.655 --> 00:34:45.865
to see if you can redesign your entire system better. If you know that you're going from let's say, a very simple example of going from SQL to NoSQL. In SQL, you have 1 to many. What does that map to in Cassandra? There are, you know I mean, it's it's not as it's there are there are no foreign keys where you can join. I mean, you have foreign keys, but there's no if it joins. So how do you how do you how do you store your list objects
345
00:34:46.245 --> 00:34:54.430
in the document store? Right? Now does that change your UI and your background picture? So when you start writing these index these index entries,
346
00:34:54.730
--> 00:35:00.350
do you have to do it manually or do you wanna do it asynchronously through CDC again? So those decisions kind of
347
00:35:00.730 --> 00:35:19.630
serve us up and you can view them from, okay, if I do that, I'm gonna incur a freshness of maybe 1 second or half a second, but I can have 10 x throughput because I'm not blocked on, you know, doing locks across tables to do a index update. So you can rethink these things as part of your architecture. Right? So when you design systems, I would say start with that mindset
348
00:35:20.035 --> 00:35:24.694
and see how you can be flexible, how you can be scalable, how you can start putting in checks for data quality in place.
349
00:35:24.994 --> 00:35:32.140
If you go really, really, really old school, queuing theory is your best friend. Ultimately, everything is, you know, pipes and and nodes.
350
00:35:32.440 --> 00:35:36.380
But some pipes get bloated, some data some some some lots of data are really gigantic
351
00:35:37.345 --> 00:35:43.605
when, how you break them, wrap them, switch through them is key. And modern systems are no different. Right? You just wanna think about it.
352
00:35:43.905 --> 00:35:44.405
Absolutely.
353
00:35:44.705 --> 00:35:45.205
And
354
00:35:45.550 --> 00:35:47.570
another aspect of data migrations
355
00:35:47.950 --> 00:35:49.730
is that some technologies
356
00:35:50.430 --> 00:35:54.555
and it seems like it's been getting better over the years, but there are definitely some technologies
357
00:35:54.934 --> 00:35:55.835
that are very
358
00:35:56.214 --> 00:35:57.355
resistant to
359
00:35:57.894 --> 00:36:13.015
facilitating that type of project where they really want just wanna lock everything into their their little ecosystem into or into their little box and trying to get anything out is like pulling teeth. Whereas other systems are designed to be interoperable, and they're designed to be maybe composable,
360
00:36:13.715 --> 00:36:22.760
in particular things like modern lakehouse architectures where you have these column oriented storage formats. You can bring different query engines to bear for different use cases.
361
00:36:23.060 --> 00:36:25.000
And I'm wondering what your
362
00:36:25.620 --> 00:36:26.920
experience has been
363
00:36:27.300 --> 00:36:34.415
in terms of the progression of years and progression of technologies as to how that maybe simplifies that work of data migrations
364
00:36:35.035 --> 00:36:38.415
and some of the areas that you see as,
365
00:36:39.260 --> 00:36:40.000
ripe for
366
00:36:40.380 --> 00:36:46.720
further investment to improve the end user experience or maybe even reduce the need for migrations
367
00:36:47.025 --> 00:36:49.765
because those technologies can be more easily composed together.
368
00:36:50.385 --> 00:36:51.585
You know, I'm gonna have something,
369
00:36:52.225 --> 00:36:53.285
controversial here.
370
00:36:53.985 --> 00:36:55.470
Alright. Let's hear it.
371
00:36:56.170 --> 00:36:59.950
And and I'm now scared to be quoted on this. Nope. I like I like controversial.
372
00:37:00.330 --> 00:37:02.510
Hey. It's really, really amazing that
373
00:37:02.810 --> 00:37:15.140
there are so many tools and so many frameworks and so many platforms that are coming up to address different parts of the data migrations. Right? Again, we talked about, you know, in the in the beginning, there was system a that had to be
374
00:37:15.440 --> 00:37:25.755
system b. So you can cut the customers off, do a snapshot, reload, and be done with it oh, and and restart the customers and be done with. And because that's no long no longer acceptable today,
375
00:37:26.055 --> 00:37:32.940
we need to do migrations, you know, at at at 0 down downtime with highest levels of consistency and the highest levels of,
376
00:37:33.480 --> 00:37:34.780
highest guarantees of
377
00:37:35.160 --> 00:37:36.220
privacy and compliance
378
00:37:36.680 --> 00:37:39.099
and disaster recovery and everything else. So
379
00:37:39.485 --> 00:38:01.105
in that context, right, that that stuff between a and b is the really complex hairy bit. Right? Now you can you can make you can make the middle bit really simple and say I'm gonna tie in any source you have via some kind of CDC or or a pool based mechanism and directly dump it to a target like s 3 or your warehouse or some, you know, target. Right?
380
00:38:01.484 --> 00:38:12.770
Or which gives you, you know, batch kind of freshness. Right? And then you can as you as you start digging into I want flexibility of which source I can pull from. I wanna do a whole bunch of transformations
381
00:38:13.150 --> 00:38:18.605
in the middle so I can take n sources and target to m, sorry, and target m destinations.
382
00:38:18.905 --> 00:38:19.405
Right?
383
00:38:19.705 --> 00:38:26.650
So your things like, for example, data flow is a really, really I mean, data flow powered by Beam, for example, right, is a really amazing
384
00:38:27.030 --> 00:38:30.650
pipelining, you know, engine for building the actual transformations
385
00:38:31.110 --> 00:38:40.984
from many sources of megahertz. It comes with a lot of it comes with its own cost and complexities. Right? Then you have, you know, the people like, you know, Debizium and I think and I believe Airbyte
386
00:38:41.285 --> 00:38:43.640
that give you a very quick and simple
387
00:38:44.440 --> 00:38:50.275
an an open source but quick and simple model for configuring these pieces. And then you have, you know, folks like Fivetran who
388
00:38:50.755 --> 00:38:54.934
who have a much more Datadog like comprehensive ecosystem
389
00:38:55.395 --> 00:39:03.299
where they, you know, they give you, I think, a 1,000 connectors and, you know, a 1000 targets and, you know, many, many pipeline options and tunable
390
00:39:03.760 --> 00:39:55.510
things. Right? As I mean, these are getting complicated because there are so many configurations to pick from. So there's no there's no
like, anyone you pick, there's gonna be obscuring that you have to do. There's gonna be some maintenance. There'll be cost on the computer storage side. Right? Now for some you know, they might say, well, we'll we'll keep it we'll give you full control, but then you gotta manage your cost on your side. Rules like everything care for you is something that becomes very expensive, right, because there's a lot of, opaque blobs that you have to deal with. So I like to go back to fundamentals, and this is where my controversial thing comes in. Right? If I don't understand it, that's always my problem. I'm not saying I'm I'm I'm you know, I should I shouldn't understand it. But if I don't understand it, I'd like to know what the trade off he is proposing and giving me. Unless I know those trade offs, I'm not gonna decide I'm not able to decide which I wanna use. And maybe it's a it's a maybe it's a I was gonna say maybe I'm pampered for having been at LinkedIn and Google for
391
00:39:55.890 --> 00:40:09.710
having those tools available to me. But I think what I consider myself lucky for is understanding how they work and why they work and when and when they don't work. Or or at least, you know, what I have to pay for making them work. I think when I have that, I'd rather fall back
392
00:40:10.330 --> 00:40:20.465
to cobbling things myself because I'm okay with it. So I can see why things are going wrong and where things are going wrong. That's not for everybody there. Data moving and data migration is not your bread and butter. It's not your, you know, core business.
393
00:40:20.845 --> 00:40:34.934
So starting off with 1 of these tools is a perfectly good option. Right? Finding the right Upwork talent to manage these tools is not such a bad option. But I would recommend understanding them, knowing what trade offs are, and not be surprised when you when you when you do get hit by the bill. And
394
00:40:35.875 --> 00:40:41.734
in your experience of working in this industry, being involved in data migration projects,
395
00:40:42.380 --> 00:40:43.039
and particularly
396
00:40:43.740 --> 00:40:46.799
in your current role of helping organizations
397
00:40:47.260 --> 00:40:48.400
automate their runbooks,
398
00:40:48.700 --> 00:40:53.395
which I imagine involves some measure of data replication or data migration
399
00:40:54.015 --> 00:40:54.515
resolutions,
400
00:40:55.455 --> 00:41:04.000
fix fixing things when they break in the process. What are some of the most interesting or innovative or unexpected ways that you've seen teams approach this question of data migrations,
401
00:41:04.300 --> 00:41:05.920
whether from understanding
402
00:41:06.220 --> 00:41:11.935
whether and how to do it to the technical elements of how they actually executed on it. Interesting.
403
00:41:12.395 --> 00:41:14.415
So our customers are not your traditional
404
00:41:14.880 --> 00:41:23.859
tech companies. Our customers are more on the l 1, l 2 support side, on the
NOC and SRE side. Right? So they don't deal as much with data as
405
00:41:24.175 --> 00:41:36.180
my as my previous teams have. Right? So in other in in a sense, we are building the tooling so that they don't have to worry about these migrations. Now the scale of our migrations are much, much lower, by the way, in this company. We actually
406
00:41:36.580 --> 00:41:44.785
again, it's surprising that we haven't used this word till now. We use AI to come up with the right runbooks to look at their data to see what kind of transformations we should create,
407
00:41:45.325 --> 00:41:48.785
for them. And these are these are, you know, these are
408
00:41:49.180 --> 00:41:52.240
low scale data migrations, low scale, low frequency data migrations.
409
00:41:52.780 --> 00:41:58.000
I would always call them data fixing because they because of their current engineering engineering practices,
410
00:41:58.955 --> 00:42:08.460
data may not always be consistent. So instead of fixing it upstream, they are falling on falling back on tools like us to apply certain rules on what what these data should look like.
411
00:42:09.020 --> 00:42:11.440
We are a 1 company that was managing bookings,
412
00:42:11.900 --> 00:42:12.640
for for,
413
00:42:13.180 --> 00:42:13.680
transportation.
414
00:42:14.619 --> 00:42:20.395
And a lot of the deletions were done using our tool, right, leading bookings that were older than,
415
00:42:20.695 --> 00:42:22.454
you know, x days. Right?
416
00:42:22.855 --> 00:42:28.859
Instead of doing it as a offline job sorry, as a offline or a nearline job running in the background,
417
00:42:29.160 --> 00:42:39.415
they would just use that tool every x minutes, hours to clean up, to do any garbage collection. So this so again, given it's a low case thing, we could almost get away with a lift and shift, by the way.
418
00:42:39.875 --> 00:42:41.095
But we're somewhere between
419
00:42:41.700 --> 00:42:46.920
a series of small batch lift and shifts on much much more partial datasets
420
00:42:47.460 --> 00:42:48.920
than a full blown migration.
421
00:42:49.984 --> 00:42:54.005
And in your experience of working on these projects
422
00:42:54.704 --> 00:42:57.365
and going through these various
423
00:42:58.240 --> 00:43:04.820
generational shifts in the tech industry, what are some of the most interesting or unexpected or challenging lessons that you've learned in the process?
424
00:43:05.565 --> 00:43:05.885
You know,
425
00:43:06.605 --> 00:43:07.885
data migration challenges
426
00:43:08.525 --> 00:43:10.545
the challenges around data migrations,
427
00:43:11.165 --> 00:43:15.630
there are the there are the technical ones. A big majority of them are actually nontechnical.
428
00:43:16.410 --> 00:43:19.905
They're about understanding the end to end ecosystem of
429
00:43:20.305 --> 00:43:27.845
where that data or that or that data store fits in. And that means that like any good engineer in any other domain, right, you have to be
430
00:43:28.320 --> 00:43:31.940
aware and interested in being aware of the broader impact
431
00:43:32.320 --> 00:43:34.100
of what this piece of data,
432
00:43:34.720 --> 00:43:40.694
is gonna have or change that or or affecting that piece of data is gonna have. Right? Which means understanding the customer experience,
433
00:43:41.315 --> 00:43:42.934
understanding the business use case,
434
00:43:43.634 --> 00:43:49.440
understanding the priorities of, you know, who wants this done when and how and where and how much do you wanna spend for it.
435
00:43:50.140 --> 00:43:58.905
So, yes, there are technical challenges. There are you know, we can do synchronization. We can do all these. You can you can read Martin Klettman's book on how to do the best in replication, you know, and charting,
436
00:43:59.205 --> 00:44:17.714
I mean, how you can apply the best charting and application approaches. But when you apply the real world, that's like maybe 5%, 10%. The why, what, who it impacts, those are those are the key. And just like that, you know, to manage cost and be efficient, how you manage, how you project manage this, how you communicate this, how you deal with real world surprises,
437
00:44:18.470 --> 00:44:24.330
Those are all kind of, you know, there. Right? I don't see them going anywhere. And it's fine. Right? That's okay.
438
00:44:25.270 --> 00:44:26.965
And in your experience
439
00:44:27.265 --> 00:44:27.765
too,
440
00:44:28.065 --> 00:44:38.370
there, I'm sure, have been cases where somebody says, we need to move this data from over here in this system to over there in this system. What what are some of the cases or what are some of the heuristics
441
00:44:38.830 --> 00:44:44.210
that you use to say, nope. That is the wrong choice. Don't do it. Man, you gotta get me in trouble now.
442
00:44:45.285 --> 00:45:03.635
Wrong wrong can mean a different it can mean different things. Right? I I like I like numbers. Right? So when I look at a system design, I think of it I mean, I I enjoy going into the numbers about, you know, how it affects latencies, how it affects cost, how it affects, know, your SLOs, how it affects, you know, throughputs,
443
00:45:04.255 --> 00:45:06.435
how it affects user metrics. So
444
00:45:06.815 --> 00:45:14.130
if you you know, your question can mean why and how. Right? The first thing is why. You know, why you wanna do this? I know it's a shinier,
445
00:45:14.910 --> 00:45:17.730
new system that might get you promoted and get you a bonus,
446
00:45:18.110 --> 00:45:21.735
but what is the value of it? Right? Now as to the how,
447
00:45:22.035 --> 00:45:30.240
are the components architecturally sound? Do they have I mean, are they just better technologies, or are there also impacts on how you're gonna maintain them, who's gonna maintain them,
448
00:45:30.779 --> 00:45:58.335
what is gonna like, what is it what is the TCO or the total cost of ownership of this new, you know, of of both the current of of the of the migration process as well as the final effort. And that's where I think being a manager kind of helps. Right? Like, you know, you kind of see that intrinsically. You ask that a lot more naturally. Right? And and I would say that almost I mean, every senior, you know, engineer in almost any area will be trained to ask these questions. And, again, to me, you know, the how is a small part rather than the why. And
449
00:45:58.955 --> 00:46:00.895
And as you continue
450
00:46:01.275 --> 00:46:01.775
to
451
00:46:02.280 --> 00:46:23.710
keep an eye on the data industry as you work in your current role as a CTO where you have to have a lot of decision making power over what systems to use, how to use them, what are some of the things that you are keeping in mind as you build out your technology stack, as you work with your customers and help them in their decision making process to
452
00:46:24.250 --> 00:46:25.310
reduce the
453
00:46:25.675 --> 00:46:28.335
current and future pain that they will experience
454
00:46:28.635 --> 00:46:30.095
due to these migration
455
00:46:30.395 --> 00:46:30.895
projects?
456
00:46:31.995 --> 00:46:43.500
I mean, these days, there are so many there are so many technologies out there that I find myself getting all rolled to. I mean and and I love technology. I love reading about things. I love learning about things. And there's just so many that that
457
00:46:43.815 --> 00:46:45.195
it's it's it's very hard to,
458
00:46:45.655 --> 00:46:46.234
you know,
459
00:46:46.695 --> 00:46:59.490
make them out through all the noise. Right? And to to your early point, we talked about how there are so many systems to to help in the middle of migration. Right? And each 1 promised a slightly you know, they promised a lot of lot of delta in certain aspects.
460
00:47:00.190 --> 00:47:03.285
And there is the marketing material and it is a reality.
461
00:47:03.825 --> 00:47:15.569
These days, it's not that practical to be able to try everything out. So you had to kind of, you know, go back to your intuition on on what do you think worked, what might work, what is the kind of foundation. I mean,
462
00:47:15.964 --> 00:47:20.464
before I go there, it's different people have different different ways of us saying what is the,
463
00:47:21.325 --> 00:47:22.525
you know, us saying what is,
464
00:47:23.530 --> 00:47:26.109
what are the strengths and weakness of of an approach. Right?
465
00:47:26.810 --> 00:47:35.745
My 1 is basically, I don't know if I can understand it, at least from a theoretical foundation perspective. Right? If I see that makes sense, then I'm a bit more willing to,
466
00:47:36.365 --> 00:47:37.025
you know,
467
00:47:37.725 --> 00:47:41.630
take a trial, take a punt on giving it, you know, on giving it a go. Right?
468
00:47:42.250 --> 00:47:45.130
Even for us, I mean, we played around with a lot of,
469
00:47:45.690 --> 00:47:46.590
vector databases,
470
00:47:47.370 --> 00:47:48.645
in our
product. Right?
471
00:47:49.984 --> 00:47:57.685
And, I mean, I I think about a year ago, or year and a half ago, there was such a frenzy about so many vector DBs on the market, like Pinecone, BB8,
472
00:47:58.850 --> 00:48:02.790
you know, because, you know, there's getting, like, you know, lots and lots of VC money. Right? And,
473
00:48:03.250 --> 00:48:08.905
again, for me, what I didn't quite get was I get it that vector databases are very optimized for
474
00:48:09.365 --> 00:48:10.585
putting and getting
475
00:48:11.125 --> 00:48:13.224
vectoring for about something. Right?
476
00:48:13.880 --> 00:48:16.859
What I didn't quite understand at the time was, look,
477
00:48:17.240 --> 00:48:17.740
Postgres
478
00:48:18.200 --> 00:48:21.579
is an amazing, amazing engine where you can where where people have written
479
00:48:21.880 --> 00:48:22.779
some of the
480
00:48:23.445 --> 00:48:24.425
most complicated
481
00:48:25.045 --> 00:48:25.545
indexing
482
00:48:26.085 --> 00:48:33.549
libraries for it. Now, yes, being that generic is not gonna make Postgres the fastest DB in the world on every benchmark.
483
00:48:34.730 --> 00:48:44.715
But it's good enough for for our use case where we can put in a 1,000 or a 1000000000 documents a month. Again, we're not we're not Google scale. Right? So do I really need something that's that that complicated?
484
00:48:45.255 --> 00:48:45.755
And
485
00:48:46.135 --> 00:48:58.730
yeah. I know. We've been going just fine with Elastic and Postgres for a lot for a while. Right? I see the same thing with our customers. Our customers get a bit starry eyed when they hear about anything. There's a bit of formal around. If I don't use it, will I miss out? Will I
486
00:48:59.175 --> 00:49:00.475
be seen as uncool?
487
00:49:01.255 --> 00:49:08.075
But really, going back to what is your business? If your business is about selling, you know, selling, you know, widgets,
488
00:49:08.490 --> 00:49:19.135
Why are you trying to use the most fanciest database out there? Right? Is is if your customer is not struggling today or or they wanted the next 10 x use case, what does this typical advance,
489
00:49:19.535 --> 00:49:20.755
advancement advance?
490
00:49:21.214 --> 00:49:21.714
Advance.
491
00:49:22.095 --> 00:49:23.474
Give you from a business perspective,
492
00:49:24.310 --> 00:49:25.770
or or from a customer experience perspective.
493
00:49:26.230 --> 00:49:29.290
So I think tying it back to business and customer kind
494
00:49:29.590 --> 00:49:32.475
of KPIs, I suppose, is always helpful. You know?
495
00:49:33.515 --> 00:49:36.575
I mean I mean, I I still feel dumb that I don't understand
496
00:49:36.875 --> 00:49:40.650
how awesome these specific, you know, vector database are.
497
00:49:41.130 --> 00:49:41.630
But,
498
00:49:42.170 --> 00:49:42.910
you know,
499
00:49:43.210 --> 00:49:44.349
I I think I'm okay.
500
00:49:45.210 --> 00:49:46.270
So the
501
00:49:46.730 --> 00:49:51.095
the long and short of this conversation is data migrations are hard, just use Postgres.
502
00:49:53.315 --> 00:49:55.015
No. No. No. I mean, you know what?
503
00:49:58.205 --> 00:50:00.300
So back at SweetHawk 15 years ago,
504
00:50:00.940 --> 00:50:05.760
the I think Lamb was the biggest thing, like, you know, Linux, Apache, MySQL,
505
00:50:06.135 --> 00:50:06.635
PHP.
506
00:50:07.895 --> 00:50:11.835
Right? And MySQL was the most, most, most, most covered,
507
00:50:12.215 --> 00:50:12.795
you know,
508
00:50:14.150 --> 00:50:16.650
start up DB, you know, for lack of a better word. Right?
509
00:50:17.750 --> 00:50:24.565
Again, I was not a database person at the time. I was, you know, an embedded guy. And when I read the 2, I couldn't tell
510
00:50:24.945 --> 00:50:25.445
why
511
00:50:25.825 --> 00:50:27.125
1 was better than the other.
512
00:50:27.905 --> 00:50:28.405
My
513
00:50:28.705 --> 00:50:35.180
switch to using Postgres was for Street Hawk was that it was more SQL compliant. That was the only thing I had to go by.
514
00:50:36.040 --> 00:50:38.140
15 years down, Postgres is still,
515
00:50:38.455 --> 00:50:41.035
I mean, you know where Postgres is today. Right?
516
00:50:42.295 --> 00:50:47.275
Again, it's it's not it's not about Postgres, but, you know, I think if you understand the fundamentals, if you if you follow your heuristics,
517
00:50:47.900 --> 00:50:50.720
it takes you a long way. You could be wrong, and you'll listen to something out of it.
518
00:50:51.099 --> 00:50:52.299
But don't be scared of,
519
00:50:53.180 --> 00:50:54.559
missing out on the shiny thing.
520
00:50:55.005 --> 00:50:55.505
Absolutely.
521
00:50:55.964 --> 00:50:57.265
Use boring technology.
522
00:50:57.805 --> 00:50:59.184
You sleep better at night.
523
00:50:59.724 --> 00:51:00.305
I mean,
524
00:51:00.684 --> 00:51:04.840
to me to me, boring is sexy. Right? To me, boring that
525
00:51:05.220 --> 00:51:09.720
has a lot of good details, you know, and doesn't make things complicated is sexy.
526
00:51:10.255 --> 00:51:11.715
Are there any other aspects
527
00:51:12.015 --> 00:51:15.155
of this overall question of large scale data migrations,
528
00:51:15.455 --> 00:51:16.355
your experience
529
00:51:16.655 --> 00:51:18.820
of working through them,
530
00:51:19.120 --> 00:51:22.580
and some of the challenges that people are,
531
00:51:23.040 --> 00:51:28.395
likely to face in the process that we didn't discuss yet that you'd like to cover before we close out the show? You know,
532
00:51:28.875 --> 00:51:30.815
I mean, this is always gonna be more. Right?
533
00:51:31.275 --> 00:51:35.295
Few things that I that I keep thinking about I mean, I've been reading a lot about data
534
00:51:35.675 --> 00:51:39.060
virtualization these days. Right? And I think if you look at data lakes and data warehouses,
535
00:51:39.520 --> 00:51:40.660
they kinda give you
536
00:51:41.440 --> 00:51:42.820
an aspect of virtualization.
537
00:51:43.360 -->
00:51:47.685
The idea being that you have all this data, why don't you just create views out of all these sources,
538
00:51:48.305 --> 00:51:49.845
without physically moving it anywhere.
539
00:51:50.305 --> 00:51:53.860
How practical it is? I'm still you know, my numbers aren't adding up.
540
00:51:54.420 --> 00:51:57.080
How I mean, there's always gonna be a use case where that works.
541
00:51:57.540 --> 00:52:05.195
If you're okay with the if you're okay with the latency guarantees in terms of let's say, for example, creating expensive create creating 1 off infrequent,
542
00:52:06.295 --> 00:52:09.355
detailed reports that need to join a 1000 data sources,
543
00:52:09.655 --> 00:52:13.500
sure. That that that that works. If you need a very live
544
00:52:13.880 --> 00:52:16.940
user, you know, dashboard being served to millions of customers,
545
00:52:17.319 --> 00:52:23.105
it's gonna start. Right? The other 1, I've been, again, hearing about and I'm trying to make the
546
00:52:23.565 --> 00:52:28.470
I'm trying to get past the labeling aspect of it is data federation. You know, again, how do you how do
547
00:52:28.849 --> 00:52:37.995
you bring different kinds of data in different transformed ways to different customers? Right? Seems like a very broad problem, but I think the details are way it's gonna be interesting.
548
00:52:38.775 --> 00:52:41.915
We spoke about replication. Right? I mean, replication is is is is
549
00:52:42.319 --> 00:52:48.420
as opposed if you everything is replication. Everything is a migration. Everything is a move. 1 thing I really wanted to talk about,
550
00:52:49.119 --> 00:52:51.859
obviously, briefly was this replication scheme,
551
00:52:52.545 --> 00:52:54.165
that I came across from Netflix,
552
00:52:55.105 --> 00:53:10.960
called DB log. Right? I mean, you have DBZM, you have Databurst, you have HVR, and all the things that do replication for you. DB log was so simple and beautiful. It it works by creating these watermarks in your in your data stream or or you get a change capture stream so that you can actually do incremental, you know, replications,
553
00:53:11.385 --> 00:53:34.075
you know, very live matter instead of waiting for a whole dump of snapshot. What definitely worth reading, I'm I'm happy to share that, link as well. And, yeah, I think I think we we haven't I don't really focus a lot on data. Right? We didn't talk that much about how should the application and application teams and, you know, double quote front end teams react to all this. I think that that's interesting thing as well, which you can yeah. So I can jump into yeah. I suppose,
554
00:53:34.555 --> 00:53:36.830
this is so so much here, really. Absolutely.
555
00:53:37.130 --> 00:53:40.430
And in particular, I like you calling out the,
556
00:53:40.890 --> 00:53:49.365
the inclusion of the application teams in the process process because if you're trying to do it all from a single point solution perspective,
557
00:53:49.984 --> 00:53:52.730
then you're going to be having everybody in the
558
00:53:53.030 --> 00:54:01.210
organization, you know, their whole weight bearing down on you waiting for this to be complete because either you're breaking things or you're making them move slower when really
559
00:54:01.665 --> 00:54:04.484
the entire organization should be moving in lockstep
560
00:54:04.785 --> 00:54:06.645
to achieve that end goal.
561
00:54:07.025 --> 00:54:09.445
And in a lot of cases, the application developer
562
00:54:10.040 --> 00:54:13.340
can simplify a lot of that work because maybe
563
00:54:13.800 --> 00:54:14.700
you don't necessarily
564
00:54:15.000 --> 00:54:19.100
have to have all of the data completely synchronized in both systems.
565
00:54:19.635 --> 00:54:28.775
Maybe they can write to 1 system and accept latency in the read from the other 1, or maybe they can handle the double rights so that you don't have to handle as much of it in the back end or
566
00:54:29.140 --> 00:54:29.640
just
567
00:54:29.940 --> 00:54:35.559
incorporating them in the overall planning steps as well can help reduce the actual complexity of the move.
568
00:54:36.260 --> 00:54:36.760
Yes.
569
00:54:37.415 --> 00:54:38.635
With a double exclamation.
570
00:54:40.535 --> 00:54:46.715
So some I mean, I know I missed this in the trigger earlier. Right? Sometimes data migrations happen because you're trying to
571
00:54:47.120 --> 00:54:54.900
save toil for certain teams. Now in a very, very small start up, that that is just 1 team. There, your problem is, should I prioritize this migration now,
572
00:54:55.775 --> 00:55:01.795
because deep net of point, can we can we wait for customers 3, 4, 5 customers to come on board and pay for this migration?
573
00:55:02.095 --> 00:55:03.555
In really large organizations,
574
00:55:04.289 --> 00:55:05.029
this problem
575
00:55:05.569 --> 00:55:08.309
is kind of I mean, the the system is similar,
576
00:55:08.690 --> 00:55:10.390
but it's framed in a different way.
577
00:55:10.690 --> 00:55:17.965
You have a 1,000 teams with, you know, their own priorities and their own road maps and their own kind of, you know, monthly, quarterly, yearly, you know, OKRs.
578
00:55:18.425 --> 00:55:26.710
It becomes a prioritization problem, like like, you know, why now and when and how and so on. So the teams coming together to walk in lockstep,
579
00:55:27.089 --> 00:55:27.830
that's ideal.
580
00:55:28.369 --> 00:55:58.705
But often, it's a bit more it's a bit more coarse. In terms that, you know, the teams have to see value for them first. Application teams might say, look, my I was using API 1 before. My application is working fine. Why do I care if you change or if you migrate this data now? Now the if the answer is purely because it'll help us manage it better, that's not a good enough answer. But if the answer is doing so will help you unlock those 5 things in your road map that you can target now rather than for 2
years. Is that all good for you? So I think that's where addressing the user concerns, the business priorities, and roadmaps, like, all that kind of all these nontechnical
581
00:55:59.085 --> 00:56:00.110
people problems kind of
582
00:56:00.830 --> 00:56:12.005
are important. Definitely. Yeah. And as technologists, it's definitely easy to fall into the trap of, I'm just going to engineer my way through this when just talking to people can save you a lot of time.
583
00:56:12.305 --> 00:56:30.625
That's right. You know, be on the latest and greatest, baby. That's right. Alright. Well, for anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest gap in the tooling or technology that's available for data management today? You know,
584
00:56:31.185 --> 00:56:32.805
there are few actually. Right? And,
585
00:56:33.585 --> 00:56:35.845
again, I know I don't keep, saying this.
586
00:56:36.320 --> 00:56:37.940
The silos in an organization
587
00:56:38.400 --> 00:56:38.900
are
588
00:56:39.280 --> 00:56:43.140
are problematic. Right? I mean, as organizations grow, as companies grow,
589
00:56:43.575 --> 00:56:44.155
you know,
590
00:56:44.855 --> 00:57:01.005
for better or worse, they find their own ways of doing things, and that changes, you know, how they manage data, how they manage data, access data, control data, process data, and so on. So, you know, effectively, you you do have data files that have to be managed and have to be worked on. Right? So tooling has to somehow
591
00:57:01.625 --> 00:57:03.885
a tooling and framework, right, have to somehow
592
00:57:04.425 --> 00:57:16.590
bridge that gap and and and and and and make it easy to, you know, work across ourselves. Also, you know, again, over time, you know, better or worse, you know, organizers end up having a lot of bespoke and manual and nonstandard process,
593
00:57:16.905 --> 00:57:36.255
you know, as they grow. I know a a very knee jerk reaction to say, hey. Let's have a central data org. This, again, has become this brings no problems. Right? You're talking about functional versus feature teams and hybrid teams. So on. You know, tools need to kind of ensure that you can take in whatever manual process you have today and onboard them into this tooling rather than force, you know, tooling force
594
00:57:36.954 --> 00:57:43.615
everyone to change their process just for tool. Right? Again, I know it sounds like a buzzword but AI actually can go a long way in in
595
00:57:44.000 --> 00:57:45.700
transforming those manual things into
596
00:57:46.000 --> 00:57:54.165
these tools. Right? The other thing is, you know, collaboration is 1 aspect. Like, I mean, a friend of mine was talking about, you know, data usability and data accessibility and data collaboratively.
597
00:57:55.025 --> 00:58:11.105
Like, how are different roles and responsibilities in your organization coming together on a single piece of, you know, unified, you know, data fabric or whatever. Right? How they're able to make sense of data in a way that's that's secure and compliant. So, and I think here's where, you know, the the idea of virtualization
598
00:58:11.405 --> 00:58:24.859
kind of makes sense. And if it is performant, it'll it'll it'll go a long way. So you can actually avoid the data moves. Cloud helps here because, you know, again, cloud is still a vehicle. It's not the final point, but, yeah, it can help you. Right? And I guess, status tooling and standard of APIs,
599
00:58:25.174 --> 00:58:29.674
you know, the other way to look forward to. But again, you have a lot of friction in the organization
600
00:58:30.295 --> 00:58:31.194
that is more
601
00:58:31.494 --> 00:58:42.280
people and process oriented, not necessarily technical oriented. Yeah. I think as these areas kind of, you know, evolve, there's definitely a lot more innovation and lot more tools and lot more frameworks because we all wanna see that world with
602
00:58:42.660 --> 00:58:43.160
seamless,
603
00:58:43.895 --> 00:58:45.755
data management and integration.
604
00:58:46.135 --> 00:58:51.035
Alright. Well, thank you very much for taking the time today to join me and share your experiences
605
00:58:51.734 --> 00:59:12.890
of going down these hard paths of data migrations and some of the lessons that you've learned in the process. It's definitely always great to hear from people who have been in the trenches and done the hard work. So appreciate all the time and energy you've put into that in your various roles and your time sharing it with us today, and I hope you enjoy the rest of your day. Thank you, Tobias. It was really, really fun being here and, you know, a great walk down memory
606
00:59:16.150 --> 00:59:16.650
lane.
607
00:59:19.245 --> 00:59:20.145
Thank you for listening.
608
00:59:20.445 --> 00:59:29.430
Don't forget to check out our other shows, podcast dot in it, which covers the Python language, its community, and the innovative ways it is being used, and the Machine Learning podcast,
609
00:59:29.970 --> 00:59:32.710
which helps you go from idea to production with machine learning.
610
00:59:33.010 --> 00:59:35.365
Visit the site at data engineering podcast.com
611
00:59:35.825 --> 00:59:45.880
to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a product from the show, then tell us about it. Email hosts at data engineering podcast.com
612
00:59:46.500 --> 00:59:47.240
with your story.
613
00:59:47.540 --> 00:59:52.685
And to help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers.