Strategies For A Successful Data Platform Migration

WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 7/6/2024 12:05:22 PM
Duration: 4192.728
Channels: 1
1
00:00:11.264 --> 00:00:15.525
Hello, and welcome to the Data Engineering podcast, the show about modern data management.
2
00:00:16.384 --> 00:00:18.164
Introducing RudderStack profiles.
3
00:00:18.705 --> 00:00:28.539
RudderStack profiles takes the SAS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable enriched data to every downstream team.
4
00:00:29.145 --> 00:00:35.485
You specify the customer traits, then profiles runs the joins and computations for you to create complete customer profiles.
5
00:00:36.280 --> 00:00:39.420
Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack.
6
00:00:43.365 --> 00:00:46.745
Modern data teams are using hex to 10 x their data impact.
7
00:00:47.285 --> 00:00:51.145
Hex combines a notebook style UI with an interactive report builder.
8
00:00:51.630 --> 00:00:57.809
This allows data teams to both dive deep to find insights and then share their work in an easy to read format to the whole org.
9
00:00:58.485 --> 00:01:02.504
In hex, you can use SQL, Python, R, and no code visualization
10
00:01:02.885 --> 00:01:05.705
together to explore, transform, and model data.
11
00:01:06.640 --> 00:01:12.740
HEX also has AI built directly into the workflow to help you generate, edit, explain, and document your code.
12
00:01:13.175 --> 00:01:19.355
The best data teams in the world, such as the ones at Notion, AngelList, and Anthropic use HEX for ad hoc investigations,
13
00:01:19.815 --> 00:01:24.060
creating machine learning models, and building operational dashboards for the rest of their company.
14
00:01:24.600 --> 00:01:30.539
Hex makes it easy for data analysts and data scientists to collaborate together and produce work that has an impact.
15
00:01:30.935 --> 00:01:33.195
Make your data team unstoppable with HEX.
16
00:01:33.495 --> 00:01:34.875
Sign up today at dataengineeringpodcast.com/hex
17
00:01:36.775 --> 00:01:52.905
to get a 30 day free trial for your team. Your host is Tobias Macy. And today, I'm interviewing Gleb Mojansky and Rob Guretsky about when and how to think about migrating your data stack. So, Gleb, you've been on a few times, but for folks who haven't listened to any of your past appearances, if you'd like to give a brief introduction.
18
00:01:53.740 --> 00:02:02.000
Hi, Tobias. Great to be back. My name is Gleb. I am CEO and cofounder of DataFold. We automate data testing.
19
00:02:03.325 --> 00:02:04.625
Before starting DataFold,
20
00:02:04.925 --> 00:02:08.465
I was a data engineer pretty much my entire career,
21
00:02:09.085 --> 00:02:10.545
companies including Autodesk
22
00:02:10.960 --> 00:02:12.260
and, Lyft.
23
00:02:12.880 --> 00:02:17.780
And, topic of migrations is near and dear to my heart as I spent 2 years
24
00:02:18.185 --> 00:02:27.085
helping Lyft migrate from 1 data warehouse to a more scalable data lake and definitely lots of lessons hard lessons learned in the process.
25
00:02:27.879 --> 00:02:29.659
And, Rob, how about yourself?
26
00:02:30.280 --> 00:02:42.055
I've been working in the field of data for over 15 years. I got my start working in big finance, for Lehman Brothers, actually, actually, when I began my career. Began working as a data integration developer, an ETL developer, a data architect. Lots of different titles over the years,
27
00:02:46.550 --> 00:02:48.730
but I have found my footing and grew
28
00:02:49.190 --> 00:03:02.655
in a path working for Major League Baseball Advanced Media for over 15 years where I got to grow out a data team there and go from being the first hands on ETL developer using graphical tools like Informatica
29
00:03:03.300 --> 00:03:06.040
to moving all the way forward to being able
30
00:03:06.420 --> 00:03:08.840
to realistically call myself a data engineer
31
00:03:09.220 --> 00:03:11.400
working with, cloud based data warehouses,
32
00:03:12.125 --> 00:03:14.305
orchestration platforms, and the like. So
33
00:03:14.765 --> 00:03:19.105
migrations have become a fact of life. It's not something that anybody particularly
34
00:03:19.800 --> 00:03:27.260
looks forward to, but if you are at the same company long enough or in this field long enough, you're going to encounter 1 sooner or later.
35
00:03:27.834 --> 00:03:31.694
It's a necessary evil and it's certainly not the thing that anybody
36
00:03:31.995 --> 00:03:39.069
is specifically excited to do but it is a means to an end that I have found the end is often where there's the real excitement,
37
00:03:39.530 --> 00:03:42.189
when you've moved forward by orders of magnitude. So,
38
00:03:42.650 --> 00:03:46.030
thanks for having me on. Excited to chat about it further today. Absolutely.
39
00:03:46.655 --> 00:03:53.795
And, Gleb, again, for folks who haven't listened to your past appearances, if you wanna give a brief recap on how you first got started working in data.
40
00:03:54.330 --> 00:03:54.489
Yeah.
41
00:03:55.209 --> 00:03:56.989
My first start was
42
00:03:57.290 --> 00:03:58.590
back in 2014
43
00:03:59.689 --> 00:04:01.470
when I joined, Autodesk's
44
00:04:02.375 --> 00:04:06.795
consumer group. At the time, it was division of Autodesk that was tasked at,
45
00:04:07.415 --> 00:04:09.034
creating a portfolio of apps,
46
00:04:09.735 --> 00:04:10.235
targeted
47
00:04:10.614 --> 00:04:11.114
at
48
00:04:11.430 --> 00:04:12.970
consumers for creative
49
00:04:13.270 --> 00:04:15.450
work, such as modeling 3 d
50
00:04:15.750 --> 00:04:17.670
objects for 3 d printing and,
51
00:04:18.070 --> 00:04:20.250
other exciting things for creativity.
52
00:04:20.715 --> 00:04:24.735
And I was pretty much tasked with creating a data platform from scratch
53
00:04:25.355 --> 00:04:32.070
and, for these portfolio of apps. And it was a really exciting time because I believe that was a year when Airflow
54
00:04:32.610 --> 00:04:36.630
was just released out of Airbnb as an open source project. And
55
00:04:37.005 --> 00:04:41.505
Snowflake, I believe, came out of stealth and we talked to them very early. And Redshift
56
00:04:42.125 --> 00:04:42.945
was the hottest,
57
00:04:43.485 --> 00:04:45.345
technology for warehousing.
58
00:04:45.970 --> 00:04:47.350
So a lot of really
59
00:04:47.650 --> 00:04:49.910
a lot of things that are currently considered
60
00:04:50.210 --> 00:04:53.270
extremely foundational were cutting edge at the time. And
61
00:04:53.765 --> 00:04:56.825
starting data platform from scratch, I was able to explore
62
00:04:57.205 --> 00:04:59.065
and try out all these different technologies.
63
00:04:59.445 --> 00:05:02.160
That was very exciting. And then I moved to Lyft where,
64
00:05:02.940 --> 00:05:11.835
we've really tested a lot of these technologies, but at a very large scale and quickly learned that we have to find more
65
00:05:12.215 --> 00:05:14.375
scalable solutions, which is,
66
00:05:14.775 --> 00:05:17.275
pretty much how I got into doing data migration.
67
00:05:17.640 --> 00:05:25.820
And, Rob, do you remember how you first got started working in data? I think like many things in life, it was probably a bit happenstance as it was sort of where I began my career,
68
00:05:26.200 --> 00:05:27.855
right after college, working,
69
00:05:28.395 --> 00:05:30.255
on a project that was involving
70
00:05:30.795 --> 00:05:31.695
some back end,
71
00:05:32.075 --> 00:05:37.610
purchasing systems, but ultimately then required data integration to be done and it was an area that I found myself
72
00:05:38.070 --> 00:05:40.010
really enjoying. And so when the opportunity
73
00:05:40.470 --> 00:05:44.375
presented itself to start a team at Major League Baseball, I jumped on that and
74
00:05:44.855 --> 00:05:46.315
after 15 years there,
75
00:05:46.775 --> 00:05:53.755
really building up the muscle for it, I recently got an opportunity to jump on as a principal engineer at Eventbrite
76
00:05:54.410 --> 00:06:11.945
And sort of the the thesis that I had was that all of the hard lessons learned over those years doing several migrations, seeing several evolutions of the stack would allow us to move even more quickly. And jumping into Eventbrite, that's certainly been the case. Been able to really accelerate the timeline around a migration and
77
00:06:12.289 --> 00:06:13.909
really see the effects of
78
00:06:14.210 --> 00:06:18.069
what moving to a modern data stack can offer an organization in
79
00:06:18.449 --> 00:06:26.625
what feels like record time. So I think the journey has been consistently about kind of knowing the change is constant and then being ready to adapt to it as needed.
80
00:06:27.240 --> 00:06:31.500
And so bringing us now to the subject of data migrations,
81
00:06:32.600 --> 00:06:39.625
migration can mean anything from I need to rename this 1 column to be something a little bit more, understandable
82
00:06:39.925 --> 00:06:43.225
to I need to move petabytes and petabytes of data
83
00:06:43.800 --> 00:06:56.715
from 1 architecture to a completely different paradigm to I need to move all of my batch data into streaming workflows. And I'm wondering if you can just start by describing, for the purposes of this conversation, what constitutes a migration.
84
00:06:57.095 --> 00:07:00.810
Yeah. I think for, like, Tobias said, there are multiple
85
00:07:01.190 --> 00:07:07.289
flavors and facets of data migration, and I think data teams are constantly moving something or
86
00:07:07.815 --> 00:07:08.875
evolving their systems.
87
00:07:09.255 --> 00:07:10.155
I think that's
88
00:07:10.854 --> 00:07:11.514
an interesting
89
00:07:11.815 --> 00:07:19.690
1 to explore today, and I know that that's kind of the scope where both Rob and I have spent a lot of time thinking about is
90
00:07:20.150 --> 00:07:23.210
when you migrate your data preparation
91
00:07:23.835 --> 00:07:26.495
workloads and your analytical consumption
92
00:07:27.354 --> 00:07:28.974
from 1 data warehousing
93
00:07:29.354 --> 00:07:30.574
system to another,
94
00:07:31.370 --> 00:07:39.150
or when you are adopting a new framework for data preparation. For example, you would be migrating from a system like Airflow
95
00:07:39.845 --> 00:07:40.585
to or Informatica
96
00:07:40.965 --> 00:07:41.785
for to
97
00:07:42.485 --> 00:07:44.025
modern frameworks like DBT,
98
00:07:45.125 --> 00:07:50.370
or sometimes you would do both at the same time to modernize your data platform.
99
00:07:50.990 --> 00:07:54.930
I think if we start there, there's definitely plenty of
100
00:07:55.315 --> 00:08:01.815
challenges and know how's discussed just within that particular scope. I don't know, Rob. Would would you agree?
101
00:08:02.490 --> 00:08:22.810
Yeah. I think on sort of defining what is a migration versus what is not, I think you have to consider that you'll be replacing 1 system with another. I think that ultimately when you roll this up to the c suite, they're gonna be looking at this in terms of cost. Right? In terms of are we are we able to replace a system that is costing us a certain amount
102
00:08:23.190 --> 00:08:27.530
in either engineering effort to keep the thing alive, which is often the case,
103
00:08:27.915 --> 00:08:35.135
or in actual spend with a vendor and what are we replacing that with. And I feel like a migration is certainly not done
104
00:08:35.459 --> 00:08:52.800
until you've cut the cord on whatever the legacy system is. So I think it's it's it's defined sort of by the sense that it is not an incremental change so much as it is a a wholesale replacement of 1 system with another, usually with 1, you know, 1 data platform for another in the case of data warehouse migrations.
105
00:08:53.500 --> 00:08:55.920
And for people who are considering
106
00:08:56.620 --> 00:08:59.040
embarking on 1 of these migration projects,
107
00:08:59.654 --> 00:09:03.515
what are some of the signals that would suggest that that is necessary,
108
00:09:04.055 --> 00:09:14.210
and what are some of the cases where it is actually possible to avoid having to do do that migration and you actually just need to invest a little bit more engineering effort into making your current platform
109
00:09:14.590 --> 00:09:15.810
maintainable and sustainable.
110
00:09:16.350 --> 00:09:29.150
Yeah. I mean, I guess I can jump in here first and say that as I said before, nobody looks forward to doing a migration. Right? I I don't know of any engineers, myself very much included, who say, oh, what I really want to do right now for the next 6, 9,
111
00:09:29.610 --> 00:09:36.670
12, 18 months is a migration project. It is something that is hard to get by in for from executives because you're essentially going to be
112
00:09:37.145 --> 00:09:51.460
considerably slowing down the velocity of your team during this time frame. So what signals point to it really are about, is there an order of magnitude improvement available since the system that you are currently working on was, created?
113
00:09:51.845 --> 00:09:58.264
That could be because there's a new emerging technology. For me, it was that we went from having, at 1 point, an on premise
114
00:09:58.565 --> 00:10:01.464
Teradata server that sat in a big server room to
115
00:10:01.850 --> 00:10:13.855
moving to cloud based data warehouses like BigQuery or Snowflake where it's a complete paradigm shift in even how you run the platform. It could be that there's major production outages that are becoming so frequent that your team is
116
00:10:14.155 --> 00:10:23.940
constantly fighting fires with a system instead of being able to be productive to it. Could be the emergence of something brand new that we haven't even thought about yet. And I think that's what's so interesting is that I've seen this pattern,
117
00:10:24.560 --> 00:10:29.165
over my career. Basically, every 5 to 7 years, there seems to be this evolution.
118
00:10:29.545 --> 00:10:45.155
And I feel like right now we're in this sweet spot where I don't think any of us can necessarily see past what the world of Snowflake and BigQuery and what's next looks like, but I'm pretty confident there'll be something. I just don't know what it is just yet. Yeah. I think if we also try to kind of come up with particular
119
00:10:45.535 --> 00:10:48.195
reasons or, like, dimensions of the decision making,
120
00:10:48.575 --> 00:10:55.460
what I've seen myself, my career, and also working with other teams, it typically comes down to probably 3 factors.
121
00:10:55.840 --> 00:10:56.980
1 is scalability.
122
00:10:58.560 --> 00:11:05.115
So can your current system support the scale of your organization in terms of the size of data, obviously,
123
00:11:05.815 --> 00:11:07.595
the complexity of data,
124
00:11:08.230 --> 00:11:09.690
and also the
125
00:11:10.389 --> 00:11:22.925
performance around consumption of data. So can, for example, the users query the datasets and your you know, something as simple as dashboards get refreshed in a meaningful amount of time. And for us
126
00:11:23.385 --> 00:11:28.820
at Lyft, when we embarked on migration from Redshift to the Hive based,
127
00:11:29.279 --> 00:11:53.020
data lake, scalability was by far the strongest reason. Back in the day, Redshift was a system that had strong coupling with between storage and compute, and that meant that whenever we had to increase the capacity of cluster to just have more data, we'd have to resize the entire cluster. That was a very scary operation that the data ops team had to perform over the weekend,
128
00:11:53.325 --> 00:11:56.945
and that took multiple hours to, like, reshuffle the data between the nodes.
129
00:11:57.404 --> 00:12:01.550
And we got to a point where we're operating 1 of the largest retro clusters in the world.
130
00:12:02.029 --> 00:12:08.370
And the other facet of scalability where we faced really, really strong pain was around consumption.
131
00:12:09.045 --> 00:12:09.545
So,
132
00:12:09.925 --> 00:12:11.225
again, back in the day,
133
00:12:11.605 --> 00:12:12.105
Redshift
134
00:12:12.725 --> 00:12:17.065
had a very concrete limit of how many queries could be executed concurrently.
135
00:12:17.860 --> 00:12:20.520
And at the time, Lyft was growing
136
00:12:20.900 --> 00:12:25.685
rapidly, and we had a massive adoption of data analysis across the company, primarily
137
00:12:26.225 --> 00:12:27.764
through tools like Looker
138
00:12:28.144 --> 00:12:29.524
and Mode at the time.
139
00:12:29.904 --> 00:12:39.360
And what happened was we'd have extremely high surges of queries starting, like, 9 AM when people came into the office, and we're refreshing the dashboards on reports.
140
00:12:39.740 --> 00:12:42.080
And that completely brought the system underwater
141
00:12:42.525 --> 00:12:56.140
to the point where data team would have days where everyone was like, let's just go get Boba because Russia will not be able to run any queries for another few hours. So that that's the scalability part. And then the other, I think, important would be interoperability
142
00:12:56.680 --> 00:13:07.875
and then general capabilities of systems. I think, Rob mentioned that sometimes you would see a completely new sets of use cases and capabilities that new system would bring as well as the ability to just have multiple
143
00:13:08.589 --> 00:13:15.970
systems interact with your data. For example, with Redshift, it was hard for us to use other systems to query that data because the system was, like, very closed
144
00:13:16.315 --> 00:13:19.135
With more data lake architecture, so even modern warehouses
145
00:13:19.675 --> 00:13:26.630
like Snowflake, that's much, much better. And then final 1 is cost, but that I think is a very tricky 1 because
146
00:13:27.089 --> 00:13:30.470
in my experience, it's really easy to get misled by how much
147
00:13:30.985 --> 00:13:33.805
your system costs, and that could be a really,
148
00:13:34.345 --> 00:13:36.685
misleading factor if that is your primary
149
00:13:37.145 --> 00:13:38.365
driver for migrations.
150
00:13:39.420 --> 00:13:42.720
Yeah. I wanna plus 1 a little bit on that on that feature set, you know,
151
00:13:43.100 --> 00:13:46.560
rationale there. I think that's something that my team certainly saw as we moved,
152
00:13:47.065 --> 00:13:53.065
at Major League Baseball from Teradata to BigQuery was, you know, we we did it primarily because of,
153
00:13:53.464 --> 00:13:57.550
performance. I mean, as you mentioned, scalability again too. Just the ability to
154
00:13:57.850 --> 00:14:08.885
not necessarily have to worry about 1 stakeholder running a bad query that brings down the entire system as in the case where a lot of systems that do couple the storage and compute have that problem today.
155
00:14:09.310 --> 00:14:13.410
So that was our initial onus for looking at a migration, but then
156
00:14:13.790 --> 00:14:20.285
the feature sets is really what ended up making it be as successful as it was. Meaning that BigQuery offered us certain certain things.
157
00:14:20.904 --> 00:14:21.644
Most notably,
158
00:14:22.665 --> 00:14:41.895
button press data sharing that at MLB we used this to set up a system that we called wheelhouse where we basically kept our our, data pipeline centralized. So we would ingest data from ticketing vendors like Ticketmaster and Tickets dotcom, bring it all together in 1 place, and then we wanted to share that back out with the 30 MLB Clubs individually.
159
00:14:42.435 --> 00:14:45.495
At 1 point, that was a system backed by good old FTP.
160
00:14:46.529 --> 00:15:06.820
And as we move forward we started moving that towards s 3 storage but being able to move to BigQuery meant that we just clicked share on the dataset and they had access to it. And so this was like a paradigm shift in how we would work. There was never gonna be a question anymore of why is my data out of sync with your data which was such a common problem with the previous system. So, it really was, yes, performance cost scalability.
161
00:15:07.360 --> 00:15:15.985
But also, like, what are the features that this system is capable of that the other 1, for whatever reason, just is not. And I think that was data sharing was a big 1 for us on BigQuery.
162
00:15:16.445 --> 00:15:34.835
You know, certain cloud data warehouses now have the ability to do time travel, which again is something that some others may not have. And that's a huge difference when you're able to not think about backups in terms of something that you have to go and ask a database administrator to pull for you, but you can just write a SQL query for. So those features completely change the way you work with the system.
163
00:15:35.615 --> 00:15:50.185
And going back to 1 of the things that you said, Rob, about you don't want to bother with doing a migration unless you're going to get an order of magnitude improvement. You listed some of the types of features that will contribute to that measurement, but often, it's not possible to fully comprehend
164
00:15:50.565 --> 00:15:57.420
the impact of the migration until after it's complete. And I'm wondering what are some of the ways that you can do some early signals gathering to
165
00:15:57.899 --> 00:16:03.579
understand, 1, whether your intended target is actually going to have the impact that you want it to,
166
00:16:04.545 --> 00:16:15.230
to what the actual overall cost difference is going to be. That's often the hardest 1 to figure out. And, just some of that early work that can be done to prove out that, yes, this is a good idea
167
00:16:15.769 --> 00:16:19.310
and get some buy in from the people who need to actually do the approvals.
168
00:16:19.930 --> 00:16:26.514
Yeah. I think it's about finding the right proof of concept use cases early. And I mean, the challenge there is identifying
169
00:16:27.454 --> 00:16:30.035
a properly representative sample of workloads.
170
00:16:30.640 --> 00:16:44.055
At Eventbrite here when I joined, the team was already underway with their evaluation of Snowflake and had already identified some key parts of our, current data pipelines that were lagging, taking a certain amount of time. And we're able to essentially
171
00:16:44.675 --> 00:17:00.165
port subsets of those over. Right? Quickly with these cloud data warehouses now at least on this generation, what's nice is you can kind of commit to very little. In fact, you know, when you first start out with BigQuery and Snowflake, you don't need to have a signed contract and a commitment for 5 years. You can just go ahead and do it on demand.
172
00:17:00.465 --> 00:17:33.525
And so I think that allows you to really experiment very cheaply at the beginning. Right? You're just using the compute resources that you bring for trying out a workload, and you can hold those workloads up against their current workload counterparts. And you can really bring almost the full size of a subset of all of your datasets over. You don't wanna, of course, bring over all your datasets. You're not actually doing the migration yet. But by finding that representative sample and bringing over the full size of that data, you can very quickly observe, hey. What is the cost of this? What is the speed of this? But, you know, as you hinted at though too, like, actually anticipating what the cost will be
173
00:17:33.905 --> 00:17:40.710
is very challenging early on, and you have to kinda keep adjusting that as you continue to port workloads over to see if you're within the
174
00:17:41.090 --> 00:17:41.909
expected range.
175
00:17:42.610 --> 00:17:45.190
Going back to to the question of cost, you mentioned
176
00:17:45.809 --> 00:17:58.120
the financial outlay of running this new system, figuring out what are the cost differences of actually operating it, but there's also the harder to account for aspects of time and opportunity cost. And I'm curious
177
00:17:59.059 --> 00:18:12.455
how you have approached that process of doing the accounting to figure out how much is it actually costing me right now, how much do I think it's going to cost me once I'm done with this, and how do we manage that differential as we move through the progress of the migration?
178
00:18:12.990 --> 00:18:14.690
I think what is interesting
179
00:18:15.470 --> 00:18:17.970
is that the cost accounting kinda changes,
180
00:18:18.990 --> 00:18:23.605
before you embark on the migration, so you're thinking is it worth doing and then as you go? Probably
181
00:18:23.985 --> 00:18:27.924
when you arrive at at the realization that you need to migrate, it probably because
182
00:18:28.429 --> 00:18:29.490
the current system
183
00:18:29.790 --> 00:18:36.049
inflicts a lot of pain on your day to day work. And, for example, in the case of Lyft, our
184
00:18:36.554 --> 00:18:43.135
losses for not doing migrations were pretty clearly expressed in the productivity of the entire analytics team
185
00:18:43.650 --> 00:18:47.990
as well as in the ability of the team to deliver insights
186
00:18:48.450 --> 00:18:49.670
and have the stakeholders
187
00:18:50.130 --> 00:18:52.070
query those insights in time.
188
00:18:52.395 --> 00:19:10.830
So 1 example is that, you know, if there are meetings scheduled to review dashboards and metric at 9 AM and you can deliver those because your system is just not able to perform all the necessary computations, then you have, you know, a huge business cost that essentially means company cannot make decisions in time for a business like Lyft
189
00:19:11.165 --> 00:19:15.505
That was, extremely high. When we were embarking on migration,
190
00:19:16.525 --> 00:19:17.985
it was clear that
191
00:19:18.430 --> 00:19:22.210
there is an enormous opportunity cost of having the actual
192
00:19:22.750 --> 00:19:33.965
analytics and data engineering team internally perform the migration because then they would not be building data products that would actually propel the business forward in terms of creating new insights and new models.
193
00:19:34.669 --> 00:19:37.809
And so I think in general, 1 of the most
194
00:19:38.350 --> 00:19:47.184
effective levers here data teams can pull is to try to outsource that work as much as they can. Because otherwise, you may lock down your entire,
195
00:19:47.645 --> 00:19:53.105
you know, analytical team in a project that doesn't necessarily have a u immediate user facing
196
00:19:53.610 --> 00:20:02.830
impacts that could be challenging both, you know, business wise, but also politically for the team. You don't wanna be in a world where for 6 months, you're just working on this infrastructure
197
00:20:03.275 --> 00:20:03.775
project.
198
00:20:04.395 --> 00:20:15.370
And I think the third 1 is, like, once you are on the new system, how do you make sure that or as you anticipated being on the new system, how do you make sure that this economically ultimately is feasible?
199
00:20:15.990 --> 00:20:18.650
And I think that is something that is easier to quantify
200
00:20:18.950 --> 00:20:20.330
with more modern
201
00:20:21.255 --> 00:20:29.355
warehouses, if we're talking about the warehouse migrations like Snowflake or BigQuery because their pricing is transparent and ultimately, it's fairly easy like Rob said to quantify
202
00:20:29.950 --> 00:20:36.610
what would it cost to move workloads between those systems. Because you know how many queries you run, you know even how much data
203
00:20:36.990 --> 00:20:38.530
those queries are scanning.
204
00:20:39.395 --> 00:20:41.575
When these things get really opaque is when,
205
00:20:42.034 --> 00:20:42.534
your
206
00:20:43.075 --> 00:20:47.495
the system you're trying to migrate is sort of like a data lake, which is
207
00:20:48.140 --> 00:20:50.400
supported by some open source
208
00:20:50.860 --> 00:20:53.679
systems like TriNet Spark that you try to run internally.
209
00:20:54.220 --> 00:20:58.174
Those can be extremely complex and high cost projects, not only because
210
00:20:58.715 --> 00:21:02.815
you can't really estimate the cost of a query because you're running infrastructure yourself,
211
00:21:03.115 --> 00:21:07.600
but also you now bake into that the cost of ownership
212
00:21:07.900 --> 00:21:11.919
in terms of, like, data ops teams and dev ops teams that now have to maintain those systems.
213
00:21:12.299 --> 00:21:12.960
That's definitely
214
00:21:13.660 --> 00:21:14.559
1 of the,
215
00:21:14.885 --> 00:21:20.905
I think, mistakes we've got we've done at Lyft is we haven't really counted the true cost of moving to such an architecture.
216
00:21:21.605 --> 00:21:23.544
And, what would be the alternative,
217
00:21:24.210 --> 00:21:26.070
for us in terms of, for example,
218
00:21:27.010 --> 00:21:29.590
embarking on a vendor that would provide a more,
219
00:21:30.690 --> 00:21:35.495
final solution, like a query, for example, like, small flake or other warehouses.
220
00:21:35.875 --> 00:21:37.735
So that's probably where the cost,
221
00:21:39.075 --> 00:21:41.335
accounting policy can be the strongest.
222
00:21:42.360 --> 00:21:54.905
Yeah. Doubling down on on that a bit, I think that that personnel cost of of keeping the the legacy platform up and running is 1 that could be very easily missed in in the overall calculations, but it is 1 that
223
00:21:55.285 --> 00:21:59.700
was certainly felt at at my current role at Eventbrite when I joined given that the team was
224
00:22:00.260 --> 00:22:01.080
self managing,
225
00:22:01.539 --> 00:22:14.075
a Hadoop based stack, based on Spark, Hive, Presto. Running it on AWS, so it's not like we were trying to run it on our bare metal, but and using tools like EMR, which, you know, do simplify some of the management of Hadoop.
226
00:22:14.455 --> 00:22:36.365
But there's still a lot there. And over time, it accumulates more and more technical debt as well in terms of, you know, sort of being pinned onto an older version of of various tool sets because of certain dependencies you may have. And so, ultimately, for for us, what we saw was that the team was just constantly burdened with on call issues. Right? On call issues that would arise day and night due to the system
227
00:22:36.800 --> 00:22:44.945
not performing the things that we just simply expected it to do every day or every hour, on a regular basis. So I think that cost was probably the most felt.
228
00:22:45.345 --> 00:23:11.580
And so I think everyone at that point was ready to welcome a managed service taking that burden off of our hands. And I think that's 1 of those that it may be a little trickier to quantify financially because it's not necessarily like system y that you move to is costing that much less than system x. But when you sort of try to bake in well, it's basically taking us 2 or 3 full time engineers time to just keep this thing's lights on. Let alone try to even improve it, which would be another set of engineers,
229
00:23:12.440 --> 00:23:17.445
you know, can we get that for free by just migrating? And that's where, again, that math starts to come into play.
230
00:23:18.485 --> 00:23:20.184
And another aspect of
231
00:23:20.565 --> 00:23:24.825
this question of migrations, when to do them, how to do them, what to migrate to
232
00:23:25.400 --> 00:23:25.900
is
233
00:23:26.360 --> 00:23:28.380
what are some of the early decisions
234
00:23:29.000 --> 00:23:39.965
in terms of the architectural design, the overall workflows that are built on top of these, the dependency chains, and and the hard requirements that you build into the overall architecture that contribute to
235
00:23:40.265 --> 00:23:45.220
the eventual need for that migration, and what are some of the lessons that you've learned in the process
236
00:23:45.760 --> 00:23:49.300
of going through these migration paths that have helped you understand
237
00:23:50.015 --> 00:23:52.035
as you reach that eventual destination,
238
00:23:52.815 --> 00:23:55.395
removing some of those potential pitfalls
239
00:23:55.775 --> 00:23:57.215
and architecting in some,
240
00:23:58.020 --> 00:24:02.600
escape hatches to be able to get some further life out of those,
241
00:24:03.059 --> 00:24:03.540
system,
242
00:24:03.940 --> 00:24:05.800
system architectures that you design
243
00:24:06.285 --> 00:24:07.985
that were, you know, previously
244
00:24:08.445 --> 00:24:10.865
major constraints on the sustainability
245
00:24:11.245 --> 00:24:12.785
of the previous platforms?
246
00:24:13.630 --> 00:24:15.490
I think while there are multiple
247
00:24:15.790 --> 00:24:16.290
dimensions
248
00:24:17.390 --> 00:24:22.210
to plan here, personally, 1 of the 1 of those that probably had the biggest,
249
00:24:22.804 --> 00:24:25.945
unfortunately, negative impact on my success is running migration
250
00:24:26.565 --> 00:24:30.745
is the the decision of whether you are doing
251
00:24:31.460 --> 00:24:33.080
lift and shift migration
252
00:24:33.380 --> 00:24:42.255
or you're trying to improve things as you go. And, just a double click into that, lift and shift essentially means that you have, let's say, 100
253
00:24:42.795 --> 00:24:54.370
workloads, like, 100, you know, transformation jobs running on your legacy system, and they can be very ugly. That could be some, like, really big performance bottlenecks. The data model is probably
254
00:24:54.909 --> 00:25:04.465
makes you wanna cry because it was built over the last 10, 15 years. But you say, you know what? I don't care. I'm just going to replicate those exact transformations on my new system.
255
00:25:04.925 --> 00:25:13.410
I'm just going to literally copy paste SQL and adjust it for or, you know, Spark or whatever. Like, adjust my code, make sure it runs on the new system,
256
00:25:13.790 --> 00:25:18.615
address any bugs. But, ultimately, I'm just aiming for the same party,
257
00:25:19.075 --> 00:25:19.575
and
258
00:25:19.875 --> 00:25:20.615
I will,
259
00:25:21.235 --> 00:25:24.934
take care of refactoring later. That's lift and shift pattern. The second pattern
260
00:25:25.235 --> 00:25:39.605
is you look at the system and you realize it is extremely inefficient. It is very badly architected because, again, it's just accumulated all your legacy work from the early days of your business. And you say, well, this is a great opportunity for us to improve things.
261
00:25:40.065 --> 00:25:44.900
And I think for data practitioners, it is very tempting to embark on the latter
262
00:25:45.280 --> 00:25:51.300
route, and this is what I have attempted to do at Lyft when I was tasked with architecting the migration.
263
00:25:52.605 --> 00:25:53.985
And that is
264
00:25:54.685 --> 00:25:56.625
a complete, in my opinion,
265
00:25:57.165 --> 00:25:58.705
road to disaster because
266
00:25:59.190 --> 00:26:02.170
even if you're doing lift and shift, it is still a very expensive
267
00:26:02.870 --> 00:26:07.370
and complex project. If you're trying to rearchitect things in the in the process,
268
00:26:08.015 --> 00:26:10.035
you are now essentially bottlenecking
269
00:26:10.495 --> 00:26:18.510
all these efforts on the ability to kind of reengineer, refactor things. But most importantly, the larger companies generate consensus about
270
00:26:19.049 --> 00:26:23.625
the, you know, new data models or spending time optimizing things, and that
271
00:26:24.165 --> 00:26:28.665
just exponentially, you know, increases the complexity of the project. So
272
00:26:29.045 --> 00:26:38.590
having burned by that, I strongly recommend everyone to do lift and shift, and I've seen, other data teams. And if we support a data full, do lift and shift,
273
00:26:38.914 --> 00:26:45.174
and being extremely successful with getting through migration way, way faster than I was able to at Lyft.
274
00:26:45.475 --> 00:26:45.794
And,
275
00:26:46.355 --> 00:26:54.870
I think the other 1 that I would also think is important is whether to outsource your work or not, which we which we touched on. And I think if you're doing lift and shift, that actually makes
276
00:26:55.924 --> 00:27:07.580
the migration workload very outsourceable because you're just saying, look, I have this system on the left. I want the same thing on the system on the right, and you can outsource that people who don't have context of your on your business.
277
00:27:08.120 --> 00:27:13.020
Again, my mistake was that we brought in consultants, and I try to employ them to
278
00:27:13.605 --> 00:27:22.745
make our data model better because, obviously, they're experts and they have seen a lot. And that was also a disaster just because having someone externally come in and work with your stakeholders
279
00:27:23.390 --> 00:27:26.610
to create better data models is just not going to work.
280
00:27:26.910 --> 00:27:31.330
Couple of different threads that I wanna go down. And so 1 of them is
281
00:27:31.765 --> 00:27:40.985
in terms of that dependency chain of, oh, we can't retire this old system because x, y, and z relies on it. 1 of the paths that lead to those,
282
00:27:41.330 --> 00:27:41.990
you know,
283
00:27:42.530 --> 00:27:44.550
very obtuse dependency chains
284
00:27:45.170 --> 00:27:52.845
is giving data consumers too much access to the internals of the system to do whatever it is that they want to do or the,
285
00:27:53.705 --> 00:28:11.245
making the initial system too clunky so people start embarking on these shadow IT projects where they'll just bring in their own tool, and they have this, you know, deep dependency on something that's in your stack you don't even know that they're touching. And so when you turn it off, everything everybody starts yelling because it's broken. So 1 of the questions is, how do
286
00:28:11.625 --> 00:28:13.165
you identify those situations,
287
00:28:13.600 --> 00:28:22.740
1, but also how do you prevent them from occurring so that you don't have these deep dependencies and you can reduce the overall surface area of what you have to maintain and refactor during the migration?
288
00:28:23.185 --> 00:28:34.790
And then maybe as a second step a second question we can move on to from there is we've been talking very broadly about this concept of migrations. You have to make sure that everything works effectively. But I'm curious what are the
289
00:28:35.330 --> 00:28:35.830
architectural
290
00:28:36.130 --> 00:28:52.970
components that you have typically seen as the major targets for these migrations, whether that's the ingest and load, the transformation layer, the warehousing layer, business intelligence. I think that that makes sense as a second question to move on to after we've addressed. How do you re how do you identify and
291
00:28:53.270 --> 00:29:00.174
possibly prevent these deep dependencies that have grown organically because of the fact that the system was so clunky to begin with?
292
00:29:00.715 --> 00:29:06.654
Yeah. I think 1 of the first things you need to do when embarking on a migration project is spend some time
293
00:29:07.220 --> 00:29:09.320
really examining your system. And
294
00:29:09.780 --> 00:29:28.340
it's something that will unearth a lot of surprises, including Tobias, as you refer to that, that shadow IT that's occurred over time. You know, tapping back into what Gleb mentioned earlier by doing lift and shift, you're at least not having to contend with the fact that you're going to have to communicate changes to downstream stakeholders at the time.
295
00:29:28.640 --> 00:29:35.345
In fact, the pattern that I've seen, and I'm 100% with you there on lift and shift. It's how I've successfully gotten through,
296
00:29:35.804 --> 00:29:39.345
3 of these in my career already is by by sticking to that. In fact,
297
00:29:39.804 --> 00:29:44.929
my my current group at Eventbrite very much uses it as a mantra that is regularly chanted whenever anybody
298
00:29:45.390 --> 00:29:51.169
suggests even changing a column name. Lift and shift. What are you doing? So I think sticking with that
299
00:29:51.595 --> 00:29:54.655
allows you that even if you do have a nested web of dependencies
300
00:29:55.595 --> 00:29:56.095
to
301
00:29:56.395 --> 00:29:59.775
minimize how much you need to really navigate that because
302
00:30:00.420 --> 00:30:08.600
you're going to be basically exposing a system that essentially from the outside from the interface of SQL, at least, assuming that we're sticking with a SQL based system,
303
00:30:08.925 --> 00:30:13.265
it looks and feels about the same to the end user. It's not to say that they won't have work to do,
304
00:30:13.565 --> 00:30:30.015
but the work can be minimized by the fact that the table names look and smell and feel the same. It's it's why 1 of the first steps that I've usually taken on any migration I've done is to set up basically a replication of the legacy system into the new 1. And when we do that, again, sticking with the lift and shift pattern,
305
00:30:30.395 --> 00:30:35.135
1 of the really big benefits there, you know, we had talked about opportunity cost earlier on,
306
00:30:35.515 --> 00:30:36.175
is that
307
00:30:36.490 --> 00:30:41.070
as soon as you have a complete replication of the core sets of tables that are in your legacy system,
308
00:30:41.690 --> 00:30:48.005
replicating into the new system, And that can be at a daily cadence or an hourly cadence, whatever makes sense, but you set that up first.
309
00:30:48.385 --> 00:30:50.965
You can switch all your consumers over to the new system.
310
00:30:51.340 --> 00:31:15.365
And this is what we've done at Eventbrite here where we basically set up Snowflake last year and set up replications from our legacy, Hadoop based platform to it. And then we basically gave all of our consumers a 3 month window to say, listen, you've got 2 systems that look and feel the same right now, But in 3 months, we're gonna cut you off from the old 1. So you've got some time, raise your concerns, raise your complaints now, or forever hold your peace.
311
00:31:15.845 --> 00:31:50.475
And that is when the, shadow IT emerged. And that is when we found out about plenty of teams that had their own 1 off Python processes running on a Windows scheduler somewhere that actually were responsible for huge pieces of our business and we just didn't know about them. And so it sort of forced those things to come to the forefront because we at least then needed to talk to them about how they were gonna switch their connection over to use Snowflake now instead. So it sort of, you know, what what was shouting from the rooftops to our organization that like, hey, this change is coming. It should be minimally impactful because things look and feel the same except things will be faster on the new system.
312
00:31:51.015 --> 00:31:58.210
But we need you to self identify now because we're gonna turn the lights off for you as a consumer on the old system early.
313
00:31:58.590 --> 00:32:02.770
Now at that point, we're not done with the migration. We haven't even talked about moving the transformation
314
00:32:03.235 --> 00:32:10.135
or ingestion layers over to the new system yet. But by doing things in that order, you make sure that you identify very early on in the process
315
00:32:10.570 --> 00:32:17.230
who are some of those dependency chains further on down your path. And then you can kind of focus on the ingestion and transformation rewrites,
316
00:32:17.850 --> 00:32:18.590
in the background
317
00:32:19.005 --> 00:32:34.450
while all your data consumers have already moved over to the new system. And in fact, the biggest challenge we have now is that everybody thinks we're done with the migration already because they're already using the new system, and they're happy on it. And we're like, actually, the data is still going through the old system and then copying to the new system.
318
00:32:34.990 --> 00:32:41.615
But from your end user perspective, you get the fast, you know, improvements and new features of a new system right away. So
319
00:32:41.915 --> 00:32:51.680
it becomes more of a carrot than a stick of, like, don't you wanna move on to the new thing? It's faster. It's better. And then eventually the stick comes of, like, alright, we're kicking you out of the old system now.
320
00:32:52.060 --> 00:32:54.080
Another element of these
321
00:32:54.380 --> 00:32:54.880
platforms,
322
00:32:55.545 --> 00:33:04.350
1 that I'm currently dealing with is the question of access control and governance, and those can be implemented very differently depending on what the underlying
323
00:33:04.750 --> 00:33:08.049
technologies are, the specific access patterns where,
324
00:33:08.750 --> 00:33:14.335
if you're piping everything through the warehouse, you can say, okay. I'm going to implement my control here, or is all of the
325
00:33:15.294 --> 00:33:27.450
business intelligence layer, or do I have to do this at some more granular level? Am I relying on some sort of single sign on, whether that's via OAuth or Kerberos or SAML? I'm curious what you have seen as
326
00:33:27.910 --> 00:33:30.230
the role that governance and,
327
00:33:30.630 --> 00:33:34.725
access control have played in the migration projects you've been involved
328
00:33:35.184 --> 00:33:44.350
with? Well, you know, I can kinda jump in first and just say lift and shift. Right? Like, I mean, whatever it was before is what we're doing on this 1. And I think it that that mantra helps you get past some of this
329
00:33:44.730 --> 00:33:53.365
inertia and say, yes. Like, we know that at Eventbrite moving to Snowflake gives us new opportunities for things like role level access control, PII redaction,
330
00:33:53.825 --> 00:33:57.924
things that we couldn't do on our previous system. We're excited about those features,
331
00:33:58.380 --> 00:34:18.680
but we want to maintain that parity right now. So that sort of becomes not part of the migration project then. Right? It becomes whatever the access patterns were before. This set of users had access to this set of tables, let's keep that for now because we wanna make sure that the migration worked successfully for those users. And then we can kind of embark on follow ons to further whittle that down. I mean,
332
00:34:19.059 --> 00:34:19.380
the,
333
00:34:20.339 --> 00:34:28.075
authentication piece as far as how they get into the system might change. Right? But the authorization piece should not. Yeah. I think the the question of governance
334
00:34:28.615 --> 00:34:29.115
actually
335
00:34:30.055 --> 00:34:36.019
is, in my experience, less important in terms of can we implement the governance on the new system,
336
00:34:36.400 --> 00:34:37.220
but rather
337
00:34:37.680 --> 00:34:39.460
how does the data governance
338
00:34:40.885 --> 00:34:42.585
affect the complications
339
00:34:43.125 --> 00:34:44.345
of migration?
340
00:34:44.964 --> 00:34:45.705
For example,
341
00:34:46.484 --> 00:34:46.984
typically,
342
00:34:47.690 --> 00:34:48.750
the vast majority
343
00:34:49.530 --> 00:34:55.150
of workloads on the system are not related to transformations, which are, let's say, more or less control
344
00:34:55.610 --> 00:34:57.150
of the data engineering team.
345
00:34:57.825 --> 00:35:01.045
But in the consumption layer. Right? So notebooks,
346
00:35:01.585 --> 00:35:04.164
1 off, create exports, BI, and dashboarding.
347
00:35:04.700 --> 00:35:11.839
And the patterns that are used by those workloads and the capabilities and the choice of those systems actually
348
00:35:12.255 --> 00:35:19.714
dictate how costly would it be to move those workloads over. For example, at Lyft, we had 2 systems at once.
349
00:35:20.335 --> 00:35:21.075
We had
350
00:35:21.630 --> 00:35:23.250
Mode, which is a very
351
00:35:23.550 --> 00:35:24.610
notebook driven
352
00:35:24.910 --> 00:35:25.410
workflow,
353
00:35:26.190 --> 00:35:27.970
and we had Looker for
354
00:35:28.484 --> 00:35:31.704
the primary dashboarding, kind of standardized reporting workloads.
355
00:35:32.325 --> 00:35:32.825
And
356
00:35:33.204 --> 00:35:35.945
moving Looker workloads over was extremely
357
00:35:36.450 --> 00:35:41.589
simple because it's mostly templated and it controlled by the internal semantic layer.
358
00:35:41.970 --> 00:36:01.845
So, yes, you had to make some adjustments, but those were minor and those were easy to do for the majority of the workloads and the majority of the consumption, you know, dashboards and other nodes. With Vowed, I remember we had a question of, well, analysts love it, so I can't really figure out, like, how many reports we need to migrate. Is it, like,
359
00:36:02.244 --> 00:36:04.105
dozens of reports or hundreds of reports?
360
00:36:04.805 --> 00:36:05.845
And I,
361
00:36:06.405 --> 00:36:15.070
did, like, a kind of a hockey lineage project where I did analysis of the API and try to figure out just the scale of dependencies. And
362
00:36:15.370 --> 00:36:16.030
I uncovered
363
00:36:16.570 --> 00:36:17.070
13, 000
364
00:36:18.085 --> 00:36:21.785
notebooks that were created and actually actively running
365
00:36:22.085 --> 00:36:22.904
on the system.
366
00:36:23.204 --> 00:36:33.190
And those notebooks had pretty much, like, no governance in the sense that if you wanted to switch them over to a new system, you had to pretty much go into every notebook and every cell that notebook and rewrite the SQL
367
00:36:33.650 --> 00:36:42.865
to query, you know, change the dialect from, like, Redshift to Snowflake. So that was not at all possible to do in any kind of centralized fashion, and so we have to pretty much resort
368
00:36:43.380 --> 00:36:51.640
to what Rob says. Right? Tell the users to migrate those over, and that was obviously a very high friction exercise for
369
00:36:52.365 --> 00:36:58.065
the stakeholders and 1 that actually really slowed down the consumption adoption of the new system.
370
00:36:58.685 --> 00:37:01.265
Another aspect of migrations is
371
00:37:01.869 --> 00:37:11.875
deciding when they are done to your point, Rob. Some people might say, oh, it's done because I'm already using the new thing. But you say, oh, I've actually still got months worth worth of work to do, so I can't actually,
372
00:37:12.415 --> 00:37:16.195
fulfill the request that you just sent me that you think is only going to take me a few minutes.
373
00:37:16.655 --> 00:37:18.595
And I'm wondering what your
374
00:37:18.930 --> 00:37:31.485
what your strategy has been at the outset of the migration to be able to define these are all the things that need that need to be done. These are the ways that we're going to validate completion, and this is what we're going to use to measure overall success.
375
00:37:31.945 --> 00:37:34.845
And then also in that path, what are some of the
376
00:37:35.470 --> 00:37:53.525
early maybe break glass situations that you have identified to say, actually, this is not going to work out the way I think it's going to be. I'm going to cut bait. I'm not going to fall victim to the sunk cost fallacy and just keep pulling forward even though it's not gonna do work and maybe, you know, step back and reconsider the overall migration effort.
377
00:37:54.180 --> 00:37:59.640
Yeah. I think in terms of, like, knowing when you're done, the definition of done, it really is that
378
00:38:00.020 --> 00:38:05.555
the legacy system is gone. Right? Gone from your budget, gone from your management,
379
00:38:06.255 --> 00:38:06.994
gone from
380
00:38:07.454 --> 00:38:13.390
being a thought in your mind. Right? And I think, like, anything short of that means that you're not done. However,
381
00:38:13.930 --> 00:38:16.349
again, as I described earlier, there are different
382
00:38:16.705 --> 00:38:19.525
perspectives that different stakeholders can have on that that
383
00:38:19.825 --> 00:38:25.685
help relieve that pressure. I mean, again, by having the replication in place early, there's not this
384
00:38:26.039 --> 00:38:29.019
built up expectation of oh, that team is working on a migration.
385
00:38:29.480 --> 00:38:39.795
It's taking forever and I can't see any results. Right? You want to minimize the time that it is something that is invisible to others and you want to start building that internal advocacy
386
00:38:40.095 --> 00:38:54.965
for it early or what you described. I haven't encountered this on my own, but but but get that internal feedback that it's not working well for a use case early on. Right? And make decisions about that. So I think it really is about by first doing this really kind of simple lift and shift replication,
387
00:38:55.505 --> 00:39:05.349
you can let all of your stakeholders and all of their workloads onto the new system and derisk that a ton because you then find out right away about any challenges, any problems they may be seeing.
388
00:39:05.730 --> 00:39:22.589
And you also then relieve a ton of pressure of, alright. I've invested in this new thing. I'm now paying a bill to Snowflake every month. How come I can't see anything from it yet? Right? How come how come nobody's using it but the data team right now? What is that? Right? So, like, you try to minimize that time that you're doing that replication stuff, which I think really helps
389
00:39:22.890 --> 00:39:34.694
and also gives you opportunities to, like, throw different types of workload at it and and navigate them. I think I remember in MLB for example we still had some of our data science team using a set of tools from SAS. I mean this was
390
00:39:34.994 --> 00:40:04.895
kind of stuff they had used prior to really Python and R becoming as as widespread as they were. And as we moved to BigQuery we kind of learned oh wait a minute the connector from SaaS to BigQuery is not really what we want it to be. And we identified a pretty big risk there of, like, alright. Well, all of these workflows really are not working the way we intend them to. Can we find a fix for that? Right? Can can we get past that? You know, is there an additional cost to license something that will get us past that? So, again, I think it's about just getting your data stakeholders in there as early as possible to at least derisk this.
391
00:40:05.434 --> 00:40:14.359
Yeah. I fully agree with Rob in terms of setting the, let's turn off the sis like, the old system as the actual goal we're aiming for.
392
00:40:14.660 --> 00:40:24.185
Because as long as the old system is running, it probably still incurs high cost in the business, and it's hard to say that, yeah, we've come you know, we've completed the project. I also think that
393
00:40:24.485 --> 00:40:24.985
because
394
00:40:25.605 --> 00:40:35.060
that kind of 0 to 1 can take months and sometimes years, it's helpful to also kind of set the goals in terms of how do we measure completion or even our progress
395
00:40:35.465 --> 00:40:40.445
within the migration. And if you wanna do that, probably the atomic
396
00:40:40.905 --> 00:40:44.525
element of migration is, you know, let's say we're migrating
397
00:40:45.069 --> 00:40:48.369
between warehouses or migrating between transformation frameworks.
398
00:40:48.829 --> 00:40:50.289
It's probably migrating
399
00:40:50.829 --> 00:40:51.329
the
400
00:40:51.630 --> 00:40:58.535
tables that are produced in the transformation process, right, as part of our kind of the transformation diagram airflow or
401
00:40:58.995 --> 00:41:02.535
DBT. So a single job that produces a single table
402
00:41:02.880 --> 00:41:05.460
kind of becomes this atomic unit of work
403
00:41:05.760 --> 00:41:06.900
that we can count
404
00:41:07.520 --> 00:41:15.735
as, you know, either done in progress or yet to do. And then some of the AI workflows as well. Right? Like, have you migrated the exact KPI dashboard
405
00:41:16.115 --> 00:41:20.010
or not? And what I found that if you are doing a migration,
406
00:41:20.870 --> 00:41:23.050
all these atomic blocks within a larger organization,
407
00:41:23.430 --> 00:41:26.810
the process that becomes really critical is user acceptance
408
00:41:27.425 --> 00:41:27.925
testing,
409
00:41:28.225 --> 00:41:31.605
which is kind of known in software. But in data, what that means is
410
00:41:32.065 --> 00:41:42.920
if you migrating an a workload, let's say, you probably have, like, a set of tables that are really, really important. Right? And you try to get business, use them. First, let's say, like Rob said, if we just replicated
411
00:41:43.515 --> 00:41:44.255
the old,
412
00:41:44.715 --> 00:41:58.660
well, the legacy produced datasets to the new system, when I try to have people switch them switch their consumption to those tables, we have to convince them that assuming the lift and shift pattern, that that's the same data. Right? They can trust it. There are no regressions. There is no
413
00:41:59.099 --> 00:42:06.065
a kind of, like, we optimize the definition of conversion in the in the process. Right? It's still it's still the same data. And that kind of user acceptance,
414
00:42:06.365 --> 00:42:09.105
creating a process for that and making sure that stakeholders
415
00:42:09.645 --> 00:42:10.145
explicitly
416
00:42:10.525 --> 00:42:12.145
get you a thumbs up,
417
00:42:12.490 --> 00:42:20.350
or you have a certain way to signal to your stakeholders, yes, this is the same data you can use it, is quite important. Back at Lyft, we were
418
00:42:20.845 --> 00:42:22.704
writing a lot of custom build
419
00:42:23.164 --> 00:42:23.664
software
420
00:42:24.125 --> 00:42:27.904
for that because we just had to like, no 1 would trust that we
421
00:42:28.329 --> 00:42:36.270
produced good data. We actually had to present this as, like, almost a dashboard that said, yes. This table is exactly the same as your
422
00:42:36.835 --> 00:42:44.855
same table but on the old system. So now can you please switch your report? Can you please switch your machine learning job to it? And that was actually 1 of the inspirations
423
00:42:45.560 --> 00:42:52.460
for me to build a tool called data diff at DataFold, which essentially produces the same analysis, but fully automatically
424
00:42:52.840 --> 00:43:01.465
for the user. And you can do this within database or across databases to actually answer the question, is your data the same for the purposes of the user acceptance
425
00:43:01.930 --> 00:43:07.550
testing? Also, I'm curious to hear how, like, you think about the UAT, Rob, given that
426
00:43:07.930 --> 00:43:14.085
you have heard a large company and, like, a lots of a lot of stakeholders, do you have to convince that you've actually completed
427
00:43:14.465 --> 00:43:15.045
the work?
428
00:43:15.425 --> 00:43:30.625
Yeah. I mean, at at our current at my current role at Eventbrite, you know, data diff, both the open source and then, you know, the commercial product of DataFold have been instrumental in something that, like Gleb mentioned, you otherwise end up engineering yourself. At Major League Baseball, we did exactly that. We didn't
429
00:43:31.005 --> 00:43:33.105
have data folder, data diffs, so we built
430
00:43:33.580 --> 00:44:00.885
tooling, you know, because we're engineers, so we always wanna build a tool to try to get through anything that we might have to do manually. We built tools that generated SQL that attempted to compare the the way the tables look. You know, really that we built a very primitive version of what data diff does. And we did this because we needed to be able to be confident in what we had been replicating. So we use in my current migration project, we've used the data diff and data fold tools in 2 different places. 1 is when we're first doing those initial replications,
431
00:44:01.425 --> 00:44:28.290
as Gleb described, to to build confidence that the old and the new systems are matching, that your replication processes themselves are valid. Eventually you gain confidence in that because your replications start to look a lot the same. If you're replicating a 1, 000 tables, once you've got something set up for it, you know that your process is working well. But it does unearth lots of problems along the way that you would have. Time zones are an old favorite. They always seem to give you an issue. Things like that, you know, character encodings, you might have missed otherwise.
432
00:44:28.590 --> 00:44:38.015
Once you're in the midst of the migration and you're actually rewriting your transformation logic, the ability to, again, have a replicated copy of what the output was from the old system
433
00:44:38.315 --> 00:45:07.380
and then compare that with the output of the newly written transformation logic. And basically look for row by row matching because you're doing a lift and shift is huge and it's something that honestly and I don't want to go on record and say that I love doing migrations but there is 1 part of it that I like which is that there is a ground truth to hold yourself to which is that if the old system is assumed to be accurate which by the way that assumption can be wrong sometimes we found plenty of bugs in the old system as we move forward but if you assume the old system is accurate
434
00:45:07.825 --> 00:45:32.545
then you actually have this ground truth to compare your results against. And I feel like that's something that working in data you very much don't always have. If you're building a brand new data pipeline the best you can do is do heuristics and metrics on it to see if you feel like it looks like it's correct. But in this case, you have an exact copy of what that data should look like to tell you that it's correct. So it's actually a really nice place to be when you're in migrations and to the point you raised earlier, why it's easier
435
00:45:32.845 --> 00:45:34.545
to sort of spread out this workload,
436
00:45:34.990 --> 00:45:40.610
to others who might not even be so familiar with the domain of the data because they can use tools like this to do an exact comparison.
437
00:45:40.990 --> 00:45:53.530
And speaking a bit more to the automation question, you talked about being able to use data diff to be able to validate that the old system and the new system are in agreement and that all of the same data is present.
438
00:45:54.070 --> 00:45:56.650
What are some of the other places where automation
439
00:45:57.110 --> 00:46:03.485
can be useful, and what are some of the ways that you have seen automation employed where it is actually a net negative?
440
00:46:03.865 --> 00:46:07.965
Speak to 1 other place where I've seen it be incredibly useful, which is in the translation,
441
00:46:08.890 --> 00:46:22.575
of a SQL dialect, for example, from 1 system to another, assuming we're talking about a migration from a SQL based data warehouse to another. At Major League Baseball, we had 100 of SQL scripts of thousands of lines of of code each sometimes
442
00:46:23.115 --> 00:46:25.835
that were written in Teradata's dialect of SQL. And,
443
00:46:26.440 --> 00:46:36.220
at that point, we, we were working with with Google Cloud, and they ended up actually acquiring a company called Compiler Works. I believe they've been on the podcast before as well. And they had a a tool that would
444
00:46:36.565 --> 00:46:45.705
understand and parse the SQL syntax tree and then turn it into another version of SQL for you. And it was pretty advanced. We were always blown away by how far it could get because there were certain features,
445
00:46:46.310 --> 00:46:53.850
SQL qualify clause that Teradata had that BigQuery just didn't have at the time, that it would then completely rewrite your query into, like, a nested
446
00:46:54.454 --> 00:46:56.395
CTE in order to get the same results.
447
00:46:57.095 --> 00:47:00.155
And that's just a game changer when you're not going through the tedium
448
00:47:00.615 --> 00:47:03.515
of like trying to find SQL differences.
449
00:47:04.070 --> 00:47:10.490
At Eventbrite, I I I was able to kind of engage in a much more, simple version of this because we were moving from PrestoSQL
450
00:47:10.790 --> 00:47:21.650
syntax to Snowflake SQL that were actually a lot more alike than they were different. And so here, I basically built an internal tool that used mostly regex expressions just to change out certain things that were slightly different.
451
00:47:21.950 --> 00:47:30.194
It certainly wasn't as advanced as something that would parse the entire SQL syntax tree. But it also didn't seem to be necessary. We got 90% of the way there with that.
452
00:47:31.214 --> 00:47:36.280
I know there's also a a tool and framework out there SQL Glot. I believe he's also been on the podcast that,
453
00:47:36.740 --> 00:47:40.840
same sort of thing. Being able to transpile SQL from 1 dialect to another,
454
00:47:41.380 --> 00:47:45.785
can save you, you know, hours of frustration in having to do something that would otherwise be very tedious.
455
00:47:46.325 --> 00:47:56.369
Yeah. Plus 1 to SQL GloD, I think. If I had this available in the early days of my data engineering career, That would be a a game changer, but I did get to use the compiler works uplift,
456
00:47:56.670 --> 00:47:58.369
which was definitely helpful.
457
00:47:58.750 --> 00:48:04.734
And, yeah, I think the other aspect of automation that can be helpful in the planning phase is data lineage.
458
00:48:05.035 --> 00:48:05.535
Because
459
00:48:05.994 --> 00:48:06.494
ultimately,
460
00:48:07.194 --> 00:48:15.930
you know, whether you're doing lift and shift or whether you try to refactor things, you have to plan and sequence the migration, and you have to decide
461
00:48:16.310 --> 00:48:18.490
of 100 and sometimes 1, 000
462
00:48:19.184 --> 00:48:19.684
of
463
00:48:19.984 --> 00:48:20.805
your jobs,
464
00:48:21.424 --> 00:48:30.330
which ones you're tackling first, which ones you're tackling last, and which ones you may not even need to tackle and you can probably deprecate because they were, like, 1 off,
465
00:48:30.870 --> 00:48:32.090
things that were built.
466
00:48:32.630 --> 00:48:40.355
And having data lineage that essentially allows you to see the dependencies within your data stack, both in terms of the transformation layer,
467
00:48:40.735 --> 00:48:41.235
how,
468
00:48:41.855 --> 00:48:43.715
models how tables are dependent,
469
00:48:44.609 --> 00:48:48.150
as well as going into the consumption layer. So understanding, well,
470
00:48:48.530 --> 00:48:56.345
is this table that gets produced actually queried by any of the BI tools? And if it is, are those, you know, tier 1 executive,
471
00:48:57.365 --> 00:49:16.045
reports querying it, or maybe those are some 1 off reports that haven't been even touched in months, Well, incredibly valuable. Back at Lyft, we did use, Compiler Works that was, at the time, you know, very cutting edge technology, and that did save us, I would say, unimaginable amounts of time of otherwise having to migrate over 10, 000 models,
472
00:49:16.585 --> 00:49:30.565
without any kind of good insight into the dependency tree. This is also a capability that we now offer as part of Dataflow that can be helpful for teams that are planning the migrations between the warehouses between the transformation frameworks.
473
00:49:31.345 --> 00:49:35.765
You know, a plus 1 about the value of lineage. Although I think the challenge that we've had
474
00:49:36.145 --> 00:49:51.474
is that you're then trying to apply lineage onto the same legacy system that you're fighting with as it is to get off of. Right? And so in the case here where we looked at, could we apply a lineage tool onto our legacy platform and get value out of it? And
475
00:49:51.855 --> 00:50:27.555
that was an example. You know, Tobias, you asked when we decided maybe that it wasn't worth it. We were looking at what it would take to actually deploy the lineage tool on the legacy system, and then what potential strain that might add. And, you know, when you're looking at a legacy system that you're trying to move away from at that point, you're afraid to look at it the wrong way or breathe on it the wrong way for fear of the whole thing toppling over. And so I think we started looking. It could be apply lineage here. Could we use some of these tools to help us untangle this? And we ended up deciding against it. We just said the the the amount of effort it might take and the risk that it poses to putting that onto the legacy system isn't worth it. So we essentially,
476
00:50:28.015 --> 00:50:37.990
in our case, basically, had to manually craft that. You definitely need that lineage graph in order to be able to do this migration. It's just about a matter of whether or not you can bring in tooling to help you do It's just about a matter of whether or not you can bring in tooling to help you do it. Because I think once you have that lineage graph, it then becomes critical
477
00:50:41.805 --> 00:50:44.785
in sequencing your order of operations during the migration.
478
00:50:45.165 --> 00:50:49.744
Most importantly, when you look at your set of data jobs as a as a web of dependencies,
479
00:50:50.329 --> 00:51:40.339
the things that you can start to migrate and turn off on the legacy system first are those that are the edges, the leaf nodes. The things that no no 1 else depends on them. So if you have table a and table b and table b depends on table a, you can't turn off table a first because table b's stuff will break. Right? So but you can if nothing else depends on table b, you can move that over and actually turn it off first. So really being able to visualize this and it doesn't have to be a fancy graph. It could just be in our case here, we're using a a Google sheet where we just keep track of this and constantly keep looking for, like, alright. Now that we've moved this thing off, these 2 things have now become edges. They've now become leaf nodes. We can now consider moving them next. And so you definitely need that. It's just about whether or not the tooling can help you with that or not. And in terms of your experiences of going through these migration projects
480
00:51:40.720 --> 00:51:50.285
and working with other people who have gone down similar paths, what are some of the most interesting or innovative or unexpected ways that you have seen these projects approached or implemented?
481
00:51:50.744 --> 00:52:26.175
You know, I'd say that, honestly, this is 1 of those places where I'm not sure that interesting is better. I think that there's sort of when you've been through a few of these, it's almost like there is sort of a process to it. You know, I kinda alluded to it of just, you know, replicate, then get consumers over, then start working through your injections and transformations, validate along the way. It feels like sort of following that playbook has has led me to success on several of these, whereas, you know, I'd be concerned about trying to deviate from that. You know, even early days just discussing how we would approach the migration here at Eventbrite. We had other ideas, but we kept coming back to this of, like, listen. This is a well trodden path that worked. So,
482
00:52:26.475 --> 00:52:34.270
I mean, I think in terms of, like, surprises that we hit while we're working on 1, 1 of them that's a very pleasant surprise usually is is how many things you can deprecate.
483
00:52:34.970 --> 00:52:41.065
You know, Gleb talked earlier, I think, about thinking of those atomic units of migration in terms of tables and
484
00:52:41.445 --> 00:53:04.740
jobs. And okay. You might start and take an inventory and say I've got 10, 000 tables and 500 jobs. But then you might think, okay, the work is now I gotta estimate out if each job is gonna take me a week. I've got 500 person weeks. But 1 great thing is you can cut down that denominator by just saying we're not gonna migrate this thing. We're just gonna kill it right now because nobody's actually using it. And I think that self reflection on a data system is something that you usually, as a data team,
485
00:53:05.040 --> 00:53:06.980
don't carve a whole lot of time out for.
486
00:53:07.280 --> 00:53:33.315
If a job isn't broken, it can be running along the way for years before anybody ever looks at it again because no one's gonna put the Jira ticket on your board to go look at something that's not broken. Right? So, like, I think the the surprise of, actually, we can cut off you know, in the case of Eventbrite, we had over 10, 000 tables in our Hive schema. We only brought over about a 1000 of them to our Snowflake data warehouse. And, you know, we were concerned in the month or 2 we talked about previously
487
00:53:33.775 --> 00:53:55.599
that people were cutting over that they might be complaints. I think we maybe got 2 or 3 hands raised of, like, this 1 table I'm missing. Right? But, like, that meant that the other 9, 000 were not providing anybody any value that we were aware of. And so, like, it was a surprise of just how much is there, how much data is cheap to store, and so you end up just storing it kind of forever in a lot of cases like that. Yeah. I strongly
488
00:53:56.140 --> 00:54:05.445
agree with Rob on on that 1. The the boring is better. Doing lift and shift and, and really, really taking discipline and not trying to solve problems,
489
00:54:06.225 --> 00:54:10.405
other problems, but to move things over is important. And that also
490
00:54:10.920 --> 00:54:13.820
with 1 exception of deprecating things that are not used
491
00:54:14.120 --> 00:54:16.540
because that's, like, a that can really massively
492
00:54:17.000 --> 00:54:21.705
if for example, Rob's example. Right? 10 x, reduce the complexity and cost
493
00:54:22.165 --> 00:54:23.625
of the the project.
494
00:54:23.925 --> 00:54:28.960
I can say from the perspective of, you know, building Datadiff and and seeing multiple
495
00:54:29.580 --> 00:54:31.520
teams leveraging it for migrations,
496
00:54:31.980 --> 00:54:35.760
I'd say 1 thing I've been generally very impressed how effective
497
00:54:36.915 --> 00:54:43.175
a well ran migration project can be. Like, if you do lift and shift, and if you are very diligent about
498
00:54:43.770 --> 00:54:53.135
kind of mapping your dependencies and getting user acceptance testing from stakeholders and you are using tools to automate the process, I I've seen incredible velocity
499
00:54:53.595 --> 00:55:02.800
achieved by teams, like, for example, at Eventbrite that I haven't been able to do in my career. That's been definitely very inspiring. And the the second aspect,
500
00:55:03.340 --> 00:55:08.164
of, I think, seeing how Datadiff is leveraged in those migrations that I haven't anticipated
501
00:55:08.545 --> 00:55:21.789
is when we built the tool initially, we thought of it as a very kind of developer first tool where, let's say, if I'm a developer and I am working in a particular table, I would convert SQL, I would run the table, then I would trigger the data diff, and I would get results.
502
00:55:22.170 --> 00:55:26.775
And we thought of it as a very developer workflow. The interesting use case for that that I've seen
503
00:55:27.155 --> 00:55:28.295
was that companies
504
00:55:28.994 --> 00:55:29.655
also basically
505
00:55:30.115 --> 00:55:30.615
ran,
506
00:55:31.075 --> 00:55:33.575
Datadiff on schedule for their entire
507
00:55:34.020 --> 00:55:35.320
scope of data models
508
00:55:35.620 --> 00:55:36.280
every day.
509
00:55:36.820 --> 00:55:43.735
And what that gave them is a pretty good estimate of, okay, what are the tables that are currently migrated?
510
00:55:44.115 --> 00:55:44.935
What are the tables
511
00:55:45.235 --> 00:55:50.135
that are in progress? And how far off we in saying we have full parity between systems
512
00:55:50.630 --> 00:55:53.190
at scale. Because then you can say, okay. Like, this
513
00:55:54.070 --> 00:55:56.090
1 of the you know, this table is, like, 99%
514
00:55:56.869 --> 00:56:00.170
there, but there's, like, 1% discrepancy, and we know where the discrepancy
515
00:56:01.065 --> 00:56:03.325
is. And pretty much, like, running this for the entire
516
00:56:03.705 --> 00:56:04.605
scope of tables
517
00:56:04.905 --> 00:56:45.100
is something that I haven't thought people would actually do, but it makes a lot of sense to to actually implement during the migration to track your progress on the, like, the overall project level. Yeah. We're 1 of those customers doing that, and we're the ones pinging you and your team, Gleb, if we're throwing 5, 000 API requests against data fold all at once and breaking the system in some way. But, yeah, we did find that we did bake it in very much as part of the validation and basically configured alerts if things were off. And hopefully, we never hear those alerts. So it becomes almost like an ongoing validation, which again is 1 of those beautiful things you can do during a migration because you have 2 different things to compare that, you know, going forward. Obviously, there's tons of talk about data observability and testing tools and it's a whole separate topic to uncover, but it's it's much tougher
518
00:56:45.400 --> 00:56:53.065
to figure out exactly what the task where is here. It's just correctness by by matching at a 100% on column by column basis that DataFold can tell us.
519
00:56:53.525 --> 00:56:55.065
In terms of your own experiences
520
00:56:55.445 --> 00:57:01.625
of going through these migration projects, what are some of the most interesting or unexpected or challenging lessons that you learned in the process?
521
00:57:02.349 --> 00:57:10.130
I think for us and that kind of speaks to the early question of how do you choose the technology to migrate to. 1 of the things where we
522
00:57:10.510 --> 00:57:11.730
struggled was
523
00:57:12.035 --> 00:57:13.015
the user experience
524
00:57:13.635 --> 00:57:25.160
of the new system. So at the time, we knew that we had to migrate from or Russia was was coupled to source compute system into data lake, which will allow, you know, much greater scalability,
525
00:57:25.700 --> 00:57:32.115
theoretically infinite scalability for both storage and compute. And the kind of the engine that we chose to
526
00:57:32.735 --> 00:57:53.865
do the majority of the lift was Hive. That was back in 20 16. So, you know, right now, some people probably haven't ever heard of Hive, but at that time, it was probably 1 of the more popular data lake engines that was very mature. They were also Presto slash Trino and Spark, but they were kind of less stable and more more cutting edge, still kind of getting getting their ground.
527
00:57:54.460 --> 00:58:06.665
And the the trouble we ran into is that Hive was fairly good at tackling jobs at scale, but it had a really poor user experience by modern standards. It was very slow. And
528
00:58:07.205 --> 00:58:15.230
because it was slow to run queries, it was also slow to fail. Like, basically, it would take sometimes 90 seconds for it just to return the
529
00:58:15.530 --> 00:58:17.070
syntax error. And
530
00:58:17.415 --> 00:58:20.715
the the trouble of that is when you're doing a migration,
531
00:58:21.015 --> 00:58:33.970
you have to be able to iterate really fast because you're, you know, converting SQL, you're running it. Let's say you're running a data diff and then you, you know, adjust things as you go. If your system is slow and
532
00:58:34.495 --> 00:58:40.035
slow to fail, slow to turn error, slow to produce results, that has a massive impact on the speed of the entire
533
00:58:40.495 --> 00:58:42.089
project. And then the other
534
00:58:42.390 --> 00:58:57.319
aspect of it is that Hive was a system that required a lot of the expertise in using it correctly. For example, it had, like, all these tuning parameters, and no 1 today probably would choose to use Hive. So I think the lesson learned is that today, you can
535
00:58:57.700 --> 00:59:01.160
choose between, you know, awesome systems of different kind,
536
00:59:01.540 --> 00:59:10.905
but really being cognizant of what is the persona you're optimizing for, who is not gonna be doing the migration, who's gonna be using the system, and making sure that you have a good match between the technology
537
00:59:11.285 --> 00:59:12.665
and their technical aptitude.
538
00:59:13.180 --> 00:59:32.670
Because if there's a steep learning curve, right, for example, maybe you just decide to run your transformations and express them in something like Flink. Right? Because it's a very powerful system. But the majority of your team may be accustomed to SQL. And so having this learning curve, having them to learn new system, and not being able to iterate quickly
539
00:59:32.970 --> 00:59:38.670
can really, really massively complicate the entire project. So that's definitely a hard lesson learned
540
00:59:39.055 --> 00:59:43.075
that I would caution modern data platform architects about.
541
00:59:43.375 --> 00:59:46.670
We touched on this a little bit, but in your experience
542
00:59:47.050 --> 00:59:48.430
of going through these processes,
543
00:59:48.970 --> 00:59:51.550
what are either some of the design patterns
544
00:59:51.850 --> 00:59:54.910
that can help prevent the need for eventual migrations?
545
00:59:55.290 --> 00:59:58.885
And what what are some of the ways that the tooling and technology
546
00:59:59.185 --> 00:59:59.685
that
547
01:00:00.065 --> 01:00:02.085
is in common use can
548
01:00:02.465 --> 01:00:03.605
be improved
549
01:00:03.905 --> 01:00:04.405
to
550
01:00:04.770 --> 01:00:07.670
make it easier to do these migration projects or,
551
01:00:08.290 --> 01:00:13.990
extend the overall runway and lifetime of the technology to prevent the need for these migrations?
552
01:00:14.565 --> 01:00:29.900
I think sometimes the tooling is almost serving the opposite effect of locking you further into what you're currently on. I mean, I look at, you know, Snowflake just announced their dynamic tables feature, and it looks amazing. But the flip side is, alright, the more we start to use the features that are specific to that platform,
553
01:00:30.200 --> 01:00:44.060
the harder ever moving away from it will be. Right? I think in in in using Google Cloud and BigQuery, similar. We got really obsessed with the ability to use connected sheets and have g seats basically be, you know, a very simple interface for our analysts to,
554
01:00:44.360 --> 01:01:02.920
do mappings of data and things like that. And I think, you know, if you're not careful you start leaning too far into a vendor specific feature set. You make yourself very tightly coupled to things that are specific to that vendor. Now luckily on the flip side of that these vendors seems to all be in an arms race where whether you call it lakehouse or not it seems like everything is kind of converging
555
01:01:03.525 --> 01:01:19.270
on a, union of all possible features. So maybe that won't be such an issue but it's something that I think you have to look at at least a little bit skeptically of like as I move away from something that is just SQL on a set of data to anything else than that, you know, what am I potentially complicating for myself down the road,
556
01:01:19.730 --> 01:01:24.494
in moving away from that if I ever need to? So it's just something to think about, you know, how do you sort of decouple
557
01:01:24.955 --> 01:01:54.900
some of what you need to do from a compute standpoint, from what the system is then offering you in terms of automation on that. Yeah. Absolutely. That that's a, that's a challenge I'm dealing with right now of, hey. This, this vendor is adding all kinds of new features on top of this underlying open source project that is what led me to choose them in the first place. But the more I lean into all these fancy new capabilities that they have, the harder it will be if I ever decide I need to just run the open source project myself.
558
01:01:55.280 --> 01:01:57.445
Yeah. I I think that today,
559
01:01:58.545 --> 01:02:00.965
the vendor lock in is almost ineligible
560
01:02:01.425 --> 01:02:03.205
at least at least in certain,
561
01:02:04.065 --> 01:02:05.125
level of your
562
01:02:05.630 --> 01:02:06.450
stack. Right?
563
01:02:06.990 --> 01:02:22.285
So, of course, you have a vendor lock in by just using a platform like AWS or Google Cloud Platform, but no 1 is really saying that I'm gonna just run my data centers and with Linux so that I don't have this vendor lock in. So but at the same time, in the other extreme, you probably don't wanna use a vendor who is has a very closed ecosystem
564
01:02:22.665 --> 01:02:26.040
that gets you know, makes it, like, hard to get data out of. So
565
01:02:26.420 --> 01:02:27.640
the probably
566
01:02:28.500 --> 01:02:32.704
the the the middle ground here is that for critical systems like warehousing
567
01:02:33.005 --> 01:02:34.464
or, you know, business intelligence
568
01:02:35.005 --> 01:02:36.144
where you
569
01:02:36.684 --> 01:02:41.010
can have a very deep vendor lock in a sense that the longer you are using a platform,
570
01:02:41.390 --> 01:02:51.835
the higher the cost system migrate out. There are probably certain dimensions that are really important to consider. Like, for example, interoperability with other systems is huge. Right? So is this is this technology
571
01:02:52.295 --> 01:02:53.835
just like a really advanced
572
01:02:54.440 --> 01:02:56.940
1 that you love as an engineer,
573
01:02:57.320 --> 01:03:10.065
or is there a large ecosystem around it? So, yes, I think with Snowflake and with Databricks and BigQuery, Google Cloud Platform, there's a huge risk of, like, if you really invest in it as a business, it's gonna be extremely expensive to getting out of it.
574
01:03:10.365 --> 01:03:29.855
But at the same time, all of those systems have enormous, you know, enormous ecosystems between them. They're also very competitive in offering features that kind of close the gaps once someone releases new feature than the other 1 will will do soon. And so I almost wanna say something, like, corny and say, like, you can't go wrong with 1 of these big players just because
575
01:03:30.240 --> 01:03:31.540
you will probably get,
576
01:03:31.920 --> 01:03:33.299
enough of what you need.
577
01:03:33.680 --> 01:03:35.619
I think when things get tricky
578
01:03:36.000 --> 01:03:49.450
is, once you kind of deviate from, like, okay, you have these foundations and then you have to build kind of less obvious parts of the stack and what technologies you choose and how that supports your business going forward. So for example, when you choose a transformation framework,
579
01:03:49.830 --> 01:03:54.810
that also has a fairly high migration cost. So if you, for example, choose, you know, DBT
580
01:03:55.585 --> 01:03:57.605
or Airflow or Daxter
581
01:03:57.905 --> 01:03:59.525
for expressing your transformations,
582
01:04:00.224 --> 01:04:04.645
moving those migrating those could be as expensive as migrating from between the warehouses
583
01:04:05.180 --> 01:04:08.480
and making sure that when you choose a framework for transformations,
584
01:04:09.020 --> 01:04:11.440
you think that it can actually support your business
585
01:04:11.900 --> 01:04:24.940
going forward. And, for example, if you start with DBT and SQL, how does that then map into your longer term road map and types of workloads you wanna do? So and specifically in the dbt's example, they also have a pretty great ecosystem and
586
01:04:25.320 --> 01:04:28.700
a lot of the tools that I would say, like, more advanced, example, Dagster,
587
01:04:29.234 --> 01:04:30.055
they can actually
588
01:04:30.355 --> 01:04:30.855
interoperate
589
01:04:31.315 --> 01:04:38.775
with dbt, import dbt project, but then allow you to do other things. So in many ways, the stack assembles itself
590
01:04:39.119 --> 01:04:40.259
quite quite nicely
591
01:04:40.720 --> 01:04:49.435
in, you know, not letting you down, but it's definitely something that you have to be careful about when you choose parts of your stack.
592
01:04:49.815 --> 01:04:55.230
Absolutely. Are there any other aspects of this subject of data platform migrations,
593
01:04:55.770 --> 01:04:57.950
the technical and organizational
594
01:04:58.330 --> 01:05:00.645
aspects thereof, and your experiences
595
01:05:00.945 --> 01:05:24.335
of embarking on these projects that we didn't discuss yet that you would like to cover before we close out the show? Let me think 1 thing. I I kind of touched on it before, but how important it is to sort of build an internal advocacy for your migration early on. It's so important because then you end up riding that wave all the way through the pain of having to validate everything along the way. And so I think that that's something that
596
01:05:24.960 --> 01:05:28.660
maybe, you know, I don't know if I'd say it was a surprise. I think it was somewhat how quickly
597
01:05:29.119 --> 01:05:36.875
when you roll out a new system like this and, you know, start initially nagging engineers to switch their things over to use it, how much it becomes less about,
598
01:05:37.175 --> 01:05:40.235
you know, sort of forcing that behavior and more about them becoming,
599
01:05:40.750 --> 01:05:57.015
you know, advocates on your behalf. And I think that's something that is just really, really important that because it gets you buy in not just from senior leadership, but from the sort of ground level up the other way too. And so I I think that's really been key to success on these things is just being able to let everyone
600
01:05:57.410 --> 01:06:26.164
see the value of it early and often. Yeah. Can't think we can't think we can't think we more with that. Alright. Well, for anybody who wants to get in touch with either of you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get each of your perspectives on what you see as being the biggest gap in the tooling or technology that's available for data management today. I feel like for me, it's still about the right development workflow, and I know this is something that everyone has different solutions for depending on the toolset you use.
601
01:06:26.724 --> 01:06:32.079
And, you know, as we move forward, I've seen evolutionary steps in this. Things like snowflake 0 copy clones
602
01:06:32.460 --> 01:06:46.040
allow workflows to be possible that just weren't possible before. But still forming that into, like, sort of a a common set of development patterns, it feels like we're still a ways off from where the software engineering community here here is on this, on being able to press a button and basically
603
01:06:46.340 --> 01:07:17.505
create a branch off of your data. And I know there are tools and technologies that do exactly this, but trying to get that in a standard way that then makes development trivial. Makes it easy to onboard developers. Makes it easy to try things out without having to incur the cost of of spinning up a full new environment per developer. I think it's something that there's still a lot of room and it's something that my team at Eventbrite has basically had to build our own version of this. And I think many companies are probably doing the same right now because there isn't a single right way to do this that everyone agrees upon just yet. Yeah. I would strongly agree with Rob. I think the developer
604
01:07:17.964 --> 01:07:18.464
experience
605
01:07:19.130 --> 01:07:20.750
on data is in general
606
01:07:21.130 --> 01:07:22.190
way way behind
607
01:07:22.650 --> 01:07:32.485
the productivity of software engineers with the tools that support them. And even if we set the kind of the recent advancements of LLM and Copilot aside,
608
01:07:32.865 --> 01:07:33.845
just in general,
609
01:07:34.145 --> 01:07:34.645
the
610
01:07:35.020 --> 01:07:37.600
basic things that software engineers could take for granted,
611
01:07:38.300 --> 01:07:41.040
are not available to, you know, data engineers
612
01:07:41.340 --> 01:07:46.015
for the most part. Like, even, you know, syntax correction for for SQL or
613
01:07:46.315 --> 01:07:46.815
dependency
614
01:07:47.115 --> 01:07:49.055
analysis that just doesn't really
615
01:07:49.515 --> 01:07:52.015
exist for the vast majority of of workflows.
616
01:07:52.790 --> 01:07:55.609
I think the it it's getting better with the
617
01:07:56.310 --> 01:08:03.125
standardization on certain frameworks like DBT where the framework is fairly opinionated, which allows to build more opinionated
618
01:08:03.425 --> 01:08:04.565
developer experiences
619
01:08:05.425 --> 01:08:06.085
and allows,
620
01:08:07.025 --> 01:08:07.925
you know, vendors
621
01:08:08.520 --> 01:08:17.900
and other consistent players, including, you know, Dataflow to plug it into that standard and then offer developer support in particular parts of the workflow. For example,
622
01:08:18.265 --> 01:08:20.125
the the concept of data diffing,
623
01:08:21.065 --> 01:08:31.900
we brought to dbt developers and allowing allow them to diff as they go to see how the code changes impact the data, which is available as a open source package and, as a Versus code extension.
624
01:08:32.280 --> 01:08:34.300
But I think we're still scratching
625
01:08:34.680 --> 01:08:36.300
overall as the community,
626
01:08:36.845 --> 01:08:44.250
the possibilities of how great the data engineer workflow can be with the right tool and support. I think we're just we're just starting on that journey.
627
01:08:44.810 --> 01:08:49.630
Alright. Well, thank you both for taking the time today to join me and share your,
628
01:08:50.010 --> 01:08:51.150
pains and successes
629
01:08:51.450 --> 01:08:53.710
in going through these data migration projects.
630
01:08:54.170 --> 01:09:00.255
As you said, it's something that all of us are inevitably going to have to deal with. So I appreciate you sharing some of your hard won,
631
01:09:00.715 --> 01:09:05.390
lessons and wisdom from that, and I hope you enjoy the rest of your day. Thank you so much. Thanks so much, Tobias.
632
01:09:11.615 --> 01:09:21.870
Thank you for listening. Don't forget to check out our other shows, podcast dot in it, which covers the Python language, its community, and the innovative ways it is being used, and the Machine Learning podcast,
633
01:09:22.330 --> 01:09:26.750
which helps you go from idea to production with machine learning. Visit the site at dataengineeringpodcast
634
01:09:27.605 --> 01:09:38.320
dotcom to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a product from the show, then tell us about it. Email hosts at data engineering podcast.com
635
01:09:38.860 --> 01:09:44.965
with your story. And to help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers.