Designing Data Transfer Systems That Scale

WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 7/6/2024 2:38:02 PM
Duration: 3837.093
Channels: 1
1
00:00:11.264 --> 00:00:15.445
Hello, and welcome to the Data Engineering Podcast, the show about modern data management.
2
00:00:16.384 --> 00:00:18.164
Introducing RudderStack Profiles.
3
00:00:18.785 --> 00:00:28.630
RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable enriched data to every downstream team.
4
00:00:29.255 --> 00:00:35.595
You specify the customer traits, then profiles runs the joints and computations for you to create complete customer profiles.
5
00:00:36.370 --> 00:00:39.510
Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack.
6
00:00:42.370 --> 00:01:04.730
You shouldn't have to throw away the database to build with fast changing data. You should be able to keep the familiarity of SQL and the proven architecture of cloud warehouses, but swap the decades old batch computation model for an efficient incremental engine to get complex queries that are always up to date. With Materialise, you can. It's the only true SQL streaming database built from the ground up to meet the needs of modern data products.
7
00:01:05.525 --> 00:01:16.970
Whether it's real time dashboarding and analytics, personalization and segmentation, or automation and alerting, Materialise gives you the ability to work with fresh, correct, and scalable results, all in a familiar SQL interface.
8
00:01:17.350 --> 00:01:18.890
Go to data engineering podcast.com/materialize
9
00:01:20.470 --> 00:01:22.090
today to get 2 weeks free.
10
00:01:22.445 --> 00:01:26.064
Your host is Tobias Macy, and today I'm interviewing Andrei Tserakav
11
00:01:26.445 --> 00:01:32.900
about in yeah. About operationalizing high bandwidth and low latency change data capture. So, Andre, can you start by introducing yourself?
12
00:01:33.280 --> 00:01:36.260
Hello. My name is Andre. I'm working as a
13
00:01:36.640 --> 00:01:38.100
as a engineer for
14
00:01:38.435 --> 00:01:46.135
quite a lot. I think it's more than 10 years now. And for last, I would say, like, maybe 6, maybe 7 years, I work for this
15
00:01:46.580 --> 00:01:48.920
mainly on distributed systems and,
16
00:01:49.300 --> 00:01:52.120
mostly focusing of data movement stuff,
17
00:01:52.740 --> 00:02:02.560
moving data from point a to point b somehow, somewhere, some with some data, and I'm actually specializing in it. I have quite a big journey
18
00:02:02.940 --> 00:02:06.140
before, but right now, I'm mostly focusing on this stuff. And,
19
00:02:07.580 --> 00:02:11.120
I think I have quite a good experience in this area
20
00:02:11.625 --> 00:02:12.685
from my past.
21
00:02:13.305 --> 00:02:16.605
And do you remember how you first got started working in data engineering?
22
00:02:17.305 --> 00:02:17.805
Actually,
23
00:02:18.105 --> 00:02:20.239
it was quite a fun story because,
24
00:02:20.860 --> 00:02:26.720
initially, I have my own startup, which is boring 1. We'd be building the bus transportation
25
00:02:27.260 --> 00:02:31.205
application, bus, in terms of, like, cross city bus,
26
00:02:31.745 --> 00:02:40.239
international buses, something like this. This laptop was fine. Everything was quite okay. And at some point, a big corporation buy out our startup. And,
27
00:02:40.879 --> 00:02:41.379
after
28
00:02:41.760 --> 00:02:51.504
a year inside of this big corporation, they decided to close a startup, and they proposed me to change my role. And, actually, after so many years working in this
29
00:02:51.970 --> 00:03:01.275
fascinating from terms of of a business area, but not so interesting in terms of a technology to change my role, and I decided to change to more technical stuff. I always,
30
00:03:01.655 --> 00:03:11.840
love working with the databases and, decide to go to the infrastructure team. Mostly, it was data infrastructure team. And, I start working on the project that
31
00:03:12.140 --> 00:03:13.360
actually quite old,
32
00:03:13.739 --> 00:03:20.735
I would say 15 years before. Once I joined, it was has already 15 years of history, something approximately like this.
33
00:03:21.114 --> 00:03:21.355
And,
34
00:03:21.995 --> 00:03:24.235
it moved a lot of data from,
35
00:03:24.555 --> 00:03:26.150
a lot of different,
36
00:03:26.849 --> 00:03:33.590
sources into 1 big data warehouse. This data warehouse was extremely big, and my goal was to
37
00:03:34.205 --> 00:03:34.705
make,
38
00:03:35.565 --> 00:03:38.925
delivery lock a bit less than it was before. And,
39
00:03:39.565 --> 00:03:40.864
this was what initial
40
00:03:41.330 --> 00:03:45.830
touch of anything in data engineering field. I think it was 2016
41
00:03:46.370 --> 00:03:47.110
or 2017,
42
00:03:47.410 --> 00:03:50.415
something about this. And, at the moment, they,
43
00:03:51.675 --> 00:03:52.735
as a product that,
44
00:03:53.195 --> 00:03:54.815
moves this data, utilize
45
00:03:55.860 --> 00:03:56.760
a lot of
46
00:03:57.459 --> 00:03:59.400
pretty old but still quite
47
00:03:59.860 --> 00:04:01.800
good concept as a MapReduce.
48
00:04:02.100 --> 00:04:05.495
So there there was, like, 2 main main pieces of this
49
00:04:06.215 --> 00:04:09.114
software. 1, it's, like, delivering data from,
50
00:04:09.655 --> 00:04:10.155
actual
51
00:04:10.935 --> 00:04:13.355
producer of of the data. It's usually application
52
00:04:13.670 --> 00:04:21.770
to a sort of a queue. It was not a Kafka because Kafka was not able to handle such load. It was internal replacement of a Kafka. It has some Kafka
53
00:04:22.224 --> 00:04:51.630
flavor. For example, topics still named as a Visa Kafka prefix, but it's implemented in the c plus plus to make it little bit faster. Right now, I don't remember what exact number, of traffic was was in 2016. But in 2018, it was about 5 gigabyte per second. It's a lot of traffic that in incoming in this in into the system. And these traffics goes from the applications. Usually, it's worth written somewhere and deployed somewhere in the cluster, and they provide the logs. And these logs need to be delivered into
54
00:04:52.090 --> 00:04:59.335
the data warehouse in a structured way as soon as as fast as possible. But initial design was like, we have this, application.
55
00:04:59.955 --> 00:05:14.479
It provide data into the queue. From the queue, we provide this in a rough format in a a data warehouse. And then we have a huge pipeline of map reduced jobs that parse this data, index this data, and publish this data as a final tables. And
56
00:05:14.780 --> 00:05:16.634
this whole pipeline was,
57
00:05:17.495 --> 00:05:21.835
quite a big scale. At the end, it generate, like, 5 or 7000
58
00:05:22.580 --> 00:05:24.660
different kinds of tables, and,
59
00:05:25.380 --> 00:05:39.354
amount of data was extremely high. It's it's a dozen of petabytes of data. So it's quite extremely loud. The problem with all this approach, it was very good in terms of scalability. It scaled quite well on horizontally
60
00:05:39.655 --> 00:05:46.480
because it utilized map reduce approach. It just chunking chunked of raw data by some side, then produce
61
00:05:46.920 --> 00:05:57.432
transform it somehow, parse it somehow, index it somehow, push it in a table, and then, get another chunk, etcetera, etcetera. But deliver the lag between the actual event on the machine, on the application, and the,
62
00:06:00.530 --> 00:06:08.074
final table, delivery lag was quite high, about 5 or 6. It was constant 1, so it's not growing
63
00:06:08.375 --> 00:06:16.730
anytime. It's always 5 or 6 hours. It's actually not doesn't matter if your table big or small. It's always 5 or 6 hours. It's like constant,
64
00:06:17.130 --> 00:06:20.669
tax for your for your data. I think this approach is more or less
65
00:06:21.050 --> 00:06:36.180
well known because it's like batch processing, big batch processing. And usually, if you have a big batch processing, you have a delivery log in in this amount. If you have a daily table, so usually to provide a daily table, you need to end the day and add 5 to 6 hours for composing
66
00:06:36.640 --> 00:06:40.025
this table at the end of the day. Something like this. And,
67
00:06:40.745 --> 00:06:45.690
I was, like, quite shocked about all this situation because before before this,
68
00:06:46.090 --> 00:06:53.610
I all only works with a a moderate amount of data. I use, like, classical databases, MS SQL, Oracle, Postgres. And they usually
69
00:06:54.284 --> 00:07:37.710
if you have, 100 gigabytes of data in it, you have already big data. But once I saw the real big data, it was, for me, a bit shocking, especially the technologies they use, the approaches they use because it's quite different. And they'll they don't use anything like actual applications. It's just a bunch of Python scripts here and there that run everything. And no 1 actually understands how it works because it's 15 years of history. A lot of people change. A lot of, people go with the name. It somehow works. You can kinda understand how it works, but, overall, it's quite complex and frail system. Yeah. And it was my first touch with a big data world, kinda like this. And given that trial by fire, figuring out how to manage
70
00:07:38.010 --> 00:07:50.390
this data transfer, the challenges associated with it, You decided that you actually wanted to invest in it even more and continue working in that space. And I'm wondering, given that context and the work that you're doing
71
00:07:50.950 --> 00:07:53.850
now, what are the core foundational
72
00:07:54.390 --> 00:07:56.250
elements of the problem of
73
00:07:56.550 --> 00:07:58.890
data transfer for purposes of
74
00:08:05.294 --> 00:08:09.360
as self solvable, I would say like this. Because in as self solvable,
75
00:08:10.060 --> 00:08:18.925
I would say like this. Because initially let me give you a big bigger context again a little bit. This whole sito the whole system was,
76
00:08:19.645 --> 00:08:56.185
front facing to actual user by a configuration in the central repository. So we have a repository. In this repository, you have a file. You have actually not 1 file, but 5 file. Files is a file 5 JSON files with all configuration for the whole companies. These files, like, 5 megabits of data. It's a big file big JSON file, which is not automatically produced. It's actually produced by users. So if user want to add 1 new item to this pipeline, you need to commit in this file. Then the CI job will trigger some Python script. This Python script will trigger another Python script, and the whole machinery starts working by magic. A user just communicate
77
00:08:56.645 --> 00:08:59.205
with all the system by changing in this pie,
78
00:09:00.019 --> 00:09:40.410
JSON file. And this is self sort of sole service approach is quite good for the data engineering because at the end, if you're providing this as a service, if you're as engineer who designed and developed the application, for you, it's very complicated systems that do a lot of stuff. It's a lot of code. But for end user, for actual data engineer who just want to see the data in the table, it's not important. They just need to somehow set up the delivery from point a to point b. And the idea that, come to my mind is actually what's actually approaching me even right now is to give the tool that can be can be used by people who don't really care about the details about implementation.
79
00:09:40.950 --> 00:09:48.375
Just 1 important thing, I need to move data from point a to point b. Just do this somehow. I don't care how. I don't care,
80
00:09:48.855 --> 00:09:59.089
I I don't care the details. The only thing I care, it's about freshness of the data and consistency of the data. Just you don't see me that consistency would be there, and
81
00:10:00.975 --> 00:10:03.235
freshness would be not so bad. And,
82
00:10:04.415 --> 00:10:08.035
idea of making this self storage approach is actually very
83
00:10:08.350 --> 00:10:09.410
core for me
84
00:10:09.710 --> 00:10:16.050
in in our product actually even right now, the ability to give this as a sole source. The only problem that,
85
00:10:16.505 --> 00:10:23.644
once you start doing this, you you start realizing that moving data from point a to point b is quite a complex task. And
86
00:10:24.149 --> 00:10:31.290
giving this as a tool rather than, like, a framework for development is a bit more makes everything complicated
87
00:10:32.115 --> 00:10:33.315
and, make,
88
00:10:33.715 --> 00:10:35.975
the configuration of all this pipeline
89
00:10:36.355 --> 00:10:38.695
extremely complex. And this actually
90
00:10:39.170 --> 00:11:00.580
very challenging task to give the user ability to, run this. It's it's a first step. Then investigate and monitor this because once you run it, you need to somehow monitor it. And once it broke, you need to, add some tools for investigation and triaging and debugging. And all these pillars actually is a foundation for good service, for providing,
91
00:11:01.200 --> 00:11:05.525
service for moving data, not just a JSON file with all configurations.
92
00:11:05.985 --> 00:11:12.245
And the the shift from this JSON file of with the configuration to some sort of a service that you can configure
93
00:11:12.649 --> 00:11:28.025
on your own and use it as a as a tool in your daily work, it's, like, quite challenging. And in your work of building these tools, trying to make a self serve capability for being able to manage those data transfers,
94
00:11:29.000 --> 00:11:40.774
what are some of the elements of the existing solutions in the ecosystem that you find to be lacking and that necessitated your investment in this problem and continuing to build out your own solutions?
95
00:11:41.154 --> 00:11:42.774
Well, we actually started
96
00:11:43.075 --> 00:11:44.454
building this solution, I think,
97
00:11:44.755 --> 00:11:50.490
2017, 2000 18. First thing that we missed is actually good library of connectors.
98
00:11:50.870 --> 00:11:54.170
The things that you need to connect multiple sources, multiple
99
00:11:55.005 --> 00:11:56.865
different kind of sources, actually,
100
00:11:57.485 --> 00:12:00.625
right now, there is a whole bunch of open source technology
101
00:12:01.245 --> 00:12:03.105
on on on the market. They
102
00:12:03.500 --> 00:12:05.600
kinda can provide you some capabilities.
103
00:12:06.140 --> 00:12:13.665
There is, like, Kafka Connect for connecting to the relational databases. There is Arbyte for connecting non relational databases.
104
00:12:14.045 --> 00:12:15.905
There is, like, Stitch, Meltano,
105
00:12:17.005 --> 00:12:18.790
anything everything, like, in this,
106
00:12:19.430 --> 00:12:29.425
world that connect allow you to grab data from somewhere is, like, connected to to your to your system. But the problem that quality of these connectors usually is unpredictable,
107
00:12:29.885 --> 00:12:30.125
and,
108
00:12:30.845 --> 00:12:34.065
performance is maybe an issue. For example,
109
00:12:34.605 --> 00:12:37.960
we have a big scale of all this stuff. We have a lot of data.
110
00:12:38.340 --> 00:12:39.640
And using the
111
00:12:40.020 --> 00:12:42.360
default connector, for example, for
112
00:12:43.705 --> 00:12:49.485
Visa, Kafka Connect was quite expensive for us. We have to scale the system not just horizontally
113
00:12:49.865 --> 00:12:58.510
and vertically, but actually be able to we should be able to run as many as as many as possible small connectors. And usually, the
114
00:12:59.075 --> 00:13:06.375
for example, the Kafka Connect 1, it's most popular implementation of changes that are captured for the open source world.
115
00:13:06.680 --> 00:13:16.220
Run of single connector usually require you a pretty decent virtual machine in, 4 gigabytes of RAM. And, if you have, 2, 000 databases
116
00:13:16.765 --> 00:13:33.009
that run these connectors, it's pretty a lot. Some of them is actually reasonable to have this 4 gigabytes of RAM inside of the cluster to allocate to this database because it's decent load. But most of them actually produce 1 or 2 rows in the in the middle. You still need to somehow
117
00:13:33.310 --> 00:13:42.455
do the transfer, And you have to think not about just scalability in terms of, like, how many instances you can run, but how cheap the cheapest instance
118
00:13:43.155 --> 00:13:56.425
is. 1 of the key reasons why we initially decide decide to go to the own implementation is to reduce the overhead that usually comes with the open source solution and make it as simple and and clean as possible,
119
00:13:56.725 --> 00:13:58.345
as and focusing on the,
120
00:13:59.285 --> 00:13:59.785
target,
121
00:14:00.725 --> 00:14:07.980
connectors to make it not like a big library of connectors, but, like, small, but really good with good scalability and good
122
00:14:09.000 --> 00:14:26.785
performance to consumption ratio because, usually, once you start something, you consume some resources. It's usually CPU and memory. Sometimes it's something else. It may also be, for example, network. But usually, you're bottlenecked in the such scenarios is CPU and memory, and you need
123
00:14:27.185 --> 00:14:28.384
might minimize them. And,
124
00:14:29.024 --> 00:14:31.764
having the generic solution like Kafka Connect,
125
00:14:32.225 --> 00:14:48.635
is not efficient because it has a lot of, extra overhead. And, initially, right now, it's not the case, but, like, 5 years ago. For example, Kafka Connect can't live without Kafka. And it was a it was a big deal because if you want to connect your database
126
00:14:49.095 --> 00:14:56.380
and read this data and put it to a data warehouse, you have to put a Kafka in between no matter what. And it's actually a bit a bit of overhead
127
00:14:56.760 --> 00:15:04.575
in some cases. And, we try to eliminate this extra overhead and connect them directly because why not? And this is,
128
00:15:05.195 --> 00:15:13.370
another point why we should why we decide to implement it on our own. In that space of data transfer, there are also a number of different
129
00:15:13.990 --> 00:15:15.050
patterns and
130
00:15:15.510 --> 00:15:26.240
requirements and constraints based on the type of the source system. So change data capture, as you mentioned, is 1 approach to reducing the overall latency of data transfer
131
00:15:36.255 --> 00:15:54.880
API endpoints where you need to be able to query the the rest API to be able to get new data. Not all of those even support having some sort of incremental sync capabilities. I'm curious how that complicates the overall problem space of saying, okay. Well, for these data sources, I can do this, and I can make sure that I've got subsecond,
132
00:15:55.495 --> 00:16:13.225
data transfer. But now when I'm dealing with my Hubspots or my Salesforces of the world, I have to do a different approach, and maybe some of these endpoints support incremental other other endpoints. I have to do a full refresh. I'm just wondering kind of how that compounds the overall problem space. It's make everything more complicated, Justin.
133
00:16:13.704 --> 00:16:15.245
That that's actually the answer.
134
00:16:16.185 --> 00:16:25.550
You can, like, try to minimize it somehow, kind of try to manage it, but overall, it's still goddamn complex. In general, we have, like, a huge matrix of of compatibility
135
00:16:26.170 --> 00:16:46.660
of different connector, like this connector compatible with these features, this connector compatible with these features. And then we can when we compile them, we, say that this transfer, like source and target endpoint pair of the source and target endpoint compatible with those features because they have, say, compatibility metrics intact. In fact in fact, we actually
136
00:16:47.635 --> 00:17:04.610
the idea that we've concentrated on a selected set of, connectors give us an ability to tune a little bit our approach. And initially, we just have 1UP. It was the first iteration of a service. You have CDC and nothing else. Then we start adding the features 1 by 1, compatibility
137
00:17:04.995 --> 00:17:07.815
features like 1 by 1. 1st, we add the backfill functionality,
138
00:17:08.195 --> 00:17:11.015
which we call a snapshot. And this backfill functionality,
139
00:17:11.315 --> 00:17:12.615
then we combine with
140
00:17:13.029 --> 00:17:31.919
replication in in CDC. So you backfill and then replicate. Then we start realizing that not everyone contains the exchange streams, and we're adding the incremental snapshot. And right now, we have, like, all set of, options available. But, initially, we start with a low a low point. Like, the first point was a change that I capture. I think in
141
00:17:32.299 --> 00:17:36.159
the most, tools, the change that I capture is a most com com complicated,
142
00:17:36.779 --> 00:17:38.559
approach because it require
143
00:17:38.865 --> 00:17:41.524
more detailed implementation of 1 particular connector.
144
00:17:41.825 --> 00:17:59.665
But we start with the hardest 1 and just adding the, anything else around it. And we build everything around this concept of change data capture. And what we try to do is utilize the same approach to any other way of reading data. So we imagine any database as a stream of changes. And this actually
145
00:18:00.445 --> 00:18:00.945
simplifies
146
00:18:01.325 --> 00:18:04.945
the design, overall design of the system and give us
147
00:18:05.720 --> 00:18:06.780
ability to build
148
00:18:07.240 --> 00:18:18.674
this extra feature easier. So stream of changes, it's a simple concept. For CDC, it's more or less natural. For snapshot, it's not so natural. But what we say is like, okay, what is the difference between a snapshot
149
00:18:19.135 --> 00:18:30.715
and a change that I capture? The only difference, like, there is a stream of changes, but at some point, it ends. That means that your snapshot is ended. You read all your data. And if you have a CDC, this is infinite
150
00:18:31.175 --> 00:18:36.955
process. The same for incremental things, actually. This incremental thing is just infinite loop of changing
151
00:18:37.335 --> 00:18:41.980
stuff. It's still a stream of changes, but these changes have more granular,
152
00:18:42.600 --> 00:18:43.500
burst nature.
153
00:18:43.880 --> 00:18:58.490
At some point, they start reading new data, burst your new piece of data in the stream, and then, like, sleep for for a while. So having this in the core of a system, this concept of a stream of events, is actually helpful for us. Another thing that we actually
154
00:18:59.110 --> 00:19:00.315
designed, it was
155
00:19:00.634 --> 00:19:09.375
quite okay concept that we chose at the beginning, and it right now would be helpful for us, is treating everything as events and, treating
156
00:19:10.240 --> 00:19:15.220
any change as a logical event, and logical event bounded to a string,
157
00:19:16.000 --> 00:19:32.200
bounded to a row, bounded to a row, 1 particular row. And this row can be bounded to a table. Initially, we think it's, like, more than enough for us. But once we start adding more features, we add a concept called technical events. And these technical events give us the ability to add extra
158
00:19:32.659 --> 00:19:46.159
features to the pipeline. So overall, we have just a pipeline or a stream of events, infinite stream of events. It can be finite if you have, like, a snapshot. But for the code itself, it's not a big deal. It's not there is no difference. But in this,
159
00:19:46.940 --> 00:19:56.655
infinite stream, there is, like, a checkpoint, for example, or maybe some events for some technical details, give a copy of data, for example, or run the DBT transformer
160
00:19:56.955 --> 00:20:05.200
or something like this. These technical events give us ability to inject functionality on the stream and give it, like, a flexibility to add the functionality.
161
00:20:05.740 --> 00:20:19.429
So the core concept, like, to summarize all of this, is treat everything as a string of of events and treat events as an event. So they can be either a row or something not related to row. Some events for
162
00:20:19.730 --> 00:20:33.515
non data related activity. Like transaction starts, transaction ends its events, but there is no data. Data is somewhere in between. But these transactional events allow us to give more guarantees for actual users, like for moving data on a transactional
163
00:20:34.020 --> 00:20:35.220
side. So we have trans
164
00:20:35.940 --> 00:20:37.960
the transaction boundaries intact,
165
00:20:38.260 --> 00:20:39.320
something like this.
166
00:20:42.165 --> 00:20:44.265
This episode is brought to you by DataFold,
167
00:20:44.725 --> 00:20:52.090
a testing automation platform for data engineers that finds data quality issues for every part of your data workflow from migration to deployment.
168
00:20:53.110 --> 00:20:58.250
DataFold has recently launched a 3 in 1 product experience to support accelerated data migrations.
169
00:20:59.045 --> 00:21:06.185
With DataFold, you can seamlessly plan, translate, and validate data across systems, massively accelerating your migration project.
170
00:21:06.990 --> 00:21:17.794
DataFold leverages cross database diffing to compare tables across environments and seconds, column level lineage for smarter migration planning, and a SQL translator to make moving your SQL scripts easier.
171
00:21:18.335 --> 00:21:21.475
Learn more about DataFold by visiting data engineering podcast.com/datafold
172
00:21:23.054 --> 00:21:23.554
today.
173
00:21:25.880 --> 00:21:27.260
Given that core
174
00:21:27.640 --> 00:21:30.140
assumption and the core logical foundation
175
00:21:30.520 --> 00:21:31.340
of these
176
00:21:31.640 --> 00:21:32.700
streams of changes
177
00:21:33.095 --> 00:22:00.030
for the situation where you do have a data source that will only give you a block of data. And every time you ask for it, it's maybe the same block of data with a few new things in it. I'm curious how you think about trying to map that into the set of changes. Do you then have to maintain history of, this is the block of data they gave us last time, and now we're just going to remove any duplicates and only add the new things to the change set? Or just curious how you think about mapping that into that problem space.
178
00:22:00.330 --> 00:22:07.130
Yeah. Act actually, it's both way. The first thing first, you need to treat every block data to block data, but sometimes you wanna,
179
00:22:07.530 --> 00:22:08.155
to short
180
00:22:08.555 --> 00:22:12.795
cut extra cost because reading this block data is not cheap. It's not for free. And,
181
00:22:14.155 --> 00:22:16.335
you need to store some server's
182
00:22:16.990 --> 00:22:20.610
course of stuff so you know the checkpoints. And these checkpoints,
183
00:22:21.310 --> 00:22:22.610
can be used as
184
00:22:22.990 --> 00:22:26.610
a part of a query or API for your source
185
00:22:27.375 --> 00:22:31.795
to not, like, just grabbing this data in the first place. And, this actually
186
00:22:32.495 --> 00:22:47.735
concept that we utilize for incremental snapshot lag. So we store in the cursor and then use this cursor to minimize the amount of data for the next iteration. So we don't just read everything. We just read smaller. Power applications is even easier. Once we
187
00:22:48.035 --> 00:23:01.365
receive an answer from a target database that everything is written, we can move the cursor of our change stream a little bit further and, continue reading. The only tricky part here is on restart.
188
00:23:01.665 --> 00:23:05.125
And this is, like, as a quickie question appears. It's
189
00:23:05.425 --> 00:23:16.540
a question about delivery semantics. It's usually an important 1. And initially, we aimed to the at least 1 delivery semantics. So every every time if something goes to the failure,
190
00:23:16.865 --> 00:23:19.285
we restart our worker, and this actually
191
00:23:19.665 --> 00:23:20.965
give us some duplicates
192
00:23:21.345 --> 00:23:25.525
of data because we send something to a target database but not receive
193
00:23:26.200 --> 00:23:28.059
acknowledgement. And this acknowledgement
194
00:23:28.360 --> 00:23:39.234
is in a gray area. It can be either not delivered because it's it's okay. It's not delivered. But sometimes it's written, but we don't know about this yet. And, they
195
00:23:39.715 --> 00:23:47.780
we we read stuff from the source database, give us a duplicate of this amount of data. What actually can be done with this?
196
00:23:48.240 --> 00:23:50.500
Actually, nothing can be done particularly.
197
00:23:50.800 --> 00:24:04.804
The only thing that can help is utilizing transaction capabilities for target database to store some sort of marks on this target database in in in a transactional way and remove these items from a write
198
00:24:05.409 --> 00:24:34.785
batch in on the target side. So you still read this data a little bit more, like excessive data, but you should not write this in a target database to prevent duplication. In most cases, it's not a big deal because you have a primary case. Primary case duplication works best for almost any cases if you have a primary case, but not every database has a primary case. For example, s 3, which we actually treat s 3 as a database, is not contains any primary case duplication. So we need to invent some sort of a logic around it. But it gets it can be implemented
199
00:24:35.165 --> 00:24:40.705
if you have the right concept in in core of this logical event. And if your logical event contain,
200
00:24:41.245 --> 00:24:42.030
some sort of
201
00:24:42.670 --> 00:25:06.580
sequence inside of it, you can easily deduplicate data based on this sequence. Most of database, most of change streams contain sort of a sequence number, and you can just utilize the sequence number to deduplicate stuff on target database. And we actually utilize it a lot. We provide for all changes the sequence number in it and just deduplicate the sequence number based on the sequence number in our target side. And now moving to the
202
00:25:06.960 --> 00:25:32.385
consumption side of a transfer service where you say, I just wanna make sure that my data gets from over here to over there. I don't care about all of these details. I just wanna be able to click a button or say, this is what I need, and everything just happens. And I'm curious as you have been working through this problem, building a product around enabling that interaction case, what are some of the sharp edges that you've had to deal with and manage,
203
00:25:33.165 --> 00:25:35.184
and and develop or an engineer around
204
00:25:35.630 --> 00:25:43.170
to be able to give that end user experience of not having to care about all of these vagaries of the underlying technical implementations
205
00:25:43.550 --> 00:25:43.975
and
206
00:25:44.375 --> 00:25:47.434
the failure cases and recovery and retries?
207
00:25:48.054 --> 00:25:51.195
I would say that, what I learned from all this stuff
208
00:25:51.630 --> 00:25:55.570
is happy path is easy. The simple stuff works pretty straightforward,
209
00:25:55.950 --> 00:26:03.225
but there is, like, edge cases, and edge cases are really hard. The first thing for problematic things that we actually, faced on,
210
00:26:03.785 --> 00:26:06.365
on on our journey to implement this self-service,
211
00:26:07.640 --> 00:26:09.820
kind of magically scaled solution is
212
00:26:10.120 --> 00:26:10.620
sometimes
213
00:26:11.400 --> 00:26:13.500
user don't want to scale. Sometimes
214
00:26:14.120 --> 00:26:15.660
the scalability should be
215
00:26:16.040 --> 00:26:16.540
more
216
00:26:17.005 --> 00:26:17.505
controlled,
217
00:26:17.805 --> 00:26:18.305
etcetera.
218
00:26:18.605 --> 00:26:24.545
What I mean by this is the transfer itself capable to read any amount of data from a source database
219
00:26:24.925 --> 00:26:27.250
just give us enough throughput.
220
00:26:27.550 --> 00:26:31.650
And the problem, if users don't think about this, like, throughput problematic,
221
00:26:32.030 --> 00:26:46.670
we can use all throughput from database. And there is no more left for any other users. In some cases, it's okay if we are only user of this data. But, I think it's rarely a case. So first thing that we I learned that actually this magic
222
00:26:47.050 --> 00:26:48.030
should be controllable.
223
00:26:48.330 --> 00:26:51.710
A user can actually control our scalability points.
224
00:26:52.090 --> 00:26:53.610
We provide some,
225
00:26:54.170 --> 00:26:57.275
insights, some settings that user can actually adjust,
226
00:26:58.155 --> 00:26:59.215
based on their needs.
227
00:26:59.675 --> 00:27:02.255
Rather small 1, so they don't want to
228
00:27:03.195 --> 00:27:08.770
run more than 1 1 or 2 concurrent queries on top of it. So they can specify,
229
00:27:09.150 --> 00:27:12.770
I wanna limit your capabilities to 2 concurrent connections.
230
00:27:13.390 --> 00:27:17.605
Deal with it. And then we should do as much as possible with this limitation.
231
00:27:17.985 --> 00:27:20.405
So users should provide us some limitation
232
00:27:20.785 --> 00:27:22.549
and we provide them,
233
00:27:23.090 --> 00:27:28.309
this ability to give this limitation. The only thing that harden this scenario is it, like,
234
00:27:28.610 --> 00:27:31.125
explain to a user how they Because,
235
00:27:31.825 --> 00:27:35.845
as as I already mentioned, most of the team just don't want to think about the
236
00:27:36.145 --> 00:27:39.045
complicated stuff like, okay. My Postgres is overloaded.
237
00:27:39.399 --> 00:27:42.620
What should I do with it? So we need, like, a lot of examples,
238
00:27:43.000 --> 00:27:52.285
best practice, and most importantly, reasonable defaults. So I would say in 80%, maybe even 90% of the case, users don't change our default,
239
00:27:52.825 --> 00:28:00.500
and this is more than enough. But having these defaults is a very hard 1. You have a lot of you should have a lot of exploitation expertise
240
00:28:00.800 --> 00:28:01.540
of the
241
00:28:01.920 --> 00:28:04.420
overall system to understand the best practice
242
00:28:04.720 --> 00:28:09.184
and put it in the right place. So we have a quite smart default
243
00:28:09.645 --> 00:28:13.345
that actually adjust it based on your target and source databases,
244
00:28:13.645 --> 00:28:14.544
based on your
245
00:28:15.049 --> 00:28:19.230
type of a transfer, amount of a tables, etcetera. And these defaults,
246
00:28:19.770 --> 00:28:20.270
actually
247
00:28:20.650 --> 00:28:22.750
hard hard to pick, hard to adjust.
248
00:28:23.105 --> 00:28:24.404
But user can be
249
00:28:24.945 --> 00:28:29.284
can adjust them if if they want. But in most cases, they shouldn't do this.
250
00:28:29.664 --> 00:28:30.164
And
251
00:28:30.540 --> 00:28:31.520
as you have
252
00:28:31.900 --> 00:28:33.680
moved beyond just the initial
253
00:28:34.380 --> 00:28:49.880
capabilities of, I have the system, it will work. I can do a, b, and c with it to now I need to actually run this at scale. I need to make sure that I can automatically provision capacity as new people come in or as their use cases grow. What are some of the challenges of operationalizing
254
00:28:50.580 --> 00:28:56.360
this complexity while still providing a an accessible and enjoyable end user experience?
255
00:28:57.035 --> 00:29:00.975
I would say that main main challenge here is,
256
00:29:02.155 --> 00:29:02.655
providing,
257
00:29:03.595 --> 00:29:08.610
providing this is easy, but you should keep track of your costs because,
258
00:29:09.150 --> 00:29:25.670
once you provide something as a service, like you're hiding some costs, you're hiding CPU, you're hiding compute under the hood. So users don't think about this, but you should think about this. So you have a should have a lot of monitorings in place. You should have a duty, shift duty for your engineers. So your engineers
259
00:29:26.550 --> 00:29:33.370
always on the call because your system became a critical 1. If something is broken in your system, a lot of people can be
260
00:29:33.915 --> 00:29:35.295
affected. So you should
261
00:29:36.315 --> 00:29:36.815
always
262
00:29:37.115 --> 00:29:41.615
take a look on this. Someone should be always online. Someone should be always
263
00:29:41.995 --> 00:29:45.559
on call. So on call is important. Also important is
264
00:29:45.940 --> 00:29:54.385
monitoring and alerts. So everything should be automated. Some weird behaviors should be automated. Right now, we don't have, like, enough,
265
00:29:54.924 --> 00:29:57.485
in my opinion, alerts. A lot of stuff just,
266
00:29:58.044 --> 00:30:00.385
flooring just by observing the monitoring.
267
00:30:00.769 --> 00:30:14.185
You see a strange pattern on the monitoring, and you then go to investigation, then maybe find some bug, maybe some interesting use case, etcetera. And another thing that is important to give a scalable experience is providing,
268
00:30:14.805 --> 00:30:16.300
building service with,
269
00:30:17.560 --> 00:30:19.340
runtime agnostic in mind.
270
00:30:19.800 --> 00:30:20.940
We build our system
271
00:30:21.480 --> 00:30:28.485
initially as a runtime agnostic 1. So anything can that can run the Docker container is our runtime,
272
00:30:28.945 --> 00:30:31.285
and we can host on top of anything.
273
00:30:31.720 --> 00:30:51.130
And the only thing that we need to care is that we have enough capacity of this underlying system. Right now, hopefully, we have AWS, GCP, Azure that, take a head of thinking about actual physical machines out of our heads. So at least this point is solved, but you still need to manage,
274
00:30:51.830 --> 00:30:54.730
alerts for costs, for amount of virtual machine, etcetera.
275
00:30:55.165 --> 00:30:56.625
This is still quite important
276
00:30:57.085 --> 00:31:00.225
because once I joined the company back in the days,
277
00:31:00.845 --> 00:31:14.105
we host all this complex stuff, complex solution on the physical machines, in the physical data centers. We have, like, several big data centers with actual machines in it, And, sometimes failures be was not because we,
278
00:31:14.645 --> 00:31:16.745
don't have, like, good coders, maybe
279
00:31:17.125 --> 00:31:26.940
not enough optimization, but sometimes just too many traffics come into the system, and we can't scale it because there is no more physical machines available in this particular data center.
280
00:31:27.240 --> 00:31:28.940
Hopefully, right now with the clouds,
281
00:31:29.445 --> 00:31:30.985
it's not so painful
282
00:31:31.285 --> 00:31:31.785
to,
283
00:31:32.165 --> 00:31:37.470
scale up and scale down because you don't need to buy and wait until the physical machines come to your
284
00:31:37.870 --> 00:31:40.049
warehouse where your data center is located
285
00:31:40.510 --> 00:31:41.010
because
286
00:31:41.390 --> 00:31:46.885
sometimes it can be quite problematic to scale a solution if you run this,
287
00:31:47.365 --> 00:31:49.545
not run time agnostic, not in the clouds.
288
00:31:52.165 --> 00:31:53.385
Data lakes are notoriously
289
00:31:53.685 --> 00:31:54.185
complex.
290
00:31:54.679 --> 00:32:10.325
For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte scale SQL analytics fast at a fraction of the cost of traditional methods so that you can meet all of your data needs ranging from AI to data applications to complete analytics.
291
00:32:10.840 --> 00:32:14.299
Trusted by teams of all sizes, including Comcast and DoorDash,
292
00:32:14.600 --> 00:32:21.085
Starburst is a data lake analytics platform that delivers the adaptability and flexibility a lakehouse ecosystem promises.
293
00:32:21.705 --> 00:32:28.365
And Starburst does all of this on an open architecture with first class support for Apache Iceberg, Delta Lake and Hoody,
294
00:32:28.720 --> 00:32:30.900
so you always maintain ownership of your
295
00:32:31.680 --> 00:32:34.580
data. Want to see Starburst in action? Go to dataengineeringpodcast.com/starburst
296
00:32:36.880 --> 00:32:37.780
and get $500
297
00:32:38.080 --> 00:32:43.395
in credits to try Starburst Galaxy today, the easiest and fastest way to get started using Trino.
298
00:32:45.590 --> 00:32:49.289
Another aspect of complexity as you scale is
299
00:32:49.590 --> 00:32:50.890
not just the
300
00:32:51.429 --> 00:32:59.605
throughput capabilities of I can handle x number of gigabytes per second or I can handle x number of users, but there's also the scaling of
301
00:32:59.905 --> 00:33:02.085
additional capabilities, additional complexity
302
00:33:03.160 --> 00:33:13.565
as you move beyond the initial focus of I need to be able to manage change data capture. I also need to be able to start adding functionality around maybe API sources.
303
00:33:14.345 --> 00:33:15.725
How how does the combination
304
00:33:16.585 --> 00:33:17.325
of operationalized
305
00:33:18.025 --> 00:33:23.100
system that your customers depend on with the need to be able to continue feature development
306
00:33:23.480 --> 00:33:25.020
and overall improvement,
307
00:33:25.880 --> 00:33:27.820
make the development process
308
00:33:28.135 --> 00:33:37.435
harder? What are the aspects that cause you to have to move slower than you did at the beginning? Just wondering how you think about the evolution of the system and being able to design and architect it in a way that doesn't cause you to
309
00:33:37.880 --> 00:33:44.140
in a way that doesn't cause you to stagnate and have to stop development to be able to make improvements?
310
00:33:44.920 --> 00:33:48.235
Yeah. That's a hard 1. Tough question, actually,
311
00:33:49.015 --> 00:33:54.635
because building is a scalable solution that can run-in the cloud, that can scale on thousands customers.
312
00:33:54.960 --> 00:33:58.340
It's not that hard if you build it, like, hard tailed,
313
00:33:59.040 --> 00:34:00.580
focused on couple connectors.
314
00:34:00.880 --> 00:34:03.780
But we actually focused with the growth of connector base,
315
00:34:04.185 --> 00:34:04.685
and,
316
00:34:05.625 --> 00:34:06.905
the the main thing that,
317
00:34:07.545 --> 00:34:11.565
actually give us this ability is 2 concepts. The first concept is pluggability.
318
00:34:12.570 --> 00:34:14.510
We define now a system in architecture,
319
00:34:15.210 --> 00:34:16.190
certain extensibility
320
00:34:16.570 --> 00:34:19.230
points that actually can be used as extension
321
00:34:19.785 --> 00:34:22.125
and build, as much as possible,
322
00:34:22.905 --> 00:34:24.125
inside of a core.
323
00:34:24.425 --> 00:34:33.809
Like, this core is working with, just an abstract concept. They don't know anything about 1 particular database, 1 particular feature, etcetera. This is
324
00:34:34.109 --> 00:34:35.970
stuff that is common for everyone.
325
00:34:36.305 --> 00:34:41.365
And then we have plugins that implement certain of aspects of connectors,
326
00:34:41.745 --> 00:34:44.349
and these, plugins in total
327
00:34:44.890 --> 00:34:50.829
compose a connector that connect to the database, to API source, to a target, to the storage, etcetera.
328
00:34:51.130 --> 00:34:53.069
And this pluggability is
329
00:34:53.555 --> 00:34:56.135
extremely important if you try to build,
330
00:34:57.234 --> 00:34:59.815
a scalable in terms of a development solution because
331
00:35:00.115 --> 00:35:02.135
scale of features is important.
332
00:35:02.700 --> 00:35:09.680
And another aspect that's actually important, build being just a pluggable system is not enough. You have to have a lot of
333
00:35:10.515 --> 00:35:13.175
infrastructure for unit testing and component tests,
334
00:35:13.475 --> 00:35:15.795
and we have extensive support of,
335
00:35:16.275 --> 00:35:20.250
tests, and we test everything. There's a lot of excessive stuff.
336
00:35:20.790 --> 00:35:21.030
And,
337
00:35:21.670 --> 00:35:22.810
initially, we,
338
00:35:23.510 --> 00:35:33.105
designed, like, a sort of a test framework for our connectors, and it give us ability to to run a test and build a new feature without trying our solution. So,
339
00:35:34.045 --> 00:35:39.030
enabling sort of extreme programming practice, which is like test driven development. We,
340
00:35:39.730 --> 00:35:41.990
in some cases, like, I would say in most cases,
341
00:35:42.290 --> 00:35:44.790
it may be easier to write a test rather than
342
00:35:45.445 --> 00:35:52.744
doing something different like debugging it locally or maybe on some development stance. Sometimes it's easier to just write a test, make it green,
343
00:35:53.330 --> 00:35:57.430
initially make it red, then make it green, and then deploy the feature. And you have
344
00:35:58.130 --> 00:35:58.950
such flexibility
345
00:35:59.250 --> 00:36:04.095
and easiness, so in which we invest invest a lot of, test infrastructure
346
00:36:04.875 --> 00:36:08.815
to improve developer experience because developer experience, if you talk about
347
00:36:09.530 --> 00:36:10.410
scaling in,
348
00:36:11.050 --> 00:36:12.110
development efforts,
349
00:36:12.410 --> 00:36:13.790
it's extremely important.
350
00:36:14.090 --> 00:36:17.505
And we invest a lot in the in development experience, to be honest.
351
00:36:17.984 --> 00:36:21.525
So 2 main pillars of this, a pluggable architecture,
352
00:36:21.825 --> 00:36:29.980
make this everything as as as pluggable as possible. And second point is a development development experience and tests test infrastructure.
353
00:36:30.440 --> 00:36:32.220
I think this is our
354
00:36:32.599 --> 00:36:42.195
approach. I would say our our current team is quite small for the amount of code that we have, and the only reason why we're still able to support this
355
00:36:42.540 --> 00:36:47.040
huge code base is, because a lot of x a lot of tests.
356
00:36:47.420 --> 00:36:51.520
Just tests give us this, assurance that everything is more or less fine.
357
00:36:51.905 --> 00:36:54.565
We have quite big code base, to be honest.
358
00:36:55.585 --> 00:37:05.910
And when I was looking at the data transfer product that you're building, I noticed that in addition to the change data capture functionality, you're also
359
00:37:06.290 --> 00:37:10.565
adding support for being able to use some of the air byte connectors capabilities
360
00:37:17.905 --> 00:37:22.440
for some capabilities for some of these source and destination connectors.
361
00:37:23.300 --> 00:37:24.040
I'm wondering
362
00:37:24.660 --> 00:37:29.105
what your overall thinking is about the current state of that ecosystem,
363
00:37:29.805 --> 00:37:38.200
what it is about the set of airbag connectors that makes it worth integrating with that, what your overall thinking is about the potential for
364
00:37:38.500 --> 00:37:44.195
future evolution and development in that space, and maybe some of the considerations or concerns
365
00:37:44.495 --> 00:37:51.715
that you have or that your customers should be thinking about as they decide which of those connectors to actually bring on board?
366
00:37:52.390 --> 00:38:03.075
I would say right now, the state of the data engineering in terms of open source is a lot better than, like, 5 years ago. Even a couple of years ago, it's it's rolling quickly. And
367
00:38:03.695 --> 00:38:06.835
I think 1 of the most important project in all of this,
368
00:38:07.775 --> 00:38:08.675
bringing in the
369
00:38:09.135 --> 00:38:09.635
engineering
370
00:38:10.570 --> 00:38:15.470
in a good quality is I think Ira Byte is a bigger biggest win in terms of open source community.
371
00:38:16.010 --> 00:38:18.270
The quality of Ira Byte evolving
372
00:38:18.570 --> 00:38:23.505
in the last year a lot it's become a lot more stable, a lot less failures,
373
00:38:23.885 --> 00:38:30.230
random failures of connectors, but yet it still, require a lot of engineering efforts to run this perfectly.
374
00:38:32.130 --> 00:38:33.110
Through my experience,
375
00:38:33.730 --> 00:38:38.875
running Arabyte connectors require you at least a couple days of investigating how it works
376
00:38:39.255 --> 00:38:39.654
and,
377
00:38:40.295 --> 00:38:43.674
reading the documentation, and then you can finally run it.
378
00:38:44.134 --> 00:38:52.590
This is, I think, is, unavoidable because some connectors is just hard to stop. Like, you need to run a tokens. You need to prepare right scopes, etcetera.
379
00:38:52.970 --> 00:38:53.710
But, overall,
380
00:38:54.010 --> 00:38:55.665
it's working, and it's good.
381
00:38:56.145 --> 00:38:57.365
The only 1
382
00:38:57.665 --> 00:38:58.165
visible
383
00:38:58.865 --> 00:39:00.885
flow from, RBITE perspective
384
00:39:01.505 --> 00:39:02.645
is extensive
385
00:39:03.105 --> 00:39:04.725
cost that come with RBITE.
386
00:39:05.059 --> 00:39:07.940
The design of Arbyte built in, this,
387
00:39:08.420 --> 00:39:09.400
extreme applicability
388
00:39:09.700 --> 00:39:10.200
idea.
389
00:39:12.020 --> 00:39:20.545
What we, we also do as a a inside of this, as a pluggable design, but we do some shortcuts for us to gain a little bit more performance.
390
00:39:20.924 --> 00:39:27.980
So IRBY itself internally wrap everything in a Docker container and communicate between the source and the target via pipe,
391
00:39:28.280 --> 00:39:29.020
or file.
392
00:39:29.345 --> 00:39:47.430
So at least you have at least 1 serialization and deserialization in between. And they use this pipe structure to communicate and decouple actually source and the target, which is good because source can be written in 1 language and wrapped in Docker container and target another language and wrapped in Docker container, which is extremely good for
393
00:39:47.765 --> 00:39:57.430
developer's call development scalability. So you can write a lot of connectors easily. You can write connectors in JavaScript, in Python, in Go, in Rust, in whatever you want.
394
00:39:58.150 --> 00:40:02.089
But this actually brings down as a point. Another pain point is that
395
00:40:02.390 --> 00:40:04.010
as a side effect of such flexibility,
396
00:40:04.470 --> 00:40:05.030
you have,
397
00:40:05.430 --> 00:40:05.930
performance,
398
00:40:06.630 --> 00:40:17.640
slowdown. You have this bottleneck, which is a file usually between source and the target, and this serialization, this serialization, especially if you have a Python connector, and most of connectors written in their bytes is Python based
399
00:40:18.020 --> 00:40:24.920
connectors. You have this bottleneck of serializing the data on the disk and then reading this data and deserializing back.
400
00:40:25.465 --> 00:40:30.605
We internally read everything in Go and Go extremely good for running performance application,
401
00:40:30.985 --> 00:40:33.485
and we don't serialize data in between. We
402
00:40:33.790 --> 00:40:43.505
use only in memory structures, and this give us huge performance boost, for for our native connectors. But writing this connector is harder because
403
00:40:43.964 --> 00:40:45.025
it's not just
404
00:40:45.325 --> 00:40:45.904
a random,
405
00:40:46.525 --> 00:40:50.144
Python code that you bring in your application and wrap in Docker container.
406
00:40:50.845 --> 00:40:51.325
This,
407
00:40:51.724 --> 00:40:52.625
limit our
408
00:40:52.990 --> 00:40:57.330
capabilities. I think in modern world, it's like a trade off right now,
409
00:40:57.870 --> 00:40:58.370
between
410
00:40:58.910 --> 00:40:59.410
scalability
411
00:40:59.950 --> 00:41:01.890
of, development and
412
00:41:02.385 --> 00:41:03.345
scalability of,
413
00:41:04.465 --> 00:41:12.820
application because Arbyte gives us extra cost, and you can't unavoid it. Another thing I would say is that right now, there is still,
414
00:41:13.360 --> 00:41:14.100
plenty of
415
00:41:14.480 --> 00:41:16.645
open questions in terms of implementation of
416
00:41:18.645 --> 00:41:21.465
change that they capture. Right now, the implementation of change that capture in Erbate is not so good.
417
00:41:21.925 --> 00:41:23.545
It still have some flaws
418
00:41:23.980 --> 00:41:25.359
and problems in,
419
00:41:25.980 --> 00:41:28.960
and it's actually still quite costly in terms of resources
420
00:41:29.260 --> 00:41:31.359
resource consumption and running the,
421
00:41:32.140 --> 00:41:32.640
classical,
422
00:41:33.285 --> 00:41:34.345
change data capture
423
00:41:34.805 --> 00:41:38.105
of a pretty cold database, which change, like, once in a while,
424
00:41:38.405 --> 00:41:39.785
is still extremely expensive.
425
00:41:40.165 --> 00:41:47.280
And, this is still a a big problem because sometimes and I would say most of the times, the open source community is mostly thinking
426
00:41:47.580 --> 00:41:57.715
about growing in a in a in a horizontal space, like improving amount of connectors. And I think it's a good focus in terms of Arrabight because they actually attract a lot of people
427
00:41:58.080 --> 00:42:01.780
by saying we have all connectors that you want, and it's actually kind of true.
428
00:42:02.320 --> 00:42:16.454
I I don't remember how exact amount of connectors they support right now. But 1 particular connector can be improved as well. And sometimes you need to invest a lot of times in these connectors. And some connectors should be extremely
429
00:42:17.420 --> 00:42:20.800
tuned fine tuned to the to the extreme because
430
00:42:21.180 --> 00:42:24.705
if you were reading data from API, you don't care much
431
00:42:25.185 --> 00:42:30.165
care a lot about the performance because API not provide, like, terabytes of data. But if you
432
00:42:30.545 --> 00:42:34.485
are reading the data from a s 3 bucket and try to move them in, like, for example, Cassandra,
433
00:42:35.170 --> 00:42:43.510
you have to think about performance because in the s 3 bucket, you usually have a terabytes or maybe petabytes of data. And moving this petabyte of data with
434
00:42:44.085 --> 00:42:49.865
code that's somewhere in between serialized and deserialized data twice with a Python code is extremely inefficient.
435
00:42:50.245 --> 00:42:52.825
And you think you still need to think about this,
436
00:42:53.480 --> 00:42:58.859
if you want to scale vertically on 1 particular pair of, of database. Yeah.
437
00:42:59.240 --> 00:43:03.165
But, overall, current state of the world is quite amazing.
438
00:43:03.705 --> 00:43:08.365
I do like a lot, RBITE as a concept, as a product, and the overall
439
00:43:09.065 --> 00:43:11.725
overall do, like, a big fan of RBITE, to be honest.
440
00:43:12.380 --> 00:43:18.079
With the work that you've done in the space of managing data transfer, trying to productionize
441
00:43:18.539 --> 00:43:19.920
and productize it,
442
00:43:20.575 --> 00:43:24.355
What are some of the lessons that you have learned in the process about,
443
00:43:24.975 --> 00:43:43.395
how best to design systems to be conducive to data transfer or how to architect analytical pipelines to maybe minimize the need to actually move data. I'm just curious. What are some of the learnings that you've had from this experience that have influenced the way that you think about how to do data engineering overall?
444
00:43:45.080 --> 00:43:53.820
I would say the main lesson I learned is, like, keep it simple. I have on my past experience a lot of over engineering data pipelines
445
00:43:54.445 --> 00:43:54.945
with
446
00:43:55.405 --> 00:43:58.465
a thousands, not even, like, maybe even tens of thousands,
447
00:43:59.085 --> 00:43:59.985
steps in
448
00:44:00.605 --> 00:44:05.640
a graph, like, out of flow graph with tens of 1, 000 steps in it just to,
449
00:44:06.180 --> 00:44:08.359
build a table. I like the
450
00:44:09.485 --> 00:44:11.885
directions that right now all data movements
451
00:44:12.285 --> 00:44:14.545
all data engineering is moving. It's
452
00:44:14.925 --> 00:44:16.705
leaning in towards the simplification
453
00:44:17.005 --> 00:44:17.665
of everything.
454
00:44:18.040 --> 00:44:23.180
I do like the shift that made, like, couple years ago from ETL to ELT,
455
00:44:23.720 --> 00:44:24.860
and I would say,
456
00:44:25.555 --> 00:44:26.835
main lessons that,
457
00:44:27.475 --> 00:44:31.335
should learn by data engineers from all of the sites, like,
458
00:44:32.035 --> 00:44:34.135
try to be as simple as possible,
459
00:44:34.680 --> 00:44:36.780
and making data transformation,
460
00:44:37.560 --> 00:44:38.060
is
461
00:44:38.839 --> 00:44:39.339
exceptionally
462
00:44:39.640 --> 00:44:40.140
hard,
463
00:44:40.440 --> 00:44:43.500
and try to avoid them as much as possible
464
00:44:44.105 --> 00:44:52.640
and try to, delay them as much as much as possible. But by by by this, I mean that don't try to build complex pipeline of transformation. Usually,
465
00:44:53.380 --> 00:44:53.880
it's
466
00:44:54.619 --> 00:44:55.920
very fragile constructions.
467
00:44:56.860 --> 00:45:02.484
Sometimes, it's necessary for sure. But if you can avoid this, it's better to be avoided. And,
468
00:45:03.265 --> 00:45:06.700
modern approach of transforming data on the target side, like,
469
00:45:07.260 --> 00:45:09.359
DBT or something like something similar,
470
00:45:09.900 --> 00:45:11.040
I think is a superior
471
00:45:11.420 --> 00:45:13.680
if you have enough computing resources.
472
00:45:14.045 --> 00:45:24.230
I do understand that modern approach with this, dbt stuff with the Snowflake and Databricks sometimes cost a lot, but not every database has a Snowflake.
473
00:45:24.850 --> 00:45:26.790
And, also, 1 another interesting
474
00:45:27.090 --> 00:45:27.590
topic,
475
00:45:28.130 --> 00:45:36.305
I would suggest to try as much as possible, build everything on top of open source stuff rather than building around the vendors
476
00:45:36.685 --> 00:45:38.065
and proprietary solutions
477
00:45:38.690 --> 00:45:39.910
Because vendors,
478
00:45:40.290 --> 00:45:46.455
even if it costs not a lot right now, it may cost a lot in the future when they understand that you're already on a hook
479
00:45:47.255 --> 00:45:48.795
and you can't avoid them.
480
00:45:49.575 --> 00:45:50.615
Vendor lock is,
481
00:45:51.175 --> 00:45:51.675
extremely,
482
00:45:52.295 --> 00:45:53.995
harmful in a data space,
483
00:45:54.360 --> 00:45:58.540
And I think right now, a lot of harm was done by
484
00:45:59.080 --> 00:46:07.644
the overuse of a Snowflake on on a lot of cases. It's not needed, to be honest. Snowflake is extremely capable solution, but it's
485
00:46:08.105 --> 00:46:10.125
shouldn't be a default 1 because
486
00:46:10.744 --> 00:46:14.569
you have you can do done a lot of stuff just in the Postgres.
487
00:46:14.950 --> 00:46:19.369
If you can fit your data in the in the Postgres, use the Postgres. Another interesting
488
00:46:19.765 --> 00:46:20.984
insight from my side.
489
00:46:21.444 --> 00:46:21.944
Because,
490
00:46:22.724 --> 00:46:26.325
I'm internally a big fan of also Postgres' screen, and,
491
00:46:26.805 --> 00:46:35.710
I think that anything below the 1 terabyte of data should live in Postgres and not leave it, like, just being just with Postgres. It's enough for you.
492
00:46:36.855 --> 00:46:38.775
My, marketing manager would,
493
00:46:39.575 --> 00:46:48.830
agree about this because we're trying to provide ClickHouse as a service. But in a lot of cases, ClickHouse is a perfect database, but not in any case every cases.
494
00:46:49.130 --> 00:46:49.630
And,
495
00:46:50.170 --> 00:46:59.705
another thing that I would suggest is, like, there is no silver silver bullet. Snowflake is not silver bullet. It's a golden bullet. It's cost a little bit more. But
496
00:47:00.085 --> 00:47:06.760
be ready to be flexible in your tools, choice of tools, and not, try to keep them simple.
497
00:47:07.300 --> 00:47:09.400
But, again, don't stick to them too much.
498
00:47:10.020 --> 00:47:13.755
The open source solution's usually easier to, abandon,
499
00:47:14.214 --> 00:47:20.795
and, having this ability to abandon your solution, if you understand that it's not work well for you, it's important.
500
00:47:21.170 --> 00:47:27.109
Just give you a little bit of freedom of of choice, like, l some some gap here.
501
00:47:27.490 --> 00:47:30.710
Try to build this, your your pipelines, your
502
00:47:31.135 --> 00:47:32.355
systems, your
503
00:47:33.215 --> 00:47:34.435
data ingestion
504
00:47:34.735 --> 00:47:35.235
workflow
505
00:47:36.335 --> 00:47:41.130
as as abundant as possible. So you can just, like, delete it and create once again
506
00:47:41.829 --> 00:47:42.570
on different,
507
00:47:42.950 --> 00:47:44.970
rails with a different technology stack.
508
00:47:45.270 --> 00:47:46.650
What else I can suggest?
509
00:47:47.444 --> 00:47:48.825
Don't overuse Python.
510
00:47:49.845 --> 00:47:50.345
And
511
00:47:50.645 --> 00:48:02.670
in your experience of working in this space, figuring out how to build these solutions, working with customers? What are some of the most interesting or innovative or unexpected ways that you've seen this challenge of data transfer addressed?
512
00:48:03.465 --> 00:48:06.685
Oh, I have a perfect example. A perfect example.
513
00:48:06.985 --> 00:48:08.445
So, I have a customers,
514
00:48:09.385 --> 00:48:12.900
which is quite interesting 1. They design against self driving cars.
515
00:48:13.599 --> 00:48:14.740
Self driving cars,
516
00:48:15.040 --> 00:48:16.339
is the
517
00:48:16.960 --> 00:48:18.260
training model, basically.
518
00:48:18.645 --> 00:48:23.785
So there is a car, a lot of fancy stuff on top of this with the leaders, with
519
00:48:24.085 --> 00:48:27.305
cameras, etcetera. And inside of it is just a big computer
520
00:48:27.760 --> 00:48:28.420
with robotic OS
521
00:48:28.880 --> 00:48:32.560
that collect all this information and somehow to analyze it. And,
522
00:48:33.040 --> 00:48:35.105
they have a challengeable challenge task.
523
00:48:35.905 --> 00:48:38.085
Each machine, each car, actually,
524
00:48:38.625 --> 00:48:40.085
like, machine car because,
525
00:48:40.465 --> 00:48:41.925
inside of it, it's just a computer.
526
00:48:42.225 --> 00:48:46.710
Each car goes around the city, collects this, telemetry information,
527
00:48:47.410 --> 00:48:49.510
and somehow they need to analyze it centrally
528
00:48:49.810 --> 00:48:51.030
on the big cluster.
529
00:48:51.545 --> 00:48:54.445
And they need to move this data from machine,
530
00:48:54.905 --> 00:49:00.740
on the ground from a car to the cloud somehow to analyze them later by machine learning guys or data engineers.
531
00:49:01.380 --> 00:49:04.119
And, how to do this? Because usually machine
532
00:49:04.820 --> 00:49:05.560
on the cities,
533
00:49:06.100 --> 00:49:12.875
has a very limited capabilities of Internet connectivity. You have, like, a 3 g, maybe 5 g in in a in a good weather,
534
00:49:13.414 --> 00:49:13.914
capability.
535
00:49:14.295 --> 00:49:16.420
Cell connectivity. Nothing else.
536
00:49:16.900 --> 00:49:17.380
So,
537
00:49:17.940 --> 00:49:27.745
they what they do, they split data into 2 pieces. Like, important 1, a pre aggregated 1 is collected on the on the on the car itself and send it via,
538
00:49:28.525 --> 00:49:30.785
mobile cell to the cloud somehow
539
00:49:31.245 --> 00:49:36.780
by simple ish typical. It was not not so complicated. So they have a challenge
540
00:49:37.240 --> 00:49:51.475
that they need to aggregate somehow information inside of a car before the sending this. So what they do, they install a click house on the machine, on the on the car itself, and start querying the click house to do aggregates of a data that collected by this right
541
00:49:51.900 --> 00:49:54.860
and sending, to the cloud. But then they need to,
542
00:49:55.340 --> 00:49:56.960
analyze this information at home
543
00:49:57.340 --> 00:50:00.125
with these rough information, not aggregated 1. But,
544
00:50:00.605 --> 00:50:03.984
how to achieve this? So they park machine in a in a garage,
545
00:50:04.525 --> 00:50:04.925
and,
546
00:50:05.565 --> 00:50:06.880
what they did, it's,
547
00:50:07.440 --> 00:50:11.460
they connect machine to the proper network, to the LAN network.
548
00:50:11.920 --> 00:50:12.240
And,
549
00:50:13.519 --> 00:50:23.975
once the machine become in the LAN network, they connect the click house the node of a click house that run inside of a machine to the cluster of a click house, and they start to replicate the data.
550
00:50:24.510 --> 00:50:25.470
And once machine,
551
00:50:25.870 --> 00:50:27.490
in the car in a in a garage,
552
00:50:28.190 --> 00:50:32.695
they replicate data. Once that is replicated, they go again for right.
553
00:50:33.175 --> 00:50:33.675
And,
554
00:50:34.055 --> 00:50:35.595
this was, like, the
555
00:50:36.055 --> 00:50:40.715
strangest cluster that I saw in my life because each machine is just a node in a cluster.
556
00:50:41.039 --> 00:50:53.545
And once the machine come back to garages, the node become a synchronized 1. And once machine in, in a in a in a city, it's just detached node. It's, like, go offline by some reason. And it's not just because,
557
00:50:53.925 --> 00:50:56.425
outage. It's because they exploring the world.
558
00:50:56.725 --> 00:50:56.965
And,
559
00:50:57.690 --> 00:50:58.750
it was very
560
00:50:59.050 --> 00:51:04.910
interesting way of, solving this problem because, initially, they tried to build it very complex, like, graph based
561
00:51:05.210 --> 00:51:06.110
Python scripts
562
00:51:06.455 --> 00:51:07.915
with the Pyflow in it,
563
00:51:08.695 --> 00:51:09.835
but it was so,
564
00:51:10.775 --> 00:51:11.915
clunky and
565
00:51:12.615 --> 00:51:17.619
not robust 1. They always failed. Like, machine come to the garage. The graph has failed
566
00:51:18.079 --> 00:51:25.505
because of some bug in the scripts, and this was hard to, implement and they don't want to work on this particular area. They
567
00:51:25.885 --> 00:51:27.025
wanna focus on,
568
00:51:27.805 --> 00:51:35.540
machine itself, on sensors, on machine learning, and all interesting stuff. And this techno technical stuff was not interesting to them.
569
00:51:36.560 --> 00:51:40.340
And they, like, come to this interesting approach of using a machine
570
00:51:40.800 --> 00:51:41.460
as a
571
00:51:41.825 --> 00:51:43.685
as a node in a cluster.
572
00:51:44.225 --> 00:51:47.525
The machine itself, like car, actual car. It's a wheel in it.
573
00:51:47.985 --> 00:51:51.365
Yeah. It's definitely a very cool and different approach to that problem.
574
00:51:52.210 --> 00:51:52.710
And
575
00:51:53.410 --> 00:52:06.775
you've definitely built a very interesting product, very capable product, but for people who are trying to figure out how do I want to get my data from point a to point b, what are the cases where the double cloud data transfer product is the wrong choice?
576
00:52:07.960 --> 00:52:11.580
That's actually interesting. I would say that if you have,
577
00:52:12.120 --> 00:52:14.940
data spurs in a lot of different systems,
578
00:52:15.575 --> 00:52:16.075
like,
579
00:52:16.855 --> 00:52:17.755
API stuff,
580
00:52:18.295 --> 00:52:21.515
I would suggest better to use a vanilla Ironbite 1,
581
00:52:22.135 --> 00:52:22.635
because
582
00:52:23.230 --> 00:52:24.369
we need to initially
583
00:52:24.670 --> 00:52:26.210
internally try to minimize
584
00:52:26.589 --> 00:52:27.410
amount of,
585
00:52:27.710 --> 00:52:33.955
support work that come to us and therefore not open every Aramite connectors that have. We just reselect,
586
00:52:35.215 --> 00:52:35.715
several
587
00:52:36.015 --> 00:52:37.795
most prominent and
588
00:52:38.175 --> 00:52:39.715
stable 1. And,
589
00:52:40.495 --> 00:52:43.170
so if you have, like, a huge
590
00:52:43.470 --> 00:52:44.930
different sources like
591
00:52:45.309 --> 00:52:46.450
Twilio API,
592
00:52:48.349 --> 00:52:53.345
some Zendesk API, or and and and so on and so on. We have dozens of such.
593
00:52:53.724 --> 00:52:55.904
I would suggest you to just use Erbyte,
594
00:52:56.605 --> 00:52:57.105
just
595
00:52:58.430 --> 00:53:00.770
because it's easier to understand
596
00:53:01.070 --> 00:53:01.390
this,
597
00:53:01.870 --> 00:53:02.370
without
598
00:53:02.830 --> 00:53:04.930
the middlemen. Because in in this case,
599
00:53:05.470 --> 00:53:06.610
these API sources,
600
00:53:06.975 --> 00:53:14.515
first of all, they will be hidden because we don't show the all API sources from everybody. And, second of all, even even exposed, we will be a middleman here.
601
00:53:14.940 --> 00:53:15.180
And,
602
00:53:15.980 --> 00:53:17.360
we're more focused on
603
00:53:17.980 --> 00:53:18.560
a classical
604
00:53:19.260 --> 00:53:21.360
movement from a more database
605
00:53:21.875 --> 00:53:23.494
style stuff rather than,
606
00:53:24.515 --> 00:53:27.335
API stuff. So if you have just an API sources,
607
00:53:27.795 --> 00:53:29.255
just use API,
608
00:53:29.954 --> 00:53:30.940
RBITE here.
609
00:53:31.500 --> 00:53:33.599
It would be better for you to evolve.
610
00:53:34.700 --> 00:53:35.200
And
611
00:53:35.900 --> 00:53:37.440
as you continue to
612
00:53:37.740 --> 00:53:49.985
build out the double cloud data transfer product, invest further in its evolution, add new capabilities? What are some of the things you have planned for the near to medium term or any particular projects or problem areas you're excited to explore?
613
00:53:50.480 --> 00:53:52.420
Well, we have a lot of in mind.
614
00:53:53.039 --> 00:53:55.859
1 of, approach that we actually investigating
615
00:53:56.160 --> 00:54:02.635
right now, it's not yet defined, but we're thinking about this quite heavily. It's open sourcing this stuff,
616
00:54:02.935 --> 00:54:05.035
open sourcing our data plane to,
617
00:54:05.975 --> 00:54:08.890
make this publicly available because code itself it's pretty
618
00:54:09.270 --> 00:54:13.369
agnostic. It can be reused by a community itself and actually get
619
00:54:13.750 --> 00:54:23.475
some love from a community and some bug fixes, to be honest, and some bug reports, to be honest, as well. Another thing that we try to invest a lot is improving in terms of,
620
00:54:23.955 --> 00:54:25.255
usability and debugability,
621
00:54:26.420 --> 00:54:28.120
allowing more information,
622
00:54:29.220 --> 00:54:34.954
about what happens inside, what can be goes wrong, how to can you can fix it. Because,
623
00:54:35.415 --> 00:54:39.035
usually, a lot of stuff can happens can go wrong here,
624
00:54:39.494 --> 00:54:41.095
and errors need to be fixed,
625
00:54:41.494 --> 00:54:50.410
mostly on a on a source or target site by actual customers. Like, password isn't correct, network connectivity is not not set up correctly, etcetera.
626
00:54:50.815 --> 00:54:55.214
And we try to improve our debug ability here. And another big,
627
00:54:55.615 --> 00:55:00.720
challenge of our improvements, what we want to do is adding the transformation
628
00:55:01.020 --> 00:55:01.520
capabilities.
629
00:55:02.300 --> 00:55:04.640
We right now introduce in a in
630
00:55:04.940 --> 00:55:06.160
a in a memory transformation,
631
00:55:07.260 --> 00:55:12.945
with SQL queries. We use a lot of, like, ClickHouse in internally as a as a internal
632
00:55:13.245 --> 00:55:13.745
engine.
633
00:55:14.365 --> 00:55:14.865
And,
634
00:55:16.125 --> 00:55:20.320
it's quite powerful tool, and we try to improve this area as well.
635
00:55:20.780 --> 00:55:21.520
And what
636
00:55:21.820 --> 00:55:23.340
another focus that we have,
637
00:55:23.740 --> 00:55:30.565
is adding more native connectors. That connectors that you read by ourself that not is inherited by Arabyte.
638
00:55:31.185 --> 00:55:31.685
That,
639
00:55:32.145 --> 00:55:32.645
designed
640
00:55:32.945 --> 00:55:34.245
specifically to transfer
641
00:55:34.545 --> 00:55:43.500
petabytes of data. And 1 of these connectors like, Iceberg, Delta Lake, and all the history based connectors, we already have pretty good compatibility
642
00:55:43.880 --> 00:55:45.935
with them, and we just want to improve this,
643
00:55:46.555 --> 00:55:47.055
because
644
00:55:47.435 --> 00:55:51.295
Ira byte itself is good with a small amount of data here.
645
00:55:51.790 --> 00:55:52.290
But,
646
00:55:52.750 --> 00:55:57.710
with a petabyte scale, it's not it's not gonna work. And, we wanted to invest,
647
00:55:58.030 --> 00:55:58.349
some,
648
00:55:59.070 --> 00:55:59.570
our
649
00:56:00.030 --> 00:56:00.530
expertise
650
00:56:01.255 --> 00:56:03.494
to provide ability to, move,
651
00:56:04.055 --> 00:56:05.595
huge extremely huge,
652
00:56:06.055 --> 00:56:18.215
delta table or isenberg tables from point a to point b. Because, usually, if you have this kind of tables, you have a lot of data. And it's not, like, gigabytes. It's not even terabytes. Sometimes it's even a petabyte.
653
00:56:18.675 --> 00:56:21.734
And, we want to move them as as fast as possible.
654
00:56:22.115 --> 00:56:24.535
Yeah. That's actually 3 our main focuses,
655
00:56:24.994 --> 00:56:26.750
right now. It's, like, capability,
656
00:56:27.370 --> 00:56:43.335
native connectors for extreme scale, and, possible open source. Yeah. Are there any other aspects of the overall challenges of data transfer and the work that you have been doing on the double cloud product that we didn't discuss yet that you'd like to cover before we close out the show?
657
00:56:43.829 --> 00:56:44.890
I would say,
658
00:56:45.670 --> 00:56:48.089
interesting aspect is running something
659
00:56:48.390 --> 00:56:49.210
in a cloud,
660
00:56:50.150 --> 00:56:52.410
especially once you strive start to,
661
00:56:52.905 --> 00:56:54.525
run this in isolation.
662
00:56:55.225 --> 00:56:56.125
I would say,
663
00:56:56.425 --> 00:56:56.925
sometimes,
664
00:56:57.305 --> 00:56:59.085
data engineers and data engineering
665
00:56:59.465 --> 00:56:59.965
people,
666
00:57:00.505 --> 00:57:01.005
doesn't
667
00:57:01.360 --> 00:57:04.100
give enough attention to the network isolation,
668
00:57:04.560 --> 00:57:06.420
and network isolation is hard.
669
00:57:07.120 --> 00:57:10.325
A lot of tools nowadays not provide you proper,
670
00:57:11.905 --> 00:57:16.405
network isolation. They run somewhere in the cloud, and you need to expose your data sources
671
00:57:16.705 --> 00:57:17.525
to the Internet.
672
00:57:18.200 --> 00:57:22.940
And I hear a lot of complaints from security people who actually,
673
00:57:24.359 --> 00:57:25.980
run this, stuff,
674
00:57:26.395 --> 00:57:30.015
that, exposing something to the Internet is a bad idea. And,
675
00:57:30.635 --> 00:57:33.855
once we implement the transfer in a in a AWS
676
00:57:34.920 --> 00:57:39.100
and GCP runtimes, we spent a lot of a heck of a lot of time
677
00:57:39.640 --> 00:57:40.040
to,
678
00:57:40.840 --> 00:57:44.275
provide the transfer and at a scale and a cheap cost
679
00:57:44.895 --> 00:57:48.895
and inside of network isolation, inside of your private nets. And,
680
00:57:49.295 --> 00:57:49.795
networking
681
00:57:50.415 --> 00:57:51.155
is hard
682
00:57:51.660 --> 00:57:53.760
even for the engineers that you,
683
00:57:54.380 --> 00:57:55.600
specialized in networking.
684
00:57:56.060 --> 00:58:13.980
But, data engineers still need to understand the network some some sort of. You still need to know that your database is not just just exposed somewhere. If you are able to access your database from your local laptop, it's not means that everyone can do so. Most probably, you have a VPN somewhere.
685
00:58:14.280 --> 00:58:14.680
And,
686
00:58:15.565 --> 00:58:18.865
connecting 2 points in the in your data
687
00:58:19.165 --> 00:58:20.704
land landmark is,
688
00:58:21.165 --> 00:58:25.650
can be hard if these 2 points in a different VPC or different cloud providers.
689
00:58:26.030 --> 00:58:29.410
Connecting cloud providers itself, it's quite a challengeable task.
690
00:58:29.869 --> 00:58:30.369
And,
691
00:58:31.475 --> 00:58:34.855
making this compliant is even more challenging task.
692
00:58:35.315 --> 00:58:36.295
Yeah. So,
693
00:58:36.995 --> 00:58:44.950
in this if if you are a data engineer nowadays, just invest a little of your time to understand what is the private nets, what is your networking
694
00:58:45.650 --> 00:58:47.190
stack of your current company,
695
00:58:47.675 --> 00:58:59.900
and, and all this stuff because networking is hard, and it still need to be managed. It still need to be handled by somehow some someone. And if it's handled not by data guy, he will just, like, forbid every access to you. And,
696
00:59:00.460 --> 00:59:04.000
it it it would be painful to enable this back again.
697
00:59:05.185 --> 00:59:20.079
Alright. Well, for anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest gap in the tooling or technology that's available for data management today.
698
00:59:22.744 --> 00:59:23.964
It's a hard question.
699
00:59:25.305 --> 00:59:26.924
I would say the biggest,
700
00:59:28.620 --> 00:59:30.320
missing biggest gap
701
00:59:30.700 --> 00:59:32.000
right now is,
702
00:59:32.540 --> 00:59:37.520
I I would say out of box cloud native solutions that you can host on yourself.
703
00:59:37.885 --> 00:59:43.585
The closest to 1 to to this area is Arbeit, but it's not, like, really cloud native.
704
00:59:43.885 --> 00:59:46.944
It's run everything on the same node on the same cluster,
705
00:59:47.390 --> 00:59:49.010
but running any anything
706
00:59:50.190 --> 00:59:52.530
something on a scale require you
707
00:59:52.830 --> 00:59:53.730
running this
708
00:59:54.405 --> 00:59:55.065
on infrastructures
709
00:59:55.365 --> 00:59:59.385
a little level, like, a bit lower level, and we don't have anything right right
710
01:00:00.005 --> 01:00:11.945
right now. Also, another thing that is a bit in a gray area right now is a data government tool. They evolved a lot in the past years, but data government is actually exceptionally hard
711
01:00:12.485 --> 01:00:12.985
because
712
01:00:13.445 --> 01:00:15.305
quality of data is important.
713
01:00:15.925 --> 01:00:24.910
And, if you just move data around, it's not enough for your business needs. You need to not just put this data on a target database. You need, actually
714
01:00:25.450 --> 01:00:26.430
put some effort,
715
01:00:26.735 --> 01:00:33.695
clean up it, make it more quality 1 for later consumption. And data quality tools, data governance tools is,
716
01:00:34.175 --> 01:00:41.609
extremely important. I would I I see some tractions in this area for the past couple years. There is a couple good projects,
717
01:00:43.109 --> 01:00:43.609
that's
718
01:00:45.035 --> 01:00:47.375
popped out in this area. I don't remember the names,
719
01:00:48.875 --> 01:00:49.375
but
720
01:00:49.995 --> 01:01:08.615
are they mostly the the the 1 of the biggest actually wins in this area that actually give a lot of flexibility is DBT tests. The DBT test is actually a good starting point for you to make a data governance. Writing tests on your data is actually important as well if you wanted to build a robust solution.
721
01:01:09.315 --> 01:01:10.135
Another interesting
722
01:01:10.595 --> 01:01:16.020
area that is evolving right now, it's not so problematic right now because we have good,
723
01:01:16.740 --> 01:01:18.840
solution on the market in open source world,
724
01:01:19.220 --> 01:01:22.345
but data cataloging calls are important because,
725
01:01:23.285 --> 01:01:26.825
seeing how does a data goes from point a to point b,
726
01:01:27.205 --> 01:01:30.905
it's extremely important on a complex system. If you have big system,
727
01:01:31.310 --> 01:01:36.690
it's really hard to track how this field was calculated in the first place. Where is the root cause?
728
01:01:36.990 --> 01:01:38.610
How what what was involved
729
01:01:38.990 --> 01:01:46.245
and what actually may broke? And if you broke something in between, what actually is affect how how big impact of this failure?
730
01:01:46.785 --> 01:01:47.285
And,
731
01:01:47.585 --> 01:01:48.507
there is 1 particular,
732
01:01:49.109 --> 01:01:55.760
particular, set of tools that help you. It's a data catalogs. I think right now most popular 1 is open metadata. So,
733
01:01:56.705 --> 01:01:58.965
this is this gap is more or less filled.
734
01:01:59.265 --> 01:02:08.090
But I think adoption of an open metadata and the data cataloging itself is not that high right now. So maybe this is would be evolved in a couple
735
01:02:08.550 --> 01:02:09.370
next years.
736
01:02:09.830 --> 01:02:10.890
The data catalog
737
01:02:11.270 --> 01:02:13.050
is not that hard to adopt,
738
01:02:13.484 --> 01:02:14.545
and it actually
739
01:02:15.005 --> 01:02:15.505
enables
740
01:02:16.765 --> 01:02:17.265
data
741
01:02:17.645 --> 01:02:24.210
inside of a company by a lot because they expose your data not just for the for data engineers but, actually for business people.
742
01:02:24.750 --> 01:02:29.330
Also, they can actually explore what actually you have for later analysis.
743
01:02:30.030 --> 01:02:31.170
So this is important.
744
01:02:31.515 --> 01:02:33.375
And I think adoption of a cataloging
745
01:02:33.835 --> 01:02:35.375
also is a is
746
01:02:35.675 --> 01:02:46.190
a is a a gap, not like tooling, but the adoption of the solution. Absolutely. Yeah. That's a space that I'm looking to invest in next myself, so I appreciate you calling that out.
747
01:02:46.570 --> 01:03:00.540
Well, thank you very much for taking the time today to join me and share the work that you are doing on the data transfer product at DoubleCloud and your overall experience working in the space and building solutions to address this very necessary problem.
748
01:03:01.079 --> 01:03:05.375
Appreciate all the time and energy that you've putting into that, and I hope you enjoy the rest of your day.
749
01:03:06.575 --> 01:03:07.615
Thank you. Thank you.
750
01:03:07.935 --> 01:03:09.795
It was a pleasure to meet you as well.
751
01:03:15.890 --> 01:03:26.185
Thank you for listening. Don't forget to check out our other shows, podcast dot in it, which covers the Python language, its community, and the innovative ways it is being used, and the Machine Learning podcast,
752
01:03:26.725 --> 01:03:29.390
which helps you go from idea to production with machine learning.
753
01:03:29.789 --> 01:03:31.170
Visit the site at dataengineeringpodcast.com.
754
01:03:32.670 --> 01:03:42.675
Subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email hosts at data engineering podcast.com
755
01:03:43.215 --> 01:03:44.035
with your story.
756
01:03:44.335 --> 01:03:49.360
And to help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers.