Addressing The Challenges Of Component Integration In Data Platform Architectures

WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 7/6/2024 2:39:19 PM
Duration: 1782.599
Channels: 1
1
00:00:11.184 --> 00:00:15.445
Hello, and welcome to the Data Engineering podcast, the show about modern data management.
2
00:00:16.064 --> 00:00:38.495
You shouldn't have to throw away the database to build with fast changing data. You should be able to keep the familiarity of SQL and the proven architecture of cloud warehouses, but swap the decades old batch computation model for an efficient incremental engine to get complex queries that are always up to date. With Materialise, you can. It's the only true SQL streaming database built from the ground up to meet the needs of modern data products.
3
00:00:39.260 --> 00:00:43.120
Whether it's real time dashboarding and analytics, personalization and segmentation,
4
00:00:43.420 --> 00:00:51.905
or automation and alerting, Materialise gives you the ability to work with fresh, correct, and scalable results, all in a familiar SQL interface. Go to dataengineeringpodcast.com/materialize
5
00:00:54.205 --> 00:00:55.905
today to get 2 weeks free.
6
00:00:56.690 --> 00:00:58.390
Introducing RudderStack Profiles.
7
00:00:59.010 --> 00:01:08.744
RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable enriched data to every downstream team.
8
00:01:09.365 --> 00:01:15.790
You specify the customer traits, then profiles runs the joints and computations for you to create complete customer profiles.
9
00:01:16.490 --> 00:01:20.510
Get all of the details and try the new product today at data engineering podcast.com/rudderstack.
10
00:01:23.265 --> 00:01:28.645
Your host is Tobias Macy, and today I'm going to be sharing an update on my own experience
11
00:01:28.945 --> 00:01:32.200
of the journey of building a data platform from scratch.
12
00:01:32.680 --> 00:01:36.460
And today I'm in particular going to be focusing on the challenges
13
00:01:36.840 --> 00:01:40.700
of integrating the disparate tools that are required to build
14
00:01:51.000 --> 00:01:53.500
able to define certain concerns
15
00:01:53.800 --> 00:01:54.780
of that platform.
16
00:01:55.880 --> 00:02:00.685
So if this is your first time listening to the show, I've been running this show for
17
00:02:00.985 --> 00:02:04.365
the better part of 6 years now, maybe close to 7.
18
00:02:04.825 --> 00:02:07.300
I have been working in technology
19
00:02:07.680 --> 00:02:14.900
for over a decade and I have been spending the past year and a half coming up on maybe 2 years
20
00:02:15.545 --> 00:02:21.645
building a data platform from scratch to get all the buzzwords out of the way using a
21
00:02:22.025 --> 00:02:24.365
cloud first data lake house architecture
22
00:02:25.030 --> 00:02:27.610
focusing on DBT for transformation,
23
00:02:28.469 --> 00:02:31.609
air bite for extract and load, Dagster for
24
00:02:31.965 --> 00:02:32.784
the full integration,
25
00:02:33.165 --> 00:02:38.465
and using Trino as the query engine for data on top of s 3. So
26
00:02:39.060 --> 00:02:46.360
recognizing that that puts me in the minority of most people who are building a data platform, particularly if they have small teams
27
00:02:47.005 --> 00:02:49.345
that has put me in the position of needing
28
00:02:50.605 --> 00:02:52.385
to figure out some of the
29
00:02:53.245 --> 00:02:53.745
interfaces
30
00:02:54.045 --> 00:02:55.025
for integration
31
00:02:55.989 --> 00:03:08.084
For a lot of people where they're just getting started with building out the data platform, they're probably going to be going with the managed platform route or picking from a selection of different vendors. The
32
00:03:08.784 --> 00:03:12.400
canonical 1 for the so called modern data stack is probably
33
00:03:12.720 --> 00:03:14.819
Fivetran, Snowflake, and DBT
34
00:03:15.519 --> 00:03:19.140
with maybe Looker as your business intelligence layer.
35
00:03:19.845 --> 00:03:24.185
I'm not gonna spend a lot of time in this episode digging into the motivations
36
00:03:24.565 --> 00:03:31.000
for why I selected those different components of the stack because I've covered it in other episodes which I will link to in the show notes.
37
00:03:31.700 --> 00:03:32.840
But it has definitely
38
00:03:33.140 --> 00:03:36.840
led to a certain amount of friction as I try to
39
00:03:37.444 --> 00:03:46.745
manage some of the different integrations out of the box. Although the story for that particular set of technologies has been steadily getting better as time goes by.
40
00:03:47.230 --> 00:03:51.570
Today, I really wanna talk about where I am in my journey currently
41
00:03:52.190 --> 00:03:56.050
of I have a core set of functioning capabilities.
42
00:03:56.445 --> 00:04:00.145
I can ingest data. I can transform it. I can query it.
43
00:04:00.525 --> 00:04:03.505
But now I'm getting to the point of needing to be able to
44
00:04:04.069 --> 00:04:07.770
onboard more people, provide a more seamless user experience,
45
00:04:08.790 --> 00:04:17.925
being able to manage some of the different means of data sharing or data delivery where maybe not everybody who's going to be accessing the data
46
00:04:18.470 --> 00:04:21.850
lives within the bounds of my team or my department.
47
00:04:23.350 --> 00:04:25.449
And there are definitely
48
00:04:25.750 --> 00:04:26.970
data sharing capabilities
49
00:04:27.350 --> 00:04:35.555
that are part of some of the different platforms, most notable being probably big query and snowflake with the ways that they manage data sharing,
50
00:04:36.500 --> 00:04:39.800
But there are a number of different ways of approaching that
51
00:04:40.340 --> 00:04:45.375
given that I am in the world of building a lake house architecture.
52
00:04:45.675 --> 00:04:51.535
I've got my data in s 3. I'm using the iceberg table format. So all of the data is already
53
00:04:52.270 --> 00:04:52.770
representable
54
00:04:53.070 --> 00:04:53.970
as a table.
55
00:04:54.910 --> 00:04:55.410
And
56
00:04:55.790 --> 00:05:18.570
so then the question is, okay, for somebody who just wants to be able to access the data, what's the best way to deliver it? Do I need to provide 1 off jobs to generate a CSV and send it to them via email? Do I need to give an s 3 bucket to be able to load things from? It all depends on what the level of sophistication is of the people who are going to be consuming the data. Maybe it's just a
57
00:05:19.005 --> 00:05:21.905
dashboard format and people just need to be able to look at the data.
58
00:05:22.925 --> 00:05:23.905
1 of the challenges
59
00:05:24.365 --> 00:05:25.505
of that handoff
60
00:05:25.940 --> 00:05:26.600
is also
61
00:05:26.900 --> 00:05:28.840
when you need to be able to
62
00:05:29.380 --> 00:05:33.160
be considerate of how that data is going to be used
63
00:05:33.575 --> 00:05:36.155
after you present it because maybe
64
00:05:36.535 --> 00:05:46.530
you want to be able to give a visual representation or give away for somebody to access the data, but you don't want them to then exfiltrate it into
65
00:05:46.910 --> 00:05:50.495
another system via means of a c s v export for instance.
66
00:05:51.135 --> 00:05:54.675
And that's where you start getting into questions of governance and
67
00:05:55.294 --> 00:05:58.514
who has access to what being able to audit data access.
68
00:06:00.014 --> 00:06:00.480
But
69
00:06:00.960 --> 00:06:06.900
harking back to 1 of the episodes I did a while ago on the idea of shadow IT in the data industry,
70
00:06:08.375 --> 00:06:11.195
the best way to prevent people from
71
00:06:12.215 --> 00:06:15.755
taking data out of the context in which you want it to be presented
72
00:06:16.120 --> 00:06:17.820
and bringing it into other tools
73
00:06:18.600 --> 00:06:19.420
is to
74
00:06:19.880 --> 00:06:21.660
reduce any friction
75
00:06:21.960 --> 00:06:23.660
or pain that they're experiencing
76
00:06:24.615 --> 00:06:28.635
accessing the data in the way that you have presented to them. Because
77
00:06:29.495 --> 00:06:29.995
if
78
00:06:30.295 --> 00:06:35.570
your options are the best option and the most accessible option for the people who are
79
00:06:36.190 --> 00:06:41.890
viewing that data, then they're not going to want to bringing it out of that system because you were giving them the best experience.
80
00:06:43.014 --> 00:06:47.595
And so that's where I am right now of figuring out what is that best experience for everybody,
81
00:06:48.055 --> 00:06:51.354
what are the requirements, how do I then manage that. And
82
00:06:52.210 --> 00:06:53.430
a lot of the
83
00:06:53.810 --> 00:06:54.310
complexity
84
00:06:55.169 --> 00:06:57.190
comes in with the elements of
85
00:06:58.530 --> 00:06:59.805
interoperability platform deliberately
86
00:07:06.349 --> 00:07:07.409
the overall platform.
87
00:07:08.030 --> 00:07:11.650
And I'm using the term platform deliberately because I am aiming for
88
00:07:12.110 --> 00:07:26.850
a holistic experience for end users versus just a number of point solutions where somebody can maybe plug something in and they do their thing and then somebody else can plug something in and do their own thing. I want to figure out what is the minimal set of interfaces
89
00:07:28.270 --> 00:07:31.970
that I need to build and support to be able to address the
90
00:07:32.285 --> 00:07:39.025
widest variety of needs while still being extensible for the case where somebody has a bespoke requirement
91
00:07:39.349 --> 00:07:42.729
that I need to be able to fulfill? How do I make sure that
92
00:07:43.669 --> 00:07:48.409
that doesn't add an undue amount of maintenance burden on myself and my team
93
00:07:48.755 --> 00:07:51.255
while still being able to deliver on that request.
94
00:07:52.675 --> 00:07:54.295
In general, the way that
95
00:07:54.835 --> 00:07:56.855
that degree of interoperability
96
00:07:57.555 --> 00:07:59.479
is managed is through adoption
97
00:08:00.340 --> 00:08:05.800
of open standards that everybody has agreed upon. So SQL is probably the
98
00:08:06.354 --> 00:08:15.574
longest lived 1 of those open standards. Although, to be fair, there are multiple different varieties of it, but at its core, SQL is understandable. So if you have a means of
99
00:08:15.880 --> 00:08:19.340
using SQL to query data, that is going to
100
00:08:20.040 --> 00:08:25.925
make it easy for a lot of different tools and people to be able to do their exploration and self
101
00:08:27.105 --> 00:08:28.485
serve. DBT has
102
00:08:28.865 --> 00:08:40.000
capitalized on that as a means of being able to build their products to be able to say, okay, the majority of structured data sources are going to be addressable by SQL. So we're going to build a tool
103
00:08:40.545 --> 00:08:42.565
that allows people to build
104
00:08:43.185 --> 00:08:44.485
a full engineering
105
00:08:45.105 --> 00:08:48.165
development and delivery flow for that SQL data,
106
00:08:48.509 --> 00:08:56.449
manage the transformations through our tool, and then add a lot of nicety around it in terms of the lineage and data documentation, etcetera.
107
00:08:57.255 --> 00:09:06.950
Because of the fact that they have invested so much in the ease of use of the tool as well as doing a lot of advocacy in the community
108
00:09:07.650 --> 00:09:08.710
to drive adoption
109
00:09:09.330 --> 00:09:11.750
that has led to a number of
110
00:09:12.305 --> 00:09:17.365
different other tools integrating with them. So they have become a de facto interface
111
00:09:17.825 --> 00:09:19.525
for managing transformations
112
00:09:19.985 --> 00:09:20.805
in a
113
00:09:21.180 --> 00:09:24.880
warehouse or warehouse like context. So there are tools such
114
00:09:25.340 --> 00:09:27.200
as preset that are
115
00:09:27.980 --> 00:09:29.040
building integrations
116
00:09:29.420 --> 00:09:29.920
into
117
00:09:30.785 --> 00:09:32.885
d b t as far as the
118
00:09:33.265 --> 00:09:35.605
data types and the metadata that they generate.
119
00:09:36.225 --> 00:09:41.570
There are entire products that are built on top of the metadata that d b t generates,
120
00:09:41.870 --> 00:09:43.330
light dash is 1 of those.
121
00:09:45.665 --> 00:09:48.005
So that has given DBT
122
00:09:48.625 --> 00:09:50.165
as a tool and as a company
123
00:09:50.705 --> 00:09:56.950
a lot of momentum, a lot of inertia, so it makes it more difficult for
124
00:09:57.330 --> 00:10:02.710
other people who maybe are targeting a similar use case, but want to add a
125
00:10:03.445 --> 00:10:07.545
step change to some of the capabilities around that tool
126
00:10:08.005 --> 00:10:14.430
to be able to actually break in and overtake that market. So the most notable 1 that I'm aware of so far is SQL Mesh
127
00:10:14.810 --> 00:10:19.870
who are actually adding a compatibility layer with DBT for being able to
128
00:10:20.515 --> 00:10:26.935
run SQL mesh on top of a DBT project where you can actually execute the transformations without having to do any code changes,
129
00:10:27.395 --> 00:10:29.175
which solves for the
130
00:10:29.690 --> 00:10:32.750
adoption step of I just wanna be able to try out this tool,
131
00:10:33.130 --> 00:10:37.230
but then there is still the challenge that they're going to have to overcome of
132
00:10:37.665 --> 00:10:41.524
building up that set of integrations with the broader ecosystem
133
00:10:42.065 --> 00:10:42.964
that DBT
134
00:10:43.265 --> 00:10:44.805
has benefited from.
135
00:10:45.330 --> 00:10:50.950
So that makes that road to adoption and the road to being a viable competitor
136
00:10:51.970 --> 00:10:52.790
a lot longer.
137
00:10:54.395 --> 00:11:00.015
Another tool that has benefited from this status of being a de facto standard and a reference implementation
138
00:11:01.050 --> 00:11:07.790
is airflow, where they have been around for a long enough time, and they have been adopted by a large enough user base that
139
00:11:08.525 --> 00:11:10.865
if you, as a new tool vendor,
140
00:11:11.325 --> 00:11:13.905
build your initial integration with airflow,
141
00:11:14.765 --> 00:11:19.490
then you have a large enough addressable market that it is worth the time of building that integration,
142
00:11:20.190 --> 00:11:22.210
but it's not necessarily worth it
143
00:11:22.670 --> 00:11:25.010
to build that same integration for
144
00:11:25.390 --> 00:11:26.850
DAXTER or Prefect
145
00:11:27.150 --> 00:11:27.650
or
146
00:11:28.014 --> 00:11:30.995
1 of the other orchestration systems that are out there.
147
00:11:31.375 --> 00:11:31.875
So
148
00:11:32.574 --> 00:11:39.220
as you are building your own platforms and doing this tool selection, it's worth considering, am I choosing something
149
00:11:39.680 --> 00:11:47.745
that is going to be able to benefit from the existing weight of the community integrations that are available for it.
150
00:11:48.285 --> 00:11:59.480
If not, then is the value that it is providing worth that additional complexity of me having to go out and build those integrations or work with other vendors to build those integrations.
151
00:12:03.685 --> 00:12:06.824
Developing event driven pipelines is going to be a lot easier.
152
00:12:07.125 --> 00:12:08.024
Meet functions.
153
00:12:08.510 --> 00:12:17.170
Memphis functions enable developers and data engineers to build an organizational toolbox of functions to process, transform, and enrich ingested events on the fly
154
00:12:17.495 --> 00:12:21.355
in a serverless manner using AWS Lambda syntax without boilerplate.
155
00:12:22.055 --> 00:12:24.670
It includes orchestration, error handling, and infrastructure
156
00:12:25.149 --> 00:12:30.450
almost any language, including Go, Python, JavaScript, dot net, Java, SQL, and more.
157
00:12:30.829 --> 00:12:32.985
Go to data engineering podcast dotcom/Memphis
158
00:12:33.765 --> 00:12:34.265
today
159
00:12:34.585 --> 00:12:35.085
to
160
00:12:35.405 --> 00:12:35.905
get
161
00:12:36.225 --> 00:12:36.725
started.
162
00:12:37.045 --> 00:12:37.785
And then
163
00:12:38.325 --> 00:12:41.705
another element of that integration question is
164
00:12:42.710 --> 00:12:50.490
the challenge of a lot of the different tools in the stack will want to be the single source of truth for certain aspects
165
00:12:50.925 --> 00:12:52.225
of your platform.
166
00:12:53.005 --> 00:12:57.824
1 of the ones that I've been dealing with recently is the question of access control and role definition
167
00:12:58.204 --> 00:13:02.110
and security and auditability of the data. So
168
00:13:03.050 --> 00:13:06.029
because I am building with a disaggregated
169
00:13:06.649 --> 00:13:11.465
database engine where the storage and the query access are separated,
170
00:13:11.925 --> 00:13:14.025
that gives 2 different locations where
171
00:13:14.485 --> 00:13:15.225
those definitions
172
00:13:16.029 --> 00:13:17.570
can be stored. So
173
00:13:18.430 --> 00:13:22.690
1 camp will say, well, the because of the fact that the storage layer
174
00:13:23.024 --> 00:13:23.765
is the lowest level
175
00:13:24.305 --> 00:13:32.645
and will have potentially multiple different points of access where maybe I'm using Trino to query it by SQL, but I may also
176
00:13:33.470 --> 00:13:36.930
just be accessing the data in s 3 directly using some Python
177
00:13:37.389 --> 00:13:43.785
tool or maybe I'm using something like Spark or Flink to do other processing approaches to it beyond just SQL.
178
00:13:44.404 --> 00:13:44.904
So
179
00:13:45.205 --> 00:13:53.120
all of that access information needs to live in the storage layer. So that's the approach that companies like Tabula are taking with the iceberg table format, where
180
00:13:54.060 --> 00:14:03.585
as long as you have the storage layer secured, then it doesn't matter what all the other layers on top of it might have to say about access control because it's going to be enforced at the table level.
181
00:14:03.965 --> 00:14:07.280
And then you go the next level up, engines like Trino,
182
00:14:07.820 --> 00:14:09.600
or the Galaxy platform
183
00:14:09.980 --> 00:14:10.800
from Starburst,
184
00:14:11.420 --> 00:14:14.560
say, we want to own the access control because
185
00:14:15.045 --> 00:14:22.585
we are going to have more visibility into the specifics of the queries, and we can have things like rollable filtering in the role definitions.
186
00:14:23.519 --> 00:14:25.860
So we should be the ones to own that
187
00:14:26.399 --> 00:14:28.259
role based access control information
188
00:14:28.639 --> 00:14:30.579
or attribute based access control.
189
00:14:31.360 --> 00:14:32.019
And then
190
00:14:32.755 --> 00:14:39.495
that's definitely another viable option, but then you go another layer up the stack into business intelligence,
191
00:14:40.380 --> 00:14:41.120
and maybe
192
00:14:41.660 --> 00:14:46.800
there's even more granularity available because you can say, oh, well, this person has access to
193
00:14:47.235 --> 00:14:51.815
this particular dataset with this bay low level filtering and they can view these
194
00:14:52.515 --> 00:14:56.980
visualizations on that data, but maybe they can't write their own queries against that data.
195
00:14:57.940 --> 00:15:00.600
So there's that challenge of, okay, well,
196
00:15:00.980 --> 00:15:04.360
do I have to define the roles in 3 different places?
197
00:15:04.805 --> 00:15:20.320
Do I have to have slightly different roles across all those 3 places? How do I align them to be able to say, well, this user in the business intelligence is the same as this user, and the query engine is the same as this user in the storage layer and make sure that they are getting a cohesive experience across those boundaries.
198
00:15:21.795 --> 00:15:22.295
So
199
00:15:22.675 --> 00:15:24.214
particularly when you have
200
00:15:24.915 --> 00:15:29.709
tool systems that maybe don't want to or that are maybe disincentivized
201
00:15:30.410 --> 00:15:30.910
to
202
00:15:31.290 --> 00:15:32.029
do that
203
00:15:32.330 --> 00:15:42.625
propagation of role information. So for instance, if I use tabular and I say I'm going to do define a role that will grant read access to this subset of tables,
204
00:15:43.485 --> 00:15:44.865
but no other visibility
205
00:15:45.245 --> 00:15:46.385
of the rest of the dataset,
206
00:15:47.610 --> 00:15:50.270
how then do I reflect that information to
207
00:15:50.650 --> 00:15:54.590
the query layer of who that user is to be able to enforce those
208
00:15:55.125 --> 00:15:58.745
permissions, and then all the way up to the business intelligence layer to say,
209
00:15:59.125 --> 00:16:03.384
from the storage layer, these are the permissions that you have in the UI. So
210
00:16:03.870 --> 00:16:06.930
that brings in the need to have some manner of
211
00:16:07.310 --> 00:16:11.730
single sign on and single source of identity for people across all of those
212
00:16:12.264 --> 00:16:12.764
boundaries.
213
00:16:13.384 --> 00:16:16.444
Beyond the question of permissions, there's another set of information
214
00:16:16.745 --> 00:16:17.644
that is
215
00:16:18.425 --> 00:16:25.300
disjoint and wants to be owned by different components within the stack. And that is the question of data flow,
216
00:16:25.759 --> 00:16:26.579
data processing,
217
00:16:27.235 --> 00:16:28.055
data lineage,
218
00:16:28.595 --> 00:16:36.615
where each tool maybe has a certain view of what the lineage graph looks like or what the processing looks like.
219
00:16:38.020 --> 00:16:43.560
But you don't necessarily have the complete end to end view of a piece of data from
220
00:16:43.940 --> 00:16:55.125
maybe where it lands in an application database all the way through to where it's being presented in a business intelligence dashboard or incorporated as a feature for machine learning model training workflow.
221
00:16:56.410 --> 00:17:01.870
So for instance, DBT has the table level lineage of the transformations that it's providing.
222
00:17:02.410 --> 00:17:04.270
Tools like airflow and Dagster
223
00:17:04.730 --> 00:17:09.585
have the view of the lineage of all of the tasks that they are responsible for executing,
224
00:17:10.365 --> 00:17:11.665
but they don't necessarily
225
00:17:12.285 --> 00:17:13.665
have information about
226
00:17:14.110 --> 00:17:15.010
out of band transformations
227
00:17:15.550 --> 00:17:24.445
that are happening by some analysts who are running ad hoc queries in the data warehouse, or they don't they maybe don't have visibility into
228
00:17:25.065 --> 00:17:32.670
the data as it's landing in a in an application database, and they only see the data once it lands in the warehouse or maybe they can see
229
00:17:33.050 --> 00:17:33.530
the,
230
00:17:33.850 --> 00:17:37.230
data integration step with an Airbyte or a Fivetran to say, okay,
231
00:17:37.605 --> 00:17:47.640
I know this table in the application database feeds into this table in the warehouse, and then these DBT flows happen, but maybe it loses sight of the data dashboards that are being generated.
232
00:17:48.900 --> 00:17:51.160
And so, again, this is the question of
233
00:17:51.780 --> 00:17:52.520
open protocols,
234
00:17:52.900 --> 00:17:53.400
interoperability.
235
00:17:53.860 --> 00:17:57.425
So tools like open lineage are designed to help address that
236
00:17:58.045 --> 00:18:04.305
where you fire events that will can be then constituted into a more cohesive lineage graph.
237
00:18:04.620 --> 00:18:08.240
And then you also have systems such as metadata platforms
238
00:18:08.779 --> 00:18:10.720
that are designed to be
239
00:18:11.500 --> 00:18:17.835
more holistic views of the entire ecosystem and incorporate things like data discovery, data governance,
240
00:18:18.615 --> 00:18:27.800
which gives you a single place to be able to view all that information, but then you're back to that question of integration of, okay, well, this can store and convey all of this information,
241
00:18:28.500 --> 00:18:34.434
but how do I get all this information into it? So each of those tools are going to have different means of
242
00:18:34.815 --> 00:18:37.315
being able to push or pull data,
243
00:18:37.730 --> 00:18:39.110
but you have to make sure
244
00:18:39.410 --> 00:18:40.630
as the platform
245
00:18:41.650 --> 00:18:42.950
designer and operator
246
00:18:43.650 --> 00:18:51.395
that those data flows are also happening. So it's an additional set of DAGs that you need to make sure are running. You need to make sure they're reliable.
247
00:18:52.335 --> 00:18:55.635
So it it's useful, but it's also an additional burden.
248
00:18:57.440 --> 00:19:01.220
So these are all things that I've been dealing with recently. And then
249
00:19:01.920 --> 00:19:02.820
in the metadata
250
00:19:03.615 --> 00:19:04.515
catalog situation,
251
00:19:05.055 --> 00:19:08.035
even if you do manage to feed all of your data into that,
252
00:19:08.655 --> 00:19:09.875
it is useful
253
00:19:10.175 --> 00:19:11.315
as a
254
00:19:11.820 --> 00:19:14.640
means of discovery or a means of being able to,
255
00:19:15.900 --> 00:19:17.520
keep tabs on what's happening,
256
00:19:17.980 --> 00:19:23.884
but then it also feeds back into, okay, well, if I want to use this as my single source of truth,
257
00:19:24.504 --> 00:19:30.284
how then do I propagate that truth back into other systems? And that's where you start to get into questions of things like active metadata,
258
00:19:30.860 --> 00:19:34.000
and then you have another set of integrations and another
259
00:19:34.300 --> 00:19:38.320
another direction of integrations where if I say, okay, I have my metadata catalog,
260
00:19:38.625 --> 00:19:50.140
this is my source of truth for role information and who can access what. Now I need to be able to push that back down into the storage layer and into the query engine and into the business intelligence dashboard,
261
00:19:50.920 --> 00:19:55.845
and I need to make sure that all of those integrations are reliable and that there are
262
00:19:56.385 --> 00:20:00.005
appropriate mappings between the different concepts throughout the different systems.
263
00:20:03.720 --> 00:20:04.940
Data lakes are notoriously
264
00:20:05.320 --> 00:20:05.820
complex.
265
00:20:06.280 --> 00:20:21.960
For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte scale SQL analytics fast at a fraction of the cost of traditional methods so that you can meet all of your data needs ranging from AI to data applications to complete analytics.
266
00:20:22.420 --> 00:20:32.575
Trusted by teams of all sizes, including Comcast and DoorDash, Starburst is a data lake analytics platform that delivers the adaptability and flexibility a lakehouse ecosystem promises.
267
00:20:33.330 --> 00:20:42.710
And Starburst does all of this on an open architecture with first class support for Apache Iceberg, Delta Lake, and Hoodie, so you always maintain ownership of your data.
268
00:20:43.365 --> 00:20:46.985
Want to see Starburst in action? Go to data engineering podcast.com/starburst
269
00:20:48.485 --> 00:20:49.385
and get $500
270
00:20:49.685 --> 00:20:55.050
in credits to try Starburst Galaxy today, the easiest and fastest way to get started using Trino.
271
00:20:57.910 --> 00:21:06.425
So this is definitely 1 of the benefits that fully vertically integrated platforms have of you don't have to fight with all of those different layers of integration.
272
00:21:07.605 --> 00:21:08.040
But
273
00:21:08.600 --> 00:21:17.020
the problem there is that you have to rely on the integrations that have been built and maybe you're constrained by what that vertically integrated platform offers.
274
00:21:17.455 --> 00:21:21.475
You don't necessarily have the means of being able to extend it into
275
00:21:21.855 --> 00:21:24.755
areas that the platform developers
276
00:21:25.710 --> 00:21:28.210
haven't had the time to implement or
277
00:21:28.590 --> 00:21:30.850
haven't been exposed to as a necessity.
278
00:21:32.750 --> 00:21:33.650
So it's
279
00:21:34.465 --> 00:21:35.684
the the continual
280
00:21:37.424 --> 00:21:38.485
story of
281
00:21:40.144 --> 00:21:46.580
software design, software development of how do we build these systems that are extensible and can be integrated,
282
00:21:46.960 --> 00:21:50.100
but also make sure that the product that you're building
283
00:21:50.615 --> 00:21:55.755
doesn't just get crushed under the weight of having to maintain all of these different
284
00:21:56.295 --> 00:21:57.115
point solutions.
285
00:21:58.380 --> 00:21:59.600
I think that we're definitely
286
00:22:00.140 --> 00:22:07.280
iterating towards these community standards. I think that tools like Open Lineage, I think that the work that the open metadata folks are doing with their
287
00:22:07.615 --> 00:22:10.195
schema first approach to metadata is interesting.
288
00:22:11.055 --> 00:22:28.595
I think that the work that DBT has done to become that de facto standard for this is how your transformations are represented, so other tools can be able to build on top of that are all valuable. So it's great to see the direction that that has that the community has taken on all of these fronts, but I do think that we are definitely still
289
00:22:29.695 --> 00:22:31.450
not to the point where
290
00:22:31.909 --> 00:22:33.450
we have a lot of the answers
291
00:22:33.830 --> 00:22:35.450
fully baked. I think that
292
00:22:36.070 --> 00:22:42.975
I think that everybody who is investing in this ecosystem, everybody who is building these tools and using these tools and giving feedback to the tool vendors
293
00:22:43.434 --> 00:22:45.775
is helping to bring us to a better place.
294
00:22:46.350 --> 00:22:46.750
But,
295
00:22:47.309 --> 00:22:48.930
as somebody who is trying to
296
00:22:50.110 --> 00:23:01.115
integrate so many different pieces and try to figure out what does that holistic platform look like, how do I build it in a way that I can maintain it with a small team but also be able to have the flexibility
297
00:23:01.655 --> 00:23:03.115
required to address
298
00:23:03.490 --> 00:23:04.150
a wide
299
00:23:04.530 --> 00:23:05.510
set of audiences.
300
00:23:06.450 --> 00:23:11.750
It's definitely still a challenge, but 1 that I've been enjoying having the opportunity
301
00:23:12.050 --> 00:23:14.284
to explore and invest
302
00:23:14.825 --> 00:23:20.125
in. So now as I have been iterating on these challenges and thinking through how best to build that
303
00:23:20.780 --> 00:23:24.080
holistic platform and something that is going to be
304
00:23:24.540 --> 00:23:27.665
enjoyable and usable by a number of different people, the next
305
00:23:28.145 --> 00:23:30.885
main architectural component that I've been
306
00:23:31.265 --> 00:23:57.679
starting to work towards is that metadata platform so that I can have that more cross cutting view, so that I can improve the data discovery story for people who aren't part of the engineering team, who just wanna see what data do you have, where is it, how do I access it. So that's where I'm going to be doing some of my next work of picking the metadata platform, getting it integrated, getting all of the data flows propagated to that tool so that we can see
307
00:23:58.139 --> 00:23:59.519
how everything is flowing,
308
00:24:00.059 --> 00:24:01.519
and then being able to
309
00:24:01.980 --> 00:24:02.480
start
310
00:24:03.075 --> 00:24:09.495
integrating single sign on as a means of identity management across the different layers and then being able to say, okay.
311
00:24:10.090 --> 00:24:16.030
You came through the metadata platform to do data discovery. Now you say, okay. Here's the dataset that I want to,
312
00:24:17.034 --> 00:24:21.695
explore, or here's the chart that I want to view, then being able to have
313
00:24:22.235 --> 00:24:31.050
a simple means of clicking a button and jumping them into the experience that they're requesting, for instance, or being able to say, okay. I need to query this database,
314
00:24:31.510 --> 00:24:37.424
and then giving them the prefilled set of credentials or the prefilled client connection to be able to run
315
00:24:38.045 --> 00:24:42.065
those queries with their tool of choice, whether that be something like a tableau or
316
00:24:42.525 --> 00:24:44.540
a SQL command line, etcetera.
317
00:24:45.480 --> 00:24:50.380
And as I have been doing this work, the most interesting or innovative or unexpected
318
00:24:50.840 --> 00:24:51.340
aspects
319
00:24:51.705 --> 00:24:53.485
have definitely been these
320
00:24:55.065 --> 00:25:05.380
integration boundaries of saying, okay. This is the tool that I have. This is the other tool that I have. I would like to be able use them together in this manner. Now, how do I go about and do that? So
321
00:25:05.920 --> 00:25:08.420
I am definitely happy that the,
322
00:25:09.605 --> 00:25:13.385
Python language has become 1 of the most widely
323
00:25:13.925 --> 00:25:14.425
adopted
324
00:25:15.525 --> 00:25:16.025
ecosystems
325
00:25:16.650 --> 00:25:31.935
for data engineering because it does simplify some of that work where you can pretty easily assume that there's going to be a Python library that does at least 80% of what you're trying to do. So in, for instance, the open metadata platform, they have a Python library for
326
00:25:32.315 --> 00:25:40.290
doing that metadata ingestion. So even if they don't have an out of the box solution for ingesting metadata from the tool that you're using,
327
00:25:40.670 --> 00:25:45.090
they do have a Python client that you can provide metadata to them
328
00:25:45.435 --> 00:25:46.815
in order to be able
329
00:25:47.195 --> 00:25:52.255
to propagate information from a system that they haven't already addressed. So I do think that that is
330
00:25:53.300 --> 00:25:55.880
a another good trend in the ecosystem of
331
00:25:56.580 --> 00:25:57.080
providing
332
00:25:57.620 --> 00:26:01.480
a good set of software clients to be able to integrate with
333
00:26:01.975 --> 00:26:02.635
their tool
334
00:26:02.935 --> 00:26:08.955
even if it's from a system that they themselves didn't already plan to integrate with. So
335
00:26:09.480 --> 00:26:14.700
while it does add a bit of extra burden to the people who are trying to use those systems, there is at least a
336
00:26:15.000 --> 00:26:22.985
path to success for it. I'd also say that that's probably the most challenging lesson that I've learned as well of figuring out
337
00:26:23.285 --> 00:26:28.320
what are the points of integration that are worth investing in and what are the ones that
338
00:26:29.580 --> 00:26:38.065
I don't have to invest in right now, and maybe I can wait to see if the ecosystem around that tool grows up. So SQL mesh is a tool that I've been keeping an eye on.
339
00:26:38.684 --> 00:26:48.460
When I first came across it, it didn't have support for Trino, so that was an obvious no for being able to use it. Now they have it, but they don't yet have an integration with DAXTER.
340
00:26:49.240 --> 00:26:58.145
And then also, as I was pointing to earlier, the existing weight of integration with DBT gives me a bit of pause of okay. Even if I do get it working with Trino and Dagster,
341
00:26:58.765 --> 00:26:59.905
what about all the
342
00:27:00.365 --> 00:27:01.665
other prebuilt integrations
343
00:27:02.045 --> 00:27:03.185
that I am
344
00:27:03.520 --> 00:27:11.695
not going to be able to take advantage of because I have decided to use a newer tool that hasn't been as widely adopted and integrated. So
345
00:27:12.095 --> 00:27:14.355
definitely a challenging aspect of that as well.
346
00:27:14.735 --> 00:27:24.020
If you have found my musings useful or informative or if they have inspired something that you would like to discuss further, I'm definitely always happy
347
00:27:24.640 --> 00:27:25.620
to take suggestions
348
00:27:25.920 --> 00:27:28.180
or bring people on the show or,
349
00:27:28.985 --> 00:27:31.325
take some feedback. So for anybody who wants
350
00:27:31.705 --> 00:27:36.285
to get in touch with me, I'll add my contact information to the show notes, but the best way is
351
00:27:36.680 --> 00:27:38.220
on data engineering podcast.com
352
00:27:38.760 --> 00:27:41.020
that has links to how you can find me.
353
00:27:41.560 --> 00:27:42.060
And
354
00:27:42.520 --> 00:27:52.195
for the final question of what I see as being the biggest gap in the tooling or technology for data management today, I definitely think it's that single source of truth for
355
00:27:52.900 --> 00:27:58.920
identity and access across the data stack of being able to figure out how do I manage
356
00:27:59.540 --> 00:28:04.825
permissions and roles across a variety of tools without having to
357
00:28:05.365 --> 00:28:05.865
build
358
00:28:06.405 --> 00:28:07.305
different integrations
359
00:28:07.605 --> 00:28:08.745
every single time.
360
00:28:10.100 --> 00:28:17.720
So definitely look forward to seeing more investment in that. Maybe even using something like open policy agent from the cloud native ecosystem.
361
00:28:18.020 --> 00:28:18.520
So
362
00:28:19.005 --> 00:28:19.505
definitely,
363
00:28:20.045 --> 00:28:31.190
definitely happy to continue investigating that as well. So thank you for taking the time to listen to me. I hope it has been, if not informative, at least entertaining. I'm very thankful
364
00:28:31.890 --> 00:28:34.365
for being able to run this show
365
00:28:34.665 --> 00:28:38.605
and bring all of these ideas to everybody who listens. So
366
00:28:38.905 --> 00:28:44.100
in honor of this being the Thanksgiving week, just wanted to share that gratitude
367
00:28:44.480 --> 00:28:54.965
to everybody who takes the time out of their life to pay attention to myself and the people I bring on the show. So thank you again, and I hope you all have a good rest of your day.
368
00:29:01.450 --> 00:29:11.735
Thank you for listening. Don't forget to check out our other shows, podcast dot in it, which covers the Python language, its community, and the innovative ways it is being used, and the Machine Learning podcast,
369
00:29:12.195 --> 00:29:15.015
which helps you go from idea to production with machine learning.
370
00:29:15.315 --> 00:29:16.615
Visit the site at dataengineeringpodcast.com
371
00:29:18.400 --> 00:29:25.415
Subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a product from the show, then tell us about it.
372
00:29:25.895 --> 00:29:27.355
Email hosts at dataengineeringpodcast.com
373
00:29:28.695 --> 00:29:29.355
with your
374
00:29:29.735 --> 00:29:35.059
story. And to help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers.