Exploring The Insights And Impact Of Dan Delorey's Distinguished Career In Data

WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 7/6/2024 2:33:19 PM
Duration: 3651.158
Channels: 1
1
00:00:13.955 --> 00:00:18.055
Hello, and welcome to the Data Engineering Podcast, the show about modern data management.
2
00:00:18.710 --> 00:00:29.085
Have you ever woken up to a crisis because a number on a dashboard is broken and no 1 knows why? Or sent out frustrating Slack messages trying to find the right dataset? Or tried to understand what a column name means?
3
00:00:29.865 --> 00:00:34.685
Our friends at Outland started out as a data team themselves and faced all this collaboration chaos.
4
00:00:35.110 --> 00:00:37.850
They started building Outland as an internal tool for themselves.
5
00:00:38.870 --> 00:00:44.795
Outland is a collaborative workspace for data driven teams, like GitHub for engineering or Figma for design teams.
6
00:00:45.675 --> 00:00:55.220
By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets and code, Atlan enables teams to create single source of truth for all of their data assets
7
00:00:55.680 --> 00:01:01.780
and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker, and more.
8
00:01:02.535 --> 00:01:03.434
Go to dataengineeringpodcast.com/outland
9
00:01:05.814 --> 00:01:09.994
today. That's a t l a n, and sign up for a free trial.
10
00:01:10.600 --> 00:01:13.820
If you're a data engineering podcast listener, you get credits worth
11
00:01:14.200 --> 00:01:15.659
$3, 000 on an annual subscription.
12
00:01:17.615 --> 00:01:24.755
When you're ready to build your next pipeline and want to test out the projects you hear about on the show, you'll need somewhere to deploy it. So check out our friends over at Linode.
13
00:01:25.650 --> 00:01:34.229
With our managed Kubernetes platform, it's now even easier to deploy and scale your workflows or try out the latest Helm charts from tools like Pulsar, Pacaderm, and Dagster.
14
00:01:35.065 --> 00:01:42.285
With simple pricing, fast networking, object storage, and worldwide data centers, you've got everything you need to run a bulletproof data platform.
15
00:01:43.240 --> 00:01:44.859
Go to data engineering podcast.com/linode
16
00:01:46.359 --> 00:02:10.110
today. That's l I n o d e, and get a $100 credit to try out a Kubernetes cluster of your own. And don't forget to thank them for their continued support of this show. Your host is Tobias Macy. And today, I'm interviewing Dan Delore about his journey through the data ecosystem as the current head of data at SoFi, prior engineering leader with the BigQuery team, and an early engineer on the Dremel project. So, Dan, can you start by introducing yourself?
17
00:02:10.565 --> 00:02:11.465
Yeah. Hi, Tobias.
18
00:02:11.845 --> 00:02:15.465
I am the vice president of data at SoFi.
19
00:02:15.924 --> 00:02:20.220
Currently, I've been here a little over a year working on building the data platform
20
00:02:20.520 --> 00:02:21.260
for SoFi.
21
00:02:21.959 --> 00:02:24.220
Prior to that, my all my
22
00:02:24.600 --> 00:02:29.564
real professional experience was at Google. I worked at Google for 14 years.
23
00:02:30.025 --> 00:02:33.165
Most of the time on Dremel and then later BigQuery
24
00:02:33.465 --> 00:02:34.844
when the teams were combined.
25
00:02:35.400 --> 00:02:38.780
So your audience is probably familiar with the Dremel paper.
26
00:02:39.080 --> 00:02:40.700
BigQuery is the externalization
27
00:02:41.000 --> 00:02:43.020
of Dremel into Google's cloud.
28
00:02:43.795 --> 00:02:48.295
Prior to that, I was doing my graduate work at Brigham Young University.
29
00:02:48.915 --> 00:02:57.860
And do you remember how you first got started working in the area of data? It was during my PhD program. Of course, back then in the early 2000, we called it data mining.
30
00:02:58.160 --> 00:03:01.845
And so I was studying empirical software engineering,
31
00:03:02.465 --> 00:03:03.285
my emphasis,
32
00:03:03.905 --> 00:03:06.165
and was in the area of
33
00:03:06.625 --> 00:03:08.165
mining software repositories.
34
00:03:08.940 --> 00:03:15.120
So what we were doing was scraping data from all of the artifacts of open source software that we could find,
35
00:03:15.739 --> 00:03:17.795
indexing it, and then trying to do interest
36
00:03:18.275 --> 00:03:19.655
analysis of that.
37
00:03:20.035 --> 00:03:23.255
When I went to Google, I worked on ads optimization,
38
00:03:23.875 --> 00:03:28.560
which was, again, sort of a use of big data problem. We were trying to build
39
00:03:28.860 --> 00:03:30.159
keywords suggestion
40
00:03:31.099 --> 00:03:35.760
so that advertisers could get help in building out their advertising campaigns.
41
00:03:36.525 --> 00:03:42.065
After 2 years on that, I had the opportunity to join the Dremel team as an engineer.
42
00:03:42.685 --> 00:04:07.140
From there, it was all big data for me. Can you start by sharing a bit about what your current relationship is to the overall data ecosystem and maybe the CliffsNotes version of how you ended up there, which you gave us a little bit, but maybe a little bit more sort of detail about the different juncture points along the way. Chris Krasner (3six 0 six): Yeah. At this point, I see myself as a consumer of the tools. That's my relationship to the ecosystem. I love all of the
43
00:04:07.760 --> 00:04:10.340
explosion of tools in the modern data stack.
44
00:04:10.775 --> 00:04:14.155
I think that we've come a long way in just the last decade.
45
00:04:14.615 --> 00:04:23.480
And so it's really exciting for me to be out here getting to use the tools and see how these things fit together. I think during my time on BigQuery,
46
00:04:24.100 --> 00:04:26.200
I had a very deep but
47
00:04:26.574 --> 00:04:28.675
somewhat narrow view of the world
48
00:04:29.134 --> 00:04:29.794
in terms
49
00:04:30.335 --> 00:04:31.875
of seeing query execution
50
00:04:32.255 --> 00:04:33.395
and data storage
51
00:04:34.150 --> 00:04:36.889
as the primary drivers of the problem.
52
00:04:37.270 --> 00:04:40.330
And really, now that I'm on the other side, I see that
53
00:04:40.710 --> 00:04:44.815
it's not really my primary concern day to day, the
54
00:04:45.435 --> 00:04:45.935
exact
55
00:04:46.635 --> 00:04:47.695
optimal performance
56
00:04:48.075 --> 00:04:49.375
of a given query.
57
00:04:49.755 --> 00:04:54.360
The bigger problems are much more finding data, making sure data is reliable,
58
00:04:54.820 --> 00:04:57.320
monitoring SLAs about data delivery,
59
00:04:57.940 --> 00:04:59.880
answering business user questions.
60
00:05:00.414 --> 00:05:08.280
And so I've seen some other pieces of the data stack that I think there's some cool innovation happening, but we still have a lot of work to do.
61
00:05:08.680 --> 00:05:15.980
And now jumping back to sort of earlier in your career, as you said, you were 1 of the early engineers on the Dremel project, and that was,
62
00:05:16.395 --> 00:05:17.295
to my understanding,
63
00:05:17.675 --> 00:05:19.215
Google's next generation
64
00:05:19.755 --> 00:05:21.055
of data processing
65
00:05:21.515 --> 00:05:26.095
after their work on MapReduce and, you know, as Hadoop was starting to become
66
00:05:26.620 --> 00:05:40.014
widespread and mainstream in the ecosystem outside of Google, Google had already moved on from that paradigm and started working on Dremel, which has come to be more of the sort of agreed upon better approach. And I'm wondering if you can just share some of your
67
00:05:40.315 --> 00:05:49.940
perspective and context on that stage of the data ecosystem and maybe some of your experience of working on Dremel and the concepts that it was
68
00:05:50.395 --> 00:05:56.735
encapsulating as compared to the MapReduce paradigm that was starting to grow and be fostered outside of Google.
69
00:05:57.169 --> 00:06:00.949
Yeah. Absolutely. And, yes, I think you've framed that exactly right.
70
00:06:01.410 --> 00:06:06.790
The original engineer, Andre Guverev, who started the Dremel project inside Google,
71
00:06:07.335 --> 00:06:10.155
He was trying to solve the problem of
72
00:06:10.695 --> 00:06:11.595
boiler plate
73
00:06:11.975 --> 00:06:13.915
code, long startup times,
74
00:06:14.535 --> 00:06:16.635
difficulty in chaining steps together,
75
00:06:17.320 --> 00:06:21.100
difficulty in writing the programs, all the things that we know about MapReduce,
76
00:06:21.560 --> 00:06:22.860
the initial ecosystem.
77
00:06:23.400 --> 00:06:32.445
At Google, there was a language called sawsall, which some folks may have heard of. You could think of it as like parallelized Python that could compile down to map reduces.
78
00:06:33.140 --> 00:06:39.640
And so he was trying to solve the problem of how could I just write a SQL query and how could I have it run really fast.
79
00:06:40.305 --> 00:06:47.125
It was fairly rewarding. In 2020, we got to write the VLDB test of time award paper
80
00:06:47.580 --> 00:06:51.520
for Dremel. The original Dremel paper was published in 2010. And so in 2020,
81
00:06:51.900 --> 00:06:53.680
it it won the test of time award.
82
00:06:54.220 --> 00:06:59.195
And when we went back and looked at that and tried to break it down, broke the contributions
83
00:06:59.575 --> 00:07:03.835
of that Dremel made to the industry down into sort of 5 categories.
84
00:07:04.375 --> 00:07:13.980
1st was bringing sequel back. People who are operating today or or have started since the kinda Dremel brought it back may not realize that in the
85
00:07:14.375 --> 00:07:16.395
late 2000, early 2010s,
86
00:07:17.255 --> 00:07:28.280
it was believed that SQL was not a proper language for big data and it wasn't gonna work. And I think now we can see with Snowflake, with Redshift, with Athena, with BigQuery,
87
00:07:28.820 --> 00:07:33.715
that everybody's all in on SQL as the right language even for very large datasets.
88
00:07:34.415 --> 00:07:38.275
The second thing for Dremel was disaggregation of compute and storage.
89
00:07:38.810 --> 00:07:48.190
The idea that I can scale 1 without scaling the other. And I would put in there along with that the horizontal scaling. The idea that I don't need ever larger computers
90
00:07:48.794 --> 00:07:51.935
to be able to handle more data, but I can have just a fleet
91
00:07:52.315 --> 00:07:55.775
of computers. A lot of that was following for the MapReduce work, obviously.
92
00:07:56.530 --> 00:08:07.034
The idea of in place data analysis that I would query data wherever it happened to reside, for us, that primarily meant in Google's distributed file system, initially GFS
93
00:08:07.574 --> 00:08:08.794
and then later Colossus.
94
00:08:09.974 --> 00:08:15.289
And the idea of serverless that the user running the query shouldn't have to worry
95
00:08:15.669 --> 00:08:23.565
about whether the servers were running or starting them up. They should just be there and be able to run the query. And then of course, columnar storage.
96
00:08:23.944 --> 00:08:32.430
That was 1 of the big innovations of the Dremel project. The idea that you could do columnar storage but still allow nested repeated data.
97
00:08:32.730 --> 00:08:37.904
For us, inside Google, that meant protocol buffers for much of the world. Now that means JSON,
98
00:08:38.285 --> 00:08:40.464
but it's the same idea. The idea that
99
00:08:40.845 --> 00:08:46.540
I could still give you an efficient query over nest bit repeated data without you having to
100
00:08:46.840 --> 00:08:47.900
normalize everything.
101
00:08:48.520 --> 00:09:08.020
The other key aspect of Dremel was that it was still a divergence from the sort of then traditional data warehouse paradigm of everything being highly structured, very rigid, but, you know, easy to be able to answer a fixed set of questions and instead being able to support unstructured datasets, querying data where it lives,
102
00:09:08.425 --> 00:09:15.485
to some degree, breaking down the concept of data gravity as a blocking factor in what you can do with the information.
103
00:09:16.070 --> 00:09:18.330
And I'm wondering if you can just talk to
104
00:09:18.790 --> 00:09:24.644
what you see as being the broader impact on the ecosystem of allowing for that sort of data federation
105
00:09:25.024 --> 00:09:28.565
working with semi structured and somewhat unstructured datasets
106
00:09:29.264 --> 00:09:35.220
and being able to then, you know, allow for this new paradigm of extract load transform that was
107
00:09:35.680 --> 00:09:37.460
prior to that point largely
108
00:09:37.920 --> 00:09:38.420
intractable.
109
00:09:39.120 --> 00:09:44.265
Yes. I think that's true. I think there were a couple of things there. 1 thing that we always focused on
110
00:09:44.645 --> 00:09:47.350
with Dremel was making the queries fast.
111
00:09:47.829 --> 00:09:50.250
It was supposed to be an interactive system.
112
00:09:50.709 --> 00:09:53.130
The majority of queries we ran,
113
00:09:53.750 --> 00:09:57.050
ran under a minute, many under 10 seconds.
114
00:09:57.655 --> 00:10:06.235
And it really changed the paradigm from a MapReduce where I would code up my couple classes, my mapper and reducer, and then I would
115
00:10:06.740 --> 00:10:26.265
kick off inside Google board job to start up however many instances of my objects I needed. And then I'd wait for my analysis to run and it might take a couple hours. And so I would go play a game of pool or something and come back later and see the answer. Once we got to the Dremel service, I should say, the initial versions of Dremel, the first couple years of Dremel,
116
00:10:27.110 --> 00:10:57.014
you did have to prepare your data ahead of time and you had to load it on the local disk on the machines and you had to start up the servers. And so that, it was not great for adoption at that point. But at the point we moved to the Dremel service where we had a standing tree of servers tree because we did aggregation trees at that time. You'll see in more recent papers that we've changed the architecture there. But the idea that you didn't have to do anything, and as long as your data was sitting in the remote file system,
117
00:10:57.315 --> 00:11:00.615
you could issue a query and that query would execute immediately
118
00:11:01.140 --> 00:11:08.040
and get you the results back so quickly that you didn't even have time to write the next query before the answer came back.
119
00:11:08.625 --> 00:11:10.964
It became really powerful. What I found is
120
00:11:11.584 --> 00:11:15.285
as people get access to tools and more data,
121
00:11:15.930 --> 00:11:19.870
it really only ever leads to more questions that they want to ask.
122
00:11:20.250 --> 00:11:26.204
And so there's always this inflection point. We saw it all the time when we were selling BigQuery into enterprises
123
00:11:26.665 --> 00:11:31.084
that they're just this inflection point in data consumption in these large organizations
124
00:11:32.264 --> 00:11:36.390
when the users get the ability to ask questions and get answers faster.
125
00:11:37.090 --> 00:11:40.390
So I think that's 1 of the big ways that it changed the dynamics.
126
00:11:41.170 --> 00:12:10.029
I'm not as up to speed with the architecture of Dremel as I am with systems such as Presto, which are, I guess, the spiritual successors at least of what you were building there. But I know that in order to be able to query across these different datasets, you know, whether they're living in s 3 or Hadoop file system or the Google file store. You need to have the metastore to understand, you know, what are the files on disk, what is the schema of those files so that I can be able to
127
00:12:10.410 --> 00:12:24.245
structure the queries and be able to, you know, build the query plan to understand, okay. These are the files that I actually need to fetch. These are the operations I need to perform in them, etcetera. And I'm curious if Dremel has that same architectural component of needing to be able to
128
00:12:24.705 --> 00:12:25.205
schematize
129
00:12:25.560 --> 00:12:37.095
and sort of maintain that metadata ahead of time before you're actually able to execute those queries or if you have a different system for being able to propagate that information to the query planner to be able to understand
130
00:12:37.635 --> 00:12:42.135
the actual sequence of operations that are necessary to be able to satisfy a given SQL structure?
131
00:12:42.630 --> 00:12:51.290
So today, the answer is yes. In what Dremel has become, BigQuery. BigQuery does have a metadata store. It does have managed storage,
132
00:12:51.834 --> 00:12:55.855
and it does use some of that information in query planning
133
00:12:56.394 --> 00:13:00.654
less than what you would probably expect relative to other systems.
134
00:13:01.660 --> 00:13:02.960
But that's because
135
00:13:04.060 --> 00:13:10.400
adrenaline, we didn't have anything like that. We never built a metadata store. The files were self describing.
136
00:13:11.214 --> 00:13:15.714
And we had this interesting dynamic for a lot of years on the internal system,
137
00:13:16.255 --> 00:13:18.515
where much of the data we were querying,
138
00:13:18.949 --> 00:13:21.050
we queried exactly 1 time.
139
00:13:21.430 --> 00:13:34.805
Meaning someone issued a query, we read that file for the very first time and the very last time we were ever gonna read that file. And that sets up a really interesting dynamic relative to the creation of this metadata and the computation of statistics
140
00:13:35.240 --> 00:13:37.980
that often get used in cost based optimization.
141
00:13:38.759 --> 00:13:41.819
If you are going to double the expense,
142
00:13:42.120 --> 00:13:46.835
right, from I'm only ever gonna read this file once to now I'm gonna read this file twice,
143
00:13:47.215 --> 00:13:48.115
you would have
144
00:13:48.895 --> 00:13:50.195
to bring some benefit
145
00:13:50.575 --> 00:13:52.115
that essentially completely
146
00:13:52.415 --> 00:13:56.640
removed the cost of of reading the file at all the second time.
147
00:13:57.100 --> 00:14:10.855
Because the first time when you're computing statistics, you're always gonna have to do a full table scan, Right? And so I think that was 1 of Andre's really clever insights and innovations when he started Dremel was just to say we're gonna make this thing performant
148
00:14:11.730 --> 00:14:14.470
without ever having to precompute anything.
149
00:14:15.009 --> 00:14:17.430
And that led to a lot of years of
150
00:14:17.889 --> 00:14:26.214
really simple usage for the users when you could just point the tool at your dataset, not have to wait for any pre computation or load or anything
151
00:14:26.680 --> 00:14:35.020
and have the query still run at interactive speeds. Now, the way we did that, of course, we didn't have magic. The way we did that was we threw tons of resources at the problem.
152
00:14:35.565 --> 00:14:43.345
Right? So you just get a lot more CPUs, you get a lot more network. Really, I think the network was the key to why Dremel succeeded.
153
00:14:43.700 --> 00:14:50.280
I know some people have asked why was Dremel so successful inside of Google, but projects like, for example, drill
154
00:14:50.660 --> 00:14:53.640
were not equally successful outside of the market.
155
00:14:54.085 --> 00:14:59.465
I think 1 of the main reasons was the innovation on the networking inside Google.
156
00:14:59.925 --> 00:15:16.265
We did everything we could to saturate the network inside the data centers we ran in. 1 of the things that people outside the Dremel team didn't know was that we were never very good citizens in terms of resource usage. But it was okay because everybody was using Dremel. So they were all benefiting from us abusing
157
00:15:16.725 --> 00:15:18.425
every loophole we could find.
158
00:15:18.805 --> 00:15:19.465
The other
159
00:15:19.980 --> 00:15:32.695
aspect that Dremel has led to is this idea of data federation where you have this query engine that, as you said, is decoupled from the storage layer, but that also gives you the opportunity to make
160
00:15:33.315 --> 00:16:01.780
the query execution pluggable so you're not necessarily just working with files on disk. You might also be, you know, translating your SQL statement into a different statement for a different relational database or a non relational database and then being able to aggregate data across multiple different systems to be able to build analyses across them so that you don't necessarily have to do this extract and load process to be able to actually answer questions. You can just say, I'm gonna query the data where it lives, and I'm wondering what you see as the
161
00:16:02.204 --> 00:16:05.024
sort of broad impact that that has had on
162
00:16:05.404 --> 00:16:06.704
analytical capabilities,
163
00:16:07.005 --> 00:16:14.730
both in terms of what tools like drill and Dremel and Presto are able to do, but also in terms of the ways that we think about
164
00:16:15.190 --> 00:16:20.089
how to build data systems and how to think about the contracts
165
00:16:20.455 --> 00:16:24.635
between data producers and owners and the downstream consumers of that data?
166
00:16:25.175 --> 00:16:27.675
Great question. Lots of stuff to dig into there.
167
00:16:28.040 --> 00:16:33.820
So 1 thing I would say is in addition to just being able to query the federated data,
168
00:16:34.200 --> 00:16:34.860
I think
169
00:16:35.320 --> 00:16:37.100
the revolution that came about
170
00:16:37.755 --> 00:16:38.655
because of
171
00:16:39.035 --> 00:16:43.775
us having done that in Dremel, at least with Inside Google, the revolution that came about was that
172
00:16:44.395 --> 00:16:45.935
suddenly people expected
173
00:16:46.699 --> 00:16:48.399
to be able to join datasets
174
00:16:49.180 --> 00:16:51.920
even if the data didn't reside in the same system.
175
00:16:52.300 --> 00:17:06.220
So prior to that, things like MapReduce the initial MapReduce, of course, didn't join anything. You had 1 dataset and you ran it. You you did a map and a reduce and you produced an output and unioning things even was hard. But joining was certainly hard.
176
00:17:06.760 --> 00:17:20.510
So Dremel, because we launched join and in order to be able to join, we had to be able to do it on the fly, We called it shuffle. That's a repartitioning operation so that you can repartition the data to get the right join keys. And,
177
00:17:20.910 --> 00:17:28.930
you know, our primary join strategy was always partition hash join. We tried a couple other. We tried lookup join. Certainly, we do have
178
00:17:29.434 --> 00:17:31.455
functionality for broadcast join
179
00:17:31.995 --> 00:17:37.775
if 1 of the datasets is small, and we did a lot of work to push small up to, like, the size of a gigabyte.
180
00:17:38.279 --> 00:17:40.700
But there were always limitations. So our primary
181
00:17:41.000 --> 00:17:43.179
mode was the shuffle hash join.
182
00:17:43.799 --> 00:17:44.299
And
183
00:17:44.840 --> 00:17:47.260
once you were able to read federated data
184
00:17:47.845 --> 00:17:49.785
and you could still do a join
185
00:17:50.405 --> 00:17:54.185
with an on the fly partition in memory and that was your core strategy,
186
00:17:55.260 --> 00:18:04.525
it opened up the ability to join data from wherever it happened to be. So for us, we use Bigtable, obviously, the the internal version of
187
00:18:05.085 --> 00:18:07.505
of HBase. But when you had a Bigtable
188
00:18:07.965 --> 00:18:17.510
that you could join to a file in Colossus, that you could join to a query result from a MySQL database or an F1 query or something like that,
189
00:18:18.130 --> 00:18:19.750
it really changed people's
190
00:18:20.134 --> 00:18:25.115
expectation and I think getting back to the idea that data just engenders more questions, it really
191
00:18:25.654 --> 00:18:27.595
opened up people's eyes to the possibilities
192
00:18:28.019 --> 00:18:31.159
of look at all these interesting analyses I could do
193
00:18:31.539 --> 00:18:36.679
that now require no work for me to pre process the data or do a multi phase
194
00:18:37.215 --> 00:18:37.715
orchestration
195
00:18:38.095 --> 00:18:41.715
with all sorts of different transformations. I can just express it
196
00:18:42.095 --> 00:18:50.860
as a common SQL statement using the join operator I'm used to. And under the hood, Dremel takes care of scheduling all of that, building the query plan.
197
00:18:51.559 --> 00:18:58.685
So I think that was 1 way in which it changed the paradigm where people were able to expect to join everything.
198
00:18:59.225 --> 00:19:01.085
And then it went to the next level
199
00:19:01.465 --> 00:19:01.965
when
200
00:19:02.320 --> 00:19:07.380
we rolled out BigQuery and now you see the same thing evolving with Snowflake's data marketplace.
201
00:19:07.840 --> 00:19:10.740
But the idea that the entire world's
202
00:19:11.535 --> 00:19:14.995
data analysis system could be 1 global system
203
00:19:15.535 --> 00:19:17.555
where all the data was joinable
204
00:19:18.495 --> 00:19:22.440
from its original place at rest. So there was no need to copy
205
00:19:23.140 --> 00:19:25.000
or get stale redundant
206
00:19:25.300 --> 00:19:27.000
versions of your data anywhere.
207
00:19:27.460 --> 00:19:29.960
I could just share my table with you
208
00:19:30.265 --> 00:19:32.045
and you would be able to query it.
209
00:19:32.585 --> 00:19:38.765
The second part though of your question, because you asked the question then what did that do to the relationship between data producers and consumers?
210
00:19:39.160 --> 00:19:47.500
I think it actually sort of broke the relationship and it's 1 of the things we're struggling with now at SoFi, figuring out how to rein that back in.
211
00:19:48.175 --> 00:19:53.075
And I didn't always see it when I was inside Google. I think in the early days,
212
00:19:53.455 --> 00:19:56.755
you could accurately describe what we built with Dremel
213
00:19:57.490 --> 00:20:08.445
as a data lake or at least an early version of a data lake. And then when we started to roll out BigQuery, we started by trying to keep that same data like paradigm.
214
00:20:09.145 --> 00:20:14.205
And we discovered after a few years that it just it wasn't working for organizations,
215
00:20:14.789 --> 00:20:17.370
and we had to become much more like a data warehouse
216
00:20:17.990 --> 00:20:18.649
in BigQuery
217
00:20:19.190 --> 00:20:21.130
moving away from the data lake paradigm.
218
00:20:21.590 --> 00:20:26.394
And I'll explain what I mean by that. In a data lake, I think the governance
219
00:20:27.255 --> 00:20:27.755
and
220
00:20:28.534 --> 00:20:33.034
the agreements between data producers and data consumers are hard to maintain.
221
00:20:33.679 --> 00:20:36.900
It lowers the barrier for entry for me as a consumer
222
00:20:37.280 --> 00:20:42.260
to just be able to query any data at rest as long as I can get access to that data.
223
00:20:42.794 --> 00:20:45.215
But it also leads to me potentially
224
00:20:45.914 --> 00:20:46.414
accidentally
225
00:20:46.875 --> 00:20:47.775
taking dependencies
226
00:20:48.955 --> 00:20:53.169
on things that the data producer has no interest in guaranteeing.
227
00:20:53.710 --> 00:20:57.009
And so schemas can change out from under me, data can disappear,
228
00:20:57.309 --> 00:21:00.450
it can be low quality or unreliable data, and
229
00:21:01.085 --> 00:21:03.985
I have no way of knowing that. And so I think there is
230
00:21:05.005 --> 00:21:09.265
now back toward the data warehouse paradigm with things like Snowflake and BigQuery,
231
00:21:09.620 --> 00:21:13.159
trying to keep all the advantages of federated data and all of that,
232
00:21:13.620 --> 00:21:16.360
but making it clear that the data that I actually
233
00:21:17.315 --> 00:21:20.294
need to be reliable because it's going into my regulatory
234
00:21:20.674 --> 00:21:26.130
filings or it's being shown back to my end users or something like that. I do
235
00:21:26.590 --> 00:21:28.850
think we're going to see a big move back to
236
00:21:29.150 --> 00:21:33.250
I want that data to be controlled and monitored and reliable.
237
00:21:34.055 --> 00:21:36.635
All of the things that I used to get from
238
00:21:37.095 --> 00:21:39.515
the really strict ETL pipelines
239
00:21:39.815 --> 00:21:42.955
owned by the central data team with guarantees about
240
00:21:43.509 --> 00:21:45.610
reliability and freshness and all of that.
241
00:21:46.309 --> 00:22:02.740
With that shift from the sort of data lake paradigm to where we are with data warehouses and now the up and coming term being the data lake house where it's sort of this hybrid of, I can dump all the data that I want, but it has to be at least semi structured and cleaned before I'm gonna bother querying it and exposing it to other people. I'm
242
00:22:03.280 --> 00:22:09.940
wondering what your view is as somebody who has helped to build a lot of the foundational tools and concepts that led to these capabilities.
243
00:22:10.575 --> 00:22:12.915
Now being a consumer of them, like, how that
244
00:22:13.375 --> 00:22:14.435
impacts your
245
00:22:14.815 --> 00:22:19.875
sort of design sense and how you think about the appropriate way to build data platforms
246
00:22:20.350 --> 00:22:28.610
at scale in a way that is sort of flexible and agile, but structured enough to be able to actually have reliable outputs and high quality data?
247
00:22:29.065 --> 00:22:29.804
That's exactly
248
00:22:30.105 --> 00:22:38.524
the right place to go with this. And I would throw 1 more thing into the mix. I know it's a bit of a fraught topic. So I'll just say the data mesh idea,
249
00:22:39.130 --> 00:22:47.870
regardless of whether you take the canonical definition or the more loose definition as I prefer, but this idea of distributed data ownership,
250
00:22:48.455 --> 00:22:54.955
that's really what we're pursuing at SoFi, and I think a lot of my peers and other organizations are pursuing as well because
251
00:22:55.780 --> 00:22:56.520
they're recognizing
252
00:22:56.980 --> 00:22:59.320
that trying to staff up
253
00:22:59.700 --> 00:23:00.200
a
254
00:23:00.580 --> 00:23:03.560
central data engineering team that owns
255
00:23:03.985 --> 00:23:10.495
all of the phases of data from production all the way to consumption isn't going to work.
256
00:23:11.210 --> 00:23:11.710
We
257
00:23:12.170 --> 00:23:15.070
are, I think, a little bit like most organizations,
258
00:23:15.770 --> 00:23:22.045
kind of fumbling in the dark. We're trying to figure out what works for us and how we can guarantee
259
00:23:22.825 --> 00:23:26.365
some of the things we need. We have being in financial technology
260
00:23:27.080 --> 00:23:30.539
and also having just gotten a bank charter. So we're not just
261
00:23:30.840 --> 00:23:34.860
like playing in Fintech, but we're an actual bank regulated by the OCC
262
00:23:35.825 --> 00:23:36.725
and other organizations.
263
00:23:37.184 --> 00:23:39.525
We have some heavier audit requirements
264
00:23:39.905 --> 00:23:40.725
and regulatory
265
00:23:41.025 --> 00:23:41.525
requirements
266
00:23:42.305 --> 00:23:44.565
than I think we were used to in the past.
267
00:23:45.020 --> 00:23:50.480
And so that's leading us to try and find ways to still be able to distribute,
268
00:23:51.020 --> 00:23:54.000
but still provide guarantees for certain datasets.
269
00:23:54.755 --> 00:23:55.495
The organizational
270
00:23:55.795 --> 00:24:00.375
structure that we've landed on is to have our engineering teams
271
00:24:01.075 --> 00:24:02.615
responsible for the production
272
00:24:03.340 --> 00:24:05.120
and ingestion of their data
273
00:24:05.820 --> 00:24:10.399
all the way into what we call the cleansed area in our data warehouse.
274
00:24:10.735 --> 00:24:11.475
We conceptualize
275
00:24:11.775 --> 00:24:13.875
our data warehouse in 4 zones.
276
00:24:14.495 --> 00:24:15.875
The first is raw.
277
00:24:16.335 --> 00:24:18.755
That's where the ingested data lands
278
00:24:19.120 --> 00:24:23.700
and there are really no guarantees. You can think of this like the data lake component.
279
00:24:24.400 --> 00:24:26.260
It's private by default.
280
00:24:26.745 --> 00:24:36.930
The data may be schema less, meaning they're just dumping JSON blobs into variant types in Snowflake and then cleaning it up as the first phase in their their ELT process.
281
00:24:37.630 --> 00:24:39.170
But our goal is for all
282
00:24:39.470 --> 00:24:41.570
analytic data or all potentially
283
00:24:42.035 --> 00:24:45.655
interesting analytic data to land in the raw zone in Snowflake.
284
00:24:46.275 --> 00:24:49.335
So we are putting all of the data inside Snowflake
285
00:24:49.930 --> 00:24:51.150
in the raw zone.
286
00:24:51.450 --> 00:24:52.110
This is
287
00:24:52.490 --> 00:25:00.885
similar in principle to what other folks are doing by putting landing in s 3 first and then transforming from there using either
288
00:25:01.185 --> 00:25:15.845
snowflakes ability to query directly from s 3 or using a query engine like Athena, something like that. For us, I wanna shut down the multiple tools, multiple sources of truth problem and so we are putting it all inside Snowflake,
289
00:25:16.385 --> 00:25:21.285
inside private schemas so that people cannot accidentally be getting to each other's raw data.
290
00:25:21.870 --> 00:25:26.210
The second zone we call cleansed. Cleansed is where you impose a schema.
291
00:25:26.590 --> 00:25:29.090
You do whatever cleaning is necessary, deduplication,
292
00:25:30.184 --> 00:25:30.684
standardization
293
00:25:31.144 --> 00:25:33.325
of data types, data content,
294
00:25:34.024 --> 00:25:37.725
probably some introduction of synthetic join keys
295
00:25:38.200 --> 00:25:40.700
to support the next phase in the process.
296
00:25:41.320 --> 00:25:45.660
And the cleansed layer is where we expect there to be a contract
297
00:25:46.015 --> 00:25:48.515
between the data producers and the data consumers.
298
00:25:48.895 --> 00:25:53.315
Meaning, if I put a field in there, I'm not gonna just pull that field away
299
00:25:53.780 --> 00:25:57.160
without some automated testing being able to catch me.
300
00:25:57.460 --> 00:25:59.880
For example, we don't give any group
301
00:26:00.260 --> 00:26:03.965
direct DDL or DML access to their cleansed schema.
302
00:26:04.345 --> 00:26:06.044
The only way you can update
303
00:26:06.345 --> 00:26:12.365
the schemas or the data inside your cleansed schema is via some automated process,
304
00:26:12.880 --> 00:26:16.500
which requires GitLab, it requires our CICD
305
00:26:16.800 --> 00:26:18.020
pipelines to run.
306
00:26:18.400 --> 00:26:23.355
And so your consumers always have the ability to detect if you're changing something.
307
00:26:23.975 --> 00:26:36.610
The next zone for us on top of cleansed, we call modeled. Pretty self explanatory, it's where you build your data models. There will be some amount of join aggregation there. We'll have all sorts of flavors in there from
308
00:26:37.230 --> 00:26:39.010
star or snowflake schemas,
309
00:26:39.485 --> 00:26:43.024
fact and dimension tables, just flat broad tables.
310
00:26:43.565 --> 00:26:51.030
And we're doing it still in a distributed way there where we do have a central core data warehouse which is the most important piece.
311
00:26:51.570 --> 00:26:54.550
But then around it, we have what are called team data marts
312
00:26:55.095 --> 00:26:59.195
where individual groups for the different vertical business units we have
313
00:26:59.575 --> 00:27:05.520
can be building their own data models. And then the last zone we have on top of all of that we call summarized.
314
00:27:06.300 --> 00:27:11.360
That's where you build your reporting layer. So those should be the tables that are built
315
00:27:11.885 --> 00:27:13.825
for optimizing some report.
316
00:27:14.205 --> 00:27:21.105
In general, the pipeline we think will flow where data science, they can prototype reports directly off the model schema.
317
00:27:21.630 --> 00:27:27.490
But then if they find the SQL query for the report they want, they'll probably turn that into
318
00:27:27.790 --> 00:27:29.250
the equivalent of a materialized
319
00:27:29.550 --> 00:27:30.050
view
320
00:27:30.485 --> 00:27:32.745
or base tables with aggregates or
321
00:27:33.125 --> 00:27:36.105
something that allows them to do very simple reporting.
322
00:27:36.644 --> 00:27:39.530
And we'd like to pull most of the business logic
323
00:27:39.850 --> 00:27:43.390
out of our business intelligence tool, which is Tableau,
324
00:27:43.770 --> 00:27:45.950
and keep it all in the data warehouse.
325
00:27:49.265 --> 00:28:00.410
So now your modern data stack is set up. How is everyone going to find the data they need and understand it? Select Star is a data discovery platform that automatically analyzes and documents your data.
326
00:28:00.870 --> 00:28:10.875
For every table in Select Star, you can find out where the data originated, which dashboards are built on top of it, who's using it in the company, and how they're using it, all the way down to the SQL queries.
327
00:28:11.335 --> 00:28:15.980
Best of all, it's simple to set up and easy for both engineering and operations teams to use.
328
00:28:16.520 --> 00:28:17.260
With SelectStar's
329
00:28:17.560 --> 00:28:22.860
data catalog, a single source of truth for your data is built in minutes, even across thousands of datasets.
330
00:28:23.565 --> 00:28:27.105
Try it out for free and double the length of your free trial at dataengineeringpodcast.com/selectstar.
331
00:28:30.045 --> 00:28:32.150
You'll also get a swag package when you continue
332
00:28:35.190 --> 00:28:54.970
The model in question is something that has been interesting to me for a while now because of the fact that since we do have these very powerful and flexible engines that allow us to be able to query across structured and unstructured datasets, and we have the power to be able to just kind of throw everything into a giant table and be able to run aggregates across it. We don't necessarily have to be as
333
00:28:55.325 --> 00:29:13.705
rigorous as we used to be when we were all relying on a, you know, a massive Oracle server and worrying about the licensing there or as, you know, Microsoft SQL Server and having to do these, you know, stars, snowflake schemas, or, you know, even data vaults. And I'm wondering what you have seen as the impact of these newer generations of technologies
334
00:29:14.085 --> 00:29:15.545
on the ways that people
335
00:29:15.925 --> 00:29:30.855
think about data modeling and the sort of rigor that they place in terms of how they're actually structuring their schemas versus just saying, okay. Well, this is all on a table. I can run a select statement that gets me what I want. Want. So, you know, good enough. I'm off to my next task.
336
00:29:31.155 --> 00:29:34.775
I think it's been a really good thing in terms of
337
00:29:35.315 --> 00:29:36.455
getting more analysis
338
00:29:36.910 --> 00:29:40.130
inside the data warehouse. As you say, now that we're not constrained
339
00:29:40.430 --> 00:29:41.330
by the resources
340
00:29:41.950 --> 00:29:50.414
of running on a single Oracle or Postgres box or whatever, and having to lock people out during certain times so core processes could
341
00:29:50.794 --> 00:29:52.815
run. That piece of it has been fantastic.
342
00:29:53.355 --> 00:29:56.015
What I'm encountering since I came over to SoFi,
343
00:29:56.539 --> 00:30:04.559
which actually never occurred to me when I was building the tools at Google, and it probably should've but it just didn't click until I was the 1 doing it.
344
00:30:05.335 --> 00:30:06.715
There were other advantages
345
00:30:07.655 --> 00:30:08.955
to the old model
346
00:30:09.335 --> 00:30:18.780
and we looked just at the cost and we said well, it was about resource constraint and now I don't have the resource constraint. So now I don't have to do any of that other stuff anymore.
347
00:30:19.800 --> 00:30:21.660
1 of the problems with that is
348
00:30:22.375 --> 00:30:29.355
if everybody is so take our model, for example. If we were to tell everybody we only have 2 zones, we have raw and cleansed.
349
00:30:29.760 --> 00:30:34.180
And then because we use Snowflake and Snowflake is so fast and we have so many resources,
350
00:30:34.640 --> 00:30:40.845
everybody's just gonna query directly against the cleansed data, which is still largely in a normalized schema.
351
00:30:41.225 --> 00:30:44.285
And then you're gonna produce your reports directly from there.
352
00:30:44.840 --> 00:30:48.060
What you get then is a proliferation of business logic
353
00:30:48.600 --> 00:30:49.500
into everybody's
354
00:30:49.800 --> 00:30:51.100
report and dashboard.
355
00:30:51.720 --> 00:30:54.935
And the bigger your organization gets and the more distributed,
356
00:30:55.635 --> 00:30:58.295
the higher the likelihood that there are gonna be disagreements.
357
00:30:58.755 --> 00:31:03.950
And we see this in spades at SoFi right now. Executives getting 3 different
358
00:31:04.490 --> 00:31:10.190
reports that all claim to be talking about the same thing but showing different numbers.
359
00:31:10.605 --> 00:31:21.830
Because either they've used different filters and so their date ranges are slightly different or their criteria for the cohorts that they're including are slightly different. Or they've joined
360
00:31:22.130 --> 00:31:29.605
in a different order or they've used a left outer join instead of a right outer join or they did an inner join where the other guy did a a right outer
361
00:31:30.145 --> 00:31:31.765
join. And so we've got
362
00:31:32.304 --> 00:31:34.725
all these different layers at which
363
00:31:35.265 --> 00:31:37.920
the reliability of the data can break down.
364
00:31:38.400 --> 00:31:43.460
And so I think right now what we're seeing is actually a pushback in the other direction
365
00:31:44.000 --> 00:31:58.410
where we went in 1 direction because we now had all this new flexibility because we had Snowflake and we had these tools that could do things really quickly. And now what we're seeing is we need to reign that in and we need to find a way to keep the goodness of being able to run
366
00:31:58.790 --> 00:32:03.050
a ton of analysis and not having data engineering be the bottleneck
367
00:32:03.405 --> 00:32:04.385
of the organization,
368
00:32:05.405 --> 00:32:10.945
but also having some sort of gatekeeping and guardrails that prevent people from
369
00:32:11.470 --> 00:32:11.970
inadvertently
370
00:32:12.350 --> 00:32:13.250
making mistakes.
371
00:32:13.870 --> 00:32:27.505
Well, clearly, the solution is that we just build a new category of tools and say that that's gonna solve all of our problems. Right? That's what the metric layer is for. That's right. Yes. The metric story is definitely what we're looking at and I would throw 1 other 1 in there, which is the data quality tools.
372
00:32:27.885 --> 00:32:32.640
That's for us sort of the most recent piece we've brought in to our system.
373
00:32:33.260 --> 00:32:40.125
And I think that as we get more rigorous about testing and monitoring data quality
374
00:32:41.225 --> 00:32:45.885
so that our end consumers are not our canaries in the coal mine,
375
00:32:46.389 --> 00:32:51.289
It's very much, I think, akin to the transformation we went through as an industry
376
00:32:51.750 --> 00:32:53.929
over the last couple decades toward
377
00:32:54.885 --> 00:32:58.025
expecting engineers to always write unit tests
378
00:32:58.325 --> 00:33:02.665
on everything they were running and expecting those tests to be run.
379
00:33:02.970 --> 00:33:10.910
I don't know other companies if they're quite the same at Google. I know mostly about Google but at Google, the system we have runs every affected test
380
00:33:11.424 --> 00:33:23.390
at every single change list. And so you always know if you broke something and it does it before you submit. You're not even allowed to submit code that would break tests. I would love to see us at SoFi
381
00:33:24.090 --> 00:33:26.510
get to a point with these data observability
382
00:33:26.890 --> 00:33:27.390
tools
383
00:33:27.930 --> 00:33:29.710
where people were not
384
00:33:30.395 --> 00:33:36.015
unknowingly pushing schema changes that broke their downstream consumers or changing data values
385
00:33:36.395 --> 00:33:38.335
that invalidated some report.
386
00:33:39.050 --> 00:33:42.190
Jumping back a little bit to the question of data federation,
387
00:33:42.890 --> 00:33:46.350
I'm also interested in your take on how you see that impacting
388
00:33:46.890 --> 00:33:50.735
the sort of questions of data governance and
389
00:33:52.075 --> 00:33:54.254
the sort of formulation and application
390
00:33:54.554 --> 00:34:01.700
of ethical priorities on how that data is being used and consumed and and remixed and some of the ways that you're able to
391
00:34:02.480 --> 00:34:09.575
guard against or account for bias in the ways that you're building your analyses because of the fact that more people have more access to more data?
392
00:34:10.035 --> 00:34:18.910
I think there's a lot in there. So federated data, I think that has been 1 of the downsides of it, that you lose some ability to govern.
393
00:34:19.369 --> 00:34:20.430
In some senses,
394
00:34:20.809 --> 00:34:21.550
that's okay
395
00:34:22.170 --> 00:34:24.990
because not all data has to be so tightly
396
00:34:25.325 --> 00:34:27.825
governed and controlled. You want there to be flexibility
397
00:34:28.125 --> 00:34:29.105
and experimentation.
398
00:34:30.125 --> 00:34:33.185
But I think there are places where it runs into problems.
399
00:34:33.645 --> 00:34:34.305
I think
400
00:34:35.069 --> 00:34:35.970
it's particularly
401
00:34:36.270 --> 00:34:38.369
hard to govern and control PII
402
00:34:38.829 --> 00:34:40.450
in a data lake environment.
403
00:34:41.069 --> 00:34:41.569
Particularly
404
00:34:41.950 --> 00:34:44.049
if that PII is sometimes
405
00:34:44.355 --> 00:34:44.855
stored
406
00:34:45.395 --> 00:34:45.635
in,
407
00:34:46.595 --> 00:34:48.775
schema lists or schema on read fields.
408
00:34:49.235 --> 00:34:50.935
So I'm saying here, for example,
409
00:34:51.510 --> 00:34:52.890
I have seen situations
410
00:34:53.190 --> 00:34:59.105
where PII is being stored inside JSON blobs, inside files written to s 3.
411
00:34:59.825 --> 00:35:04.244
That becomes a real danger for governance and for compliance
412
00:35:05.185 --> 00:35:05.685
because
413
00:35:06.280 --> 00:35:09.020
if it's stored at least in a structured environment,
414
00:35:09.640 --> 00:35:12.940
what I can do is read the first few rows
415
00:35:13.395 --> 00:35:14.214
of a dataset
416
00:35:14.835 --> 00:35:15.655
and identify
417
00:35:16.115 --> 00:35:18.855
it looks like the values in this field
418
00:35:19.474 --> 00:35:23.095
contain a social security number or address or something.
419
00:35:23.950 --> 00:35:25.970
But if I've got JSON blobs,
420
00:35:26.270 --> 00:35:27.410
which do not
421
00:35:27.869 --> 00:35:29.490
have strictly defined schemas,
422
00:35:29.950 --> 00:35:39.865
I can't read just part of that data. I have to read all of that data and I have to parse every JSON block and look through every field to say, do any of these look like PII?
423
00:35:40.510 --> 00:35:42.530
That becomes a cost prohibitive
424
00:35:43.230 --> 00:35:45.090
expense, I think, for many organizations.
425
00:35:45.870 --> 00:35:52.675
So I think 1 of the important way you asked about ethical considerations. 1 of the important things we have to think about is
426
00:35:53.055 --> 00:35:57.075
as we take advantage of these new tools that have huge scale
427
00:35:57.520 --> 00:35:58.340
and the ability
428
00:35:58.640 --> 00:36:00.340
to give us greater flexibility,
429
00:36:01.360 --> 00:36:03.780
we have to think about whether that flexibility
430
00:36:04.080 --> 00:36:09.295
is being applied to data which is safe or data which could potentially leak.
431
00:36:09.995 --> 00:36:19.230
I'll throw 1 more on that if I can. It's a little divergence from the question you asked but getting back to this idea of joins, I think that has been
432
00:36:19.770 --> 00:36:20.830
the real innovation
433
00:36:21.450 --> 00:36:22.510
in the last decade
434
00:36:22.994 --> 00:36:26.214
since Dremel. The idea that I can join data
435
00:36:26.515 --> 00:36:28.454
from all sorts of different datasets.
436
00:36:28.994 --> 00:36:29.815
And I think
437
00:36:30.319 --> 00:36:32.819
historically, when people think about data breaches
438
00:36:33.440 --> 00:36:35.299
or misuses of their data,
439
00:36:35.760 --> 00:36:37.059
they think primarily
440
00:36:37.359 --> 00:36:40.974
about leaks, about exposing this is like the Equifax
441
00:36:41.435 --> 00:36:47.400
thing. Right? Someone's gonna get a hold of my 140, 000, 000 records, and then they'll know all the data that I need.
442
00:36:47.880 --> 00:36:59.565
But I think what we're seeing now is the actual bigger danger is not that they get the data that you had, but that they get your data and they join it to some other data set that you hadn't even considered.
443
00:36:59.945 --> 00:37:03.885
We've seen leaks like this for example, right? The the idea that
444
00:37:04.420 --> 00:37:08.760
American military personnel wearing fitness trackers that were accidentally
445
00:37:09.060 --> 00:37:10.760
hooked up still to Strava
446
00:37:11.305 --> 00:37:16.125
and continuing to do their exercise regimen even though they're deployed to secret bases
447
00:37:16.425 --> 00:37:17.405
around the world,
448
00:37:17.945 --> 00:37:19.565
suddenly draws a map
449
00:37:19.900 --> 00:37:26.160
on a publicly accessible map that says, hey, there is something going on in this region of
450
00:37:26.700 --> 00:37:29.775
pick your favorite foreign country that you don't want
451
00:37:30.154 --> 00:37:37.360
adversaries knowing about military bases in. Or the New York taxi data that was released and then read it found,
452
00:37:37.740 --> 00:37:49.105
hey I can join this data with paparazzi photos and I can find out where famous people are taking taxi rides to and from and where they tend to be at a given date and time.
453
00:37:49.484 --> 00:37:54.865
So I think we need to get in the mindset not just of saying I secure my own data but
454
00:37:55.369 --> 00:38:06.785
what is the absolute worst thing someone could do if they had my data and random dataset x that they could join it to? Yeah. And that's giving rise to another sort of subindustry
455
00:38:07.165 --> 00:38:10.225
of this idea of privacy engineering where you have
456
00:38:10.579 --> 00:38:17.480
before you actually expose any dataset, even whether internally or externally and share it, you actually go through and
457
00:38:17.935 --> 00:38:22.915
obfuscate the data or, you know, mask it or, you know, add in some random noise
458
00:38:23.215 --> 00:38:24.595
to make sure that you
459
00:38:25.750 --> 00:38:36.715
can guard against with reasonable expectation against these re identification attacks or, you know, these, you know, data joining attacks where you say, you know, this 1 piece of information is innocuous in isolation,
460
00:38:37.015 --> 00:38:42.075
but when I join that with, you know, US census data, actually, I can see exactly who this person is.
461
00:38:42.455 --> 00:38:46.109
Yep. Yep. And there's another thing. So you mentioned the metrics
462
00:38:46.490 --> 00:38:56.744
store. That's definitely a piece of technology we're looking at for SoFi's data platform and actively trying to figure out what we're doing there. Another 1 is this idea of a privacy vault
463
00:38:57.125 --> 00:38:58.744
that some people are talking about.
464
00:38:59.070 --> 00:39:04.130
And I think there are some really innovative approaches there. What we're thinking about is
465
00:39:04.670 --> 00:39:09.174
we call it shifting left. We would like to be able to shift the obfuscation
466
00:39:09.635 --> 00:39:10.535
of PII
467
00:39:11.315 --> 00:39:14.135
as close to the point of production as possible.
468
00:39:14.680 --> 00:39:21.660
And the way you could do that is, I think 10 years ago, the hope would have been for homomorphic encryption.
469
00:39:22.225 --> 00:39:27.685
We tried we implemented a couple homomorphic operators in Dremel and in BigQuery, but
470
00:39:28.065 --> 00:39:33.080
it just hasn't panned out yet. I'm not smart enough on the math to know if it ever will.
471
00:39:33.380 --> 00:39:42.395
I don't think that fundamentally that's gonna be our saving grace. And we may put a note in the show notes if folks are not familiar with the idea of homomorphic encryption.
472
00:39:42.775 --> 00:39:45.390
But essentially it is, I can encrypt
473
00:39:45.770 --> 00:39:46.589
2 values
474
00:39:47.049 --> 00:39:48.750
and then I can do operations
475
00:39:49.130 --> 00:39:50.510
on the encrypted values,
476
00:39:50.970 --> 00:39:56.495
which give me the encrypted version of the result I would have gotten from doing those operations
477
00:39:56.955 --> 00:40:04.460
on the unencrypted values. So say, I take 2 numbers, I encrypt them, I add them together. The answer is the encrypted version
478
00:40:04.920 --> 00:40:10.145
of the sum of those 2 numbers. But there's no way for me to back out at any point
479
00:40:10.525 --> 00:40:13.345
from the encryption and get back to the original values.
480
00:40:13.725 --> 00:40:19.539
So if homomorphic encryption can't save us from this, I think the next best thing is
481
00:40:19.920 --> 00:40:20.420
tokenization
482
00:40:21.200 --> 00:40:23.059
where I can get a deterministic
483
00:40:23.359 --> 00:40:24.900
token for my value.
484
00:40:25.395 --> 00:40:28.295
I can store that in a secure place
485
00:40:28.755 --> 00:40:31.815
and only pass the token around within my ecosystem.
486
00:40:32.480 --> 00:40:39.940
And if the tokens are deterministic, I can at least do equality comparisons between them. So this now gives me join and aggregation
487
00:40:40.640 --> 00:40:42.020
without having to decrypt.
488
00:40:42.625 --> 00:40:46.724
And if I need to do other operations, I can have differential privacy
489
00:40:47.425 --> 00:40:49.845
in the vault at the point of access.
490
00:40:50.310 --> 00:40:59.610
So now I don't have to worry about who gets access to my token. I can hand the token out willy nilly and when they go try to exchange it with the privacy vault,
491
00:40:59.914 --> 00:41:02.255
the privacy vault can be the 1 that decides
492
00:41:02.714 --> 00:41:14.470
do you get all the data? Do you get masked data? Do you get no data? Do you get access just to the privacy vault doing comparisons for you? This would be the model where I say, I know a token.
493
00:41:14.895 --> 00:41:20.435
I know another thing. I want you to tell me if these 2 are the same thing and it can give you yes, no answers,
494
00:41:20.735 --> 00:41:33.415
but it can't give you back the original data. Yeah. I've I've been reading about that a little bit recently myself as well. I've had somebody contact me with the potential of being on the show to talk about that idea as well, so definitely
495
00:41:33.875 --> 00:41:35.175
spending some focus there.
496
00:41:35.555 --> 00:41:46.380
And so now digging a bit more into your experience building at SoFi, you've mentioned a bit about some of the stack that you've built up, some of the things that you're looking towards. And I'm wondering if you can just talk to
497
00:41:46.744 --> 00:41:47.645
how your experiences
498
00:41:48.345 --> 00:41:51.885
working at Google on building Dremel, helping to grow BigQuery,
499
00:41:52.505 --> 00:41:54.285
building some of the broader ecosystem
500
00:41:54.585 --> 00:41:58.599
of data analysis and data tooling for the Google Cloud Platform,
501
00:41:59.060 --> 00:42:04.280
how that has influenced the way that you think about platform design, organizational structure,
502
00:42:04.845 --> 00:42:06.385
and sort of the prioritization
503
00:42:07.005 --> 00:42:07.505
of
504
00:42:07.805 --> 00:42:09.265
features and projects
505
00:42:10.685 --> 00:42:26.875
and the valuation of data at SoFi and some of the ways that that earlier experience has led into where you are right now and the ways that you think about things. Yeah. There's 1 more thing about the time at Google that I would add that I think has probably had the biggest impact on how I'm thinking about things at SoFi.
506
00:42:27.414 --> 00:42:31.434
So shortly after we launched the Dremel service back in 2010,
507
00:42:32.130 --> 00:42:39.510
we began on a project which we eventually named Plex inside Google. It was p l x because, I guess,
508
00:42:39.815 --> 00:42:52.120
vowels are expensive or something. The Plex system folks may have read. There there are some papers we published a little bit externally about it. But the idea was to build a unified data analytics platform.
509
00:42:52.660 --> 00:42:58.055
And even back then, we had identified a number of components like a data catalog,
510
00:42:58.835 --> 00:43:00.375
like data observability
511
00:43:00.835 --> 00:43:02.295
tools, a robust
512
00:43:02.755 --> 00:43:05.415
ad hoc query UI, dashboarding tools,
513
00:43:05.910 --> 00:43:13.930
all the things that we would need to build. And of course, being Google and being that it was early in the evolution of these things, we built everything
514
00:43:14.390 --> 00:43:14.890
ourselves.
515
00:43:15.434 --> 00:43:20.015
And some pieces succeeded more than others and some took a long time to get to
516
00:43:20.474 --> 00:43:23.694
to various points. I would say Plex is now very successful
517
00:43:24.450 --> 00:43:25.270
inside Google.
518
00:43:25.890 --> 00:43:27.830
And the reason I bring that up, I think
519
00:43:28.210 --> 00:43:30.950
I have taken 2 lessons away from that experience
520
00:43:31.730 --> 00:43:32.230
that
521
00:43:32.875 --> 00:43:35.375
have shaped the way I'm thinking about it at SoFi.
522
00:43:36.395 --> 00:43:37.135
1 is
523
00:43:37.755 --> 00:43:43.020
I think we did a really good job of identifying the important components on the Plex project.
524
00:43:43.560 --> 00:43:47.340
And when I came over to SoFi and as we have been building,
525
00:43:48.040 --> 00:43:51.585
it has very much shaped my view of what's missing.
526
00:43:52.125 --> 00:44:04.030
And so as I've come in, I've identified along the way when I first got here, there were 4 specific things I identified. There were gaps in our offering that we needed to fill in. We've since added a couple more.
527
00:44:04.330 --> 00:44:07.295
But that idea of thinking about it in terms of
528
00:44:08.395 --> 00:44:09.615
component systems
529
00:44:10.155 --> 00:44:11.695
and how do they fit together,
530
00:44:12.235 --> 00:44:14.255
and how do I build a complete
531
00:44:14.670 --> 00:44:16.690
solution out of these component systems?
532
00:44:17.230 --> 00:44:18.609
I think that was 1 thing.
533
00:44:18.910 --> 00:44:25.835
But then on the other side, the negative example I took away from our Plex experience in shaping the way we approach things at SoFi is
534
00:44:26.455 --> 00:44:27.835
it was extraordinarily
535
00:44:28.615 --> 00:44:29.115
expensive
536
00:44:29.495 --> 00:44:31.755
to try and build all that stuff ourselves.
537
00:44:32.480 --> 00:44:43.415
Now, if you're Google, you can afford that. I mean, Google, you can even afford to build multiple redundant systems and compete with yourself in the internal market, which we did way more than we should have.
538
00:44:43.715 --> 00:44:50.535
At a place like SoFi, I can't afford it. I can't afford to be building so much infrastructure. I think it's great when I look at
539
00:44:51.010 --> 00:44:54.470
my web first peers like Lyft and like Airbnb
540
00:44:55.170 --> 00:44:57.110
and Square and some of these that
541
00:44:57.570 --> 00:45:01.605
I think have a little bit more of the luxury of building infrastructure
542
00:45:01.905 --> 00:45:02.645
and tooling.
543
00:45:03.105 --> 00:45:09.610
And that's great and I hope they keep doing it because we're really benefiting from their blogs and their learnings and the open source
544
00:45:09.910 --> 00:45:13.210
and the startups that they're spinning out that we get to buy from.
545
00:45:13.990 --> 00:45:17.210
But I think there are a lot more companies who are in buy position
546
00:45:17.815 --> 00:45:19.994
at SoFi where building infrastructure
547
00:45:20.695 --> 00:45:23.675
is not our business. Our business is Fintech.
548
00:45:24.055 --> 00:45:26.155
And so we need to, I think,
549
00:45:26.680 --> 00:45:30.380
be more in the business of putting together tools.
550
00:45:31.160 --> 00:45:36.060
Yeah. And I think 1 more maybe, this is not a lesson I take away but
551
00:45:36.425 --> 00:45:47.369
it's something that's a change for me from the time when I worked trying to sell BigQuery. Because for a lot of years, my job was trying to sell BigQuery to the biggest companies on the planet and convince them that we had all of their solutions.
552
00:45:47.910 --> 00:45:52.329
And we very much our model was building a unified platform
553
00:45:53.030 --> 00:45:59.595
that would that everyone could just leverage. And now being on the outside, I don't think that's reality. I think
554
00:45:59.895 --> 00:46:05.195
it's much more the case than at least in the phase of development we're in right now,
555
00:46:05.880 --> 00:46:07.099
everybody needs bespoke
556
00:46:07.400 --> 00:46:07.900
solutions.
557
00:46:08.359 --> 00:46:12.539
Everybody's making different trade offs. There are things that are right for SoFi
558
00:46:12.855 --> 00:46:16.315
or that are necessary for SoFi, choices I have to make
559
00:46:16.615 --> 00:46:18.795
that would not be right for
560
00:46:19.335 --> 00:46:20.315
even a Robinhood
561
00:46:20.615 --> 00:46:31.974
or a Chime or some of the ones that people might look at and say, well, they're pretty close to SoFi so they probably have the same requirements. But as I said, being a chartered national bank,
562
00:46:32.434 --> 00:46:37.850
we have different requirements than they have and I'm gonna have to bring in heavier weight process
563
00:46:38.150 --> 00:46:39.850
that would be appropriate for them.
564
00:46:42.310 --> 00:46:55.540
RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility
565
00:46:55.840 --> 00:46:56.580
and control.
566
00:46:57.200 --> 00:47:11.224
Their SDKs make event streaming from any app or website easy, and their state of the art reverse ETL pipelines enable you to send enriched data to any cloud tool. Sign up for free or just get the free t shirt for being a listener of the data engineering podcast
567
00:47:13.470 --> 00:47:13.970
atdataengineeringpodcast.com/rudder.
568
00:47:16.910 --> 00:47:21.755
In addition to the work that you're doing at SoFi and the work that you've done at Google
569
00:47:22.214 --> 00:47:28.155
and actually tying that to sort of what you've done at Google and some of the ways that that has influenced the trajectory
570
00:47:28.535 --> 00:47:31.170
of the ecosystem of data tools, it has
571
00:47:31.630 --> 00:47:40.025
opened up the idea of data marketplaces with 1 of the first ones being actually in BigQuery and being able to have public and shareable datasets where
572
00:47:40.325 --> 00:47:41.545
I can host a dataset
573
00:47:41.845 --> 00:47:45.990
and you can query it, but you're gonna be the 1 paying for the queries so that I don't
574
00:47:46.370 --> 00:47:52.870
have to. And that has been copied as a model for Snowflake, and there are actually whole businesses being built up around
575
00:47:53.215 --> 00:47:57.955
having shareable datasets to build up this sort of data sharing economy. And I'm wondering if you can
576
00:47:58.255 --> 00:48:01.075
share your thoughts on what you view as some of the
577
00:48:01.589 --> 00:48:04.650
benefits and potential future of these data marketplaces
578
00:48:05.030 --> 00:48:05.930
as the technology
579
00:48:06.470 --> 00:48:08.490
grows to support them more natively.
580
00:48:09.185 --> 00:48:13.045
I am personally really excited about the idea of data marketplaces
581
00:48:13.744 --> 00:48:14.244
and
582
00:48:14.625 --> 00:48:15.445
the opportunity
583
00:48:15.905 --> 00:48:17.110
I think we have
584
00:48:17.510 --> 00:48:18.810
to turn the economics
585
00:48:19.270 --> 00:48:20.010
of data
586
00:48:20.790 --> 00:48:22.090
sort of on its head
587
00:48:22.470 --> 00:48:24.490
and better align the incentives
588
00:48:24.790 --> 00:48:30.325
of the data generators. Like you and me, the human beings who are going around using a software.
589
00:48:31.184 --> 00:48:32.404
And the data
590
00:48:33.080 --> 00:48:42.935
providers, the people who are collecting it and aggregating it and then reselling it, which is also me being at SoFi, we're collecting a lot of data. Right? We have a lot of data. So
591
00:48:43.715 --> 00:48:44.775
we had a system
592
00:48:45.155 --> 00:48:47.655
inside Google that could allow
593
00:48:48.195 --> 00:48:49.095
data owners
594
00:48:49.635 --> 00:48:50.135
to,
595
00:48:50.660 --> 00:48:53.240
in real time, at execution time,
596
00:48:53.700 --> 00:49:12.990
have a check on the sorts of queries that were being run over their data. Now obviously that had to be automated. It couldn't be a human being sitting there saying yes, no to every query. Though there were times where we had things that were that level of detail where, say, an ad agency wanted some really specific data and we would have human beings who could review
597
00:49:13.450 --> 00:49:17.265
those queries and say yes or no. We will allow you to see this result set.
598
00:49:17.965 --> 00:49:19.505
The reason that was important
599
00:49:19.965 --> 00:49:21.265
in my mind is
600
00:49:22.045 --> 00:49:23.905
I can have much more confidence
601
00:49:24.520 --> 00:49:26.220
in sharing my data with you
602
00:49:27.400 --> 00:49:30.220
if we're in the world we're in now. So it's no longer
603
00:49:30.760 --> 00:49:53.145
I have to FTP you all of the data I have and then just trust that that will work, right? This was the initial temps attempts at data marketplace. I remember Amazon had 1 and Microsoft had 1. I can't remember who else might have. But it was effectively an FTP site with a credit card reader in front of it. And so I would have important data and I would put it on the site and then you would swipe your credit card and now you've got my data.
604
00:49:54.025 --> 00:49:54.925
And that never
605
00:49:55.305 --> 00:50:02.500
worked. And the reason it never worked is because if you try to sell data in that model and it's valuable data, you get to sell it 1 time.
606
00:50:02.980 --> 00:50:10.520
And then I sell it to you and it ends up on a torrent site and now nobody else ever pays me for my data again and I've lost all control of my data.
607
00:50:11.035 --> 00:50:13.135
And it also really only works for
608
00:50:13.675 --> 00:50:15.375
primarily static datasets.
609
00:50:15.755 --> 00:50:19.455
If the dataset updates every month, then you gotta pay me every month.
610
00:50:19.950 --> 00:50:23.410
It becomes a much less interesting proposition for most people.
611
00:50:23.950 --> 00:50:38.215
So now we've solved the problem of saying, I don't have to release my data to you. I can have a view, say, in front of my table and I can allow you to query that view and so I know you're only getting to these fields that I want and only with these particular
612
00:50:38.780 --> 00:50:39.760
filters on them.
613
00:50:40.700 --> 00:50:43.680
But at the next level then would be an idea where
614
00:50:44.060 --> 00:50:49.955
I can let you run a query, but maybe you run a query and your query would only aggregate
615
00:50:50.335 --> 00:50:51.795
over 3 individuals.
616
00:50:52.630 --> 00:50:54.809
And I say, that's not enough. You could deanonymize
617
00:50:55.269 --> 00:51:09.535
from 3 individuals. That's not gonna work. I need there to be at least a 100 people. And so you send a query and I say, nope, that query can't run. And then there are debates and discussions about how much information I give back to you when the query can't run
618
00:51:09.840 --> 00:51:14.100
because sometimes you can learn things just from sending repeated failed queries,
619
00:51:14.640 --> 00:51:16.260
but we can negotiate that.
620
00:51:16.640 --> 00:51:28.345
So then if you think about extending that to the next level, imagine that individual consumers could determine what they were comfortable with their data being used for in various analyses.
621
00:51:29.210 --> 00:51:35.869
And obviously, most people are not gonna have the technical wherewithal to figure this out. You solve that with associations,
622
00:51:36.330 --> 00:51:48.869
groups basically who can say these are our principles. This is what we're okay with. This is what I'm not okay with. And then I can join these groups and say, okay, I'm okay with my data being used according to the principles of this group.
623
00:51:49.410 --> 00:51:53.109
Once we get into a world like that, we can then allow
624
00:51:53.490 --> 00:51:57.095
the data owners, the human beings who produced the data
625
00:51:57.395 --> 00:51:59.015
to share in the benefits
626
00:51:59.635 --> 00:52:08.930
of having their data resold. And share can come in all sorts of forms. The easiest 1 to think about is monetary. I can make some money by letting my data
627
00:52:09.310 --> 00:52:18.265
go out there. Right? My data is just another asset that I own that can be put to work for me and the amount it gets used in queries that can come back to me.
628
00:52:19.045 --> 00:52:19.545
So
629
00:52:20.085 --> 00:52:23.865
I think that's the direction we could head with these data marketplaces.
630
00:52:24.550 --> 00:52:33.210
There's a little bit more technology work obviously the ability for these on the fly at query execution time analyses to be injected
631
00:52:33.744 --> 00:52:37.925
into the query processing stream. And then the idea of
632
00:52:38.385 --> 00:52:38.885
these
633
00:52:39.345 --> 00:52:39.845
associations
634
00:52:40.464 --> 00:52:42.165
of like minded individuals.
635
00:52:42.880 --> 00:52:45.220
For me, that's what excites me about data marketplaces.
636
00:52:45.839 --> 00:52:50.315
That's definitely a very interesting approach to the overall question of how
637
00:52:50.795 --> 00:53:01.119
to be able to bring that sort of monetization factor or the capture value back to the individual user because there have been debates about that for several years now, and most of the approaches that people
638
00:53:01.420 --> 00:53:06.160
share are either impractical or impossible or just make no sense whatsoever.
639
00:53:06.664 --> 00:53:11.805
But it there's definitely an interesting perspective on it. And then the other question too of being able to say,
640
00:53:12.105 --> 00:53:27.065
you know, I'm not gonna answer your query because it's far too pointed. You know? You you need to have a more general query that is going to give you some value, but it's not going to give you exact information about this, you know, very small cohort of people that you're trying to target. So that could also help to
641
00:53:27.445 --> 00:53:43.960
mitigate some of the issues that we have about things like ad retargeting and the, you know, very detailed dossiers that all of these ad agencies are able to build up about individuals. So it helps to solve a couple of those problems at the same time and definitely be interested to see how that might manifest in the years to come.
642
00:53:44.425 --> 00:53:45.965
Yeah. I think it would also
643
00:53:46.665 --> 00:53:49.244
probably put our industry on a better footing
644
00:53:49.545 --> 00:53:51.325
relative to government regulation.
645
00:53:52.010 --> 00:53:54.109
Right now, I think it's a shame that
646
00:53:54.569 --> 00:54:00.430
we really only offer 2 options. Right? Either you give me all your data and I do whatever I want with it,
647
00:54:00.985 --> 00:54:04.285
or you don't give me any data and you don't use my services.
648
00:54:06.025 --> 00:54:29.160
And I think that that's a very hard choice. We've seen the laws that try to come in like, I forget the name of it. But whatever the law that makes me click allow all cookies on every single website I visit, every time I visit it. And I I think very few of us ever say, well, I'm not gonna use this website because I had to click that button. And so really it was that a benefit to anyone that we made that change. So I think that if we can get into
649
00:54:29.780 --> 00:54:30.920
a world where
650
00:54:31.540 --> 00:54:32.200
our incentives
651
00:54:32.500 --> 00:54:35.240
are aligned with the data generators incentives,
652
00:54:35.954 --> 00:54:37.655
then we will see much more sensible
653
00:54:38.035 --> 00:54:38.535
regulation.
654
00:54:39.234 --> 00:54:39.734
Absolutely.
655
00:54:40.674 --> 00:54:47.059
And in your time working in industry and helping to build a lot of the systems that helped to
656
00:54:47.760 --> 00:54:52.500
catalyze the broader ecosystem and now being a consumer and integrator of those systems,
657
00:54:52.865 --> 00:54:55.765
what are some of the most interesting or innovative or unexpected
658
00:54:56.385 --> 00:55:00.405
data processes or data systems that you've encountered or had the privilege to build?
659
00:55:00.880 --> 00:55:04.900
Well, I think at least 1 of those for me is
660
00:55:05.280 --> 00:55:07.220
this notion of data
661
00:55:07.600 --> 00:55:13.045
observability, data testability, whatever you wanna call it. The idea that we're going to automate
662
00:55:13.984 --> 00:55:14.484
expectations
663
00:55:14.785 --> 00:55:19.365
about data and bring machine learning to kind of watch all of our tables
664
00:55:19.930 --> 00:55:20.910
and make sure
665
00:55:21.530 --> 00:55:23.470
that we know when things are changing.
666
00:55:23.930 --> 00:55:29.755
I think that's an innovative 1. The notion of the privacy vault that we talked about, I think that's really
667
00:55:30.295 --> 00:55:31.835
innovative and really important
668
00:55:32.214 --> 00:55:34.155
that we find a solution to that problem.
669
00:55:34.694 --> 00:55:43.420
Obviously, I think a lot of the stuff that we did on Dremel was very innovative at the time. But this industry, it just moves so fast that I think what we think is innovative
670
00:55:43.880 --> 00:55:46.140
today, we just end up taking for granted
671
00:55:46.535 --> 00:55:50.235
a few years from now. I go back and recruit at universities
672
00:55:50.615 --> 00:55:51.195
a lot.
673
00:55:51.655 --> 00:55:52.875
And over
674
00:55:53.415 --> 00:55:55.515
15, 16 years of doing this,
675
00:55:56.010 --> 00:55:58.590
it's just been really interesting to observe
676
00:55:58.970 --> 00:56:02.270
how much the world has changed and how much things that
677
00:56:02.890 --> 00:56:05.550
used to seem like the impossible blocker
678
00:56:05.915 --> 00:56:11.775
when I was there. Like network was always going to be a problem for distributed computing. And
679
00:56:12.234 --> 00:56:21.910
at some point in the last decade, it just became not a problem anymore. We've got more than enough of it. I have a hard time saying, I guess, what the innovative things are because
680
00:56:22.345 --> 00:56:30.525
it's all changing so fast that I will probably leave something out that really was innovative at the time. And now I just think it's commonplace.
681
00:56:31.510 --> 00:56:39.210
In terms of your experience of building and supporting these data systems, what are some of the most interesting or unexpected or challenging lessons that you've learned?
682
00:56:39.565 --> 00:56:44.945
I think the challenging lessons that I've seen are all people and business problems.
683
00:56:45.565 --> 00:56:46.945
The technical stuff,
684
00:56:47.420 --> 00:56:51.119
1 of my favorite engineers I worked with at Google had had a habit of
685
00:56:51.579 --> 00:57:03.345
saying it's software. We can do literally anything. Anytime someone would would say, well, I don't know if we can do that. He would his rejoinder would be his software. We can do absolutely anything we want. We just have to decide whether we want to do that.
686
00:57:03.930 --> 00:57:08.750
So the difficult things for me or the complex challenges are
687
00:57:09.610 --> 00:57:12.190
figuring out the structures of organizations,
688
00:57:12.734 --> 00:57:13.555
figuring out
689
00:57:14.335 --> 00:57:17.555
how we can get people all pushing in the same direction
690
00:57:18.015 --> 00:57:19.234
on our data problems
691
00:57:19.990 --> 00:57:24.490
and really make it easier to do the right thing than to do the wrong thing.
692
00:57:24.950 --> 00:57:27.690
Because if you make it hard for
693
00:57:28.515 --> 00:57:29.895
engineers to produce
694
00:57:30.195 --> 00:57:33.895
reliable analytic data or you make it hard for data scientists
695
00:57:34.195 --> 00:57:34.695
to
696
00:57:35.155 --> 00:57:35.655
consume
697
00:57:36.035 --> 00:57:36.935
from the
698
00:57:37.540 --> 00:57:38.040
canonical
699
00:57:38.740 --> 00:57:39.880
blessed tables,
700
00:57:40.660 --> 00:57:44.359
they'll just do something else. And then all your work on the technology
701
00:57:44.660 --> 00:57:45.160
side
702
00:57:45.565 --> 00:57:51.825
to try and make things good just goes out the window because the human beings are gonna do what they're incentivized
703
00:57:52.125 --> 00:57:52.785
to do.
704
00:57:53.250 --> 00:57:58.150
Are there any other aspects of your work on Dremel and at BigQuery and SoFi
705
00:57:58.450 --> 00:58:04.625
or any other forward looking pontifications that you'd like share that we didn't discuss yet that you'd like to cover before we close out the
706
00:58:04.925 --> 00:58:07.185
show? Yeah. I don't think so. I'm not a real pontificating
707
00:58:07.565 --> 00:58:10.625
kind of guy. You need other people for that. I think
708
00:58:11.090 --> 00:58:14.790
what I will say is we are at a great time
709
00:58:15.090 --> 00:58:15.590
in
710
00:58:15.890 --> 00:58:17.990
data analytics. The cost
711
00:58:18.705 --> 00:58:20.565
of launching new
712
00:58:21.025 --> 00:58:22.565
data focused businesses
713
00:58:23.345 --> 00:58:26.859
is as low as I've ever seen it with this competition
714
00:58:27.240 --> 00:58:29.180
from Snowflake and BigQuery.
715
00:58:30.359 --> 00:58:36.675
The number of tools that are out there is so great that I think we're really at a place where
716
00:58:37.455 --> 00:58:38.595
there's a big opportunity
717
00:58:38.895 --> 00:58:39.395
for
718
00:58:39.695 --> 00:58:41.555
some names to be made here
719
00:58:41.869 --> 00:58:46.290
and for a decade from now for us to be talking about the people who revolutionized
720
00:58:47.710 --> 00:58:56.485
data governance or data cataloging or data quality or or whatever it is. I'm just really excited to see where that goes. And I'm kind of excited to be
721
00:58:56.785 --> 00:59:04.020
now seeing it from the other side of the sideline, being the person who's using it rather than the person who's having to worry about building it.
722
00:59:04.480 --> 00:59:18.615
Alright. Well, for anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as a final question, I'd like to get your perspective on what you see as being the biggest gap in the tooling or technology that's available for data management today?
723
00:59:19.060 --> 00:59:25.320
The biggest 1 we are seeing right now is that privacy vault that we discussed or some comparable solution.
724
00:59:25.700 --> 00:59:27.160
But something that makes
725
00:59:27.665 --> 00:59:29.125
handling PII correctly
726
00:59:29.425 --> 00:59:31.285
easier than handling it incorrectly.
727
00:59:31.585 --> 00:59:33.925
That is the 1 I would love to see people solve.
728
00:59:34.385 --> 00:59:59.055
Well, thank you very much for taking the time today to join me and share the work that you've been doing over the past decade plus and the work that you're doing now. So I appreciate all the time and energy you've put into helping to grow the community and inspire the surrounding ecosystem and now being a consumer of said ecosystem. So thank you again for all of the time and energy you've put into that, and I hope you enjoy the rest of your day. Alright. Thanks, Tobias. Talk to you later.
729
01:00:04.315 --> 01:00:05.055
For listening.
730
01:00:05.355 --> 01:00:08.640
Don't forget to check out our other show, podcast dot in it@pythonpodcast.com
731
01:00:10.060 --> 01:00:14.480
to learn about the Python language, its community, and the innovative ways it is being used.
732
01:00:14.780 --> 01:00:16.160
And visit the site at dataengineeringpodcastdot
733
01:00:17.255 --> 01:00:27.040
com to subscribe to the show, sign up for the mailing list, and read the show notes. If you've learned something or tried out a project from the show, then tell us about it. Email hosts at data engineering podcast.com
734
01:00:27.500 --> 01:00:32.880
with your story. And to help other people find the show, please leave review on Itunes and tell your friends and coworkers.