An Introduction To Data And Analytics Engineering For Non-Programmers

WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 7/6/2024 1:41:13 PM
Duration: 3013.923
Channels: 1
1
00:00:13.554 --> 00:00:17.575
Hello, and welcome to the Data Engineering Podcast, the show about modern data management.
2
00:00:18.640 --> 00:00:25.779
When you're ready to build your next pipeline and want to test out the projects you hear about on the show, you'll need somewhere to deploy it. So check out our friends over at Linode.
3
00:00:26.644 --> 00:00:35.225
With our managed Kubernetes platform, it's now even easier to deploy and scale your workflows or try out the latest Helm charts from tools like Pulsar, Packaderm, and Dagster.
4
00:00:36.150 --> 00:00:43.370
With simple pricing, fast networking, object storage, and worldwide data centers, you've got everything you need to run a bulletproof data platform.
5
00:00:44.295 --> 00:00:45.915
Go to data engineering podcast.com/linode
6
00:00:47.415 --> 00:00:57.290
today. That's l I n o d e, and get a $100 credit to try out a Kubernetes cluster of your own. And don't forget to thank them for their continued support of this show.
7
00:00:58.470 --> 00:01:10.745
The only thing worse than having bad data is not knowing that you have it. With Bigeye's data observability platform, if there is an issue with your data or data pipelines, you'll know right away and can get it fixed before the business is impacted.
8
00:01:11.510 --> 00:01:16.810
Bigeye lets data teams measure, improve, and communicate the quality of your data to company stakeholders.
9
00:01:17.510 --> 00:01:20.145
With complete API access, a user friendly interface,
10
00:01:20.605 --> 00:01:25.905
and automated yet flexible alerting, you've got everything you need to establish and maintain trust in your data.
11
00:01:26.440 --> 00:01:28.140
Go to data engineering podcast.com/bigeye
12
00:01:29.480 --> 00:01:32.060
today to sign up and start trusting your analyses.
13
00:01:32.520 --> 00:01:46.840
Your host is Tobias Macy, and today I'm interviewing Brian McMillan about building data products in his book to introduce the work of data analysts and engineers to non programmers. So Brian, can you start by introducing yourself? Thanks for having me on. My name is Brian McMillan. And professionally,
14
00:01:47.220 --> 00:01:48.040
I'm a longtime
15
00:01:48.660 --> 00:01:51.000
enterprise architect working in large corporations
16
00:01:52.204 --> 00:01:55.664
and primarily been focused on data and analytics problems
17
00:01:56.365 --> 00:01:57.664
within those big companies.
18
00:01:58.579 --> 00:02:05.960
Even though EDS and HP are technology companies, a lot of what they're doing is just provide the enterprise services to others.
19
00:02:06.340 --> 00:02:12.055
And they're definitely old school companies. And then more recently, I'm working at a major defense contractor.
20
00:02:13.075 --> 00:02:15.250
I left my job in
21
00:02:15.570 --> 00:02:16.630
October 2020
22
00:02:17.250 --> 00:02:18.630
to write a book about
23
00:02:19.090 --> 00:02:20.230
building data products
24
00:02:20.610 --> 00:02:24.790
called Building Data Products, Data and Analytics Engineering for Non Programmers.
25
00:02:25.525 --> 00:02:28.745
And do you remember how you first got involved in the area of data management?
26
00:02:29.285 --> 00:02:30.665
Yeah. So my
27
00:02:31.045 --> 00:02:36.310
degree is in economics, not computer science. And I am a business guy.
28
00:02:36.610 --> 00:02:38.549
That's really colored a lot of my
29
00:02:39.170 --> 00:02:55.180
career going forward. So they have to go back to the mid nineties. I was a data analyst. It was my first real job. I was a data analyst working for electronic data systems in a General Motors manufacturing plant. And 1 of the things that was kind of unique to the business at that time is that they had actual
30
00:02:55.480 --> 00:02:57.660
the IT people were embedded in
31
00:02:58.145 --> 00:03:02.965
the customer's organization. So I got to participate in the daily management stand ups.
32
00:03:03.425 --> 00:03:06.165
My job was to really run their manufacturing war room.
33
00:03:06.864 --> 00:03:07.364
And
34
00:03:07.890 --> 00:03:12.550
the biggest thing was that they introduced me to the theory of constraints.
35
00:03:13.410 --> 00:03:16.870
It was an engine plant. They were going through, you know, they had old technology.
36
00:03:17.845 --> 00:03:25.545
GM was going through a lot of turmoil at that time, and the plant was always on the verge of being shut down. They decided to go all in on theory of constraints
37
00:03:26.644 --> 00:03:30.810
and packed on me because I saw, you know, basically a failing company
38
00:03:31.270 --> 00:03:35.530
turn themselves around, and it kept that plant open for almost another 10 years.
39
00:03:36.175 --> 00:03:38.595
Next 15 years are basically spent architecting,
40
00:03:39.055 --> 00:03:52.840
product managing, people managing, and hands on building a variety of enterprise data management systems for EDS, HP, and Raytheon. Just because you've asked us in a number of other podcasts, I've been working remotely for most of the last 15 years,
41
00:03:53.540 --> 00:03:55.160
which I think is pretty unique.
42
00:03:55.735 --> 00:03:59.915
So having to having to go, you know, COVID sending us all home was actually
43
00:04:00.215 --> 00:04:01.755
very pleasant for me.
44
00:04:02.135 --> 00:04:03.835
Yeah. Levels the playing field.
45
00:04:04.410 --> 00:04:22.350
I've worked remote off and on throughout my career. And so when everything went fully remote, just no ifs, ands, or buts about it. It was kinda nice for me because it gave me an excuse to go back to being full time remote instead of being primarily in the office with part time remote. Yep. And there are a lot of
46
00:04:22.670 --> 00:04:24.690
really significant downsides to that,
47
00:04:25.390 --> 00:04:28.290
but a lot of positives and, hey, we didn't have a choice. Exactly.
48
00:04:29.525 --> 00:04:33.065
And so as you mentioned, you recently wrote this book.
49
00:04:33.604 --> 00:04:38.905
You made the decision to quit your job to spend the time on it. So I'm wondering if you can talk to some of the
50
00:04:39.250 --> 00:04:44.470
motivation for deciding that this book was necessary, that this was the time to do it, and
51
00:04:44.850 --> 00:04:54.115
that you wanted to actually have that full time focus on it, and just some of the overall story behind how you came up with the idea, the motivation around it, and how it came to be?
52
00:04:54.574 --> 00:04:57.555
So when I started working for Raytheon,
53
00:04:58.010 --> 00:05:01.230
I didn't hire into the IT department. I hired into the quality department.
54
00:05:01.850 --> 00:05:03.790
And, you know, it was pretty clear
55
00:05:04.330 --> 00:05:09.854
when I hired on, they had a really big data problem that they were trying to solve. So that's so I came in to build a
56
00:05:10.235 --> 00:05:13.694
data warehouse for them for their quality data. And
57
00:05:14.634 --> 00:05:16.895
having done this a whole bunch of other times,
58
00:05:17.350 --> 00:05:25.530
this was the first time in a long time that I had actually been really hands on. You know, architects don't typically you know, there's a certain point in an architect's career where they're no longer
59
00:05:25.830 --> 00:05:30.155
allowed to touch things. That was pretty frustrating to me, not being able to you know, I was
60
00:05:31.095 --> 00:05:35.675
feeling like I wasn't able to touch things and work on things. So this is a great opportunity.
61
00:05:36.169 --> 00:05:41.630
They had had a lot of trouble with the IT department getting you know, the idea of getting a database
62
00:05:42.490 --> 00:05:44.110
for a business
63
00:05:45.095 --> 00:05:45.595
organization
64
00:05:46.134 --> 00:05:52.235
to do their own database work was not very popular. So they had managed to get themselves a database.
65
00:05:52.710 --> 00:05:57.770
They had some really good ideas of projects they wanted to work on and problems they wanted to solve,
66
00:05:58.230 --> 00:06:01.130
but they needed somebody to actually come in and
67
00:06:01.444 --> 00:06:12.910
do that work. So got hired in, started to do that work for them. It was pretty clear that it was way more work than 1 person could do, which is never a good idea anyways, but that's where you always start.
68
00:06:13.530 --> 00:06:18.990
I had got the opportunity to do a lot of training on that team with people who had never touched a database before.
69
00:06:19.925 --> 00:06:21.945
And they had good domain knowledge,
70
00:06:22.565 --> 00:06:24.665
but they didn't have the technical skills.
71
00:06:25.045 --> 00:06:26.745
That started a ball rolling.
72
00:06:27.169 --> 00:06:31.430
And I guess probably 1 of the biggest things that I learned was how important
73
00:06:32.130 --> 00:06:35.030
that domain knowledge really is, that domain expertise.
74
00:06:35.465 --> 00:06:42.205
So 1 of the first things we did was, well, what's our production yield? We don't have a good way to look at our production yields,
75
00:06:43.360 --> 00:06:49.860
and that should be pretty simple. We'll just go in. We'll look at the data from the warehouse, and we'll, you know, just write some reports.
76
00:06:50.514 --> 00:06:55.335
That wasn't what it turned out to be at all. And what we quickly found out was that they had
77
00:06:56.115 --> 00:07:07.285
hundreds of parts, that they were serialized parts that they were recycling through the production process. So you go to do your recursive query to figure out what your bill of materials looks like, and you can't do it because you've got loops.
78
00:07:08.085 --> 00:07:10.745
So, you know, as a technical person,
79
00:07:11.285 --> 00:07:12.745
what you do is you'd start
80
00:07:13.205 --> 00:07:15.305
chasing all these rabbit trails
81
00:07:15.680 --> 00:07:19.780
to try to figure out how could this possibly be happening because this doesn't make sense.
82
00:07:20.320 --> 00:07:24.340
Instead, we were really lucky on that team. We had somebody who had been in manufacturing
83
00:07:24.640 --> 00:07:26.884
for, oh, jeez, let's just say decades,
84
00:07:27.504 --> 00:07:31.205
and he had a story for everything. So I learned the story of Swaptronics.
85
00:07:32.224 --> 00:07:36.020
So unlike a lot of businesses, you know, Raytheon, they are rocket scientists,
86
00:07:37.520 --> 00:07:38.020
and
87
00:07:38.400 --> 00:07:41.460
all of the work that they do is just bleeding edge.
88
00:07:41.914 --> 00:07:43.935
And the sensor systems in particular
89
00:07:44.875 --> 00:07:56.430
are tough to test, so they fail Tesla on. They pull it out. They put it back on the shelf. It eventually goes back in another device, and sometimes it matches up with the rest of the components and sometimes it doesn't.
90
00:07:57.285 --> 00:08:09.040
I never would have known that that's what was going on unless I had talked to someone who had been on the plant floor and said, oh, yeah. We do this all the time. It's no problem. Well, it is a big problem, and we need to stop doing it.
91
00:08:09.500 --> 00:08:13.600
That pivoted our whole work to trying to figure out how to solve that problem.
92
00:08:14.539 --> 00:08:15.039
And
93
00:08:15.595 --> 00:08:17.455
it became very clear
94
00:08:18.235 --> 00:08:22.895
that domain knowledge is absolutely the most important thing you can do
95
00:08:23.970 --> 00:08:34.145
to solve the problems you're at. Domain knowledge is the most important thing you could have in order to solve business problems you have. There's no way around it. You can't algorithm your way out of it.
96
00:08:34.525 --> 00:08:37.505
The other thing that I motivated me was
97
00:08:37.965 --> 00:08:41.899
I just couldn't believe it literally took that team 7 years to get
98
00:08:42.279 --> 00:08:42.940
a production
99
00:08:43.800 --> 00:08:45.019
SQL Server database
100
00:08:45.720 --> 00:08:49.180
on the network that they had access to, but it didn't come with any
101
00:08:49.655 --> 00:08:50.475
ETL tools.
102
00:08:50.855 --> 00:08:52.155
Like, what good is that?
103
00:08:52.535 --> 00:08:57.035
So the big thing that prompted me is I've been in this business a long time. And as an industry,
104
00:08:57.600 --> 00:09:04.980
I would like to think that at some point we'd be ready to come to terms with the fact that we keep doing the same things over and over again.
105
00:09:05.405 --> 00:09:07.025
We keep reinventing the wheel.
106
00:09:07.485 --> 00:09:09.265
Most of our projects still fail,
107
00:09:10.045 --> 00:09:10.545
and
108
00:09:11.325 --> 00:09:18.060
we tend to collect a lot of data that we don't know what to do with just because it's fairly easy to collect a lot of data.
109
00:09:18.759 --> 00:09:20.139
But that ends up being
110
00:09:20.519 --> 00:09:22.175
generated a lot of technical debt.
111
00:09:22.575 --> 00:09:23.235
And then
112
00:09:23.935 --> 00:09:26.195
for enterprise companies in particular,
113
00:09:26.815 --> 00:09:29.075
their whole operational model is all centralized.
114
00:09:29.540 --> 00:09:37.080
You know, we have centralized IT departments, and you go to the data model, and you go to the ETL person, and you go to the report developers. No. We've gotta have
115
00:09:37.415 --> 00:09:38.235
project managers
116
00:09:38.535 --> 00:09:39.915
managing the whole thing,
117
00:09:40.455 --> 00:09:42.635
and it just doesn't work.
118
00:09:43.175 --> 00:09:45.675
And outside of big enterprise companies,
119
00:09:46.410 --> 00:09:48.110
we are doing things to
120
00:09:48.810 --> 00:09:51.630
fix that problem. But there's a huge opportunity
121
00:09:52.170 --> 00:09:55.635
inside big enterprise companies to solve that. It's definitely interesting how,
122
00:09:56.415 --> 00:10:05.529
you know, the current buzzword is the modern data stack of everything as a service. It's easy to just get a new database. You just throw a credit card at it, but that's only the case for,
123
00:10:06.070 --> 00:10:19.505
you know, a certain subset of the industry. And as you pointed out, in the enterprise, yeah, you have these procurement paths. Like, you can't just throw a credit card at something because that credit card is, you know, being held by a gatekeeper that has, stockade of paperwork to
124
00:10:19.885 --> 00:10:21.085
fend you off. And so
125
00:10:22.210 --> 00:10:25.510
yeah. That's a big nut to crack. I don't know.
126
00:10:26.130 --> 00:10:28.070
I don't know how to solve that problem.
127
00:10:28.530 --> 00:10:30.630
I mean, I know how to solve it in.
128
00:10:31.315 --> 00:10:42.360
I know how to solve it in a subversive way. Quite frankly, that was 1 of the motivators of the book. There is a bit of subversion in the book. Like, here's a whole bunch of free software that you can do that you can implement
129
00:10:43.060 --> 00:10:43.720
to do
130
00:10:44.020 --> 00:10:46.600
basically everything you'd wanna do. Orchestration,
131
00:10:47.755 --> 00:10:48.255
serving,
132
00:10:49.035 --> 00:10:49.535
storage,
133
00:10:50.315 --> 00:10:54.014
you name it. It's all in here. You can do that if you want. You
134
00:10:54.510 --> 00:10:56.050
may or may not wanna do that.
135
00:10:56.750 --> 00:11:19.790
Sometimes you have no choice. Right. Yeah. I mean, there's definitely the double edged sword of shadow IT of, k. You know, these people are unblocked. They're able to get their job done, but they're not necessarily doing it in the most effective way or they're, you know, reinventing their own wheel that's already been solved by somebody else in the organization. And so there is that problem of being able to connect up all the people who have the right problems and solutions.
136
00:11:20.475 --> 00:11:23.135
Yeah. I lucked out as an architect.
137
00:11:23.995 --> 00:11:33.660
1 of the side jobs I've always had is go find the shadow IT teams. Go find them, find out what they're doing, and decide what we should do about them. And a lot of times, it's
138
00:11:33.960 --> 00:11:43.475
we're gonna give you funding. We're gonna give you additional resources to scale your solution up. Yep. You've got a great start of day server monitoring platform
139
00:11:43.935 --> 00:11:44.755
in Australia.
140
00:11:45.350 --> 00:11:51.370
We're going to take that, but we're gonna have to rewrite the entire thing, and you get a central role to help build it.
141
00:11:52.070 --> 00:11:55.065
That isn't what a lot of IT organizations
142
00:11:55.444 --> 00:11:55.944
do.
143
00:11:56.245 --> 00:11:58.665
The first position is almost always
144
00:11:59.365 --> 00:12:04.709
shut these teams down, make it harder for them so that they can bring us requirements,
145
00:12:05.410 --> 00:12:06.870
and we'll work on those requirements
146
00:12:07.329 --> 00:12:17.225
and rebuild their thing for them. But that isn't what they want. They just want to, you know, meet the business problems they have. And, generally, they have a very good idea of what their business problems are.
147
00:12:17.685 --> 00:12:28.180
As IT folks, we we tend not to know what the real business problems are we should be focusing on because we're focused on technology, because we like bright shiny things. Yeah. And I think that that's probably why
148
00:12:28.715 --> 00:12:30.175
the data ecosystem
149
00:12:30.555 --> 00:12:34.175
has been going through such a long and cyclical
150
00:12:34.875 --> 00:12:37.135
route of self discoveries because
151
00:12:37.650 --> 00:12:46.550
it's never just the technical solution, and it's never just the business problem. And it's always hard to get both of those sides in the same room and agreeing
152
00:12:46.895 --> 00:13:00.569
with each other and speaking even in the same language. So I think that that's probably why we keep going through these, oh, well, we'll build this new evolution of this technology platform, and that's going to solve our problems. And, like, nope. Still have the same problem.
153
00:13:01.670 --> 00:13:02.490
Yeah. Absolutely.
154
00:13:02.870 --> 00:13:03.529
You know,
155
00:13:04.069 --> 00:13:04.569
enterprise
156
00:13:05.625 --> 00:13:06.605
application integration.
157
00:13:07.385 --> 00:13:09.485
Well, it smells a lot like Kafka.
158
00:13:10.024 --> 00:13:10.524
Absolutely.
159
00:13:11.545 --> 00:13:28.555
And now that we have Kafka, we're still running into the same problems of, okay. Well, we just put everything into this service bus, but now we don't actually know everything that's using it. Or, oh, we just broke the contract for this data structure because it was being used by this other service that is actually mission critical now.
160
00:13:29.255 --> 00:13:43.485
Yeah. Yeah. It always has been and probably always will be. And then there's that pendulum between centralized and decentralized. Like, you know, right now, the big thing is, you you know, so there's the modern data stack, the 100 vendors that are inside of that,
161
00:13:44.025 --> 00:13:47.405
and all the overlap and whatnot. And then you have the decentralized pendulum
162
00:13:47.709 --> 00:13:48.450
swinging pretty hard,
163
00:13:50.430 --> 00:13:58.785
and, you know, with things like data mesh. Probably talk later about that. But that centralized decentralized pendulum has always been swinging around.
164
00:13:59.325 --> 00:14:03.584
And, you know, it feels to me like right now, that pendulum is, like, kind of peaked
165
00:14:04.180 --> 00:14:09.079
or close to peaking, and a lot of the modern data stack vendors are starting to get big
166
00:14:09.779 --> 00:14:13.485
and start to be more centralized. And I think that's 1 of the things we're gonna see is
167
00:14:14.125 --> 00:14:14.625
some
168
00:14:15.165 --> 00:14:19.505
pretty aggressive consolidation in the business. Yeah. Absolutely. And another interesting
169
00:14:19.885 --> 00:14:20.385
trend
170
00:14:20.740 --> 00:14:35.144
is the repackaging of the modern data stack with companies such as Mozart Data of okay. So you've got 5 different tools that you need to use to do to this 1 thing. So we're just going to be that 1 bill for you, and we're gonna run those 5 tools on your behalf.
171
00:14:35.925 --> 00:14:36.425
Yeah.
172
00:14:36.964 --> 00:14:38.425
Which I think to
173
00:14:39.520 --> 00:14:40.500
enterprise customers
174
00:14:41.120 --> 00:14:43.300
is gonna be very appealing. Absolutely.
175
00:14:43.839 --> 00:14:48.180
Because the people who are buying those systems are not the people who are actually going to use them.
176
00:14:48.995 --> 00:14:52.695
So that value proposition that we can take this complex,
177
00:14:53.475 --> 00:14:54.695
intertwined architecture
178
00:14:55.154 --> 00:15:01.890
and package it up, and we can guarantee you, knock on wood, that everything works together nicely.
179
00:15:02.510 --> 00:15:03.885
But, again, the problem is
180
00:15:04.445 --> 00:15:07.025
you've got all these tools that people need to learn.
181
00:15:07.405 --> 00:15:08.145
You've got
182
00:15:08.605 --> 00:15:10.785
business problems that you need to try to solve.
183
00:15:11.405 --> 00:15:15.420
And all of that's very difficult to deal with. Absolutely.
184
00:15:15.720 --> 00:15:24.895
And so in terms of the book itself, I'm wondering as you were setting out to write it, what were you keeping in mind as your target audience and the
185
00:15:25.275 --> 00:15:29.295
primary goals that you were trying to achieve and help them realize
186
00:15:29.850 --> 00:15:34.430
through the creation of this book and the kind of core lessons that you're trying to impart?
187
00:15:34.890 --> 00:15:38.830
Well, 1 is, you know, architects at traditional IT departments,
188
00:15:40.235 --> 00:15:44.415
you know, used to the typical traditional IT stack, you know, okay, we have
189
00:15:45.115 --> 00:15:47.535
Oracle databases or Microsoft databases.
190
00:15:48.020 --> 00:15:49.240
We're using Informatica
191
00:15:49.700 --> 00:15:51.160
or integration services.
192
00:15:51.860 --> 00:15:56.120
We may have stood up, you know, if you're a Microsoft stack, you know, analysis services,
193
00:15:56.900 --> 00:16:01.105
we've got a whole bunch of other web apps that we may have built internally.
194
00:16:02.285 --> 00:16:07.265
If you're fortunate enough to have people who are doing application development in your organization,
195
00:16:08.279 --> 00:16:12.139
they probably switched a long time ago to building things, you know,
196
00:16:12.759 --> 00:16:13.259
in
197
00:16:13.560 --> 00:16:16.300
a 30 year old but modern software development.
198
00:16:16.714 --> 00:16:18.334
They're doing continuous integration,
199
00:16:18.714 --> 00:16:22.574
continuous deployments. They're test driven development. All of those
200
00:16:23.194 --> 00:16:29.759
things that help you build a better software product, They're doing that. But on the data side,
201
00:16:30.220 --> 00:16:51.875
we tend to not do that. We gave up on testing data a long time ago. Just too complicated. We can't do it. You know, if the file doesn't show up and the dashboard breaks, well, then we'll go fix the somebody will call us. The executive will call us and say the dashboard hasn't updated. Where's the fix? Then we'll go fix it. And we really shouldn't be behaving that way. Know, people figure these problems out.
202
00:16:52.654 --> 00:16:53.154
So
203
00:16:53.615 --> 00:16:54.675
architects need
204
00:16:55.055 --> 00:16:58.035
to know that there are alternate ways of doing things.
205
00:16:58.690 --> 00:17:01.510
And I'm presenting to the book kind of a ridiculous,
206
00:17:02.610 --> 00:17:03.990
minimum viable product
207
00:17:04.290 --> 00:17:09.005
that's got the entire stack. You know, if we're building in the technical part of the book, we're building a little cupcake
208
00:17:09.865 --> 00:17:11.485
for a data product,
209
00:17:11.945 --> 00:17:18.220
looking at sales data. And the data is actual sales data from a company in the UK. It's very messy data.
210
00:17:18.919 --> 00:17:19.419
And
211
00:17:20.120 --> 00:17:21.340
at the end of the day,
212
00:17:21.960 --> 00:17:25.340
that data gets exposed as APIs and a web GUI
213
00:17:26.035 --> 00:17:28.215
on, you know, Google's cloud infrastructure.
214
00:17:28.675 --> 00:17:35.095
And you can do that. And it's not really that complicated. And you can treat everything as code, and you don't need to resort to gooey applications.
215
00:17:36.060 --> 00:17:46.955
And that may not be where you're at right now, but you need to start thinking about that. The second target, I've talked briefly about this before, are those shadow IT teams. You know, the teams who are just fed up, and
216
00:17:47.415 --> 00:17:49.035
maybe they need to learn some
217
00:17:49.895 --> 00:17:57.470
better tools to really take it to the next level. And then maybe the IT departments will come along and help them out.
218
00:17:57.850 --> 00:17:59.550
In terms of the
219
00:18:00.090 --> 00:18:06.515
contents of the book, you mentioned that you're working with real sales data. You're iterating through building out a
220
00:18:07.135 --> 00:18:23.725
small scoped data product from that information. I'm wondering if you can talk to the overall approach that you took for deciding what was the structure of the book, what were the kind of main technological choices you were going to lean on for determining this is how I'm going to impart these core lessons
221
00:18:24.265 --> 00:18:38.924
and some of the ways that you are able to talk through these are the technologies that we're using, but these are the actual fundamental principles that we really care about here. So, you know, not to get too bogged down into the specifics of tool x or y. Well, that was the first thing. Don't
222
00:18:39.304 --> 00:18:45.085
try to not make it too tool centric. 1 thing I didn't wanna do is I didn't wanna make this a DBT snowflake
223
00:18:45.700 --> 00:18:46.200
text.
224
00:18:47.700 --> 00:18:51.960
So the start of it was really getting back to those core business concepts,
225
00:18:52.740 --> 00:18:54.360
starting with things like
226
00:18:54.715 --> 00:18:55.855
product life cycle.
227
00:18:56.155 --> 00:18:57.615
All products go through
228
00:18:57.915 --> 00:19:01.215
this s curve product life cycle. They start out in,
229
00:19:01.650 --> 00:19:04.390
and I'm using, Kent Beck's model here, 3 x,
230
00:19:04.690 --> 00:19:09.670
that he says he developed at Facebook, but, you know, it's in the extreme programming book already.
231
00:19:11.165 --> 00:19:16.145
Yep. Things start out in exploration. In an exploration, you don't know what's going to work.
232
00:19:16.525 --> 00:19:20.145
You don't know what's gonna bring value. So what you should be trying to do is
233
00:19:20.740 --> 00:19:22.200
run lots of little experiments.
234
00:19:22.820 --> 00:19:28.520
And as you get traction on real business value, do the next step, then do the next step. And, eventually,
235
00:19:28.955 --> 00:19:33.055
you'll start to make this transition where you're you're expanding.
236
00:19:33.835 --> 00:19:39.100
You're bringing on more users. You're delivering more value. The products getting more complicated.
237
00:19:39.640 --> 00:19:40.299
And then
238
00:19:40.679 --> 00:19:44.940
you make another transition at some point where the value starts to level off.
239
00:19:45.375 --> 00:19:47.715
And you're in what he calls extract,
240
00:19:48.414 --> 00:19:50.035
where you're really in
241
00:19:50.575 --> 00:19:51.955
a process of
242
00:19:52.414 --> 00:19:54.355
making a good tool better,
243
00:19:54.820 --> 00:19:55.560
not necessarily
244
00:19:56.260 --> 00:19:57.480
making big improvements
245
00:19:58.100 --> 00:20:01.080
to it, but you're just making small incremental improvements
246
00:20:01.940 --> 00:20:03.480
and starting to
247
00:20:04.145 --> 00:20:09.605
just extract whatever value you can out of that. You're not really investing a whole lot of new
248
00:20:09.905 --> 00:20:20.120
into the product, but you're just getting the most out of it. And then the thing that isn't in the 3 x cycle is the exit phase, which is critically important. At some point, you have to start winding that product down.
249
00:20:20.655 --> 00:20:22.995
Hopefully, it's gonna be replaced by something else.
250
00:20:23.855 --> 00:20:28.435
Or your environment may say, you know, we don't need this thing anymore, and it drops off a cliff.
251
00:20:29.400 --> 00:20:34.860
So that's the first thing people need to understand. And that product lifecycle, if you know where you
252
00:20:35.400 --> 00:20:37.820
prefer to sit, you know, if you're a builder,
253
00:20:38.205 --> 00:20:42.305
and you like challenges, and you like fighting fires, you know, you're an expander.
254
00:20:42.925 --> 00:20:46.145
If you're somebody who's more concept based, you're probably an explorer.
255
00:20:46.850 --> 00:20:50.310
And there are plenty of people who just wanna make things run like clockwork.
256
00:20:50.850 --> 00:20:56.755
That's the first big business concept. The second 1 is that you need to understand how your company makes money.
257
00:20:57.235 --> 00:21:02.375
You need to understand the value chain, and this is probably where all these projects and teams need to start.
258
00:21:02.755 --> 00:21:04.935
Do we know how our company makes money?
259
00:21:05.340 --> 00:21:07.120
And a great example of that was
260
00:21:07.580 --> 00:21:10.000
when HP and EDS merged,
261
00:21:10.379 --> 00:21:13.815
I was the architect for the availability capacity organization, and
262
00:21:14.375 --> 00:21:15.835
both companies were
263
00:21:16.774 --> 00:21:17.274
supporting
264
00:21:17.575 --> 00:21:21.835
about 200, 000 servers a piece, you know, each. So we were gonna
265
00:21:22.270 --> 00:21:26.530
put 400, 000 servers worth of performance data into a single data warehouse,
266
00:21:27.550 --> 00:21:28.050
and
267
00:21:28.590 --> 00:21:40.495
it wasn't working very well. We had lots of arguments about how to do that and what was important. And the key came down to understanding that value chain. And as it turns out,
268
00:21:41.070 --> 00:21:41.809
about 90%
269
00:21:42.190 --> 00:21:45.090
of electronic data systems business was in
270
00:21:45.710 --> 00:21:51.365
hands on capacity management. We had people looking at server performance and making recommendations
271
00:21:51.905 --> 00:21:53.125
for how you should
272
00:21:53.425 --> 00:21:54.245
manage your
273
00:21:54.625 --> 00:21:55.765
servers, where HP
274
00:21:56.065 --> 00:21:57.125
was only 10%
275
00:21:57.940 --> 00:21:58.840
advanced management.
276
00:21:59.300 --> 00:22:00.840
Totally different business models.
277
00:22:01.220 --> 00:22:03.240
The solutions in those 2 cases
278
00:22:03.860 --> 00:22:05.800
are going to have to be different solutions.
279
00:22:06.375 --> 00:22:08.075
The money is being made differently,
280
00:22:08.615 --> 00:22:10.955
and it behooves you to
281
00:22:11.975 --> 00:22:13.595
figure that out and treat
282
00:22:13.960 --> 00:22:14.460
those
283
00:22:14.919 --> 00:22:16.539
2 value chains separately.
284
00:22:17.159 --> 00:22:17.980
The advanced
285
00:22:18.280 --> 00:22:21.100
level of that is starting to look at things like wordly maps
286
00:22:21.565 --> 00:22:23.665
and breaking that value chain down into
287
00:22:24.045 --> 00:22:29.105
the product life cycle phase they're in and treat those pieces of your solution differently.
288
00:22:29.680 --> 00:22:33.620
Yeah. Wordly maps are fantastic. And then the 3rd business concept is
289
00:22:34.400 --> 00:22:38.820
that just like products have life cycles, companies have financial life cycles.
290
00:22:39.145 --> 00:22:44.205
Like a start up, your job is to just figure out how to get somebody to pay you something.
291
00:22:44.585 --> 00:22:46.525
May not be the product you just wanna build,
292
00:22:47.260 --> 00:22:49.519
probably won't be the product you intended to build.
293
00:22:49.820 --> 00:22:52.080
But your job is find a way to make money.
294
00:22:52.380 --> 00:22:54.240
And then eventually, you get customers,
295
00:22:54.784 --> 00:22:57.125
You're profitable, but your profits are fluctuating.
296
00:22:57.505 --> 00:23:00.885
And there are different metrics that you can look at. The either
297
00:23:01.424 --> 00:23:04.005
the academic name for that is throughput economics.
298
00:23:04.580 --> 00:23:12.280
There's some good work in there from the theory constraints folks about what metrics you should be measuring. You know, when is it appropriate to do things like
299
00:23:13.060 --> 00:23:13.560
measure
300
00:23:14.385 --> 00:23:15.525
your throughput
301
00:23:16.225 --> 00:23:19.924
and when does it not? You know, and throughput's kinda central to theory constraints,
302
00:23:20.545 --> 00:23:28.450
but it's not always appropriate to measure that. So that's on the business side, and that's really the first third of the book. And then the second third of the book is about
303
00:23:29.710 --> 00:23:32.130
demonstrating some key technical skills,
304
00:23:32.645 --> 00:23:34.505
how to write some basic stories,
305
00:23:34.885 --> 00:23:39.065
not do a lot of planning or do little to no planning, write some basic stories,
306
00:23:39.365 --> 00:23:46.890
get familiar with command line tools. You know, most, particularly in traditional IT departments, were very GUI application focused,
307
00:23:47.429 --> 00:23:53.054
and we are terrified of the command line. But there are a lot of great utilities out there
308
00:23:53.674 --> 00:23:57.054
that just require you to write, you know, 5 words,
309
00:23:57.595 --> 00:23:58.735
and you can be deployed
310
00:23:59.275 --> 00:24:00.414
on Google's infrastructure.
311
00:24:00.880 --> 00:24:06.900
It's really amazing. And once you start to see that, you go, oh, wait a second. I can work in a completely different way.
312
00:24:07.200 --> 00:24:08.340
SQL, of course,
313
00:24:08.765 --> 00:24:14.065
there's been a renewed realization that just plain chain SQL gets you a lot a long way
314
00:24:14.605 --> 00:24:15.585
into your problems.
315
00:24:16.190 --> 00:24:19.090
Well, yeah, I'll I'll say that into your problems.
316
00:24:20.750 --> 00:24:21.250
Yes.
317
00:24:22.030 --> 00:24:25.915
It brings some problems with it. And it, you know, it can be used inappropriately.
318
00:24:26.615 --> 00:24:27.915
Automation and orchestration.
319
00:24:28.615 --> 00:24:39.950
You know, that's a big problem, particularly for like the shadow IT teams. Yeah. Great. We built this great thing. You know, we've got a great data model. We've got beautiful reports, but I still need to come in on Monday and push buttons
320
00:24:40.375 --> 00:24:45.115
and babysit things. You don't need to do that. Like in the book, I use make
321
00:24:46.135 --> 00:24:47.595
and use make file for everything.
322
00:24:48.419 --> 00:24:51.480
And that 1 hung around in my head for a long time, you know,
323
00:24:51.860 --> 00:24:57.635
watching a presentation on airflow years ago. And the description was, well, it's just like make for data.
324
00:24:58.035 --> 00:24:58.855
That's interesting.
325
00:25:00.035 --> 00:25:16.005
You know, I've compiled data with make. I've compiled programs with make, but I never really thought about using it for data. It's awesome for that. The syntax is easy. You know, if everything you're doing is in command line, you're just stringing a bunch of command lines together without having to write some nasty
326
00:25:16.545 --> 00:25:19.205
Python script or some nastier bash script,
327
00:25:19.585 --> 00:25:25.789
just really clean. And then 1 thing that I don't really talk about in the book, but critically important is version control.
328
00:25:26.409 --> 00:25:28.590
You've got your code in version control.
329
00:25:29.929 --> 00:25:31.710
Stop putting it on file servers.
330
00:25:32.325 --> 00:25:34.345
Yes. I know you are doing that.
331
00:25:35.044 --> 00:25:37.065
Stop it. Don't do that.
332
00:25:37.365 --> 00:25:38.745
You know, even if it's just
333
00:25:39.125 --> 00:25:47.870
check-in and sync up, start doing, you know, do trunk mail, do trunk based development because when you get to CICD, it'll make that a lot easier. Those kind of things,
334
00:25:48.725 --> 00:25:49.225
they're
335
00:25:49.525 --> 00:25:52.425
new concepts to most, you know, enterprise
336
00:25:53.445 --> 00:25:55.945
data people. How do you treat a tableau
337
00:25:56.870 --> 00:26:00.410
a tableau report as text or as code? You can't.
338
00:26:00.790 --> 00:26:01.450
It's hard.
339
00:26:04.405 --> 00:26:10.185
Today's episode is sponsored by prophecy dot I o, the low code data engineering platform for the cloud.
340
00:26:10.565 --> 00:26:17.290
Prophecy provides an easy to use visual interface to design and deploy data pipelines on Apache Spark and Apache Airflow.
341
00:26:17.910 --> 00:26:21.210
Now all the data users can use software engineering best practices.
342
00:26:21.670 --> 00:26:25.835
Git tests and continuous deployment with a simple to use visual designer.
343
00:26:26.215 --> 00:26:35.809
How does it work? You visually design the pipelines and prophecy generates clean Spark code with tests and stores it in version control. Then you visually schedule these pipelines on Airflow.
344
00:26:36.190 --> 00:26:39.935
You can observe your pipelines with built in metadata search and column level lineage.
345
00:26:40.895 --> 00:26:53.470
Finally, if you have existing workflows in AB Initio, Informatica, or other ETL formats that you want to move to the cloud, you you can import them automatically into prophecy, making them run productively on Spark. Learn more at dataengineeringpodcast.com/prophecy.
346
00:26:57.835 --> 00:27:00.335
Given the fact that you are trying to cover these
347
00:27:00.875 --> 00:27:01.934
technical practices
348
00:27:02.635 --> 00:27:07.455
and keep it rooted in the business requirements, and I'm just wondering,
349
00:27:07.980 --> 00:27:08.960
you know, as technologists,
350
00:27:09.340 --> 00:27:21.674
we are often kind of tainted by the hubris of, oh, well, this is technology and it's hard, so I'm not going to kind of expose that level of detail to the business users. I'm just going to try to
351
00:27:22.054 --> 00:27:30.630
make it as pretty as possible so that you know, give them some pretty pictures so that they can make their own decisions, and, you know, we'll use that as the handoff. And I'm wondering what you see as being the kind
352
00:27:30.985 --> 00:27:32.365
of realistic expectations
353
00:27:32.745 --> 00:27:35.565
for business users to actually adopt the
354
00:27:35.945 --> 00:27:36.445
core
355
00:27:36.905 --> 00:27:37.405
technological
356
00:27:37.785 --> 00:27:38.285
tools
357
00:27:38.585 --> 00:27:44.590
and approaches that we as engineers have become accustomed to and vice versa of
358
00:27:45.130 --> 00:27:47.150
both getting engineers to actually
359
00:27:47.774 --> 00:27:53.075
understand the various business concepts and the economic modeling and all of these
360
00:27:53.934 --> 00:28:03.510
more sort of process and organizational concerns beyond the realm of the technologies and tools that we're using to make bits fly around the ether?
361
00:28:04.625 --> 00:28:05.524
Yeah. Well,
362
00:28:05.904 --> 00:28:12.725
by far, the hardest thing is to get technical people to understand the business because the business people have enough trouble with it themselves.
363
00:28:13.210 --> 00:28:16.830
If you're in the marketing department, you're focused on marketing things.
364
00:28:17.929 --> 00:28:19.549
You're not focused on,
365
00:28:20.065 --> 00:28:23.285
is it possible to build the thing that you're trying to sell?
366
00:28:23.665 --> 00:28:28.565
So business people have this trouble as well. But the more people who could understand,
367
00:28:29.389 --> 00:28:31.009
you know, how to think about
368
00:28:31.309 --> 00:28:32.610
the value chains
369
00:28:32.990 --> 00:28:42.975
holistically for the company you're in, the better off everyone will be. It will be much easier to have conversations about what's valuable or not because we spend a lot of time suboptimizing
370
00:28:43.275 --> 00:28:43.775
things
371
00:28:44.475 --> 00:28:48.335
and just making busy work for ourselves because we have to be busy. Right?
372
00:28:49.510 --> 00:28:52.169
So getting technical people understand the business is
373
00:28:52.470 --> 00:28:53.690
first the most difficult.
374
00:28:53.990 --> 00:28:57.370
Getting the business focused people to understand the technical stuff,
375
00:28:57.725 --> 00:29:03.745
you have to keep your eyes open. There are a lot of people who a lot of analysts who are really good SQL developers
376
00:29:04.365 --> 00:29:06.545
or could be if you had taught them
377
00:29:07.120 --> 00:29:09.460
common table expressions and window functions.
378
00:29:09.920 --> 00:29:13.060
There are lots of analysts out there, you know, writing their Tableau reports
379
00:29:13.520 --> 00:29:14.020
who
380
00:29:14.394 --> 00:29:15.294
can write subqueries
381
00:29:15.995 --> 00:29:26.440
and, you know, they can write the Getty code subquery thing. If you just spend 5 minutes teaching them how to do a common table expression and pop that logic out into an independent piece that they can eyeball
382
00:29:26.900 --> 00:29:28.280
and maybe someday test,
383
00:29:29.299 --> 00:29:32.360
they will be ecstatic because it now gives them
384
00:29:32.684 --> 00:29:35.105
some confidence to start doing other things.
385
00:29:35.645 --> 00:29:41.825
If you can get people comfortable with SQL and you can get people comfortable with checking it in
386
00:29:42.210 --> 00:29:49.430
to a centralized repository that's shared with everybody else. You know, try to do some kind of teaming, you know, pair programming,
387
00:29:49.730 --> 00:29:58.895
that kind of thing so that the knowledge is spread around. Eventually, you're gonna find someone who, you know, really wants to know how to write Python and R,
388
00:29:59.300 --> 00:30:10.044
and they've got the basics on. Now you could get a case where you've got someone who knows how to get the data to use that they wanna use, and now they have the technical skills to start doing more data science y type things.
389
00:30:10.424 --> 00:30:14.684
Lord knows there are a lot of data scientists who can't get data out of databases,
390
00:30:15.145 --> 00:30:18.260
which blows my mind, but it happens a lot.
391
00:30:18.560 --> 00:30:31.945
They're just not comfortable with it. So it goes both ways, which gets to a bigger question about you have to start with what you have. A big part of data literacy is just trying to figure out where you are right now. What skills do you have?
392
00:30:32.485 --> 00:30:33.225
What capabilities
393
00:30:34.680 --> 00:30:37.660
could you get the team up to quickly?
394
00:30:38.120 --> 00:30:39.260
That's a hard problem.
395
00:30:39.720 --> 00:30:41.980
That's really the tricky part, data literacy.
396
00:30:42.375 --> 00:30:45.835
Yeah. It's definitely the interesting aspect of it is,
397
00:30:46.295 --> 00:30:49.355
you know, beyond just the technical bits is understanding
398
00:30:49.655 --> 00:30:50.475
how do you
399
00:30:50.840 --> 00:31:00.380
create and propagate context of the information that you're actually using? How do you understand the kind of statistical and semantic elements of manipulating the data?
400
00:31:00.745 --> 00:31:08.205
What are the, you know, downstream impacts of these mutations as far as what you can and can't do with it after you've made them?
401
00:31:08.580 --> 00:31:12.360
So it's definitely a large and complicated sea of
402
00:31:12.980 --> 00:31:16.600
complexity and concepts no matter what your background is.
403
00:31:17.125 --> 00:31:19.625
Yeah. I think the big thing is there aren't enough
404
00:31:20.325 --> 00:31:21.705
data people to go around,
405
00:31:22.245 --> 00:31:25.625
and we have to make more data people. Absolutely. Yeah.
406
00:31:26.330 --> 00:31:32.669
Yeah. That's really what it boils down to. There's a presentation, and I referenced this presentation in another interview recently
407
00:31:33.394 --> 00:32:04.680
by Jez Humble from, you know, early on in the process of the DevOps adoption saying, you know, stop trying to hire your DevOps people and create them instead. There's something to that effect. And we're we're definitely in that same phase with data where we're not gonna hire our way out of these problems. We have to start educating everybody who's already working on the problem to understand it more thoroughly so that they can, you know, do the things that need to get done rather than trying to hire the next, you know, data scientist or data engineer who already has all the skills,
408
00:32:05.140 --> 00:32:08.440
you know, but in a tool that we're not actually using right now.
409
00:32:08.980 --> 00:32:12.165
Yeah. That's a great point. You know that these DevOps practices
410
00:32:12.705 --> 00:32:13.845
are 98%
411
00:32:14.145 --> 00:32:17.525
applicable to data. Like, I hate the term data ops.
412
00:32:17.860 --> 00:32:21.000
It's the same thing, people. We're shooting for the same target.
413
00:32:21.460 --> 00:32:22.920
We want reliable,
414
00:32:24.180 --> 00:32:25.400
reproducible products.
415
00:32:26.180 --> 00:32:27.945
That's all we're looking for. Absolutely.
416
00:32:28.325 --> 00:32:34.025
And so to that point, another thing that has come up in some of my conversations recently is,
417
00:32:34.550 --> 00:32:39.850
you know, maybe the idea of the data engineer or the analytics engineer
418
00:32:40.150 --> 00:32:42.865
or what have you is starting to
419
00:32:43.165 --> 00:32:45.905
be on the wane. And we don't actually need these
420
00:32:46.525 --> 00:32:56.460
specific job titles, but we really just need our developers who understand how to work with data. And so that just needs to become the kind of baseline status quo of if you're an engineer,
421
00:32:57.000 --> 00:33:04.855
everybody needs to understand these concepts because it's just becoming more ubiquitous, and so we need to generalize and not specialize in these regards.
422
00:33:05.395 --> 00:33:11.730
Yeah. You know, it's that Conway's law thing. If you've got data engineers and analytics engineers and data scientists
423
00:33:12.610 --> 00:33:19.585
oh, and then you've got the platform folks. You know, you've got your site reliability engineers and whatnot. You're going to end up with
424
00:33:20.285 --> 00:33:20.785
siloed,
425
00:33:21.325 --> 00:33:22.385
centralized systems.
426
00:33:23.005 --> 00:33:26.050
And there's always gonna be a space for centralization
427
00:33:26.350 --> 00:33:31.170
and, you know, people with deep course you know, that t shaped skills thing. You know? They're gonna be people who who
428
00:33:31.550 --> 00:33:32.770
absolutely have to be
429
00:33:33.150 --> 00:33:35.255
t shaped, deep, long t's
430
00:33:36.435 --> 00:33:38.375
because this stuff is complicated.
431
00:33:39.075 --> 00:33:41.975
But we need more people to be more generalists.
432
00:33:42.755 --> 00:33:43.255
And
433
00:33:44.179 --> 00:33:45.559
if we can combine
434
00:33:45.860 --> 00:33:48.120
people who have more general skills
435
00:33:49.140 --> 00:33:49.640
with
436
00:33:50.020 --> 00:33:52.760
less requirements, you know, and the and the big 1 is
437
00:33:53.155 --> 00:33:55.655
cutting down in the amount of data you have to process.
438
00:33:56.195 --> 00:33:59.255
Like, if you can narrow down the data set you're working with,
439
00:33:59.555 --> 00:34:07.070
suddenly that makes a lot of very complicated things a lot easier to deal with. So start doing more of that. Get more general.
440
00:34:07.745 --> 00:34:13.285
You're going to naturally end up in a more distributed system with a more distributed environment.
441
00:34:14.225 --> 00:34:15.205
And hopefully,
442
00:34:15.859 --> 00:34:18.200
you're going to end up with teams that are
443
00:34:18.660 --> 00:34:30.825
really knowledgeable in the domain and the problem that's interesting to them. You know, that old adage, look for people who are concerned about a problem and let them do things, which gets to another thing that we that we do a poor job of.
444
00:34:31.240 --> 00:34:33.900
We've got to get more diversity in these teams.
445
00:34:34.360 --> 00:34:35.820
You know, we need to hire
446
00:34:37.000 --> 00:34:37.820
for more diversity,
447
00:34:38.235 --> 00:34:42.415
and we need to assemble teams with more diversity in mind. Because whenever you assemble a team that's,
448
00:34:42.955 --> 00:34:50.130
you know, you're looking for these t shaped skills and you're looking for people with lots of experience, and you don't want that. What you really want is you want
449
00:34:50.670 --> 00:34:54.050
people with a wide range of skill sets and a wide range of backgrounds.
450
00:34:54.910 --> 00:35:01.974
They make better problem solvers. There are some cool studies that I referenced some of them in the book about that. You know, having a more diverse team
451
00:35:02.355 --> 00:35:09.119
makes it easier for you to solve problems you don't already know the answer to. And we need to get better at that. In terms
452
00:35:09.660 --> 00:35:10.400
of the
453
00:35:10.940 --> 00:35:14.160
kind of scoped problem of building a data product,
454
00:35:14.915 --> 00:35:25.660
You know, in the book, you kind of focus on the use case of a small team and working in a smaller group to be able to conceptualize and iterate on and build and produce these data products.
455
00:35:26.200 --> 00:35:33.045
And I'm wondering what you see as being the kind of core practices that are necessary
456
00:35:33.345 --> 00:35:34.885
for that venue
457
00:35:35.185 --> 00:35:36.965
and some of the ways that those practices
458
00:35:37.585 --> 00:35:38.085
either
459
00:35:38.710 --> 00:35:43.690
change or mutate or when you start needing to bring in other concepts or specializations
460
00:35:44.309 --> 00:35:48.235
as that team starts to scale and can no longer be considered small
461
00:35:48.615 --> 00:35:55.595
and maybe speak to what that tipping point happens to be, whether it's actually in terms of quantity of people or complexity of problem,
462
00:35:56.299 --> 00:36:11.880
etcetera? Oh, ew. Boy, I think that the answer to that depends. I mean, the short answer is pretty straightforward. 2 pizza teams. Right? You know, no more than 5 to 8 people. Keep the number of people odd so that you can have a tie breaker when you have to decide some, you know, all of that kind of stuff.
463
00:36:12.519 --> 00:36:14.059
That's the good short answer.
464
00:36:14.839 --> 00:36:15.339
But,
465
00:36:15.640 --> 00:36:17.099
you know, the reality is
466
00:36:17.720 --> 00:36:21.295
you can only hold so much stuff in your head at once.
467
00:36:21.675 --> 00:36:23.375
It's difficult to think about
468
00:36:24.075 --> 00:36:27.055
a wide ranging problem with any kind of complexity
469
00:36:27.930 --> 00:36:30.750
by yourself. So you need other people to help you do that.
470
00:36:31.770 --> 00:36:33.870
But the more people you have,
471
00:36:34.410 --> 00:36:37.550
the problem that people think they're working on is gonna diverge,
472
00:36:37.855 --> 00:36:42.835
and that's okay. Maybe you need to split that team off. And, again, this is a centralized versus decentralized
473
00:36:43.215 --> 00:36:45.315
thing. This is 1 of the big
474
00:36:45.695 --> 00:36:46.835
drivers behind
475
00:36:47.869 --> 00:36:50.290
the, I guess, philosophy of
476
00:36:50.670 --> 00:36:57.785
data mesh is that you know, same thing with microservices. I mean, again, I think about data mesh as being microservices for data.
477
00:36:58.245 --> 00:36:59.225
And sometimes,
478
00:37:00.245 --> 00:37:01.065
that's probably
479
00:37:01.900 --> 00:37:09.280
the most appropriate way to solve the problem. Other times, it's not. 1 area where I think is the most appropriate way to solve the problem is when you're in exploration,
480
00:37:09.795 --> 00:37:15.095
When you're just exploring the problem, you've got couple of people who are dedicated to solving a particular problem.
481
00:37:16.035 --> 00:37:20.000
Give them an exploration platform that lets them do their job
482
00:37:20.300 --> 00:37:21.760
as efficiently as possible.
483
00:37:22.540 --> 00:37:29.235
And if it doesn't work, you package it up and you put it on the shelf. If it works, you probably will need to completely change
484
00:37:29.775 --> 00:37:31.315
the way that thing is implemented.
485
00:37:31.695 --> 00:37:33.910
And at that point, you're gonna need to bring in
486
00:37:34.470 --> 00:37:34.970
specialists
487
00:37:35.430 --> 00:37:36.569
to help you do things.
488
00:37:36.950 --> 00:37:38.890
You're gonna need to bring in people with
489
00:37:39.589 --> 00:37:40.410
deeper technical
490
00:37:40.789 --> 00:37:41.289
experience
491
00:37:41.990 --> 00:37:50.635
than the team probably has. And then that's where you start to figure out what's the most appropriate team structure because we're doing something different. We're growing.
492
00:37:51.230 --> 00:37:52.770
As far as the
493
00:37:53.630 --> 00:38:01.095
overall state of the data ecosystem, we mentioned earlier about the modern data stack, and there have been various evolutions
494
00:38:01.475 --> 00:38:10.680
of ETL and ELT tools. And every, you know, few months or every couple of years, there's some new product category that people are exploring.
495
00:38:11.059 --> 00:38:13.720
You know, some of the recent ones are data catalogs
496
00:38:14.099 --> 00:38:14.599
and
497
00:38:14.900 --> 00:38:17.400
now data quality and data observability.
498
00:38:18.425 --> 00:38:20.205
And with all of that activity,
499
00:38:21.145 --> 00:38:25.005
and, you know, a lot of this is stuff that we've had in some variety or another
500
00:38:25.579 --> 00:38:29.040
for years. It's maybe just repackaging it with a particular
501
00:38:29.660 --> 00:38:31.280
focus. And I'm just wondering
502
00:38:31.980 --> 00:38:33.520
what you see as
503
00:38:33.900 --> 00:38:34.400
the
504
00:38:34.905 --> 00:38:35.725
kind of challenges,
505
00:38:36.265 --> 00:38:39.885
particularly for business people who aren't steeped in this every day to
506
00:38:40.505 --> 00:38:43.085
extract useful signal from all of the noise.
507
00:38:43.440 --> 00:38:44.900
And if there's anything
508
00:38:45.200 --> 00:38:55.775
in all of this kind of, you know, funding laden hype of these different elements of the data ecosystem that is actually truly new and innovative and not just
509
00:38:56.155 --> 00:38:59.935
revisiting the same concepts with a, you know, shinier brand?
510
00:39:00.599 --> 00:39:01.900
The first thing is,
511
00:39:02.359 --> 00:39:02.859
again,
512
00:39:03.160 --> 00:39:07.660
you've got to understand what problem you're trying to solve. Why do you bring in a data catalog?
513
00:39:08.135 --> 00:39:09.755
First off, you need to realize that
514
00:39:10.214 --> 00:39:12.315
we've always had data catalog efforts.
515
00:39:12.934 --> 00:39:18.250
You know, for 20 years, there's been a push, you know, every 5, 10 years for
516
00:39:18.789 --> 00:39:19.930
new data catalogs.
517
00:39:20.470 --> 00:39:34.560
Well, why is that? Because we don't have any visibility to the data. We start collecting the data. We don't manage the data well. It's difficult to expose it because we lock it away in databases, and we don't let people get access to those databases. So
518
00:39:35.040 --> 00:39:38.340
it's a legitimate business need. But the problem is,
519
00:39:39.040 --> 00:39:42.260
again, I'll I'll go back to, you know, 1 of the central themes.
520
00:39:42.945 --> 00:39:45.525
Why on earth is your data system so large
521
00:39:46.145 --> 00:39:47.125
and so complicated
522
00:39:47.745 --> 00:39:53.170
that you need to put a data catalog over everything? And how long is it going to take you
523
00:39:53.550 --> 00:39:58.050
to deploy that data catalog and to catalog all of your assets and make them
524
00:39:58.430 --> 00:40:02.685
make them visible? Your system's so big, where it might be better to
525
00:40:03.225 --> 00:40:04.285
distribute that
526
00:40:04.665 --> 00:40:05.405
work out
527
00:40:05.705 --> 00:40:17.000
and have the teams that really understand it figure out, no. Okay. This is valuable. This is not valuable. Let's figure out how to prune the system, you know, get rid of that technological debt. It's difficult
528
00:40:17.845 --> 00:40:18.984
for business people
529
00:40:19.365 --> 00:40:20.904
who are making these buying decisions
530
00:40:21.684 --> 00:40:26.904
to focus on that because they get sold on, oh, we need to have a catalog, and we need to
531
00:40:27.339 --> 00:40:34.240
have APIs on everything, and we need to do this, and we need, you know, to hire a team of 12 data scientists
532
00:40:34.540 --> 00:40:35.200
to do
533
00:40:35.525 --> 00:40:37.065
machine learning models, and
534
00:40:37.685 --> 00:40:51.610
we don't stop to think about why, and do we really need to do that right now. And more importantly, are we even prepared to do that? What would we do with that? I mean, in manufacturing, you see this all the time. You show someone who's working on a production line, statistical process control chart,
535
00:40:51.910 --> 00:40:55.204
XPR and R chart, they're gonna look at you like deer in headlights.
536
00:40:55.505 --> 00:41:04.560
You have to go back and explain variation to them. All they need to know is, am I doing okay today? Maybe a dashboard. The equivalent would be a dashboard. That's probably where you need to be.
537
00:41:05.980 --> 00:41:07.475
And leave the fancy stuff
538
00:41:08.195 --> 00:41:10.455
to come after the training you provide them
539
00:41:10.835 --> 00:41:12.215
to understand why,
540
00:41:12.515 --> 00:41:14.455
you know, this is more valuable.
541
00:41:15.050 --> 00:41:18.830
You know, why is a scatter plot more valuable than a bar chart,
542
00:41:19.290 --> 00:41:21.310
let alone, you know, a pie chart?
543
00:41:21.849 --> 00:41:30.155
We're pretty immature. And as technologists, we need to step back and go, are people ready for the level of maturity that I'm pitching here?
544
00:41:30.855 --> 00:41:33.595
Chances are probably not. So what do you do?
545
00:41:33.970 --> 00:41:36.390
And so in terms of
546
00:41:36.690 --> 00:41:37.190
the
547
00:41:37.970 --> 00:41:40.710
kind of work of building data products and
548
00:41:41.155 --> 00:41:46.375
being able to bring business people into the planning and execution of that
549
00:41:47.315 --> 00:41:48.935
and being able to bring
550
00:41:49.240 --> 00:41:49.900
engineering teams
551
00:41:50.520 --> 00:41:55.500
into the business problems, what do you see as the biggest risks to
552
00:41:55.880 --> 00:42:01.285
those engineering and data teams as they're starting to embark on these products or starting to
553
00:42:01.905 --> 00:42:03.365
evolve their capabilities
554
00:42:03.825 --> 00:42:06.885
to kind of work more closely with the business needs?
555
00:42:07.359 --> 00:42:13.380
We don't get alignment on the problem. We don't clearly articulate what the problem is and have a shared understanding of that.
556
00:42:14.000 --> 00:42:18.165
Take the time to get a shared language around the problem. That's really the core.
557
00:42:18.465 --> 00:42:19.205
And again,
558
00:42:19.665 --> 00:42:23.685
we don't understand the business. Next thing that flows from that is we don't understand the business problem.
559
00:42:24.465 --> 00:42:24.965
We
560
00:42:25.265 --> 00:42:26.805
think too big too quickly.
561
00:42:27.410 --> 00:42:31.190
And this goes for both sides of the fence, the business folks and the technical folks.
562
00:42:32.050 --> 00:42:36.069
We want to jump to a conclusion. We want to pay
563
00:42:37.095 --> 00:42:39.835
for some product to magically take our problems away.
564
00:42:40.535 --> 00:42:44.875
And that's where we first jump. We need to slow down a little bit
565
00:42:45.494 --> 00:42:46.715
and think about
566
00:42:47.720 --> 00:42:49.580
what we are really trying to accomplish.
567
00:42:50.040 --> 00:42:53.900
That's just always a big risk. You know, it's always people in process problems.
568
00:42:54.520 --> 00:42:55.260
And then
569
00:42:55.744 --> 00:42:57.204
I do think that there's
570
00:42:57.825 --> 00:43:03.510
a problem trying to collect all the data, you know, for folks in the data warehousing space,
571
00:43:04.070 --> 00:43:05.770
A famous quote by the
572
00:43:06.550 --> 00:43:08.010
CTO of General Motors,
573
00:43:08.310 --> 00:43:13.210
who was the CTO of Hewlett Packard, he was really famous for saying, I wanna know everything about everything.
574
00:43:13.945 --> 00:43:15.165
Like, that is the most
575
00:43:15.625 --> 00:43:16.125
insanely
576
00:43:16.425 --> 00:43:20.765
crazy thing you could ever say. We're just going to build a big pile of technical debt
577
00:43:21.350 --> 00:43:21.850
that
578
00:43:22.150 --> 00:43:23.770
no 1 will be able to use.
579
00:43:24.150 --> 00:43:24.650
Absolutely.
580
00:43:25.830 --> 00:43:30.570
And so in terms of the work that you've been doing on the book and some of the
581
00:43:31.015 --> 00:43:38.955
kind of core lessons that you've put in there, what are some of the most interesting or innovative or unexpected ways that you've seen those ideas applied or
582
00:43:39.440 --> 00:43:49.535
some of the useful or interesting feedback you've gotten on the book now that it's out? So my wife works for an educational publisher. You know? You know what the educational market looks like right now.
583
00:43:50.475 --> 00:43:51.695
It's just exploding.
584
00:43:52.875 --> 00:43:55.455
And they're trying to deliver their own new products.
585
00:43:56.395 --> 00:44:02.750
And as she's explaining, you know, what's going on at work, it's like, oh, let me tell you about Explore, Expand and Extract.
586
00:44:03.130 --> 00:44:07.515
And I got invited to come and do what I affectionately call a man explaining minute,
587
00:44:09.015 --> 00:44:09.994
lunch and learn
588
00:44:10.615 --> 00:44:11.115
about
589
00:44:11.655 --> 00:44:16.590
their transition because where they were at, they were exiting out of exploring, going into expand.
590
00:44:17.370 --> 00:44:18.110
And that
591
00:44:18.410 --> 00:44:24.755
piece there, you know, it's the crossing the chasm spot. You know, from that famous book, I call it the crappy chasm of doom
592
00:44:25.775 --> 00:44:26.275
because
593
00:44:26.575 --> 00:44:29.315
it's really horrible to make that inflection point.
594
00:44:29.695 --> 00:44:30.595
Nothing works.
595
00:44:31.130 --> 00:44:33.790
You know, everybody's working way too much overtime.
596
00:44:34.810 --> 00:44:45.045
And there's no sign that anything's ever going to get better. And 1 of the big messages that was really nice to say because I've been thinking about this. I've been applying it to where other places where I've worked.
597
00:44:45.425 --> 00:44:47.970
I've been through this transition point a lot,
598
00:44:48.530 --> 00:44:50.150
and it's always horrible.
599
00:44:50.450 --> 00:44:53.190
And it was nice to explain that to non IT people
600
00:44:54.609 --> 00:45:00.395
that, you know, it's going to be okay. You just need to give it a few more months, and you will be past this,
601
00:45:00.935 --> 00:45:07.900
and you'll be back to normal again. And that was really satisfying because these concepts are really universal.
602
00:45:08.280 --> 00:45:13.125
Anytime you try to build things, the worst thing that can happen to you is people actually like it,
603
00:45:13.525 --> 00:45:17.625
and it takes off. Nobody's prepared for that. That was a nice thing
604
00:45:18.005 --> 00:45:19.625
that I got out of the book.
605
00:45:19.925 --> 00:45:24.780
In terms of your experience of writing the book and sort of formulating the
606
00:45:25.160 --> 00:45:32.465
ideas behind it and the core kind of mission of it, what has been the most interesting or unexpected or challenging lessons that you've learned in the
607
00:45:39.985 --> 00:45:40.485
development,
608
00:45:41.100 --> 00:45:45.520
and throw in a whole bunch of business theory at the same time was just ridiculous.
609
00:45:47.260 --> 00:45:55.494
But I couldn't think of another way to do it. I couldn't think of you know, if you learn how to use a particular set of tools, you don't know what to use them for.
610
00:45:55.795 --> 00:46:00.990
If you know where to point a tool, you have to know how to use the tool. And so I tried
611
00:46:01.369 --> 00:46:04.430
to weave a fine line between both of those
612
00:46:04.730 --> 00:46:05.790
difficult situations,
613
00:46:06.569 --> 00:46:10.125
and it was hard. And I'm I'm not really sure I did a great job of it,
614
00:46:10.505 --> 00:46:12.525
but I did the best job I could do.
615
00:46:12.984 --> 00:46:19.250
The other thing that was really good is I learned that I had about 20 years of frustration locked in my head,
616
00:46:19.790 --> 00:46:21.410
and getting it out on paper
617
00:46:21.870 --> 00:46:28.744
was very cathartic. Like, I can sleep at night. I don't think about data problems at night now, and I was before.
618
00:46:29.204 --> 00:46:30.825
They were keeping me up at night.
619
00:46:31.125 --> 00:46:32.505
Lot of imposter syndrome
620
00:46:32.950 --> 00:46:33.850
that I encountered
621
00:46:34.630 --> 00:46:36.170
still have, so
622
00:46:36.630 --> 00:46:52.260
I think everybody who puts themselves out there really suffers from that. Yeah. Those were really the most valuable pieces of learning that I got out of doing this. It was a lot harder than I thought. It was 1 of the hardest things I ever done, And to crank out basically a textbook in 8 months
623
00:46:52.800 --> 00:46:53.300
was
624
00:46:53.840 --> 00:46:56.500
nuts. That was ridiculous to try to do that.
625
00:46:56.880 --> 00:46:58.740
Well, now that you've done it, what's next?
626
00:46:59.585 --> 00:47:00.565
You know, I
627
00:47:00.865 --> 00:47:12.530
don't know. I'm kind of looked at when I left. You know, when I took my resignation leave, I kind of thought about it as I've been planning this for a while, you know, saving up my money for a while. So right now, I'm just looking for new opportunities.
628
00:47:13.630 --> 00:47:16.290
I don't know that I have another book in me right away.
629
00:47:18.235 --> 00:47:24.415
And just trying to think about how to solve these problems. So if anybody else feels that they're interested in these kind of problems,
630
00:47:24.869 --> 00:47:31.049
feel free to get in touch with me. I'd love to talk to you about it. It's something I'm really passionate about and have been passionate for a long time,
631
00:47:31.349 --> 00:47:33.690
and I don't really know what's gonna come next.
632
00:47:34.155 --> 00:47:48.640
Alright. Well, for anybody who does wanna get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as a final question, I'd like to get your perspective on what you see as being the biggest gap in the tooling or technology that's available for data management today.
633
00:47:49.005 --> 00:47:51.424
I think there's a lot of potential in data mesh,
634
00:47:51.724 --> 00:47:52.704
both for good
635
00:47:53.005 --> 00:47:56.385
and for just total disaster. We had the same thing with microservices.
636
00:47:56.910 --> 00:48:00.130
Go look at how microservices panned out, learn from that,
637
00:48:00.910 --> 00:48:03.010
then start applying it to data and hopefully
638
00:48:03.470 --> 00:48:05.090
make fewer mistakes.
639
00:48:05.575 --> 00:48:11.195
That would be 1 thing. The second thing is, man, I wish there was something like SQL for visualization
640
00:48:12.055 --> 00:48:14.235
that didn't require you to write Python code.
641
00:48:14.535 --> 00:48:17.740
Because if you hand a page of Python code
642
00:48:18.040 --> 00:48:22.220
to someone who's not a developer, their eyes glaze over, and
643
00:48:22.520 --> 00:48:28.474
it's terrifying to them. And we need something that's simpler. In the book, I used Vega Lite, and that was fantastic.
644
00:48:28.935 --> 00:48:32.075
But pretty arts and charts, not a whole lot of interactivity.
645
00:48:32.830 --> 00:48:36.290
Some places, that's exactly what you want. Other places, not.
646
00:48:36.750 --> 00:48:39.010
So I would like to see something that's easy
647
00:48:39.550 --> 00:48:41.490
for people to grok as SQL
648
00:48:41.975 --> 00:48:44.635
for visualizations. And I'm not I maybe have a suggestion
649
00:48:44.935 --> 00:48:55.380
for something to take a look at. I haven't found 1 yet. Alright. Well, thank you very much for taking the time today to join me and share the work that you've been doing on writing this book and some of the
650
00:48:55.680 --> 00:48:59.575
thoughts and experiences that went into it. Definitely a very interesting
651
00:49:00.035 --> 00:49:04.375
problem space, and it's definitely great to see people trying to bring business people
652
00:49:04.915 --> 00:49:08.760
more in the fold of working with the data and helping to
653
00:49:09.140 --> 00:49:21.565
educate engineers on the business concepts and requirements that go into what they're actually trying to build for. So appreciate all the time and energy you have put into that, and I hope you enjoy the rest of your day. Great. Thank you. I really appreciate the opportunity.
654
00:49:27.420 --> 00:49:30.400
Listening. Don't forget to check out our other show, podcast.init
655
00:49:30.975 --> 00:49:31.475
atpythonpodcast.com
656
00:49:32.895 --> 00:49:37.235
to learn about the Python language, its community, and the innovative ways that is being used.
657
00:49:37.615 --> 00:49:38.995
And visit the site at dataengineeringpodcastdot
658
00:49:40.020 --> 00:49:48.975
com to subscribe to the show, sign up for the mailing list, and read the show notes. If you've learned something or tried out a project from the show, then tell us about it. Email hosts at dataengineeringpodcast.com
659
00:49:50.315 --> 00:49:55.500
with your story. And to help other people find the show, please leave a review on Itunes and tell your friends and coworkers.