TRANSCRIPCIÓN 🔗
WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 8/4/2024 7:34:19 PM
Duration: 3210.341
Channels: 1
1
00:00:11.264 --> 00:00:15.445
Hello, and welcome to the Data Engineering Podcast, the show about modern data management.
2
00:00:16.064 --> 00:00:25.050
Your host is Tobias Massey, and today, I'm welcoming back Chris Bergh to talk about his tireless quest to simplify the lives of data engineers. So, Chris, can you start by introducing yourself?
3
00:00:25.590 --> 00:00:30.025
So my name is Chris Bergh. I'm a CEO of a company called Data Kitchen in Massachusetts.
4
00:00:30.884 --> 00:00:32.745
I have a long history in technology,
5
00:00:33.285 --> 00:00:46.824
going back to the nineties. So I worked at companies like NASA and MIT Lab. I spent a lot of time doing I was doing AI back when it wasn't cool, and your, machine learning class had only 6 people in it. So I'm that old.
6
00:00:47.125 --> 00:00:54.870
And, then about 2,005, I had done a bunch of things in software and managing teams, and I thought the best thing to do would do data and analytics.
7
00:00:55.430 --> 00:00:57.370
And so I I started managing a team,
8
00:00:57.750 --> 00:01:12.635
and I thought my kids were young. They're, like, 8 and 5, and I thought, well, this will be easy. I'm a big software engineer. This data stuff, no problem. And, it turned out my life was hell. Things kept breaking. We could never go fast enough. Everyone wanted their own tool.
9
00:01:13.000 --> 00:01:21.100
And that was a really good experience. So we spent 8 years doing a lot of work. We eventually sold the company. And then about 11 years ago, we started this company, DataKitchen,
10
00:01:21.625 --> 00:01:25.564
to really focus on those sets of problems for data and analytics teams, which are
11
00:01:25.865 --> 00:01:28.204
sort of generated from our experience that
12
00:01:29.300 --> 00:01:40.575
stuff's breaking left and right. You can never go fast enough. And, you know, your customers are just incredibly demanding. And and how do you live in that world and and sort of not get depressed and and and and somewhat,
13
00:01:41.055 --> 00:01:41.955
want a therapist.
14
00:01:43.295 --> 00:01:51.040
You mentioned a bit about what DataKitchen is, some of the story behind how you got into it. What is it about that particular space of
15
00:01:51.580 --> 00:01:59.815
trying to address the problems of data engineers that has kept you engaged and focused on that problem for so long. Should should you have solved it by now?
16
00:02:00.354 --> 00:02:04.135
Yeah. Yeah. Well, I wake up in the middle of the night and say I should have. Right? So,
17
00:02:04.810 --> 00:02:05.310
well,
18
00:02:05.610 --> 00:02:10.910
this will be an interesting discussion because why isn't it solved? Right? Why aren't those problems solved? And
19
00:02:11.450 --> 00:02:12.110
I think
20
00:02:12.475 --> 00:02:16.575
they really are, in some ways, getting somewhat better, but also getting worse.
21
00:02:17.035
Load full transcript
--> 00:02:18.575
And so a lot of people
22
00:02:19.115 --> 00:02:21.375
will build data and analytics systems.
23
00:02:22.020 --> 00:02:27.000
Right? They'll spend their they'll focus on the 1st day. They'll go into Azure or Amazon.
24
00:02:27.380 --> 00:02:32.725
They'll go into GitHub and go, look at all this stuff I got I can do. And they'll put together something and
25
00:02:33.185 --> 00:02:34.805
have some insight for a customer.
26
00:02:35.345 --> 00:02:44.250
And and that's great. And and they'll focus a lot on building new things, but they won't focus on kind of the 2nd or 3rd day, which is I have to run this.
27
00:02:45.590 --> 00:02:46.970
So you mentioned
28
00:02:47.405 --> 00:02:54.625
that you ended up starting DataKitchen because of the problems that you were seeing as a manager for a data team, and
29
00:02:55.049 --> 00:03:13.660
you were one of the people who helped to define and popularize the concept of data ops. And I'm wondering if you can talk to some of the ways that your journey has progressed and where things are getting better and maybe some of the ways that things have stayed the same and the pressures of the complexities of data
30
00:03:14.040 --> 00:03:14.780
have resisted
31
00:03:15.080 --> 00:03:19.340
everyone's efforts to move into this bright new future where everything works all the time?
32
00:03:20.165 --> 00:03:21.385
Yeah. I think
33
00:03:22.805 --> 00:03:26.105
we did a survey about 2 years ago of 700 data engineers.
34
00:03:26.730 --> 00:03:29.150
And it came back where 78%
35
00:03:29.530 --> 00:03:30.990
of them were so frustrated.
36
00:03:31.370 --> 00:03:33.870
They wanted their job to come with a therapist,
37
00:03:34.325 --> 00:03:36.985
and over 50% of them were thinking of quitting.
38
00:03:37.364 --> 00:03:42.344
And so why is that, right, when data is great, we have all these tools,
39
00:03:42.800 --> 00:03:44.340
Why are people so upset?
40
00:03:45.120 --> 00:03:47.220
And I think it comes down to the fact that
41
00:03:48.319 --> 00:03:50.099
we're disconnected from our customers.
42
00:03:50.560 --> 00:03:52.340
We're not achieving our goals.
43
00:03:52.855 --> 00:03:53.355
And
44
00:03:54.295 --> 00:03:55.595
from an industry standpoint,
45
00:03:57.335 --> 00:03:58.235
we popularize
46
00:03:58.615 --> 00:04:00.235
a lot of buzzwords.
47
00:04:00.540 --> 00:04:02.720
And I've been part of that, popularizing
48
00:04:03.020 --> 00:04:07.360
data ops. But they they mutate, and they become almost meaningless.
49
00:04:08.535 --> 00:04:10.395
So these terms that go out there,
50
00:04:11.255 --> 00:04:15.115
I'm not quite sure, but they're like a gas. They sort of expand to fill everything.
51
00:04:15.480 --> 00:04:18.620
And so what happens to any term like data ops or
52
00:04:19.400 --> 00:04:20.620
data mesh or
53
00:04:21.639 --> 00:04:22.620
data observability
54
00:04:23.000 --> 00:04:28.414
or data products or data lakehouse, they they start off with a fairly precise definition.
55
00:04:29.034 --> 00:04:36.600
But then people end them to whatever they're currently doing and saying, I'm doing data mesh. But in fact, what they're just doing is plain old DTL.
56
00:04:36.980 --> 00:04:48.985
Or I'm I have a data product tool. And, well, what what is that? Well, it's really, a governance platform. And so the pollution of the name space in data and analytics is it makes it hard to get through that.
57
00:04:49.445 --> 00:04:56.729
And, you know, we've tried. We wrote a manifesto. We've written 2 books on on data ops. We've tried to stay pure to kind of the 2
58
00:04:57.385 --> 00:05:09.169
concepts because of of what DataOps is. And and one is that in production, you're running a factory of data and analytics, and that factory is taking data from one side and passing it through a bunch of tools
59
00:05:09.550 --> 00:05:12.610
and giving insight on the other side. And and so
60
00:05:13.525 --> 00:05:17.544
the ability for you to run a factory with low errors and low bottlenecks,
61
00:05:18.324 --> 00:05:23.430
how you do that takes a lot of inspiration from industrial process control and demoing, etcetera.
62
00:05:23.970 --> 00:05:35.125
And then the second part is you have a factory that you want to change very quickly because your customers are demanding. And so that takes a lot of the same ideas that came from agile software development because the change is a software.
63
00:05:35.824 --> 00:05:42.919
You know, terms are not built by engineers. They're distorted by marketers. And so it's hard to make hard to make sense of anything nowadays.
64
00:05:43.379 --> 00:05:43.879
Absolutely.
65
00:05:44.985 --> 00:05:47.485
On that note of the misappropriation
66
00:05:48.185 --> 00:05:54.425
of the term DataOps, it seems to have gone through a similar journey to the concept of DevOps that started
67
00:05:54.970 --> 00:05:57.790
I guess, it must be close to 15 years ago now where
68
00:05:58.169 --> 00:06:04.510
it started with a very specific use. It then ballooned to mean everything that had to do with
69
00:06:09.805 --> 00:06:16.760
vernacular. Still not used correctly in most cases, but there's less of a argument about the the misappropriation
70
00:06:17.140 --> 00:06:21.220
of it. And it seems that DataOps has gone through a similar cycle of
71
00:06:21.935 --> 00:06:27.955
it was introduced. People started using it to mean everything having to do with data quality, data observability,
72
00:06:28.895 --> 00:06:29.635
data testing,
73
00:06:30.015 --> 00:06:43.275
and now has started to fade a bit more into the background in favor of the latest cycle of hyped up terms. And I'm wondering if you can give your sense of what you see as the state of DataOps today,
74
00:06:43.735 --> 00:06:45.915
however you want to define that.
75
00:06:47.260 --> 00:06:48.800
Well, I agree with your definition.
76
00:06:49.100 --> 00:06:51.840
I think people even use DataOps as,
77
00:06:53.180 --> 00:06:57.755
a moniker
for ETL or or data transformation or databases. Right? It's everything.
78
00:06:58.215 --> 00:07:00.634
And I don't agree with that way of thinking.
79
00:07:01.014 --> 00:07:04.870
I I do think it's proceeded a lot slower than software,
80
00:07:05.650 --> 00:07:11.170
adoption of DevOps. And I think it's really interesting why. So if, you know, when I
81
00:07:12.435 --> 00:07:13.655
like, 20 years ago,
82
00:07:14.035 --> 00:07:19.575
one way I would change code in production is I'd literally go on the production machine and and change some code.
83
00:07:20.110 --> 00:07:22.610
And nobody on my team went, ew.
84
00:07:23.070 --> 00:07:31.005
What are you doing? That's gross. Don't do that. Don't change things on production. In fact, it was seen as kind of cool. Like, oh, he's he's a cowboy.
85
00:07:31.785 --> 00:07:34.604
And so what hasn't happened yet with data people
86
00:07:34.905 --> 00:07:36.205
is they're not saying
87
00:07:36.505 --> 00:07:37.164
eww enough.
88
00:07:37.850 --> 00:07:45.870
Like, you just change your code in production, and you don't know if it works. You don't automate deployment. You don't have testing.
89
00:07:46.555 --> 00:07:49.375
You don't have monitoring. You're kinda hoping everything works.
90
00:07:50.155 --> 00:07:54.095
And that's unfortunate because I think as a team, one of the reasons why
91
00:07:54.910 --> 00:08:05.015
we're so depressed and so upset is because of these death of a 1000 knives. We're not automating enough. We're not testing enough. We're not saying, oh, to each other enough.
92
00:08:05.475 --> 00:08:11.175
And we're sort of accepting that heroism is part of it. Or on some teams, they're the exact opposite.
93
00:08:11.720 --> 00:08:18.940
They're living such fear of breaking anything that they've made. They don't wanna touch it, and then they have this sort of procedural defensiveness.
94
00:08:20.605 --> 00:08:22.625
So along with that misappropriation
95
00:08:23.085 --> 00:08:29.265
of the concept of data ops, there were a sequence of subsequent trends that came along,
96
00:08:29.699 --> 00:08:30.919
notably data observability,
97
00:08:31.220 --> 00:08:32.680
data reliability engineering.
98
00:08:33.540 --> 00:08:35.720
Those all try to encapsulate
99
00:08:36.180 --> 00:08:39.160
at least a portion of the idea of data ops.
100
00:08:39.795 --> 00:08:40.695
And I'm wondering
101
00:08:41.235 --> 00:08:48.950
how those subsequent trends from when you first started DataKitchen, first started trying to popularize DataOps, wrote the DataOps manifesto,
102
00:08:49.890 --> 00:08:54.550
how those different cycles of adoption and technical evolutions
103
00:08:54.850 --> 00:09:28.814
have influenced the way that you think about the work that you're doing at DataKitchen? Actually actually, quite a bit. And I think, you know, 10 years ago when I was first talking to DataOps, I'd go to conferences and bring up Git, and I had to explain version control. And sort of 1 out of 10 people, 1 out of a 100 knew what Git was, knew what CI CD was. The thought of actually coding SQL freaked a lot of people out. And so I think there's some tools that have really helped the world, like like the existence of of DBT and its integration with Git and its way of writing template at SQL. You know, those sort of 600 or $700,000,000
104
00:09:30.555 --> 00:09:36.254
in data observability investment capital and the massive marketing budgets, that that's also how people have data observability
105
00:09:36.555 --> 00:09:37.454
projects now,
106
00:09:38.060 --> 00:09:44.080
or data reliability projects. And I also think the fact that, like, more and more tools actually have
107
00:09:44.785 --> 00:09:46.965
Git and CI and CD integration.
108
00:09:47.345 --> 00:09:52.780
And so I think those things are not as far out as they once were, which is, you know, which is fantastic.
109
00:09:53.160 --> 00:10:04.375
And I think they've actually have influenced. And I think in order to answer that question, you know, I'm a technical founder. It took me a while to start a start a company, but I was intended to do it kind of based
110
00:10:05.225 --> 00:10:13.850
on my values, based on always being profitable. So we've run a company that's never had any investment, either our own or not. We've just sort of been quietly bootstrapping.
111
00:10:14.584 --> 00:10:18.845
And I think it's really influenced me because I thought sort
112
00:10:19.225 --> 00:10:28.440
of 2, maybe 4 years ago and and and even before that, that the way this change would happen in the world, that way people would adopt the principles of data ops,
113
00:10:28.980 --> 00:10:30.600
would be from the top down.
114
00:10:31.035 --> 00:10:41.510
And I went to a conference, a DevOps conference 6 or 7 years ago, the DevOps Enterprise Summit, and there were all these enterprise teams saying, we're adopting agile. We're adopting DevOps. They were so enthusiastic.
115
00:10:41.890 --> 00:10:42.710
They were so
116
00:10:43.170 --> 00:10:49.325
into it. And I'm like, okay. This is gonna happen. People are gonna start to believe these ideas just like they believe DevOps.
117
00:10:49.945 --> 00:10:54.589
And it turns out that's not the way change is happening. And, unfortunately,
118
00:10:54.970 --> 00:10:57.470
the people who run data and analytic teams
119
00:10:57.930 --> 00:11:03.885
have, like, a 18 to 24 month, sometimes two and a half year life cycle. CDOs, VPs,
120
00:11:04.345 --> 00:11:06.445
they're hired, they're fired so quickly.
121
00:11:06.825 --> 00:11:07.325
And
122
00:11:07.785 --> 00:11:09.084
when you try to have a
123
00:11:09.520 --> 00:11:18.580
top down change like that and your sort of foundation is a mess, it's really hard to make that happen. And so what we've switched to is is is really focusing on
124
00:11:19.135 --> 00:11:20.195
individual contributors
125
00:11:20.654 --> 00:11:24.115
and kind of starting over again. And we have a you know, we still
126
00:11:24.575 --> 00:11:28.435
support talking to CDOs, but we've open sourced a whole bunch of software
127
00:11:28.879 --> 00:11:30.180
targeted at a single
128
00:11:30.560 --> 00:11:33.220
data engineer, data quality person who
129
00:11:33.759 --> 00:11:52.280
is tired of, you know, getting be getting beaten up for having the errors in production or just tired of being able to go too slow and then wants to have more satisfaction on their job. And so it's really influenced how we think because, you know, how does change happen in the world? Right? When when we started, we had to say, here's the idea,
130
00:11:53.115 --> 00:11:59.214
wrote wrote books, videos, talked about it. You know, we had to run a company. But really, we've invested, I don't know, $6,000,000
131
00:11:59.515 --> 00:12:11.894
in software development, which is a lot for a company like ours, and we've completely open sourced it. It's all full featured Apache 2 0. And we're excited about this sort of opportunity to continue to try to push the the market forward.
132
00:12:12.834 --> 00:12:14.295
On that note of
133
00:12:14.915 --> 00:12:16.535
how to drive change
134
00:12:16.915 --> 00:12:19.815
and the turnover cycles of
135
00:12:20.149 --> 00:12:20.970
senior leadership
136
00:12:21.589 --> 00:12:22.730
in the data space,
137
00:12:23.589 --> 00:12:26.970
I think that that speaks to as well the fact that
138
00:12:27.365 --> 00:12:34.425
companies think, oh, data is going to solve all my problems. We're going to throw a bunch of money at it. We'll become data driven, and everything will be great, will be profitable.
139
00:12:34.980 --> 00:12:38.199
But then they don't factor in the realities of
140
00:12:39.540 --> 00:12:40.600
how complicated
141
00:12:41.540 --> 00:12:43.320
the data workflow is,
142
00:12:43.860 --> 00:12:44.360
the
143
00:12:45.015 --> 00:12:52.315
cycles of how long it takes to actually build out the data warehouse and then be able to gain real value from it. I'm actually reading
144
00:12:52.740 --> 00:12:58.120
the Bill Inman, how to build the data warehouse book recently, which has a lot of useful insight there about,
145
00:12:58.740 --> 00:13:09.384
unlike software, you don't start from requirements and build the product. You build the product to then get the requirements, and then you go do it again. And I'm wondering if you're seeing some of those same cycles of
146
00:13:10.129 --> 00:13:12.790
senior leadership comes in. They make all kinds of promises.
147
00:13:13.410 --> 00:13:19.829
They can't deliver in the time frame that they've said, and then they get kicked out, and then the next person comes in and makes different promises.
148
00:13:20.515 --> 00:13:21.815
And that's and
149
00:13:22.195 --> 00:13:29.255
those promises are a 100% buzzword compliant. They go to the latest Gartner conference, pick up some buzzwords, and then they're gonna do these buzzwords.
150
00:13:29.769 --> 00:13:32.750
And then their foundation is utterly unreliable.
151
00:13:33.130 --> 00:13:38.110
And so, you know, I feel sorry for these men and women who are in these roles because they're trying
152
00:13:38.575 --> 00:13:49.140
and they're just thinking about it the wrong way. They're thinking about the data and not the processes acting upon data. They're not thinking about their people and process. They're thinking about sort of trend following.
153
00:13:49.680 --> 00:13:57.140
And I I, you know, I I think we've built a lot of good things. We have a lot of great tools, but we've got to focus on,
154
00:13:57.575 --> 00:14:01.435
as I said, the day 2 or 3 problem, we've got to focus on.
155
00:14:01.895 --> 00:14:08.930
Another way to think about it is what data ops means is you've got 2 23 year old problems and 1 46 year old problem.
156
00:14:09.310 --> 00:14:14.130
So another way to think about this is you hire a 23 year old on your team, and he or she's got
157
00:14:14.444 --> 00:14:22.305
a degree in computer science, really smart. Maybe they've done some data work and as a as a co op student, and you bring them in.
158
00:14:22.920 --> 00:14:26.140
Can they make a change something? Can they make a small change?
159
00:14:27.400 --> 00:14:34.285
You know, a line of SQL, a line of Python. And can that get into production in their 1st week where you know,
160
00:14:34.745 --> 00:14:44.769
that they can make that change and they don't have to be babysat by your by your senior people? Can they make a change with very low risk? And most data teams don't benchmark themselves that way.
161
00:14:45.070 --> 00:14:51.495
You know, they'll they'll make a change and maybe you'll hope it works or it works in their box. And if you just think about you change
162
00:14:51.955 --> 00:14:59.040
a column name, what's the ripple effect on that on all the other tables, on all the reports, on all the models, on all the governance?
163
00:14:59.580 --> 00:15:06.320
Can you have a button next to them? And, you know, Tobias, I have a son who's 25, and he took a job at Amazon
164
00:15:07.295 --> 00:15:19.580
a few years ago as a software engineer, like, in their b to b payments place, and and he was able to do that. And, you know, I love my son, and I I'm sure he's a good engineer. But if you look at his room when he was, you know, when he was 10 or,
165
00:15:20.200 --> 00:15:28.214
his room when he was 16, which was scary, I wouldn't trust him to make a change. And so where does that trust come from? Well, they built a system
166
00:15:28.675 --> 00:15:40.385
next to their 23 year olds, you know, to help them deploy fast with low risk, next to their 23 year olds to help them observe the system to make it right. And then finally, the 46 year old on the team
167
00:15:40.765 --> 00:15:47.420
actually has a set of metrics that measure things like error rates and cycle time and customer satisfaction and productivity.
168
00:15:47.959 --> 00:15:50.459
And so we've got to focus on that, you know,
169
00:15:50.760 --> 00:15:55.500
on not so much on let's let's build it, but how do we run it? How do we change it? How do we measure
170
00:15:55.925 --> 00:15:58.585
our success? And those things, I think, are
171
00:15:59.365 --> 00:15:59.865
really
172
00:16:00.325 --> 00:16:03.145
the the key to gaining back a lot of time
173
00:16:03.750 --> 00:16:08.890
and the key to sort of minimizing the immense amount of waste that happens in data and analytic teams.
174
00:16:09.670 --> 00:16:11.370
I think that that note on
175
00:16:11.735 --> 00:16:18.235
the customer impact is also interesting in the context of the data space because I think that for a long time,
176
00:16:18.880 --> 00:16:22.500
there wasn't a clear view on who the customer is of the work that you're doing.
177
00:16:23.120 --> 00:16:29.355
When you're an operations engineer, you know that your customers are the software engineers who need to be able to run run their applications.
178
00:16:30.135 --> 00:16:37.995
The software engineers know that their customer is the end user of the application, whether that's internal or external to the organization.
179
00:16:39.450 --> 00:16:40.510
On data teams,
180
00:16:41.370 --> 00:16:48.805
theoretically, the customer is the CEO, but only if the CEO knows what he's looking at and how to interpret it. And I think that that's another
181
00:16:49.425 --> 00:16:57.440
evolution that we've gone through over the past 10 years is starting to understand more who is the customer of the data that you're producing,
182
00:16:58.220 --> 00:16:58.720
particularly
183
00:16:59.100 --> 00:17:03.040
as we move into the ideas of data as a product,
184
00:17:03.935 --> 00:17:04.835
data contracts,
185
00:17:05.535 --> 00:17:07.155
data tests, things like that?
186
00:17:07.855 --> 00:17:11.395
Yeah. Yeah. And I I think there's one salient thing that's
187
00:17:12.390 --> 00:17:22.875
it doesn't matter 10 years ago, 20 years ago, you don't know what your customer wants. And, like, you may be the smartest person in the world, but, like, you don't have the whole business or the whole organization in your head.
188
00:17:23.175 --> 00:17:23.915
And so,
189
00:17:24.615 --> 00:17:29.275
the best way to do that is to deliver small bits of value to your customer iteratively.
190
00:17:29.780 --> 00:17:34.440
And so to do that, you need to be able to make changes quickly with low risk.
191
00:17:34.820 --> 00:17:38.440
To do that, you need to think and interact with your customer directly.
192
00:17:38.755 --> 00:17:47.815
And also you need to to develop a sense of trust. And and I think what happens with a lot of data teams is you talk to a customer once you say, okay. I think you want 10 things.
193
00:17:48.120 --> 00:17:48.940
And they instantiate
194
00:17:49.640 --> 00:17:57.340
that and however they do that. And then they go off for weeks or months, and they come back and say, here, I built these 10 things. And the customer says, oh, that's great.
195
00:17:57.674 --> 00:18:19.385
I only want 3 of those. You know, really, those 7, I don't really need. And by the way, here's 5 more. And so the waste that happens and building things that aren't right for your customers is huge. And then the waste of keeping things running that aren't used anymore by your customers is huge. And so I'm I'm all about let's recapture that waste. And and if you look at it from a number standpoint,
196
00:18:19.924 --> 00:18:31.260
you could argue that 70% of the time is waste in data and analytic teams. In in some cases, it's 90% of the time. And so let's recapture that waste. Let's not focus on
197
00:18:31.705 --> 00:18:43.250
being able to write SQL faster because that's only in the the 30% of the time. Let's focus on the system that allows you to put SQL into production faster and monitor it because that's where you're gonna recapture all your time.
198
00:18:44.029 --> 00:18:44.529
Another
199
00:18:45.150 --> 00:18:49.090
note from what you were saying earlier about the DevOps Enterprise Summit
200
00:18:49.725 --> 00:18:50.225
is
201
00:18:50.605 --> 00:18:51.185
that DevOps
202
00:18:51.885 --> 00:18:53.345
went through a very long cycle.
203
00:18:54.125 --> 00:18:59.265
Even now, there are companies in the enterprise who are just starting to adopt those concepts.
204
00:19:01.230 --> 00:19:12.164
New things take a long time to percolate through into the enterprise, and I'm wondering what you see as some of the aspects of data ops, the iterative development and deployment of
205
00:19:12.544 --> 00:19:15.125
data and data products that
206
00:19:15.585 --> 00:19:17.590
are still lacking that
207
00:19:17.990 --> 00:19:19.210
enterprise capability
208
00:19:19.510 --> 00:19:24.010
and some of the things that you see as an industry that we need to address to be able to actually
209
00:19:24.390 --> 00:19:35.770
make that leap from, hey. We've got all these shiny tools and everybody in start ups and small companies is using them to all of these tools are robust and enterprise ready and will actually drive adoption at large scale.
210
00:19:37.030 --> 00:19:38.790
Yeah. I guess I just have this
211
00:19:39.270 --> 00:19:42.730
you know, you might disagree with me. I just don't think it's about the tools,
212
00:19:43.105 --> 00:19:48.325
Whether they're small scale or enterprise scale, I think it's about the system in which you work with those tools.
213
00:19:48.865 --> 00:19:54.809
And so I think we need to work on that. And so we've had enough tools. And honestly, I've been
214
00:19:55.190 --> 00:19:57.450
and it's fantastic that we've got,
215
00:19:57.910 --> 00:19:58.570
you know,
216
00:19:59.175 --> 00:20:09.070
a ton of orchestrators out there. They're all really cool. We've got a bunch of SQL templating tools. We've got a bunch of GUI tools to build things. We can now chat to an LLM and get some SQL.
217
00:20:09.690 --> 00:20:16.795
We can write Python. There's there's 50 different ways to take data
from one place, to put it to another and do some transformation on it.
218
00:20:17.255 --> 00:20:19.595
I just don't think that that's from a
219
00:20:19.975 --> 00:20:20.955
problem standpoint.
220
00:20:21.495 --> 00:20:21.995
If
221
00:20:22.420 --> 00:20:26.360
70% of your time is waste even more, you're improving the 30%.
222
00:20:26.660 --> 00:20:33.705
That's not where we're gonna make gains. Work on where the big problems are. And the big problems are the systems of work that people have
223
00:20:34.164 --> 00:20:35.544
and how they actually
224
00:20:36.245 --> 00:20:36.745
enable
225
00:20:37.365 --> 00:20:40.264
you to make rapid changes in small increments,
226
00:20:40.630 --> 00:20:49.290
run systems so you don't have these emergencies and war rooms where everyone on the team's kinda doing this group grope to find out where the problem is because the CEO's
227
00:20:49.985 --> 00:20:50.485
upset.
228
00:20:51.185 --> 00:20:52.165
And so I think
229
00:20:52.465 --> 00:20:59.205
in some ways, we've got it's not about the tools. It's about how you work with your team and those tools.
230
00:20:59.570 --> 00:21:04.470
And in some ways, it's not about the data. It's more about the processes acting upon data.
231
00:21:05.250 --> 00:21:10.805
Digging more into that, I think that one of the symptoms as well is that all of the tools are built
232
00:21:11.265 --> 00:21:11.765
for
233
00:21:12.145 --> 00:21:20.270
data engineers, data analysts. They're not built for other people in the business to actually look at them, interact with them in any fashion.
234
00:21:21.370 --> 00:21:27.215
There are some tools that are starting to move in that direction of being a hub of collaboration
235
00:21:27.674 --> 00:21:28.735
across the organization,
236
00:21:29.035 --> 00:21:31.855
which I think helps to enable that use case.
237
00:21:32.190 --> 00:21:39.650
And I'm wondering if you can talk to some of the ways that the massive growth over the past 7 years and recent consolidation
238
00:21:40.590 --> 00:21:41.090
has
239
00:21:41.554 --> 00:21:42.775
influenced the ways
240
00:21:43.395 --> 00:21:44.615
that data teams
241
00:21:44.915 --> 00:21:45.815
relate to
242
00:21:46.115 --> 00:21:52.380
the rest of the organization and the ways that they think about the purpose of the work that they're doing beyond just, hey. Isn't this really cool?
243
00:21:53.720 --> 00:21:56.700
Yeah. Well, if you actually, like, open up Azure, there's
244
00:21:57.105 --> 00:21:59.284
10, I think now 12 ways to
245
00:21:59.745 --> 00:22:06.725
do to integrate data, change data. And some of them have nice UIs that are targeted business users, some of which are more technical,
246
00:22:07.090 --> 00:22:10.790
some of which work on only some databases, some of which are open source.
247
00:22:11.170 --> 00:22:13.110
And so of those ten ways,
248
00:22:13.810 --> 00:22:18.695
every data and analytics system has this characteristic of where do you put the logic.
249
00:22:19.115 --> 00:22:19.615
And
250
00:22:20.035 --> 00:22:27.669
do you so let's say you get a new question from your customer. I want us can you segment my customer base? I wanna look at them this way.
251
00:22:27.970 --> 00:22:49.240
One way to do it is actually do it in the chart and graph tool. Another way is to do it in the front end of the chart and graph tool like like in Alteryx. 3rd way is have analytic engineers do it in the database. A 4th way is to is to have data engineers do it more on ingest. There's and and so where the logic lives in your system is actually a really interesting question. And so I I I'm a believer in in
252
00:22:50.325 --> 00:23:06.125
making things prove that they're worthy. And so from a logic standpoint, the farther you put it down the stack, the more general it is. But you don't know whether that segmentation is worthy or not. You just be wanna be able to run it. So maybe it should be in the BI tool. Maybe it should be the
253
00:23:06.924 --> 00:23:07.905
or maybe the
254
00:23:08.285 --> 00:23:12.625
analytic engineer should do it. It's a question and a discussion that you have with your customer
255
00:23:13.220 --> 00:23:14.200
about how you
256
00:23:14.900 --> 00:23:20.520
maximize the amount of learning and then either take something and push it into something that's a little bit more
257
00:23:21.135 --> 00:23:29.635
tested, a little bit more deployable, or or or maybe not. And so we're in the ideas and service business and and the promoting people to make quick changes.
258
00:23:30.250 --> 00:23:30.830
And then
259
00:23:31.530 --> 00:23:42.405
once it's proved morally, having a discussion about how we can industrialize this, how we can make it safe, how we can eat eat the technical debt that we may have created. Because not everything is worth
260
00:23:43.025 --> 00:23:44.245
testing and automating
261
00:23:44.700 --> 00:23:45.919
because it may not.
262
00:23:46.220 --> 00:23:46.720
So,
263
00:23:47.500 --> 00:23:50.399
I I think it's really about that discussion and and
264
00:23:50.860 --> 00:23:53.520
taking things like technical debt and testing
265
00:23:54.035 --> 00:24:06.039
and observability and deployment and getting all those things and and putting them on the table with your customer and saying, look. I wanna put this on the table as what we do as well as doing new things. And if you spend my time
266
00:24:06.419 --> 00:24:11.685
telling my team all to build new things, we're gonna build up so so much technical debt and so many problems that,
267
00:24:12.645 --> 00:24:13.785
guess what's gonna happen?
268
00:24:14.085 --> 00:24:19.865
People are gonna start to quit. We're gonna start to go slower. Everyone's gonna get frustrated, and you're gonna get frustrated.
269
00:24:20.300 --> 00:24:22.080
And so, to me, it's
270
00:24:22.700 --> 00:24:23.840
it's really about
271
00:24:24.620 --> 00:24:30.415
having a good relationship with your customer and a and a discussion and
making some things that aren't are invisible,
272
00:24:30.875 --> 00:24:36.015
like testing and automating and deploying visible to your customer and and where things live in this the stack.
273
00:24:36.555 --> 00:24:39.540
And, that's a a management technique that I think,
274
00:24:40.500 --> 00:24:41.880
you know, we all need to learn.
275
00:24:42.500 --> 00:24:45.960
I certainly learned it, and it's just hard because your customers don't really care
276
00:24:46.420 --> 00:24:47.020
kind of,
277
00:24:47.460 --> 00:24:48.760
you know, where you are.
278
00:24:49.645 --> 00:24:51.905
You know, they don't really care about that how things work.
279
00:24:54.765 --> 00:24:56.705
Data lakes are notoriously complex.
280
00:24:57.020 --> 00:25:07.575
For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst is an end to end data lake has platform built on Trino, the query engine Apache Iceberg was designed for.
281
00:25:08.355 --> 00:25:13.575
Starburst has complete support for all table formats, including Apache Iceberg, Hive, and Delta Lake.
282
00:25:13.900 --> 00:25:18.080
And Starburst is trusted by teams of all sizes, including Comcast and DoorDash.
283
00:25:18.620 --> 00:25:21.840
Want to see Starburst in action? Go to data engineering podcast.com/starburst
284
00:25:23.385 --> 00:25:29.405
today and get $500 in credits to try Starburst Galaxy, the easiest and fastest way to get started using Trino.
285
00:25:32.520 --> 00:25:33.500
Another aspect
286
00:25:34.280 --> 00:25:36.700
of this space that I think
287
00:25:37.560 --> 00:25:40.860
acts as a distraction is that with every new
288
00:25:41.755 --> 00:25:48.575
breakthrough or every new capability, there's another hype cycle of, hey. Hey. Look at the shiny new tool. It's gonna make you move faster and everything will be easier.
289
00:25:49.190 --> 00:25:54.169
Obviously, the latest one is large language models and the ways that they could be used to generate
290
00:25:54.549 --> 00:25:55.690
anything that you want.
291
00:25:56.205 --> 00:26:00.625
And as somebody who has been in the space for a while, I'm curious if you could maybe
292
00:26:01.005 --> 00:26:09.470
put this LLM and generative AI hype cycle in the context of other hype cycles that you've seen and some of the ways that you see it as maybe
293
00:26:09.930 --> 00:26:13.355
being beneficial and some of the risks that come along with it?
294
00:26:13.995 --> 00:26:17.054
I think it's beneficial. I just don't think it's revolutionary. I mean,
295
00:26:17.434 --> 00:26:17.934
so
296
00:26:18.315 --> 00:26:28.330
in 1990, I learned to program at NASA with man pages and a bunch of c c programs that of other people. And so and then I would go if I didn't know it, I'd go to this thing called a nerd bookstore
297
00:26:28.630 --> 00:26:36.715
that had all which is now extinct that had all, books, and there used to be quantum books in Cambridge and SoftPro. There was a one in
in the West Coast.
298
00:26:37.095 --> 00:26:39.835
And then, you know, it came along in the 2000, and
299
00:26:40.340 --> 00:26:53.935
people started to use Google search. Right? And and people started to to post answers to questions on the Internet. And and what happened is a whole bunch of new people became enabled to be programmed. You didn't have to be, like, a master's degree dude at NASA to figure it out.
300
00:26:54.235 --> 00:27:02.600
And I think what LLMs are gonna do are just gonna enable another realm of people to start programming more. And I think that's a good thing. But with anything
301
00:27:03.140 --> 00:27:08.120
with any kind of technical activity, it's complexity and how do you manage that complexity,
302
00:27:08.715 --> 00:27:17.534
especially when you have more people creating it? And then you you end up with this question. Is it worthy to be tested and observed? Where in the stack should it fit?
303
00:27:17.860 --> 00:27:37.080
It maybe we should throw it out. How do we generalize it? What's the right schema? These technic these technical debt questions that we need to continue to work on. And we've gotta stop building systems where you're rushing to get stuff in your customer's hands, and then you have no idea if it's gonna work the second day and no and no ability to change it the 3rd.
304
00:27:37.860 --> 00:27:39.480
So moving now into
305
00:27:39.780 --> 00:27:42.975
the tools that you were mentioning earlier, you recently open sourced
306
00:27:43.455 --> 00:27:47.475
a pair of utilities, DataOps test gen and DataOps observability.
307
00:27:48.655 --> 00:27:54.020
I'm wondering if you can talk to the specific outcomes that you're trying to drive with those projects
308
00:27:54.320 --> 00:27:57.300
and the motivation for releasing them as open source.
309
00:27:58.135 --> 00:28:02.235
I guess the number one motivation is that I talked to I've talked to
310
00:28:02.935 --> 00:28:05.435
100, thousands of companies over over the years,
311
00:28:05.895 --> 00:28:06.395
and
312
00:28:07.049 --> 00:28:11.870
they build systems and they're having errors in production and their customers are finding problems.
313
00:28:12.490 --> 00:28:21.205
And so their customer may be an an analyst team and maybe the CEO, but there's customer visible problems. And then they have to react. So they're reactive
314
00:28:22.120 --> 00:28:23.980
to customer problems. So
315
00:28:24.520 --> 00:28:29.820
our hope in this is that we can make them find problems before their customers see them.
316
00:28:30.165 --> 00:28:33.545
And there's 2 technical challenges that we built this on.
317
00:28:33.925 --> 00:28:37.690
Number 1 is that you have to find problems in data.
318
00:28:38.169 --> 00:28:40.509
And so finding problems in data
319
00:28:41.289 --> 00:28:53.665
means you have to check the data. And so how do you check the data? Well, you might wanna recount and look at anomalies over time. Will that make sense? But you also want
to actually dig into the the syntax of the data itself.
320
00:28:53.965 --> 00:28:55.585
And so our test gen tool
321
00:28:55.940 --> 00:28:59.320
profiles your data and then has has this sort of algorithmic
322
00:28:59.620 --> 00:29:18.570
AI engine that builds a whole set of data quality validation rules for you so you don't have to build them. And then exposes them through a nice UI, through, you know, through a YAML file so you can you can manage them. So the the problem that we're trying to solve with DataOps test gen is is automatic creation of a whole wide variety
323
00:29:19.110 --> 00:29:21.610
of data quality rules because data engineers
324
00:29:22.255 --> 00:29:27.395
either don't have the business context to understand what they what they are or they don't have the time.
325
00:29:27.934 --> 00:29:30.995
And so the reason we open source the second tool,
326
00:29:31.620 --> 00:29:32.120
observability,
327
00:29:32.500 --> 00:29:33.720
is that we think
328
00:29:35.300 --> 00:29:38.600
the problems in production are not just with data.
329
00:29:38.945 --> 00:29:48.950
You can have perfect data and still have a problem, a customer visible problem. You can have perfect raw data and perfect integrated data, but the model may not be right
330
00:29:49.330 --> 00:29:53.590
or the export to the UI or your reverse ETL may break.
331
00:29:53.970 --> 00:30:07.924
So you need something that sits on top of it, and we call that a data journey. And think of it as the fire alarm control panel for your data and analytic systems. You know, it's it tries to point you saying, hey, the fire's in this room in this corner.
332
00:30:08.450 --> 00:30:14.070
And it's we think that that concept of a data journey is missing from data and analytics systems.
333
00:30:14.370 --> 00:30:19.155
It's certainly missing if you want to regress a system. It's certainly missing in production.
334
00:30:19.535 --> 00:30:31.710
So the 2 really reasons we built these is, hey, testing data is complicated. You don't have time, so you need a a tool to help you generate tests. And then observing the entire complexity of production during production
335
00:30:32.330 --> 00:30:45.820
is something that you need to do. And we have this concept of a data journey that we've done. And both are available, both are Apache 2 0, both are completely full featured. And so the outcome that we're trying to do is just use these tools and don't, you know,
336
00:30:46.600 --> 00:30:47.260
say when
337
00:30:47.880 --> 00:30:55.174
you have a problem in production, start saying, oh, let's not have that. Let's try to find the way to fix it so that never happens again.
338
00:30:55.554 --> 00:31:01.750
And that's I think these things can accelerate. And we put a lot of work into making them sort of easy to install,
339
00:31:02.210 --> 00:31:06.690
easy to use. And we've been working on, you know, test gen for we I've been
340
00:31:07.490 --> 00:31:14.445
my team and I have been doing data testing for close to 20 years. And so we've got a whole bunch of best practice baked into it. And so,
341
00:31:15.245 --> 00:31:19.630
really excited. We've only open sourced it, like, 2 2, 3 months ago. So it's it's
342
00:31:21.630 --> 00:31:23.970
still quite new. The observability
343
00:31:24.350 --> 00:31:25.570
piece in particular,
344
00:31:26.510 --> 00:31:34.144
there are a number of tools that try to address at least a portion of that scope. And I'm wondering if you can talk to
345
00:31:34.524 --> 00:31:39.389
the ways that you see your DataOps observability and DataOps test gen tools
346
00:31:39.769 --> 00:31:40.269
overlapping
347
00:31:40.570 --> 00:31:41.470
with other
348
00:31:42.490 --> 00:31:43.710
capabilities in the ecosystem
349
00:31:44.235 --> 00:31:47.055
and some of the ways that they are differentiated
350
00:31:47.595 --> 00:31:53.055
either because of their ease of use or their capabilities and just some of the ways that you think about their position in the market?
351
00:31:53.730 --> 00:31:54.230
Yeah.
352
00:31:54.690 --> 00:31:55.429
And so,
353
00:31:55.809 --> 00:31:56.950
first of all, I think
354
00:31:58.529 --> 00:31:59.669
whatever you do,
355
00:32:00.465 --> 00:32:04.725
don't it's not acceptable to hope that your production system works.
356
00:32:05.184 --> 00:32:13.559
And you need to test data and everything that's acting upon data from a principal standpoint. And then you need to pinpoint where the problem is quickly.
357
00:32:14.100 --> 00:32:14.840
And so
358
00:32:15.140 --> 00:32:18.500
those kinds of outcomes, I think, are are are really good. And if you've got
359
00:32:19.835 --> 00:32:26.815
so the person who's listening to this is add some tests yourself. You know, write some SQL tests. Just just do it. Don't wait for permission.
360
00:32:27.140 --> 00:32:31.560
Just start putting tests in. You don't need a tool to do this. And every time you have a problem,
361
00:32:31.940 --> 00:32:40.794
try to find a way to automate it or test it so it never happens again. Those kinds of ideas aren't you don't need to compete with someone. And so I think the market itself
362
00:32:41.174 --> 00:32:47.960
is for data observability tools. Is very focused on ingest teams and kind of pulling databases for changes.
363
00:32:48.260 --> 00:32:51.799
And I think that's that's a good use a good place to start for some teams. Right?
364
00:32:52.455 --> 00:32:54.235
Proving that the ingest is right.
365
00:32:54.695 --> 00:33:14.315
But there's other use cases in data and analytics beyond that. And we do that. We pull databases. You know, we'll find anomalies in databases. That's fine. But, also, I think you need to to look at the full production cycle of data and analytics and test not just raw data, but integrated data. Actually, you need to test data before you even put it into production. You need to profile data.
366
00:33:14.695 --> 00:33:20.580
You need to be say, is this data even worthy of
being in production? Should you just push back on
367
00:33:21.360 --> 00:33:26.580
your data engineers or patch it up before it gets in production? So we have a sort of a more broader scope
368
00:33:26.915 --> 00:33:28.775
of how people do observability.
369
00:33:29.235 --> 00:33:32.615
But to me, I think start checking and testing your data
370
00:33:32.995 --> 00:33:34.935
manually, start looking at products.
371
00:33:35.310 --> 00:33:41.730
And I think what all the data observability vendors, I think we all agree on, it's just utterly unacceptable to have embarrassing
372
00:33:42.110 --> 00:33:44.610
errors on your customer in front of your customer.
373
00:33:45.085 --> 00:33:54.465
And there you don't have to live with it. It's not a good way. You're not a hero. You're just you're just putting a bunch of waste in the system that's making everyone's lives,
374
00:33:54.970 --> 00:33:55.470
crappy.
375
00:33:56.490 --> 00:33:59.470
So for teams who are looking to
376
00:33:59.929 --> 00:34:08.375
adopt these tools, integrate them into their work flow. I'm wondering if you can give an overview about how to actually get them set up and
377
00:34:08.675 --> 00:34:10.695
some of the ways that the
378
00:34:11.970 --> 00:34:13.990
test gen and observability tool
379
00:34:14.290 --> 00:34:17.030
are going to be interacted with throughout a typical Workday.
380
00:34:18.050 --> 00:34:25.495
Yeah. Yeah. So it's just you you go to GitHub and, you know, we have an installer program. It takes about 10 minutes to install on your machine.
381
00:34:25.795 --> 00:34:36.950
And so let's say the first thing that you have is I've got a new file that I need to put into my data and analytics system. Well, test gen will profile it, and then it'll give you a bunch of data hygiene checks on it to say, well,
382
00:34:37.330 --> 00:34:49.390
this is this this column's blank. Maybe you should did you mean to have that? Or you've got 5 different ways to say the same thing. And these are kind of heuristic rules to help you say, maybe this data isn't good enough.
383
00:34:50.190 --> 00:34:57.484
And then as you get something in production, we can start pulling the database test gen and observability to find out if their things are
384
00:34:57.785 --> 00:35:02.125
wrong. And then as your data moves through its other tools, like models, visualization
385
00:35:02.425 --> 00:35:02.925
tools,
386
00:35:04.000 --> 00:35:07.780
export, reverse ETL tools. We have integrations to monitor its progress
387
00:35:08.400 --> 00:35:08.900
and,
388
00:35:09.200 --> 00:35:15.415
to check it if it's right. And so the full data journey as it moves from kind of source to destination we're checking
389
00:35:15.875 --> 00:35:17.494
on. And so all these tools,
390
00:35:17.795 --> 00:35:19.255
every if you look at
391
00:35:19.635 --> 00:35:20.454
where problems
392
00:35:21.075 --> 00:35:22.934
are in data and analytic projects
393
00:35:24.180 --> 00:35:26.840
and kind of think of them like a like a graph. Oftentimes,
394
00:35:27.540 --> 00:35:28.840
problems start with
395
00:35:29.140 --> 00:35:31.835
bad raw data. But then over time,
396
00:35:32.135 --> 00:35:35.275
once you work with your supplier, once you patch up the data,
397
00:35:35.735 --> 00:35:57.280
the data doesn't itself doesn't have problems, but also it's interaction with other datasets. And you start to learn more about the problems. And then as it goes into production, it's the problem isn't the data. The problem is someone changed some code acting upon data, and your system's not right. And then you need to start thinking about regression and and and other testing. And so I think
398
00:35:58.140 --> 00:36:02.320
where problems lie in your data and analytic team or data and analytics systems
399
00:36:02.704 --> 00:36:04.704
often change with the maturity of,
400
00:36:05.105 --> 00:36:08.805
of your system and the maturity of the dataset and how how complex it is.
401
00:36:09.670 --> 00:36:17.050
What are some of the most interesting or innovative or unexpected ways that you're seeing the observability and test gen utilities applied?
402
00:36:17.454 --> 00:36:19.714
Well, we have one organization who,
403
00:36:20.575 --> 00:36:21.795
they use Databricks,
404
00:36:22.255 --> 00:36:23.714
they did all their ETL
405
00:36:24.255 --> 00:36:26.770
and it's sort of product analytics. So,
406
00:36:27.410 --> 00:36:28.870
manufacturing product analytics.
407
00:36:29.410 --> 00:36:36.915
And so they have all these sort of stream of notebooks that does all this work, all these algorithms, and they had no idea if it ran. It had its own complex interactions.
408
00:36:37.615 --> 00:36:41.210
And so they just send a huge amount of data to our system, complex
409
00:36:46.550 --> 00:36:56.930
system in notebooks that they went ahead again, spent day 1 building a lot of. Complex system in notebooks that they went ahead again, spent day 1 building a lot of, and are able to find it. And and was a very fast and and easy case to do it.
410
00:36:57.490 --> 00:37:04.630
Another case is we just have a a customer who has sort of, I think, in the order of 80 datasets that are updated
411
00:37:05.355 --> 00:37:05.855
often
412
00:37:06.315 --> 00:37:11.535
hourly. And they're just very important, very high value. And and what they were able to do was was
413
00:37:11.835 --> 00:37:16.840
instrument the whole system of tasks by just sort of scanning it. So they got a a big step up
414
00:37:17.220 --> 00:37:23.160
on data quality validation testing because they sort of point to test gen at it and then auto automatically generated tests.
415
00:37:23.645 --> 00:37:25.825
Now as time went on, they had to
416
00:37:26.285 --> 00:37:29.345
write tests that were sort of specific to their business domain,
417
00:37:30.125 --> 00:37:34.830
tests that looked across other tables, tests that compared with history, tests that were very specific.
418
00:37:35.290 -->
00:37:43.055
And so they could instantiate those and and test gen. In fact, they they wrote some of them themselves, and and they run ran them their own way. And so we're very open
419
00:37:43.435 --> 00:37:46.575
in integrating data quality tests that are from from other tools.
420
00:37:46.875 --> 00:37:55.760
And so, really, it comes down to you can build it in whatever tool, but we can capture and observe it, and and we can actually generate data quality tests quickly.
421
00:37:56.724 --> 00:37:57.464
Once teams
422
00:37:57.765 --> 00:38:00.265
adopt these tools and they have better visibility
423
00:38:00.724 --> 00:38:11.290
into their end to end flow, how are you seeing that impact the way that they think about building new systems and some of the ways that that drives more of that
424
00:38:11.590 --> 00:38:12.090
organizational
425
00:38:12.390 --> 00:38:12.890
change?
426
00:38:14.525 --> 00:38:15.425
It lowers
427
00:38:16.204 --> 00:38:16.944
the fear.
428
00:38:17.724 --> 00:38:20.865
It lowers the shame that happens in organizations.
429
00:38:21.210 --> 00:38:24.109
And a lot of teams, they're sort of fear and shame bound,
430
00:38:24.650 --> 00:38:26.750
and that locks them up from
431
00:38:27.369 --> 00:38:28.270
making changes.
432
00:38:28.595 --> 00:38:34.935
And so the the it just becomes lighter, and you enjoy your work more. And I found that personally. Like,
433
00:38:35.849 --> 00:38:50.455
you know, I had to take back in 2005, I had to take my team through this, and I didn't know it. I was reading Deming at the time and thinking about manufacturing. But, you know, Deming had this thing called a quality circle where you sat around and said, well, what were our problems? Let's put them in a spreadsheet.
434
00:38:50.994 --> 00:38:54.860
How can we find the root cause? How can we fix it so it never happens again?
435
00:38:55.240 --> 00:38:55.740
And
436
00:38:56.040 --> 00:38:58.460
I think teams are lighter. They're happier.
437
00:38:58.760 --> 00:39:04.185
They're more able to make changes. They're more able to create. They're they're not wasting so much time.
438
00:39:04.565 --> 00:39:05.065
And,
439
00:39:05.445 --> 00:39:06.025
you know,
440
00:39:06.725 --> 00:39:09.305
it it you know, it's still early in the,
441
00:39:09.765 --> 00:39:15.280
release of these tools, so I don't have a 100 cases. But I do have the case where people have done
442
00:39:15.660 --> 00:39:23.585
with our automation tool. We've seen that a a lot of times to be able to, make these things. And and to me, that's that's why I'm still doing this. Is it just
443
00:39:23.885 --> 00:39:26.625
my life sucks. I see teams that are just really unhappy,
444
00:39:27.244 --> 00:39:29.105
and it's not about your tools.
445
00:39:30.150 --> 00:39:32.410
And it's not gonna get any better with whatever
446
00:39:32.790 --> 00:39:39.735
AI enabled super Gartner buzzword tool. It's just not gonna change until you start thinking in systems and thinking
in
447
00:39:40.295 --> 00:39:42.155
how you actually make your life better.
448
00:39:43.095 --> 00:39:44.715
In your journey of
449
00:39:45.255 --> 00:39:46.875
trying to drive
450
00:39:47.329 --> 00:39:52.309
the adoption of data ops and the principles go that go along with it and drive this organizational
451
00:39:52.690 --> 00:39:55.829
shift to make the data work
452
00:39:56.135 --> 00:39:56.875
more impactful,
453
00:39:57.415 --> 00:39:58.315
less wasteful?
454
00:39:58.615 --> 00:40:03.035
What are some of the most interesting or unexpected or challenging lessons that you've learned in that journey?
455
00:40:04.829 --> 00:40:12.130
Well, first of all, I think I suck at it because if you would ask me 10 years ago, I I I thought it would be all be done by now. Right? I thought
456
00:40:12.454 --> 00:40:14.474
at the at the most
457
00:40:14.855 --> 00:40:17.035
data and analytic teams are 5 years behind.
458
00:40:17.415 --> 00:40:20.315
And so it's a lot slower than I thought. And
459
00:40:21.059 --> 00:40:23.160
I'm not sure, you know, how,
460
00:40:23.940 --> 00:40:29.640
you know, our latest swing at the bat is to go away from leaders and to go to individual contributors.
461
00:40:30.075 --> 00:40:32.494
We spent a lot of years building a tool that
462
00:40:32.875 --> 00:40:34.095
that sort of encapsulated
463
00:40:34.395 --> 00:40:35.214
the entire
464
00:40:35.755 --> 00:40:39.615
value stream of data and analytics. It was an orchestration test enhanced
465
00:40:40.609 --> 00:40:41.109
orchestrator
466
00:40:41.810 --> 00:40:44.470
to try and move things into production quickly.
467
00:40:44.770 --> 00:40:48.150
And it was really hard to get that to to make money with it,
468
00:40:48.635 --> 00:40:54.095
honestly, because people would go ahead and build a whole bunch of stuff. And we had to catch them right at the beginning of their projects.
469
00:40:54.635 --> 00:40:55.135
And
470
00:40:56.010 --> 00:40:58.830
so for me and I don't know. I asked you, Tobias,
471
00:40:59.770 --> 00:41:01.550
it's slower than I thought. And
472
00:41:02.170 --> 00:41:04.325
and am I am I being negative?
473
00:41:04.785 --> 00:41:05.605
Am I being
474
00:41:05.905 --> 00:41:14.960
too hopeful? Do you think it's never gonna be solved? Do you think this is what do you think of the whole idea of DataOps and its maturity and impediments? What what what's your stand on it?
475
00:41:15.500 --> 00:41:18.240
I think that we have come a long way.
476
00:41:18.619 --> 00:41:19.839
I think that DevOps
477
00:41:20.205 --> 00:41:22.225
took a decade before it
478
00:41:22.525 --> 00:41:29.345
even reached any modicum of widespread adoption. And as you noted, it still hasn't made its way through everywhere.
479
00:41:30.480 --> 00:41:31.380
I think that
480
00:41:31.680 --> 00:41:33.380
from a tooling perspective
481
00:41:33.839 --> 00:41:38.020
and an industry perspective, there has been a lot more focus on
482
00:41:38.465 --> 00:41:43.605
making sure that the data that we build and deliver is actually reliable and trustworthy
483
00:41:44.385 --> 00:41:52.010
and that we have a way of being able to actually test changes before they make their way into production and to end users.
484
00:41:52.789 --> 00:41:53.770
I think that
485
00:41:54.630 --> 00:42:01.494
there are a lot of different players in the space, and so it makes it hard to get everybody aligned. It makes it hard for everything to
486
00:42:01.954 --> 00:42:05.654
integrate effectively to be able to actually make sure that
487
00:42:06.140 --> 00:42:06.880
your testing
488
00:42:07.579 --> 00:42:08.079
environment
489
00:42:08.460 --> 00:42:10.480
is actually able to be
490
00:42:11.099 --> 00:42:12.319
run end to end
491
00:42:12.619 --> 00:42:13.920
because there are certain
492
00:42:14.475 --> 00:42:24.790
point solutions of, okay, my warehouse can have 0 copied tables, and I can make some changes to that without impacting production. But can I do the same thing with my,
493
00:42:25.490 --> 00:42:32.310
business intelligence environment and just some of the ways that that end to end integration can hamper the
494
00:42:32.675 --> 00:42:33.175
testing
495
00:42:34.195 --> 00:42:40.455
and validation of the changes that you're making before you pull the trigger and say, okay? Put all of this into production all at once.
496
00:42:41.270 --> 00:42:42.250
Yeah. Yeah.
497
00:42:42.630 --> 00:42:45.849
Yeah. I I think there is a lot of progress. And, like and I think
498
00:42:46.390 --> 00:42:57.875
I think more and more people are are seeing it. And I'm talking to teams that are sort of agility is sort of expected now and not. But of course, I talk to other teams where it's it's rare. And
499
00:42:58.480 --> 00:43:01.299
and I it's very interesting that you would think
500
00:43:02.160 --> 00:43:14.545
you know, it's not the usual suspects. Right? The sort of small companies who are unagile. They're big companies who are agile. It's it's just very interesting, the the the layout. And and from from my standpoint, I think the other
501
00:43:14.880 --> 00:43:26.335
challenging thing, and this is from a corporate standpoint, our our company, I'm just really grateful to have that we've never gotten investors, that we've always been profitable because, like, getting investors would have given us a timeline.
502
00:43:26.875 --> 00:43:27.855
And, like, my
503
00:43:28.235 --> 00:43:30.895
I never really trusted my ability to predict the future.
504
00:43:31.355 --> 00:43:37.599
Certainly with human adoption of ideas, I always thought I would, like, get it wrong. And so building a profitable company,
505
00:43:38.059 --> 00:43:39.359
investing in our employees,
506
00:43:39.819 --> 00:43:43.359
making our customers satisfied, and not having to, like, make these,
507
00:43:43.915 --> 00:43:44.415
you
508
00:43:44.875 --> 00:44:00.330
know, timelines that I'm not sure if I could keep, I think are good. Because we're gonna
still be here. We're still profitable. We're still gonna be around. You know, hopefully, a decade from now, we'll still be I'll still be I'll still be talking to you, Tobias, at 10 years. I'll probably still be talking about the same thing. But, hopefully, it'll be a lot better.
509
00:44:02.055 --> 00:44:05.755
Yeah. I I think another element of this too is that
510
00:44:06.695 --> 00:44:12.590
we talk about this idea of data teams who are doing their work, but I think a lot of it too is that
511
00:44:12.970 --> 00:44:19.070
you have a company, they start with a software team, they start to realize, oh, we need to be able to do these
512
00:44:19.385 --> 00:44:19.885
longitudinal
513
00:44:20.185 --> 00:44:24.045
analytics. So they just say to the software team, hey. Go build me something.
514
00:44:25.145 --> 00:44:25.645
And
515
00:44:26.025 --> 00:44:31.800
despite the fact that it's all still software that you're working with, there is a lot more dimensionality
516
00:44:32.180 --> 00:44:41.075
to data problems, and there's still a generalized lack of understanding of how that changes the way that you have to think about the problem space.
517
00:44:41.454 --> 00:44:45.460
And I think that there just aren't enough people in the ecosystem
518
00:44:45.760 --> 00:44:47.220
who have that context
519
00:44:47.680 --> 00:44:49.460
to be able to make that
520
00:44:50.685 --> 00:44:51.665
a widespread
521
00:44:52.045 --> 00:44:52.545
availability
522
00:44:53.165 --> 00:44:53.665
of
523
00:44:54.045 --> 00:44:56.705
understanding how to tackle the problem properly
524
00:44:57.810 --> 00:45:07.270
beyond just I write some software and I put it into production, and, oh, now I've gotta go back and keep fixing things and fixing things because they don't understand it from a holistic perspective.
525
00:45:08.435 --> 00:45:10.135
Yeah. Yeah. And and I guess
526
00:45:10.675 --> 00:45:27.875
the change I'm seeking is that individual contributors who are making that change and going back and fixing it aren't gonna passively quit. Right? They're gonna say, no. We can actually fix this. Here's the steps, and they're gonna start to talk to their manager and get their get their manager bought into it. Because I think that's
527
00:45:28.515 --> 00:45:37.089
I saw that with agile adoption. There were individual contributors that said, look, I don't wanna spend 6 months building something and then no one does this. Can we have a can we actually build something in 2 weeks?
528
00:45:37.390 --> 00:45:56.050
We're gonna learn a lot more. And I think what it comes down to is it's just gonna your life's gonna be better, and you're gonna have a lot more fun in your job if you can iterate quicker. You can run things with low errors. You can measure your work. And I think those things, I think, have to come from individual contributors because your bosses are just you know,
529
00:45:56.830 --> 00:46:00.690
they're they got a shotgun to their head, and they're getting shot every 18 months.
530
00:46:01.335 --> 00:46:03.355
And so it it's it's unfortunate,
531
00:46:03.975 --> 00:46:07.355
but it's gotta come from the bottom. And I think it's a real opportunity for
532
00:46:07.790 --> 00:46:11.650
people new to their career. Because I one of the things that makes me sad is people who've
533
00:46:12.109 --> 00:46:16.369
gone to school. They've got a master's degree. They they came of age when,
534
00:46:17.025 --> 00:46:21.685
you know, data scientist was the sexiest job in the world, and then they're just like they're they're depressed.
535
00:46:22.065 --> 00:46:32.450
They don't like their job. They're like, why am I why am I stuck fixing and re fixing things? Why am I not doing anything innovative? And that's just you know, I'd like that I'd like that to change. But I think, collectively,
536
00:46:33.335 --> 00:46:38.555
you know, we've gotta start sort of holding our teams to higher standards and and how we work together.
537
00:46:39.815 --> 00:46:53.865
As you continue to build and invest in the tools that we discussed and your overall mission at DataKitchen, what are some of the things you have planned for the near to medium term or any particular projects or problem areas you're excited to dig into?
538
00:46:54.325 --> 00:46:56.105
Well, I I'm excited to dig into
539
00:46:57.525 --> 00:47:03.220
the role of a data quality person on a data team because I think they've been underinvested on. And I think,
540
00:47:03.520 --> 00:47:16.385
you know, data observability is very focused on the ingest team and making sure the ingest is good, but the data quality person is is really interesting. And we're actually doing some market research now, you know, to to talk to a bunch of data from data quality people because
541
00:47:17.325 --> 00:47:20.930
it it impacts, you know, having good data, but also
542
00:47:21.550 --> 00:47:22.050
having,
543
00:47:22.510 --> 00:47:44.140
their ability to influence the team. And so I think we're gonna look at that. And then I think one of my favorite things that I hope we're gonna get to, probably won't be this year, but I think the ability to capture analytics about your your develop about your your data and analytic team, their amount of product the amount of work they're doing, the amount of deploys they make, the amount of errors they make in production,
544
00:47:44.599 --> 00:47:50.865
their customer sat, like, sort of looking at the value streams that people create in production and deployment with their customers.
545
00:47:51.405 --> 00:47:56.119
It's amazing how many data and analytic teams don't metric themselves.
546
00:47:56.740 --> 00:48:03.319
And, you know, they produce metrics. They produce numbers, but they kinda don't know what's going on with their team from a numeric standpoint.
547
00:48:03.645 --> 00:48:05.744
I think that's the, you know, that's the you know,
548
00:48:06.285 --> 00:48:21.180
we spent our first product solving the 23 year
old deployment problem. Now our second and third products were solving the 23 year old monitoring and making sure there's low errors in production. And I think our hopefully, our last product will be solving the 46 year old problem. How do you run your team
549
00:48:21.575 --> 00:48:26.154
in an analytic way? And so, those are the things that we're we're thinking about.
550
00:48:26.694 --> 00:48:29.434
I think that that is noteworthy too because
551
00:48:30.240 --> 00:48:34.660
in that concept of the parallel between DevOps and DataOps
552
00:48:35.200 --> 00:48:36.660
is the concept of
553
00:48:37.039 --> 00:48:39.140
being on call, error monitoring,
554
00:48:39.585 --> 00:48:43.765
trying to drive down mean time to resolution, mean time to detect failures.
555
00:48:44.865 --> 00:48:49.520
I think that that is one of those pieces of the data ecosystem that is still
556
00:48:49.900 --> 00:48:59.325
in the process of being figured out of what does it even mean for a data person to be on call, and what does it mean for there to be an incident in the data platform?
557
00:49:00.265 --> 00:49:04.765
Yeah. Yeah. And what is it? And is it number of incidents? What's the impact?
558
00:49:05.305 --> 00:49:06.765
And I think it's
559
00:49:07.450 --> 00:49:09.390
important to measure those things because
560
00:49:10.410 --> 00:49:15.265
if you look at it from a manufacturing line, your error rates and how you manufacture
561
00:49:15.565 --> 00:49:18.785
are and bottlenecks are huge impediments to success.
562
00:49:19.245 --> 00:49:39.865
And so, like, you don't need fancy software to do it. Like, I encourage every team to get a spreadsheet and every time you have any production error, put it down in that spreadsheet. And then just every 2 weeks, sit with your team and say, what's the one thing we can do to automate this so it never happens again? What's the script? What's the test? What's the fix we can do? And so
563
00:49:40.650 --> 00:49:41.869
I I think that sort
564
00:49:42.250 --> 00:49:46.829
of being able to reflect and and think of it think of it as love your errors.
565
00:49:47.289 --> 00:49:56.645
Try to have a way where errors aren't cause you sort of shame and blame, but let's love them and try and fix them and improve them. And I think that will help your team
566
00:49:57.410 --> 00:50:01.829
be more productive and get better. And you don't have to do any software to do that. You just have to have a spreadsheet.
567
00:50:02.770 --> 00:50:09.085
Are there any other aspects of this overall ecosystem of DataOps, the work that you're doing at DataKitchen,
568
00:50:09.545 --> 00:50:11.565
the specifics of the observability
569
00:50:11.865 --> 00:50:16.180
and test gen tools that we didn't discuss yet that you'd like to cover before we close out the show?
570
00:50:18.000 --> 00:50:19.380
No. I I, you know,
571
00:50:19.760 --> 00:50:23.835
I may sound negative. Right? Like, people are depressed. They wanna quit.
572
00:50:24.375
--> 00:50:32.090
You know, it's frustrating. But I think in this change, and I think the fact that, like, we've had this great burst of tool creativity
573
00:50:32.550 --> 00:50:34.570
and the fact that we're seeing some consolidation.
574
00:50:34.950 --> 00:50:35.770
And I think
575
00:50:36.870 --> 00:50:45.714
I think it's actually a really opportunity for people to reflect on how they do work and improve it. And I think there's so many new people and and they're running into the same similar sets of problems
576
00:50:46.255 --> 00:50:47.555
that I'm excited
577
00:50:48.100 --> 00:50:51.240
at the future. And I think it's it's really gonna be a good thing when,
578
00:50:51.780 --> 00:50:53.080
you know, you don't have to
579
00:50:53.860 --> 00:51:00.234
have these huge data and analytic teams where you could have a smaller one that's more in that does more work, does better work,
580
00:51:00.535 --> 00:51:04.630
and the teams are happier and and people aren't quitting. And I'm looking forward to
581
00:51:05.109 --> 00:51:06.170
helping sort
582
00:51:06.470 --> 00:51:07.690
of nudge that along
583
00:51:08.069 --> 00:51:08.809
in the future.
584
00:51:09.670 --> 00:51:18.164
Alright. Well, for anybody who wants to get in touch with you and follow along with the work that you and your team are doing, I'll have you add your preferred contact information to the show notes.
585
00:51:18.464 --> 00:51:25.970
And as the final question, I'd like to get your perspective on what you see as being the biggest gap in the tooling or technology that's available for data management today.
586
00:51:26.750 --> 00:51:33.925
Yeah. I think it's that last one. I think it's the 46 Euro problem of how do you measure the success of your data and analytic team. I think both definitionally,
587
00:51:34.705 --> 00:51:41.120
what is a successful team look like? You know, what are the numbers? There's no in software, there's Dora Metrics,
588
00:51:41.820 --> 00:51:46.960
DevOps Research Association that they have sort of a set of metrics that are really sort of software
589
00:51:47.635 --> 00:51:48.455
deployment focused.
590
00:51:48.755 --> 00:51:56.080
I think those, and we have a set of deployment metrics. We have a set of production error metrics. We have productivity, customer sat.
591
00:51:56.480 --> 00:52:05.825
I think we need to the biggest gap is really measurement of our team and what those measurements are. And, like, you know, I've often thought we need a Dora group for data and analytics.
592
00:52:06.305 --> 00:52:20.390
What's the common way to benchmark teams in terms of how well they're doing, how successful they are, and what things that we should measure. And so that's another challenge, I think. So I think some smart entrepreneur will will pick up or someone someone should work on.
593
00:52:20.930 --> 00:52:29.985
Alright. Well, thank you very much for taking the time today join me and share the work that you're doing on the observability and test gen tools, your perspective
594
00:52:30.285 --> 00:52:31.425
on the evolution
595
00:52:31.725 --> 00:52:32.465
of DataOps
596
00:52:32.930 --> 00:52:33.910
and its adoption,
597
00:52:34.369 --> 00:52:41.565
and the ways that we are trending as an industry. So I appreciate the insights and the time, and I hope you enjoy the rest of your day.
598
00:52:42.045 --> 00:52:43.025
Alright. Thank you.
599
00:52:50.349 --> 00:52:54.130
Thank you for listening, and don't forget to check out our other shows. Podcast.netcoversthepythonlanguage,
600
00:52:55.950 --> 00:53:03.585
its community, and the innovative ways it is being used, and the AI Engineering Podcast is your guide to the fast moving world of building AI systems.
601
00:53:04.365 --> 00:53:14.060
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email host at data engineering podcast.com
602
00:53:14.440 --> 00:53:15.260
with your story.
603
00:53:15.560 --> 00:53:20.734
Just to help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers.