Adding An Easy Mode For The Modern Data Stack With 5X

WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 7/6/2024 10:45:27 AM
Duration: 3372.259
Channels: 1
1
00:00:11.264 --> 00:00:15.445
Hello, and welcome to the Data Engineering Podcast, the show about modern data management.
2
00:00:16.384 --> 00:00:18.164
Introducing RudderStack Profiles.
3
00:00:18.785 --> 00:00:28.630
RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable enriched data to every downstream team.
4
00:00:29.255 --> 00:00:35.595
You specify the customer traits, then profiles runs the joints and computations for you to create complete customer profiles.
5
00:00:36.370 --> 00:00:39.510
Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack.
6
00:00:42.370 --> 00:00:43.510
Data lakes are notoriously
7
00:00:43.890 --> 00:00:44.390
complex.
8
00:00:44.915 --> 00:01:00.579
For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte scale SQL analytics fast at a fraction of the cost of traditional methods so that you can meet all of your data needs ranging from AI to data applications to complete analytics.
9
00:01:01.015 --> 00:01:04.555
Trusted by teams of all sizes, including Comcast and DoorDash,
10
00:01:04.855 --> 00:01:11.200
Starburst is a data lake analytics platform that delivers the adaptability and flexibility a lakehouse ecosystem promises.
11
00:01:11.900 --> 00:01:18.560
And Starburst does all of this on an open architecture with first class support for Apache Iceberg, Delta Lake and Hoody,
12
00:01:18.895 --> 00:01:21.315
so you always maintain ownership of your data.
13
00:01:21.935 --> 00:01:24.755
Want to see Starburst in action? Go to dataengineeringpodcastdot
14
00:01:26.100 --> 00:01:47.280
com slash starburst and get $500 in credits to try Starburst Galaxy today, the easiest and fastest way to get started using Trino. Your host is Tobias Macy, and today I'm welcoming back Tarush Agarwal to talk about what he and his team at 5x Data are building to improve the user experience of the modern data stack. So, Tardush, can you start by introducing yourself for folks who haven't heard any of your past appearances?
15
00:01:48.140 --> 00:01:53.275
Of course. Hey, Tardush. Good to be on the show again. I think this is my 3rd time. My background,
16
00:01:53.735 --> 00:02:06.820
I'm the founder of 5 x. My backgrounds have been very focused in the data space, got to be 1 of the first data engineers at Salesforce, 1 of the first data engineers in the world, most recently ran data for WeWork, been very, very narrowly focused
17
00:02:07.440 --> 00:02:10.660
on what used to be called the modern data stack or data infrastructure.
18
00:02:11.715 --> 00:02:12.215
And,
19
00:02:12.835 --> 00:02:14.935
you know, we we sort of started 5x
20
00:02:15.875 --> 00:02:16.375
about
21
00:02:16.675 --> 00:02:20.230
about 2 and a half years ago, and I'm sure we'll get all into that.
22
00:02:20.629 --> 00:02:24.010
And, again, for folks who haven't heard your past appearances,
23
00:02:24.470 --> 00:02:35.275
do you remember how you first got started working in data management, data engineering, and what it is that's held your attention this long? Yeah. Absolutely. You know, obviously, we've been through many, many pivots. And
24
00:02:35.735 --> 00:02:38.155
when we first started, we were actually building
25
00:02:39.470 --> 00:02:41.170
a course or a program
26
00:02:41.470 --> 00:02:41.970
for
27
00:02:42.750 --> 00:02:50.185
data for sort of companies wanting to invest in data. You know, post WeWork, I would get a lot of calls from CEOs,
28
00:02:50.564 --> 00:02:53.144
you know, looking to bring in a head of data, and this was
29
00:02:53.540 --> 00:02:59.720
SMBs, midsized enterprise, real estate, banks, SaaS, ecommerce, you know, across the entire spectrum.
30
00:03:00.180 --> 00:03:03.775
And everyone was talking about, you know, monetizing their data and, you know,
31
00:03:04.395 --> 00:03:05.935
and sort of data is the new oil.
32
00:03:06.395 --> 00:03:19.180
And in reality, all of those conversations went like, hey. You know, I think, you know, you're sitting on a gold mine of data, but the reality is is you have to go build this data platform, and these are the vendors to speak to. And, you know, it's not rocket science, but it's highly contextual.
33
00:03:19.815 --> 00:03:25.355
And, you know, there's this thing called data modeling and, you know, BI and self-service, which are really the first steps.
34
00:03:25.895 --> 00:03:29.435
And so out of that, you know, we just
35
00:03:30.930 --> 00:03:31.830
figured that
36
00:03:32.290 --> 00:03:37.910
many, many businesses are trying to get started and, you know, expertise in this space didn't quite exist.
37
00:03:38.365 --> 00:03:40.945
So the first version of 5 x was really,
38
00:03:41.725 --> 00:03:50.650
a course to actually you know, this is kind of what we've learned in the industry over the last 10 years. This is the playbook on how to go do it, and that failed miserably.
39
00:03:51.189 --> 00:03:54.489
We barely had anyone buy it. But, yeah, that was
40
00:03:55.565 --> 00:03:59.584
v 1 of back then, we were sort of 5x data. Now just 5x.
41
00:04:00.045 --> 00:04:05.025
In terms of what you're building at 5x, you gave a little bit of the backstory, how it came to be.
42
00:04:05.709 --> 00:04:10.690
And the last time we spoke about your work was in March of 2022,
43
00:04:11.069 --> 00:04:13.330
so we're at about a year and a half now.
44
00:04:13.665 --> 00:04:16.165
Wondering what are some of the notable changes
45
00:04:16.705 --> 00:04:30.240
in the overall product focus, what you're building, how you're pitching it, and just the ways that the drastic shifts in the data infrastructure ecosystem over that time have influenced the ways that you're thinking about that problem?
46
00:04:30.585 --> 00:04:39.885
Yeah. I know. It's sort of great question. You know, a few years ago, you know, pre modern day stack, like, we had everyone using Informatica.
47
00:04:40.800 --> 00:04:45.300
And and still today, you know, many, many, many enterprises are on Informatica.
48
00:04:46.080 --> 00:04:47.940
And some big advantages
49
00:04:48.715 --> 00:04:51.375
of Informatica was it's an end to end platform.
50
00:04:51.835 --> 00:05:00.390
You can do everything on it. You can ingest and transform and, you know, you have all of your capabilities inside a single platform, and that was really powerful.
51
00:05:00.930 --> 00:05:01.430
Now
52
00:05:01.890 --> 00:05:02.630
the users
53
00:05:03.225 --> 00:05:10.710
of, you know, sort of data teams hated the product because it just isn't a very robust product, very easy to use. It's it's,
54
00:05:11.669 --> 00:05:13.130
and on the banks of that,
55
00:05:13.509 --> 00:05:17.449
you know, we had all of these different vendors go after individual pieces.
56
00:05:17.885 --> 00:05:25.185
So we had the 5 Trans and the Snowflakes and the DBTs and the BIs and, you know, reverse ETL and sort of 15, 20 other categories.
57
00:05:25.930 --> 00:05:28.190
And fast forward, you know, a few years,
58
00:05:28.490 --> 00:05:32.110
you know, we have 500 vendors in sort of 30 different categories.
59
00:05:32.810 --> 00:05:33.310
And
60
00:05:33.945 --> 00:05:34.445
today,
61
00:05:35.145 --> 00:05:38.445
what's happened, especially in the last 18 months, is that
62
00:05:39.305 --> 00:05:49.580
blank checks to your data team just because your investors wanted you to invest in data are no longer a thing. And CFOs are getting involved. And, you know, data teams
63
00:05:49.965 --> 00:05:52.384
have been hit pretty hard, and they need to show ROI.
64
00:05:53.085 --> 00:05:58.305
And the analogy we use it is what happened from, you know, the 1 platform is,
65
00:05:58.630 --> 00:06:10.005
you know, the analogy we used today is all of these vendors today are selling car parts. And if you walked into a Honda and instead of selling you a Honda Civic, if they sold you an engine, we would have much fewer cars on the street
66
00:06:10.465 --> 00:06:13.045
because building your car and maintaining your car
67
00:06:13.505 --> 00:06:14.245
are significant
68
00:06:14.545 --> 00:06:15.045
overhead.
69
00:06:15.680 --> 00:06:19.860
And so we see sort of data teams today, we sort of struggle to go manage
70
00:06:20.320 --> 00:06:21.620
multiple different vendors,
71
00:06:22.375 --> 00:06:26.875
especially at the enterprise stage, like, you know, that lunch with sort of Snowflake and Looker,
72
00:06:27.415 --> 00:06:28.875
10 lunches every month
73
00:06:29.490 --> 00:06:34.790
sort of sort of starts to become too much. Right? And everyone's trying to go optimize for their for their part of the puzzle.
74
00:06:35.490 --> 00:06:38.470
So, you know, we're entering this world which is
75
00:06:39.145 --> 00:06:41.884
do more with less and and sort of data teams
76
00:06:42.185 --> 00:06:46.125
are are getting back to really sort of going and focusing on ROI.
77
00:06:46.840 --> 00:06:57.365
So what 5 x is is how do we give you that Informatica experience, that end to end platform experience, having 1 neck to choke, unified billing, single provisioning, a single place to operate,
78
00:06:57.985 --> 00:07:05.820
except it's built on top of it's built in a very modular way. So depending on your industry, on your use case, on your size, on your budget,
79
00:07:06.280 --> 00:07:12.755
we can build you a data platform. And as you grow in scale and requirements change and you need and you need new capabilities,
80
00:07:13.375 --> 00:07:16.595
we can add on and sort of swap out existing pieces,
81
00:07:16.900 --> 00:07:19.639
and you get the entire end to end experience.
82
00:07:20.340 --> 00:07:21.880
So, you know, we're really,
83
00:07:23.060 --> 00:07:27.675
and I think, you know, when we spoke in sort of a year and a half ago, we were in early stages of that.
84
00:07:27.975 --> 00:07:36.910
But sort of today, we're seeing, you know, what this looks like at the, you know, more and more at the enterprise scale and sort of the difference it's making and making
85
00:07:37.449 --> 00:07:38.990
and making data teams
86
00:07:39.610 --> 00:07:41.710
hilariously more efficient. Another
87
00:07:42.815 --> 00:07:43.315
interesting
88
00:07:44.095 --> 00:07:50.735
wrench in the works of the ways that people are thinking about data, particularly in the past 6 months to a year, is the,
89
00:07:51.750 --> 00:07:58.730
frenetic pace of AI and the fact that every company has decided that they need to implement some sort of AI or LLM
90
00:07:59.294 --> 00:08:02.354
capability into their product, whether it makes sense or not,
91
00:08:02.735 --> 00:08:04.895
and that also adds additional,
92
00:08:06.370 --> 00:08:07.110
strain and
93
00:08:07.570 --> 00:08:12.150
infrastructure requirements around the data platform. And I'm curious how you've seen that
94
00:08:12.530 --> 00:08:18.764
influencing the ways teams are thinking about data, talking about data, building around data, and
95
00:08:19.145 --> 00:08:28.810
whether the outside view of, oh, everybody's adding AI is realistic or if it's more just that maybe there are a couple of toy projects that are maybe Skunk Works,
96
00:08:29.110 --> 00:08:42.400
and the core of the business is still just standard, you know, let's get our business and reporting and maybe a couple ML models, and we're focused on just this core capability and AI is somewhere off in the wings, maybe it'll become a thing later on.
97
00:08:42.940 --> 00:08:43.920
That's a great question,
98
00:08:44.300 --> 00:08:47.600
and I think there are a few different ways to look at it. I think,
99
00:08:48.524 --> 00:08:50.065
for sure, we are
100
00:08:50.764 --> 00:08:52.625
seeing data teams being
101
00:08:53.084 --> 00:08:53.584
pressured
102
00:08:54.285 --> 00:08:56.225
to have an AI strategy.
103
00:08:57.090 --> 00:09:01.910
So it's it's sort of very real. It's sort of data teams that sort of bring this up all the time.
104
00:09:02.690 --> 00:09:04.710
We're obviously very much
105
00:09:05.235 --> 00:09:06.055
in the infancy
106
00:09:06.355 --> 00:09:35.660
of AI inside the data world. You know, we have some text to SQL. You know, what I'm actually very, very excited about is is sort of conversational BI. In order to get there, things like the data modeling layer, things like the semantic layer become even more important, you know, to essentially give AI sort of context on a business. Because things like your sort of definition of what is MRR, you know, how many daily active users you have are extremely nuanced. Right? So, you know, we think the semantic layer is really
107
00:09:36.040 --> 00:09:39.740
is sort of gonna be our best shot at sort of getting into conversational AI.
108
00:09:40.115 --> 00:09:42.375
Having said all of these things, if you just look
109
00:09:42.754 --> 00:09:44.774
at the data life cycle in general,
110
00:09:45.475 --> 00:09:49.175
it's I think there are 2 different aspects to consider. What is,
111
00:09:49.770 --> 00:09:52.830
you know, sort of data practitioners and people in the industry
112
00:09:53.370 --> 00:09:57.935
sort of really talk about and, you know, VCs, what they're investing in, and a lot of that is like
113
00:09:58.495 --> 00:10:00.595
future sort of state of data.
114
00:10:01.135 --> 00:10:03.555
And, you know, what I find really interesting is
115
00:10:04.095 --> 00:10:04.595
companies,
116
00:10:05.350 --> 00:10:09.930
you know, just the core data stack, right, the 5 trans, the dbts, the sort of snowflakes,
117
00:10:10.550 --> 00:10:13.370
they're, you know, just getting into sort of
118
00:10:14.035 --> 00:10:15.895
adoption at the enterprise level.
119
00:10:16.274 --> 00:10:26.410
They're just now getting, you know, sort of I think Snowflake is a little bit ahead, but all of the other vendors, just the core vendors, you know, are finally starting to be sort of adopted now.
120
00:10:27.270 --> 00:10:32.885
So, you know, there's a big delay in, you know, what practitioners and and what's happening in the industry
121
00:10:33.265 --> 00:10:40.885
versus really enterprise adoption looks like. So I think, you know, with that lens in mind, you know, from the enterprise standpoint,
122
00:10:41.860 --> 00:10:42.440
we are
123
00:10:43.060 --> 00:10:46.440
quite far away from this. And I think from the SMB standpoint,
124
00:10:47.140 --> 00:10:58.875
you know, we're looking at something in the next, you know, I think end of next year too. Still quite early, but I think 2025 is, I think, when sort of AI and data will will sort of will sort of start to get really interesting.
125
00:10:59.590 --> 00:11:00.090
And
126
00:11:00.550 --> 00:11:05.050
in terms of the modern data stack, that was a term that started getting thrown around,
127
00:11:05.350 --> 00:11:09.175
I'd say, probably in the 2020 time frame in particular,
128
00:11:09.634 --> 00:11:17.620
and that was also around the time that the venture investment in data infrastructure and data startups was at an all time high.
129
00:11:18.160 --> 00:11:22.819
They would throw money at anybody that had the word data somewhere in their pitch, and
130
00:11:23.754 --> 00:11:26.735
now that has dramatically shifted where,
131
00:11:27.595 --> 00:11:28.415
data infrastructure
132
00:11:28.954 --> 00:11:35.560
is, you know, the previous generation of interest. There's still work being done there. There's still successful businesses, but it's not
133
00:11:36.100 --> 00:11:45.725
a situation where there is room for everybody with an idea to get funding and run a business, and so we're starting to see a cycle of consolidation in the ecosystem.
134
00:11:46.680 --> 00:11:54.940
And I'm wondering how you're seeing that influence the ways that people think about what actually constitutes the modern data stack, what are the capabilities that are actually
135
00:11:55.605 --> 00:12:01.865
necessary and required and which were the ones that were perhaps frivolous or maybe just a feature of a larger product?
136
00:12:02.779 --> 00:12:07.279
Yeah. You know, I think having 15 different categories is probably
137
00:12:07.980 --> 00:12:08.880
it was
138
00:12:09.420 --> 00:12:18.134
an overkill of, you know, going from an end to end ecosystem to a fragmented ecosystem. And we're gonna see a a sort of lot of these categories become features,
139
00:12:19.170 --> 00:12:25.430
which can be adopted by sort of other categories. So consolidation has to happen. Right? Like, the way the world stands, we're also looking at
140
00:12:25.935 --> 00:12:29.315
we're looking at a lot of data companies today, which will sort of struggle,
141
00:12:29.935 --> 00:12:33.235
in the next 12 to 18 months as our runway dries up.
142
00:12:33.950 --> 00:12:35.570
So we are gonna see consolidation.
143
00:12:36.190 --> 00:12:37.890
And I think that's something which
144
00:12:38.270 --> 00:12:41.170
with sort of 5 x, which has always been the sort of consolidation
145
00:12:41.515 --> 00:12:43.935
of the data stack, is actually very excited about
146
00:12:44.475 --> 00:12:44.975
because
147
00:12:45.435 --> 00:12:48.255
what starts to happen when you start to consolidate
148
00:12:48.635 --> 00:12:49.135
again
149
00:12:49.675 --> 00:12:50.175
is
150
00:12:50.870 --> 00:12:59.290
optionality needs to exist. Like, 1 BI tool, you know, it's very difficult for Snowflake to go acquire 1 BI tool because
151
00:12:59.795 --> 00:13:03.495
they then get locked into the very specific
152
00:13:04.115 --> 00:13:09.175
use cases which that BI tool is good at. And so different companies have different use cases.
153
00:13:09.700 --> 00:13:17.640
So, you know, I think it's very healthy for those sort of 15 categories to potentially come down to, you know, 6, 7, 8 core categories.
154
00:13:18.175 --> 00:13:26.195
But inside these categories, you're still gonna have a bunch of dominant players. Right? Even if you just look at the data warehousing space, which is very, very mature,
155
00:13:26.589 --> 00:13:32.450
you know, we have Snowflake. We have Databricks. We have BigQuery. You know, we still have companies on Redshift.
156
00:13:33.390 --> 00:13:34.270
Sort of DuckDB,
157
00:13:35.175 --> 00:13:39.834
is a company which, you know, has recently made huge headlines, you know, to think of
158
00:13:40.135 --> 00:13:47.240
a 5th warehouse right now getting funded in a big big way in a in an already pretty sort of mature space is just
159
00:13:47.700 --> 00:13:58.285
more confirmation that they're gonna be many, many different use cases. And depending on what what sort of cloud you are and what type of use cases you are, you're gonna have solutions which make more sense than others.
160
00:13:58.620 --> 00:14:00.240
So while consolidation
161
00:14:00.620 --> 00:14:01.120
helps
162
00:14:01.500 --> 00:14:03.360
decrease the overall footprint
163
00:14:03.899 --> 00:14:07.600
even in a world with 8 different categories you're gonna want to have optionality.
164
00:14:08.464 --> 00:14:12.325
5x is, you know, the answer to that. And especially when you look at, you know, enterprise,
165
00:14:12.785 --> 00:14:15.365
it's not data stack, it's data stacks.
166
00:14:15.670 --> 00:14:17.210
You know, you have different
167
00:14:17.750 --> 00:14:20.890
set of subsidies, acquisitions, or even different departments
168
00:14:21.350 --> 00:14:34.269
sort of using different tools. Right? Like, you know, what sort of what do you find in large enterprise companies? It's not are they Snowflake or Databricks? Very often, they're both. And when you look at, again, you know, how do you take now
169
00:14:35.209 --> 00:14:56.230
extremely disjointed space? What what sort of 5x allows you to do is we have this concept of workspaces and a workspace can belong to a department or a subsidiary or, you know, the core data team. And you're just looking at what are the vendors which make sense for you. You have a single ID experience to operate those vendors without logging into different tools. I can go forecast my cost
170
00:14:56.665 --> 00:15:00.765
of what does my spend on, you know, my particular snowflake instance
171
00:15:01.145 --> 00:15:05.325
sort of look like. You know, you you can have shared tools across different workspaces.
172
00:15:05.960 --> 00:15:14.620
You know, some might have their own. And these things at the enterprise level become really, really difficult to manage. Even sort of even sort of digital identity. Right? Access
173
00:15:15.045 --> 00:15:24.745
to all of these different tools, the sort of permissions. And, again, how do you do this across subsidiaries and who has access to what and how do you audit this, right, from a central team perspective? So these are a whole
174
00:15:25.190 --> 00:15:26.170
suite of tools,
175
00:15:27.350 --> 00:15:29.930
which haven't been ever addressed
176
00:15:30.470 --> 00:15:38.334
by the sort of fragmentation of the space. Like, you would have to kinda go do this yourself and then go back to this sort of building your own car analogy.
177
00:15:38.955 --> 00:15:51.130
You know, at an SMB, great. You know, the many tutorials on how do you go spin up the the core 3, 4 layers in, like, in, like, 2 or 3 hours, this starts to balloon extremely quickly. So, you know, for all of these different types of use cases,
178
00:15:51.584 --> 00:16:21.339
even in inside a consolidated world, there's gonna be a huge need for this, especially at the mid market enterprise level. The identity and permissioning aspect is something that is very poignant for me because I'm dealing with that in my own data stack efforts of the the situation of the modern data stack. Bring all your tools, but then every tool wants to own that permissioning piece, and none of them are really want to talk to each other. There's no central control plane. There there's no agreement about how that should even manifest.
179
00:16:21.640 --> 00:16:22.300
That that's
180
00:16:22.760 --> 00:16:35.780
definitely a big problem and 1 that I can see being compounded when you're in an enterprise and you don't even necessarily have consistency across the tooling in the different business units, and then you have to figure out who has permission to what data, where, and why.
181
00:16:36.240 --> 00:16:36.980
Yeah. Exactly.
182
00:16:37.440 --> 00:16:42.340
And, you know, Oktaar and all of those sort of digital identity solutions are great, but
183
00:16:42.915 --> 00:16:50.935
they've worked more in a in software engineering context where they're managed by IT teams, which sit sort of separately from engineers, then it becomes fairly straightforward.
184
00:16:51.440 --> 00:17:03.615
But the world of beta is way more nuanced. Right? Like, all analysts don't get the same permissions. You have analysts with different permissions, and you have analysts in different departments. So when you add on that other layer of, you know, nuance,
185
00:17:04.075 --> 00:17:08.895
it goes away from a tool which is which is primarily managed by your IT department.
186
00:17:09.440 --> 00:17:14.740
Is something which you wanna have a robust solution which can be managed by your sort of data teams or,
187
00:17:15.120 --> 00:17:22.995
more operational teams. So we think, you know, removing digital identity in the data space, you know, we can go support Okta and work with them,
188
00:17:23.535 --> 00:17:24.550
but having that
189
00:17:25.030 --> 00:17:25.930
that pain,
190
00:17:26.550 --> 00:17:29.930
actually managed in your data platform becomes really essential
191
00:17:30.390 --> 00:17:31.530
at the enterprise level.
192
00:17:31.845 --> 00:17:32.905
In terms of
193
00:17:33.525 --> 00:17:34.185
the consolidation,
194
00:17:35.045 --> 00:18:01.919
the shifts in the data economy of we actually need to make sure that the money we're putting into this is worth it, and we're not just gonna throw money at it because everybody says we should throw money at it, and someday it'll be useful going back to the, era of Hadoop of just throw data in there, and, eventually, that data will become useful. Now we're starting to narrow down a little bit more where the last cycle was well, we actually only wanna collect the data that we know is useful because, otherwise, we have liability problems.
195
00:18:02.460 --> 00:18:09.675
Now it's we only wanna invest in the data infrastructure and the data capabilities that we know are gonna be useful because, otherwise, we're gonna have money problems.
196
00:18:10.055 --> 00:18:13.080
And I'm curious how you're seeing that calculus
197
00:18:13.460 --> 00:18:19.080
start to influence the work that data teams are doing, the ways that they think about building their infrastructure,
198
00:18:19.515 --> 00:18:20.815
you know, whether to,
199
00:18:21.434 --> 00:18:37.710
say yes to all of the data requests that are coming their way, or maybe there is a little bit more pushback about, well, why are you asking for this? I'm just wondering how how that is influencing the way Yeah. The data teams are operating. So I think there are 2 elements to that. Right? Like, when you bring on when you bring in sort of fin ops into the conversation,
200
00:18:38.645 --> 00:18:55.650
what everyone obviously talks about is the cost of data infrastructure. Right? Like, again, what is the cost of all of these tools? And they and they add up pretty quickly. But the sort of second piece, which has been sort of spoken about less, is the people in data. And we have apart from building a very fragmented ecosystem, we've also
201
00:18:56.255 --> 00:19:03.795
invented job titles faster than, you know, than sort of universities could even sort of keep up with sort of actually training people in some of these different professions.
202
00:19:04.850 --> 00:19:10.309
So there's consolidation on the infrastructure, which which we have been talking about, which is happening. But I think we're also
203
00:19:10.850 --> 00:19:13.110
entering the rise of the sort of data generalist
204
00:19:13.705 --> 00:19:20.285
and having, you know, do more with less is a theme which is which is being sort of universally applied.
205
00:19:21.059 --> 00:19:23.399
And we're seeing with consolidation,
206
00:19:24.100 --> 00:19:26.200
there is more of a need for,
207
00:19:26.659 --> 00:19:30.039
you know, people working in the data realm to be able to,
208
00:19:30.404 --> 00:19:47.775
you know, go manage the platform, do ingestion, do modeling, do BI. And, you know, we're going back to, you know, sort of consolidation across roles. And, you know, we're gonna see the rise of, like, much leaner teams. And, again, you know, in the sort of 2019 sort of 2020 is and I was guilty of this,
209
00:19:48.175 --> 00:20:10.924
myself for show. And I was running data teams. It's, you know, in some ways, the the sort of metric between sort of data leaders was, you know, the size of your data team and, you know, how big, you know, you know, what teams do you have and, you know, what different roles are you bringing in, like, and what are all these different use cases and these people would sort of vary with with sort of very special skill sets. I think that's, like, reversing. And, you know, you you're gonna see the
210
00:20:11.225 --> 00:20:13.644
that sort of rise of, like, lean teams,
211
00:20:14.020 --> 00:20:15.720
which are just way more efficient,
212
00:20:16.420 --> 00:20:25.304
because you don't you know, you're not paying the most expensive tax in people, which is the communication tax as you build, like, sort of larger and larger teams and just being able to do more,
213
00:20:25.845 --> 00:20:44.865
because, you know, sort of finely tooling is at a point where it's more mature. And, again, with sort of with sort of the consolidation, just 1 person is just able to do a lot. It's to be way more end to end. So I think we're we're actually seeing it from both these different perspectives in terms of they've been layoffs. And I think data teams have been hit particularly hard,
214
00:20:45.245 --> 00:20:46.544
as well as more pressure
215
00:20:47.085 --> 00:21:02.325
on bringing cost down across infrastructure as a whole. Particularly as you start talking about enterprise and coordination across teams and across business units, then you start bringing in the conversations around things like data fabric, data mesh, these architectural principles,
216
00:21:02.625 --> 00:21:14.150
data as a product. I'm wondering if you've seen that start to come to fruition and teams are actually building that, and it's they're realizing the value that is promised by these approaches or if it's largely been,
217
00:21:14.770 --> 00:21:25.415
something that is maybe interesting and not as effective or harder to put into practice and a lot of confusion. Just wondering how that how these architectural ideas are also influencing the ways the teams are thinking about building their systems.
218
00:21:25.720 --> 00:21:30.700
I think at the risk of being very sort of controversial over there, like, are these features
219
00:21:31.160 --> 00:21:31.900
are these,
220
00:21:32.200 --> 00:21:35.180
you know, just sort of processes and thinking about
221
00:21:35.625 --> 00:21:52.155
how to go structure your data teams, or are these actually entire categories? And just going into a few of them, like, you know, we don't have any any sort of problem with your next catalog tool or observatory tool or mesh or semantic layer. They make sense as sort of features. But the idea is introducing
222
00:21:52.615 --> 00:22:01.570
1 more tool into the equation, 1 more place where your team has to log in, 1 more vendor you deal with, 1 more, you know, sort of platform where you have to go sort of do digital identity.
223
00:22:01.870 --> 00:22:14.605
The answer of, hey, we wanna go solve this problem, and we're gonna build a new platform on top of your existing platforms to go do it, That's that's that's sort of that's sort of not the answer. And I think a lot of these will potentially be rolled into sort of bigger platforms
224
00:22:15.320 --> 00:22:16.779
in in sort of consolidation.
225
00:22:17.399 --> 00:22:25.065
But, again, going back into the enterprise use cases and, again, you know, we get to speak with some of the biggest sort of data consultancies in the world.
226
00:22:25.544 --> 00:22:26.044
And
227
00:22:26.424 --> 00:22:29.404
what they tell us is 80, 90% and
228
00:22:29.705 --> 00:22:34.524
and a lot of these consultancies obviously do a lot of referrals and do a lot of reseller.
229
00:22:34.960 --> 00:22:51.615
And sort of speaking to them, would it you know, 80, 90% of their referrals are going into just core 4 layers, ingestion, storage, modeling, reporting. So, you know, sort of going back to kind of what I said earlier, you know, at least some massive enterprise companies which are implementing
230
00:22:51.995 --> 00:22:53.294
sort of these sort of solutions,
231
00:22:53.610 --> 00:23:04.565
I think as we get into this year and next year and the year after that, I think some of those renewal conversations are gonna be extremely difficult conversations for some of these platforms to have because the hype cycle of great,
232
00:23:05.105 --> 00:23:06.965
you have a huge mess.
233
00:23:07.505 --> 00:23:09.925
And this 1 tool is gonna come and solve everything.
234
00:23:10.225 --> 00:23:10.725
Unfortunately,
235
00:23:11.529 --> 00:23:14.669
having a layer on top of everything else is not the answer to that.
236
00:23:16.970 --> 00:23:39.460
You shouldn't have to throw away the database to build with fast changing data. You should be able to keep the familiarity of SQL and the proven architecture of cloud warehouses, but swap the decades old batch computation model for an efficient incremental engine to get complex queries that are always up to date. With Materialise, you can. It's the only true SQL streaming database built from the ground up to meet the needs of modern data products.
237
00:23:40.185 --> 00:23:43.005
Whether it's real time dashboarding and analytics, personalization
238
00:23:43.305 --> 00:23:44.045
and segmentation,
239
00:23:44.345 --> 00:23:51.620
or automation and alerting, Materialise gives you the ability to work with fresh, correct, and scalable results, all in a familiar SQL interface.
240
00:23:52.000 --> 00:23:53.540
Go to data engineering podcast.com/materialize
241
00:23:55.120 --> 00:23:56.740
today to get 2 weeks free.
242
00:23:58.880 --> 00:23:59.380
And
243
00:23:59.735 --> 00:24:09.434
talking to maybe the individual team scope and building the infrastructure components, I'm curious what are the main points of friction or the most difficult
244
00:24:09.880 --> 00:24:12.940
decisions that they have to make as far as how to
245
00:24:13.240 --> 00:24:14.940
implement their data platform?
246
00:24:15.480 --> 00:24:50.700
Yeah. You know, I think what's happening now is we're seeing companies who have a platform. You know, they have they they sort of built everything sort of actually coming to us to go consolidate. Right? Being like this is becoming really difficult to manage. In reality, we just want 1 neck to choke. So can you actually go take over all of these different vendors and really go consolidate this? Right? And so we're doing this exercise across, you know, mid market, across enterprise. And what we're seeing is, you know, the first question really is, are you are you using the right vendors? Because the reality of it is this is our full time job. Right? Like, we have you know, our entire team is is sort of is sort of tracking the space. This is what we do on a daily basis, and we can barely keep up with what's happening.
247
00:24:51.105 --> 00:24:56.405
So to think that, you know, a data team at an enterprise company is going to make
248
00:24:56.865 --> 00:24:58.245
necessarily the best decisions
249
00:24:58.650 --> 00:25:08.525
on what vendors to use. And number 2 is even after what vendors to use, are you using them in the right way? You know, how they're being set up, like, building your entire modeling layer inside LookML,
250
00:25:09.625 --> 00:25:42.570
isn't probably the best thing as you think about now reusing a lot of that modeling layer and, you know, pushing it into data science, so you are sort of sort of sort of using reverse ETL. Again, as some of these tools start to overlap, right, like, something which we see in enterprise companies all the time is, you know, a portion of the jobs are built inside 5 tran, inside their modeling, you know, inside their dbt integration, and some are built, you know, traditionally on dbt. Some are built inside Looker's semantic layer. Creates this sort of huge mess all over the place. So the idea of having sort of very consistent workflows that you can do inside a single UI is, again, becoming
251
00:25:42.885 --> 00:25:43.945
it's really simplifying.
252
00:25:44.485 --> 00:26:06.975
It's sort of preventing a big part of that mess in the first place. And we think the solution to this is is not so much gonna be on a layer on top, but it's really how do you, you know, set things up in a way which is, you know, again, give you an end to end experience so you don't have a mess in the first place. So a lot of these data teams, again, is they've spent all this money building this platform, and now they're starting to question the ROI of it. You know, unfortunately,
253
00:26:07.355 --> 00:26:19.355
we are gonna have many of them going back to the drawing boards and being like, which 1 of these decisions were the right decisions? And, actually, I think the bigger questions coming up is are we even the right people to make a decision on what tool makes sense,
254
00:26:19.735 --> 00:26:28.770
sort of sort of for us? So I think, again, I think it's just gonna be I I think we're the direction which we're heading is sort of going back to the basics
255
00:26:29.150 --> 00:26:30.210
of what is
256
00:26:30.590 --> 00:27:00.899
fluff and what is, you know, all of these solutions, which actually actually promised us they can fix it. People sort of waking up and realizing that a layer on top isn't the answer. We have to kinda go back to the drawing board and figure out how do we do this correctly. Absolutely. And now digging into what you're building at 5 x, ways that you are thinking about how to unify that experience, what are the appropriate tools and vendors, and how to integrate them. Wondering if you can talk through some of the learnings that you've gone through as you're building out your current iteration of the platform
257
00:27:01.200 --> 00:27:06.580
and some of the choices that you've made and how you've approached that tool and vendor selection.
258
00:27:07.005 --> 00:27:08.705
Yeah. You know, in terms of vendor selection,
259
00:27:09.085 --> 00:27:09.825
we are
260
00:27:10.365 --> 00:27:25.625
more like the apple of the space, you know, instead of the Android, you know. A year and a half ago, we were looking at opening up our ecosystem such that any vendor can kinda go integrate with us. But, really, what we've, you know, decided to do is is go partner with 15, 20,
261
00:27:26.245 --> 00:27:32.970
sort of core vendors. We work extremely closely with them. We integrate with them at a deep level. We're integrated into the sales process.
262
00:27:33.270 --> 00:27:40.090
You know, we have API. We have access to APIs, which are not public APIs and really provide, you know, a sort of much sort of deeper experience
263
00:27:40.924 --> 00:27:50.145
and have been you know, we think of the world inside capabilities. Right? Like, what are the capabilities which your data team needs? Like, you know, we think reverse ETL is a capability.
264
00:27:50.500 --> 00:27:52.360
We think sort of catalog is a capability.
265
00:27:52.740 --> 00:28:10.399
BI is a capability. So when we look at the conversations in your use case and sort of capabilities, we're not just looking at it at an individual sort of vendor level, but what are the set of vendors which work really well together? And we are sort of doubling down on existing vendors, so we're taking opinions on, you know, which ones we think, you know, work really well.
266
00:28:10.700 --> 00:28:24.685
We have optionality inside categories. So, you know, we don't pick 1 vendor across the category. But, you know, depending on the needs of our customers, you know, we we sort of try to have an open source offering. We try to have the, you know, the sort of the sort of commercial,
267
00:28:25.065 --> 00:28:36.980
you know, sort of non open source sort of products. And, you know, we think about what are the different types of use cases and what for those use case, what would be the best vendor? And we're coming up with these sort of golden paths. Right? Like, you know, this set of vendors
268
00:28:37.345 --> 00:28:39.525
work really well together. And increasingly,
269
00:28:40.305 --> 00:28:58.205
more people are paying us for our opinion on based on my use cases, what works really well together instead of we just hired someone who came from x y z vendor, and we're gonna go use x y z vendor. And, you know, I have no idea about the layer before this and the layer after this, and, you know, we'll just figure it out. From the perspective of that integration
270
00:28:59.225 --> 00:29:00.925
and unifying the experience,
271
00:29:01.385 --> 00:29:02.525
what are the
272
00:29:03.120 --> 00:29:03.860
core elements
273
00:29:04.160 --> 00:29:06.740
of making sure that there is a cohesive
274
00:29:07.200 --> 00:29:09.940
platform field to these disparate tools
275
00:29:10.295 --> 00:29:20.510
and the engineering work that you and your team have had to do to be able to give that a more contiguous flow? Yeah. So, you know, we have this concept of ID. You know, we call it super ID
276
00:29:20.890 --> 00:29:23.070
or sort of also unified ID,
277
00:29:23.690 --> 00:29:37.085
where you can go operate all of these different vendors inside a single product. So, you know, you have an ID on top of your warehouse. You can go, you know, go sort of sort of edit your dbt jobs. You can, you know, ingest data.
278
00:29:38.170 --> 00:29:58.200
So we have each sort of product becomes an app inside our experience. And so some of them get embedded. You know, we're able to work with the vendors on, like, on, like, you know, sort of removing parts of their product inside the embedded experience so to see a more focused embedded experience. There's some categories like, you know, the IDE, the core, you know, sort of SQL IDE,
279
00:29:58.580 --> 00:30:25.965
which we'll go build ourselves. And the idea is, you know, if you're using a sort of a sort of Fivetran or you're using, hypothetically, an air bite, the apps will be slightly different because they don't map apples to apples to each other. So the pipeline has got a concept of jobs and airflow might have a sort of different concept. So the individual apps available inside the IDE are sort of slightly different from each other, but you have a single place to go do this. And why this is really powerful is that
280
00:30:26.265 --> 00:30:38.655
is that, you know, sort of data teams can now go figure out what are the ideal workflows. Sort of going back to that example of, are you doing some of your transformations inside Fivetran and then some inside DBT and then some inside sort of LookML?
281
00:30:39.115 --> 00:30:44.255
You know, in sort of logging into 5 different tools, it becomes really hard to go police this because
282
00:30:44.630 --> 00:31:07.910
sort of vendors aren't giving you feature flags that you put this that you put this sort of feature off. Like, that's you know, why would they do that? They want more adoption. They want more engagement across each of their products. Whereas inside 5 x, you know, we could still log in to your vendors. But, again, the goal is can we so can we give this to you as a single ID experience? So whatever your data team decides the path you want to do, we can make those apps available
283
00:31:08.290 --> 00:31:12.575
which go power those workflows. So this leads to a highly consistent
284
00:31:12.955 --> 00:31:13.455
experience
285
00:31:13.835 --> 00:31:15.215
because, you know, 80%
286
00:31:15.675 --> 00:31:27.340
of where someone is sort of logging in on a daily basis and doing their work sort of becomes part of, like, you know, this sort of becomes this sort of golden workflow of how things happen. So, you know, from our perspective,
287
00:31:27.799 --> 00:31:28.539
we decided
288
00:31:29.085 --> 00:31:32.465
to, you know initially, you know, we were a platform to go provision.
289
00:31:32.925 --> 00:31:41.490
We would handle the procurement, billing, legal, all of those things. We had the single digital identity layer so you could go manage your users. We had utilization
290
00:31:42.110 --> 00:32:02.180
to go look at your spend forecast and, you know, more tools and optimizing spend. We had the security, so audit logs, all these tools are going into a single place that are great for your c I s o and your all and compliance teams. So know, we always had those sort of fundamentals. I think a big shift for us has really come in sort of focusing on this unified
291
00:32:02.480 --> 00:32:03.460
ID experience
292
00:32:03.784 --> 00:32:16.130
sort of such that, you know, the data teams on a daily basis can go do all of this from 1 tool. And along with just making their life easier because they're not logged into 5 different tools, it promotes a lot of hygiene
293
00:32:16.510 --> 00:32:19.250
in terms of best practices, which can be more standardized.
294
00:32:19.710 --> 00:32:31.950
And, you know, you can have more guardrails up as to this is how we wanna do things as opposed to a free for all. Any vendor, just go log in and do it, in that way. And then from that unified IDE perspective,
295
00:32:32.330 --> 00:32:40.675
there are couple of interesting elements to that. 1 is engineers are very opinionated about the tools that they want to use for doing the work that they do, And I'm wondering
296
00:32:41.055 --> 00:32:45.555
what are some of the ways that you make that IDE experience
297
00:32:46.175 --> 00:32:50.260
customizable so that they can feel at home doing the work in that context?
298
00:32:50.800 --> 00:32:51.620
And, also,
299
00:32:52.160 --> 00:32:57.300
maybe some of the ways that you're thinking about how do we extend that experience
300
00:32:57.795 --> 00:33:08.149
into the tools that people are already using? Yeah. That's a great question. I think, you know, what we're gonna see is a lot more apps on sort of 5 x IDE. Right? And apps are,
301
00:33:08.610 --> 00:33:15.945
experiences on how you want to basically go operate them. So, you know, we are, again, going and building, you know, some of the core experiences.
302
00:33:16.325 --> 00:33:27.370
Again, you know, we see that in larger businesses, again, there's a little bit less, you know, inside enterprise again. You know, I think, again, at the SMB stage, you know, there's a lot of flexibility.
303
00:33:27.750 --> 00:33:31.049
People are free to kinda use their own tools. And, you know, again,
304
00:33:31.565 --> 00:34:18.670
some it's just they're way faster to move and adapt. And, you know, we see sort of circle and we see hex and, you know, different people are kind of using different things, and that's kind of all fine. Right? It's all very manageable. You can speak to 5 different people and you know it. Again, at enterprise, this becomes sort of very different. Right? Like so there is a certain level of of, like, flexibility and sort of customization. And, you know, there's a lot more stuff which we've planned in terms of the ID experience to, you know, go make it sort of more flexible. But I think, you know, what we're really focusing on now is what are some of these core sort of use cases which teams are really focused on, and how do we provide a really solid sort of unified way to go do this. So that's really what the focus is now, but we're gonna see a lot more, you know, app a a a sort of lot more experiences to go sort of modify this in a way makes sense for you while still having some guardrails,
305
00:34:18.970 --> 00:34:22.670
which the company wants to have in order to have a consistent experience.
306
00:34:23.345 --> 00:34:26.165
And from that consistent experience perspective
307
00:34:26.465 --> 00:34:26.965
to
308
00:34:27.505 --> 00:34:44.195
what is the ideal flow that users will experience when they say, I have this data problem I need to resolve. Either I need to onboard this data or I need to build this report or I need to ensure that these transformations are running. I'm curious what are the different stages
309
00:34:44.495 --> 00:34:50.980
of that development flow, some of the ways that you're thinking about how to manage versioning and change management,
310
00:34:51.360 --> 00:34:57.620
the auditability streams that you're integrating into that experience for managers or administrators,
311
00:34:58.305 --> 00:35:10.299
and just some of the key touch points in that overall experience and the ways that you're thinking about building this into a cohesive product? Yeah. You know, it's a great question. I think we don't want to go create
312
00:35:10.760 --> 00:35:12.059
the underlying
313
00:35:12.519 --> 00:35:40.849
sort of tooling. Right? Like, the reason we go partner with everyone in this space is we think they are, you know, very sort of robust solutions out there, which do a sort of phenomenal job in in sort of what they're doing. So we don't necessarily wanna go reinvent the wheel. Right? So a lot of the versioning, a sort of a sort of a lot of the branches, you know, sort of we think dbt does a great job in a lot of these different layers. And, again, you know, our ID supports sort of dbt natively. We have, our own version of sort of dbt core, which we've deployed, you know, more for, like, for sort of smaller customers.
314
00:35:41.150 --> 00:36:02.170
But we continue to be partners with dbt at the enterprise level. And, you know, we integrate into we integrate into that. So the whole premise is you get to have the underlying components are sort of powered by a solution which makes sense for your business. Then, you know, in the future, they could be coalesced. They could be sort of battalion. They could be, you know, whatever
315
00:36:02.550 --> 00:36:15.360
the tool which you want it underneath to basically go power the experience. We wanna create we wanna expose the functionality of that tool and the feature set of that tool to make this really robust for you. And as you are trying
316
00:36:15.980 --> 00:37:19.809
to simplify the experience of the end users, you're papering over some of the sharp edges of the different tools and platforms. I'm curious what are some of the edge cases that you've had to engineer around or some of the ways that you need to think about building in escape hatches for the case where somebody really needs to deep reach into the guts of the system that you are trying to prevent this nice interface to and some of the engineering work to be involved there? You can always log in directly into the tools. We never take that away. We're not OEMing any of this. We don't it's we don't have 5x branding on, like, every different piece. In fact, we go out of our way to make sure you can see what are the, you know, the tools which are actually going and powering this. And admins can sort of enable or disable the login button directly from 5 x. So you could go to 5 x. You could go over a tool, and you could push a button which says login. And you can go directly inside the tool and go do whatever you need to. We're not trying to take away any of that functionality. There are always gonna be edge cases, new features which a vendor sort of launches, which might not be integrated into IDE, sort of instantly
317
00:37:20.109 --> 00:37:23.250
parts of IDE, which we currently don't support.
318
00:37:23.630 --> 00:37:28.369
You know, we're looking at the world and, like, you will be able to go embed some of these experiences.
319
00:37:28.715 --> 00:37:36.815
So you wanna go do something on Snowpark and you wanna go run Spark jobs or you wanna go run Python jobs, you can go embed that,
320
00:37:37.480 --> 00:38:00.440
sort of that page into IDE so that, you know, it's an easy way which people can go sort of cooperate that. So, you know, our goal is to not, you know we aren't OEM ing any of this. So you have, you know, sort of full functionality. What we think about is for, you know, your sort of day to day job, you should be able to do majority of this inside a single ecosystem. The other aspect of building this unified
321
00:38:00.900 --> 00:38:03.880
experience is that you want to give people this
322
00:38:04.244 --> 00:38:24.885
nice easy flow, but you're also working with companies who probably have already made investments into their data platform that maybe they don't want to get rid of or maybe there's going to be a long deprecation path. I'm curious what that overall integration and migration process looks like where they say, we've already built a bunch of stuff, but we also wanna be able to have this unified experience.
323
00:38:25.425 --> 00:38:41.205
How how do you help to bridge that gap? Yeah. That's a great question. So the sort of 5 x platform doesn't care if you already have a vendor or you wanna go you want us to go manage your vendor relationship and go buy that vendor from us. You can either buy vendors from us and, you know, sort of simplify sort of billing,
324
00:38:41.585 --> 00:38:45.125
or you can go, import your own vendors and have you on vendor relationships.
325
00:38:45.619 --> 00:38:59.725
So it doesn't sort of make a difference. We have a bunch of hybrid where, you know, they might have a bidding relationship with a few vendors, and we introduce a few others. And that works great too. What we're seeing more is we are very sort of deliberate on the partners we choose to work with. We want
326
00:39:00.105 --> 00:39:02.365
extremely adopted partners on our ecosystem
327
00:39:02.820 --> 00:39:08.520
Because, again, we sort of realized we're not gonna go integrate with 500 different vendors. We're gonna, you know, have,
328
00:39:09.220 --> 00:39:17.945
sort of some vendors which we think are sort of extremely widely adopted. That these vendors are gonna stand the test of time, and we bring them on. So what we see is that it's
329
00:39:18.245 --> 00:39:19.705
rare unless they're
330
00:39:20.300 --> 00:39:32.714
not using a warehouse and they're doing stuff on s 3, and they have spark jobs on top of it. And then, you know, it's a completely different paradigm shift. In general, people on the warehouse world and, you know, we support all all 4 of the big warehouses. We support Snowflake, BigQuery,
331
00:39:33.015 --> 00:39:47.795
sort of Redshift, and sort of NextEer integrating with with sort of Databricks is a big focus of us. So, you know, we support all of the, you know, big players out of the box. So, yeah, we support all of the big players out of the box. So very often when companies want to move to us, it's
332
00:39:48.095 --> 00:39:49.875
not that they are that
333
00:39:50.255 --> 00:39:57.680
you know, for the most part, we're able to support most of the vendors which they are already working with. That's 1 piece. But, again, I think what we're seeing
334
00:39:58.060 --> 00:40:19.885
more and what we're gonna see a lot more next year is sort of companies coming to us and be like, this is our data platform. Can you actually go consolidate all of this, even the sort of even the sort of vendor relationships? And, you know, we want to have 1 neck to choke. So you handle all of that, and we can spend 100% of our time focused on actually delivering data without having to manage the vendor relationships.
335
00:40:20.585 --> 00:40:22.445
In terms of those vendor relationships
336
00:40:23.224 --> 00:40:23.630
and
337
00:40:24.029 --> 00:40:31.650
the onboarding work, the integration work, you mentioned that you decided fairly early on that it wasn't just going to be an open ecosystem.
338
00:40:32.029 --> 00:40:33.410
Anybody can come in
339
00:40:33.835 --> 00:40:35.295
and be part of this experience.
340
00:40:36.315 --> 00:40:42.700
What are your criteria for deciding which tools, which vendors will be incorporated into that platform,
341
00:40:43.000 --> 00:40:45.100
the work that you have to do to be able
342
00:40:45.800 --> 00:40:49.180
to integrate and expose that vendor and hook them into
343
00:40:49.615 --> 00:41:15.695
the overall experience. And I'm particularly interested in how that factors in for those vendors that don't fit cleanly into 1 category or another, and there's overlap between them? Yeah. That's a great question. We evaluate vendors on, like, 4 different criteria. Obviously, number 1 is the technology. We know what they're solving, what their product is. Number 2 is their road map. So where are they going in the future? What's becoming more and more relevant for them? Number 3 is
344
00:41:17.390 --> 00:41:52.930
is to sort of is is to sort of partnership. Like, do we have alignment inside? Are we you know, do we have a deep partnership? You know, the sales sort of level, do we have other sales customer support at the sort of at the product level? And number 4 is just there's some sort of categories which we haven't entered. Right? Like, we think they still might be relevant, but we don't have a really good opinion on on where that category is going and, you know, how so how do we look at it? Sort of sort of sort of fourth 1 is sort of how do they fit in into the sort of general macroeconomic sort of climate. We use this sort of criteria. Again, we wanna partner with some vendors which we think are on the uprise.
345
00:41:53.230 --> 00:41:57.890
And we wanna partner and we wanna build sort of deep integrations into these vendors,
346
00:41:59.055 --> 00:42:01.394
And so we can go provide the best experience
347
00:42:02.174 --> 00:42:03.714
for our customers on 5x.
348
00:42:04.734 --> 00:42:05.714
And I think for
349
00:42:06.174 --> 00:42:07.154
for, like, vendors,
350
00:42:07.829 --> 00:42:11.210
you know, who are across multiple different categories,
351
00:42:11.510 --> 00:42:14.329
you know, we are starting to see that. Right? For example,
352
00:42:15.105 --> 00:42:20.485
we are starting to, you know, sort of partner heavily with RudderStack, and they do
353
00:42:21.345 --> 00:42:22.565
the CDP stuff,
354
00:42:23.000 --> 00:42:25.660
but they also have a reverse CTL offering.
355
00:42:25.960 --> 00:42:26.460
So
356
00:42:26.840 --> 00:42:27.500
I think,
357
00:42:28.360 --> 00:42:37.175
you know, all of this is on a case to case basis. We're partnering with with an end to end called peak dotai, which is, you know, an end to end experience across data science.
358
00:42:37.715 --> 00:42:40.775
And we sort of mainly focus on the data engineering
359
00:42:41.630 --> 00:42:43.250
and sort of analyst personas.
360
00:42:43.790 --> 00:42:46.850
So, you know, we our partnership with peak is, you know,
361
00:42:47.390 --> 00:42:49.170
more so that a lot of the customers
362
00:42:49.655 --> 00:42:58.235
who wanna do data science need a sort of data engineering persona, and and sort of a lot of our customers who have infrastructure from 5 x ask about a data science platform.
363
00:42:58.550 --> 00:43:07.130
So it's, you know, a little bit sort of disjoint. But, again, we're all we're only starting to see, you know, sort of multiple different categories overlap. For example, workflow manager
364
00:43:07.475 --> 00:43:14.135
is something which is sort of consistent across both. Right? You know, both sort of data science use cases, as well as data entering use cases
365
00:43:14.435 --> 00:43:16.455
sort of need sort of workflow manager.
366
00:43:16.890 --> 00:43:20.190
So as we really sort of now get into
367
00:43:20.890 --> 00:43:24.510
going deeper into all of this, you know, it's not gonna be
368
00:43:25.195 --> 00:43:44.905
as clean. It sort of never is. They I it's gonna be overlap, and I think our product and ensuring teams are sort of I I sort of started to think about these things. In your work of building the 5 x data platform, working with your customers, working with vendors, what are some of the most interesting or innovative or unexpected ways that you've seen your product used?
369
00:43:45.845 --> 00:43:51.385
So what we very intentionally decided to do a few months ago is sort of double down on our consultancy.
370
00:43:52.000 --> 00:43:55.540
We've always had a small consultancy to sort of co help customers.
371
00:43:56.160 --> 00:43:59.140
Again, as we look at the enterprise landscape,
372
00:43:59.835 --> 00:44:01.375
there's a lot of fragmentation
373
00:44:01.675 --> 00:44:02.734
there too because
374
00:44:03.195 --> 00:44:12.259
a particular vendor is gonna go set is gonna go sell you their product. And when you ask for help on it, they're gonna introduce you to an SI integrator or a sort of consultancy.
375
00:44:13.039 --> 00:44:13.539
And
376
00:44:13.975 --> 00:44:17.195
if we really want to make sure that people are doing things correctly,
377
00:44:17.575 --> 00:44:21.915
you know, being able to actually offer services and, you know, help out with some of these implementations
378
00:44:22.375 --> 00:44:24.200
or as needed bring in the expertise.
379
00:44:24.660 --> 00:44:26.760
We just think it's it's just
380
00:44:27.780 --> 00:44:31.075
part of helping our customers go get value from data.
381
00:44:31.395 --> 00:44:34.214
So, you know, we're partnering with a bunch of our our
382
00:44:35.075 --> 00:44:42.510
vendors. You know, again, a deep partnership where we're also become we're also becoming SI providers and can go, you know, do the implementations.
383
00:44:44.090 --> 00:44:44.910
And I think
384
00:44:45.425 --> 00:44:55.030
so we have a subset of of of sort of customers where we're doing end to end data as a service. We we sort of give you the platform, and, you know, they're using us to,
385
00:44:55.650 --> 00:44:58.550
go build the sort of reporting layer on top of it,
386
00:44:59.090 --> 00:45:04.355
and we sort of recently did something with this sort of restaurant chain, you know, they have
387
00:45:05.234 --> 00:45:07.494
50 to a 100 different sort of locations.
388
00:45:08.115 --> 00:45:09.895
They are inside the QSR category.
389
00:45:10.435 --> 00:45:10.755
They,
390
00:45:12.579 --> 00:45:14.119
for lack of expertise
391
00:45:14.420 --> 00:45:17.480
and, you know, not having sort of data people in there,
392
00:45:17.780 --> 00:45:21.000
they were used to, you know, the analytics they got from,
393
00:45:21.345 --> 00:45:23.204
you know, Uber Eats and Restaurant 365
394
00:45:23.505 --> 00:45:30.885
and, you know, sort of Postmates and, you know, all these different things. And, you know, their entire marketing agencies, which just go focus on that category,
395
00:45:31.490 --> 00:45:35.030
and they do what they do and you know, obviously, it's it's a big business
396
00:45:36.210 --> 00:45:39.830
and what we were able to do with them in just a few months is that the sort of
397
00:45:40.130 --> 00:45:44.695
level of of sort of data and, you know, the insights we could get in the analytics perspective,
398
00:45:45.635 --> 00:45:46.935
it was something which
399
00:45:47.315 --> 00:45:51.250
they'd they'd sort of never seen before. So much so that sort of marketing agency,
400
00:45:51.790 --> 00:45:54.290
which, you know, works with with sort of 1700
401
00:45:54.590 --> 00:45:55.410
different restaurants,
402
00:45:56.270 --> 00:46:03.925
was, you know, just completely sort of shocked and blown away by it. And they now wanna sort of go do this across all of these different verticals.
403
00:46:04.305 --> 00:46:12.230
I think we're getting exposed to, you know, sort of use cases and and sort of company types, which are really interesting, was, you know, sort of very large. But, again,
404
00:46:12.930 --> 00:46:13.430
haven't
405
00:46:14.825 --> 00:46:15.805
had the appetite
406
00:46:16.185 --> 00:46:16.685
or
407
00:46:17.225 --> 00:46:17.725
the
408
00:46:18.585 --> 00:46:19.085
conviction
409
00:46:19.545 --> 00:46:23.245
or the expertise to actually go make those investments in data.
410
00:46:23.720 --> 00:46:28.540
Because as a data industry, we haven't made it easy to go make those investments in data.
411
00:46:29.080 --> 00:46:35.925
Instead of giving the fragmentation, we're even just giving, like, you know, the support and implementation needed to actually go get value from your product.
412
00:46:36.865 --> 00:46:37.365
So
413
00:46:37.825 --> 00:46:41.205
we're seeing a lot more of these use cases, which we don't where
414
00:46:41.760 --> 00:46:44.660
industries which previously haven't entered the ecosystem,
415
00:46:45.839 --> 00:46:48.819
are now able to, you know, be completely disrupted
416
00:46:49.555 --> 00:46:52.375
because what we're able to do is something which
417
00:46:52.835 --> 00:46:56.280
is, for lack of better words, no 1 else is able to do.
418
00:46:56.920 --> 00:46:59.980
And another interesting aspect of what you're building
419
00:47:00.280 --> 00:47:02.060
is that because you have this
420
00:47:02.360 --> 00:47:16.460
unified experience, the incremental cost of adoption for new tools or new capabilities is much lower for teams than it would be if they had to go out and do that evaluation process, do the integration process themselves where it turns from,
421
00:47:16.840 --> 00:47:18.780
oh, I wanna use DBT
422
00:47:19.320 --> 00:47:21.660
or some other tool or vendor.
423
00:47:22.065 --> 00:47:39.905
I just say, click a button, log in, start working with it versus, oh, now I have to spend 6 months going through that whole process. And then maybe if you are in in a larger organization, you also have to do some selection of source paperwork, get the funding, etcetera, etcetera. I'm wondering how that influences the ways that teams are,
424
00:47:40.285 --> 00:47:45.744
approaching that process of saying, oh, I wanna add this new capability. I wanna start doing this new thing.
425
00:47:46.119 --> 00:47:47.900
Yeah. You know, it's a good question, and
426
00:47:48.839 --> 00:47:50.460
I wanna answer it in
427
00:47:50.920 --> 00:47:52.780
once you get used to buying a car,
428
00:47:53.295 --> 00:47:56.435
it becomes very difficult to go by car parts ever again.
429
00:47:57.055 --> 00:47:57.955
So once
430
00:47:58.255 --> 00:47:59.155
sort of customers
431
00:47:59.535 --> 00:48:06.900
see how easy it is to go onboard a vendor and have our expertise on, you know, some making some of these decisions and helping out with implementation,
432
00:48:07.920 --> 00:48:08.420
they
433
00:48:09.840 --> 00:48:10.340
have
434
00:48:10.720 --> 00:48:12.180
a much lower barrier
435
00:48:13.105 --> 00:48:18.005
to, hey. What about the next tool? Because it's not as daunting of a process every single time.
436
00:48:18.465 --> 00:48:18.965
So
437
00:48:19.265 --> 00:48:27.400
we have sort of companies which sort of which sort of start small, and they add capabilities as they need them with time. And they're able to, you know I like to use the word
438
00:48:27.780 --> 00:48:30.359
do it in a hilariously more efficient manner,
439
00:48:31.065 --> 00:48:38.045
as opposed to, you know, the way companies sort of with with the way companies do this today. And, again, you know, for us, what's very important is
440
00:48:38.390 --> 00:48:51.275
being able to look at this very holistically. Right? Not just from the sort of vendor standpoint and the decision making standpoint, but also the implementation and making sure they're getting value from it is how do we provide sort of customers with the most amazing experience
441
00:48:51.815 --> 00:48:53.675
to to basically go do this.
442
00:48:54.055 --> 00:49:05.500
And, you know, we're playing a long game over here. Right? Like, we are we we wanna make sure customers are getting value from data because, you know, the way we look at the world is we can go sell all of these different tools and
443
00:49:05.905 --> 00:49:08.725
you can go do that. But if customers are not getting value from data,
444
00:49:09.185 --> 00:49:10.165
at some point,
445
00:49:10.865 --> 00:49:15.730
it's gonna come back. Right? Like, either those tools are gonna get axed. The the data team is gonna get axed.
446
00:49:16.850 --> 00:49:33.490
And we're not looking to go make a short term buck by by by sort of going, and sort of adding a new tool over there. We wanna make sure that, you know, we're looking at this very sort of holistically, and and and we're just playing a sort of longer game, and we wanna create the best experience for our customers and data teams.
447
00:49:34.430 --> 00:49:35.890
And in your experience
448
00:49:36.270 --> 00:49:45.255
of building this product, building the company, working with customers, what are some of the most interesting or unexpected or challenging lessons that you've learned in the process?
449
00:49:46.835 --> 00:49:47.735
You know, people
450
00:49:48.595 --> 00:49:50.615
sort of 2 years ago called us crazy
451
00:49:51.280 --> 00:49:51.780
because
452
00:49:52.080 --> 00:49:56.020
we want building a new category. We want building what
453
00:49:56.560 --> 00:49:58.180
VCs call a product,
454
00:49:58.695 --> 00:50:02.875
which is this is a category and, you know, this is how you go sell. And, you know, you have
455
00:50:03.335 --> 00:50:10.440
you know, you sell your product and, you know, this is your ACV. And we are something completely different to how the entire industry work.
456
00:50:10.820 --> 00:50:12.200
And everyone called
457
00:50:12.740 --> 00:50:20.635
this idea of going and consolidating the stack in, like, by building a layer on top of it, all sorts of names and all sorts of all sorts of craziness. And
458
00:50:21.015 --> 00:50:21.675
I think
459
00:50:22.220 --> 00:50:24.960
the last, you know, few months have been really exciting because
460
00:50:25.579 --> 00:50:30.555
a lot of people are really seeing the value in this now. So yeah. I mean,
461
00:50:31.255 --> 00:50:39.970
it was it was sort of it was quite challenging to go get our first integrations. It was sort of challenging to, you know, explain data teams when you had unlimited funding
462
00:50:40.430 --> 00:50:42.130
why they don't want to
463
00:50:42.670 --> 00:50:45.490
go and build their own platform and manage this forever.
464
00:50:45.855 --> 00:50:46.355
And,
465
00:50:47.295 --> 00:50:48.835
you know, it's been really
466
00:50:50.015 --> 00:50:50.515
reassuring,
467
00:50:50.815 --> 00:50:54.930
and I don't even think that's the right word, but it's been it's just very
468
00:50:55.410 --> 00:50:56.710
just sort of very grateful
469
00:50:57.490 --> 00:50:59.190
for people to see the value,
470
00:50:59.890 --> 00:51:01.349
in what we're doing now.
471
00:51:02.645 --> 00:51:08.825
So, yeah, it just makes a lot of the sort of decisions we made to stay on our path back then.
472
00:51:09.360 --> 00:51:09.860
Just
473
00:51:10.160 --> 00:51:14.180
just you know, we see a new wave of people really excited to show up to work every day.
474
00:51:14.560 --> 00:51:18.474
And for people who are in the process of tool evaluation,
475
00:51:18.775 --> 00:51:26.555
maybe they're doing their own integration work across different vendors in the modern data ecosystem, what are the cases where 5 x is the wrong choice?
476
00:51:27.779 --> 00:51:36.200
Great question. 5x is not for you if you are not using a warehouse or don't plan to use a data warehouse. We are built on a data warehouse first approach.
477
00:51:37.185 --> 00:51:41.685
Apart from that, I think we're relevant across the entire life cycle,
478
00:51:42.305 --> 00:51:44.725
of your data team. And what I mean by that is
479
00:51:45.369 --> 00:51:50.750
for SMB companies, we have a program where we waive the cost of the 5 x platform.
480
00:51:51.369 --> 00:51:55.085
You can still take advantage of provisioning vendors from us
481
00:51:55.645 --> 00:51:57.425
at list price or even cheaper,
482
00:51:57.885 --> 00:52:05.790
and then using 5 x to go operate all your vendors. So in some ways, it's a no brainer if you're building a platform from scratch today to build it on 5 x.
483
00:52:06.410 --> 00:52:08.750
For mid market and enterprise companies,
484
00:52:10.089 --> 00:52:12.430
other advanced tools in our suite,
485
00:52:13.015 --> 00:52:17.275
as we spend this as we spend discussing on this podcast become extremely relevant.
486
00:52:18.295 --> 00:52:22.680
So, you know, again, a no brainer to use us to make your data teams more efficient.
487
00:52:23.539 --> 00:52:25.880
Overall, I think if you are using
488
00:52:26.260 --> 00:52:27.799
a warehouse first approach,
489
00:52:28.464 --> 00:52:32.565
using 5 x to really simplify the management and operation of your platform,
490
00:52:33.345 --> 00:52:34.805
is something worth doing.
491
00:52:35.860 --> 00:52:37.160
And as you continue
492
00:52:37.460 --> 00:52:44.825
to build and iterate on your product and business, what do you have planned for the near to medium term or any particular projects or problem areas
493
00:52:45.125 --> 00:52:46.345
you're excited to explore?
494
00:52:47.685 --> 00:52:49.065
I think, you know, we're
495
00:52:49.525 --> 00:52:53.065
we're we're just focused on the basics. Right? Like, sort of getting our
496
00:52:57.580 --> 00:53:05.585
daily active users to go use to go operate inside our platform. You know, big focus for us, you know, sort of enterprise and enterprise
497
00:53:05.885 --> 00:53:06.385
readiness,
498
00:53:06.765 --> 00:53:08.865
you know, sort of everything which happens
499
00:53:09.325 --> 00:53:12.460
as everything you need to basically go after that segment.
500
00:53:12.920 --> 00:53:16.780
We are not looking you know, apart from Databricks, next year,
501
00:53:17.320 --> 00:53:21.095
we're not really looking to expand our vendor footprint drastically.
502
00:53:21.955 --> 00:53:34.480
We wanna double down on our existing partnerships and, you know, build the best experience all the way from sort of SI work to, you know, integration into, you know, the sales teams, into integration into their product and engineering.
503
00:53:35.135 --> 00:53:39.475
And at a macro level, we're super focused on, sort of, profitability
504
00:53:40.335 --> 00:53:43.315
as a business. So, you know, we're playing a long game,
505
00:53:43.615 --> 00:53:44.115
and,
506
00:53:45.309 --> 00:53:51.329
I came from WeWork. I came from a company which raised a lot of money. And, you know, some of my lessons over there is
507
00:53:51.635 --> 00:53:52.375
you wanna
508
00:53:52.915 --> 00:53:56.935
go figure out what business you have sooner rather than later. So
509
00:53:57.235 --> 00:53:59.975
we wanna be around for a long time. We're playing a long game,
510
00:54:00.569 --> 00:54:03.549
and we wanna get the business in a position which we can go to that.
511
00:54:04.250 --> 00:54:19.850
Well, for anybody who wants to get in touch with you, follow along the work that you and your team are doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest gap in the tool in your technology that's available for data management today?
512
00:54:21.030 --> 00:54:24.490
I think, again, you know, we spoke about a lot of those things about
513
00:54:25.072 --> 00:54:28.895
if you look at AWS, AWS is is a collection of 50 different services,
514
00:54:29.195 --> 00:54:35.599
but they give you a really cohesive experience to go manage it. Right? A single place to provision, single digital identity, billing, migration.
515
00:54:36.220 --> 00:54:37.440
They give you cost optimization,
516
00:54:38.460 --> 00:54:40.720
and they give you role based access control.
517
00:54:41.075 --> 00:54:45.895
All of these different things have just been hilariously missing inside the data ecosystem.
518
00:54:46.355 --> 00:54:53.870
So, again, the glue which actually connects all of this together, you and I still continue to think that that's 1 of the biggest missing pieces in the space.
519
00:54:54.330 --> 00:55:02.224
Well, thank you very much for taking the time today to join me and share the work that you're doing at 5 x data. It's definitely a very, interesting
520
00:55:02.685 --> 00:55:15.250
problem area that you're trying to address, interesting product that you're building around it. So appreciate all the time and energy that you and your team are putting into making the modern data ecosystem a more tractable and approachable
521
00:55:15.550 --> 00:55:24.660
problem space. So thank you for the work you're doing there and for your time, and I hope you enjoy the rest of your day. Thank you so much for having me. Hopefully, we've added some we've added some value to your listeners today.
522
00:55:31.059 --> 00:55:41.355
Thank you for listening. Don't forget to check out our other shows, podcast dot in it, which covers the Python language, its community, and the innovative ways it is being used, and the Machine Learning Podcast,
523
00:55:41.860 --> 00:55:44.680
which helps you go from idea to production with machine learning.
524
00:55:44.980 --> 00:55:46.360
Visit the site at dataengineeringpodcast.com
525
00:55:47.780 --> 00:55:55.215
to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a project from the show, then tell us about it.
526
00:55:55.515 --> 00:55:57.855
Email hosts at data engineering podcast.com
527
00:55:58.395 --> 00:56:04.620
with your story. And to help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers.