Re-Bundling The Data Stack With Data Orchestration And Software Defined Assets Using Dagster

WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 7/6/2024 12:52:33 PM
Duration: 3493.985
Channels: 1
1
00:00:13.125 --> 00:00:17.225
Hello, and welcome to the Data Engineering Podcast, the show about modern data management.
2
00:00:17.880 --> 00:00:24.699
When you're ready to build your next pipeline or want to test out the projects you hear about on the show, you'll need somewhere to deploy it. So check out our friends at Linode.
3
00:00:25.075 --> 00:00:28.775
With their new managed database service, you can launch a production ready MySQL,
4
00:00:29.235 --> 00:00:36.260
Postgres, or MongoDB cluster in minutes with automated backups, 40 gigabit connections from your application hosts, and high throughput SSDs.
5
00:00:37.200 --> 00:00:38.820
Go to data engineering podcast.com/
6
00:00:39.635 --> 00:00:49.640
lunode today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services. And don't forget to thank them for their continued support of this show.
7
00:00:50.340 --> 00:01:06.045
Your host is Tobias Macy. And today, I'm welcoming back Nick Schrock to talk about software defined assets and his work at Elementl on Dagster to improve the developer experience for data orchestration. So, Nick, can you introduce yourself for folks who haven't heard of you before?
8
00:01:06.500 --> 00:01:11.800
Yeah. Thanks for having me, Tobias, and thanks for that intro. As you said, my name is Nick Schrock. I'm the CEO and founder
9
00:01:12.260 --> 00:01:14.760
of Elementor, which is the company behind Dagster.
10
00:01:15.425 --> 00:01:20.645
Before Elementl, my career was mostly spent at Facebook from 2009, 2017.
11
00:01:22.145 --> 00:01:31.325
And there, I founded this team called product infrastructure, which was to make our application developers more efficient and productive. And kind of the most notable piece of work that came out of that
12
00:01:31.725 --> 00:01:36.865
was that I was personally involved in was GraphQL, and I'm 1 of the cocreators of that. And
13
00:01:37.405 --> 00:01:39.585
then moved on from Facebook 2017
14
00:01:40.365 --> 00:01:42.720
and got into this entire domain
15
00:01:43.020 --> 00:01:45.280
and started working on data orchestration.
16
00:01:45.980 --> 00:01:48.240
And in terms of the
17
00:01:48.895 --> 00:01:50.354
overall DAXTER project,
18
00:01:50.655 --> 00:01:56.755
for folks who wanna understand more about what it is and some of the history, I'll point them to some of the other interviews that we've done.
19
00:01:57.110 --> 00:02:05.130
But the last time we spoke was last November, and I'm wondering if you can just give a quick update on what's new in DAXTER and
20
00:02:05.675 --> 00:02:09.615
its reaction to the evolution of the data space since the last time we spoke.
21
00:02:10.075 --> 00:02:11.695
Yeah. So since November,
22
00:02:12.475 --> 00:02:14.895
we've been making a ton of progress. So
23
00:02:15.580 --> 00:02:21.200
in December, the month after that interview, we launched our cloud product to early access,
24
00:02:21.739 --> 00:02:28.295
and that's been going better than I anticipated, actually. And we have tons of users across
25
00:02:28.754 --> 00:02:32.055
all sorts of companies all the way from startups to Fortune 500 companies.
26
00:02:32.730 --> 00:02:38.990
And as we'll talk about later in the episode, we're actually, you know, gonna be launching that to general availability to the open public
27
00:02:39.425 --> 00:02:42.005
next month on August 9th. So that's super exciting.
28
00:02:42.545 --> 00:02:44.405
In terms of the open source project,
29
00:02:45.025 --> 00:02:50.950
we had talked about it on that podcast, but we've really been doubling down on this software defined assets direction.
30
00:02:51.410 --> 00:02:51.910
And
31
00:02:52.210 --> 00:03:04.165
in November, it was a super early prototype. Not super early. It was a fairly early prototype being used by a few early design partners. And now, you know, adoption and development has accelerated massively.
32
00:03:04.770 --> 00:03:07.030
And we've launched it with a stable API.
33
00:03:07.330 --> 00:03:08.390
And then we're actually
34
00:03:08.770 --> 00:03:16.155
also launching Daxter 1 0 next month at Daxter Day along with our Cloud GA launch. So it's been a huge 6 months for us.
35
00:03:16.535 --> 00:03:18.395
And so in terms of
36
00:03:18.695 --> 00:03:19.195
the
37
00:03:19.575 --> 00:03:26.590
overall software defined asset concept and some of the ways that it manifests, I'm wondering if you can talk to a bit of
38
00:03:27.050 --> 00:03:36.215
what the motivation is and what the meaningful impact is for data teams who are working with software defined assets as their primary abstraction
39
00:03:37.075 --> 00:03:48.580
rather than the core abstraction of this task based DAG that folks have become familiar with since the early days of data orchestration and especially exemplified by the Airflow community?
40
00:03:49.200 --> 00:03:50.900
Yeah. So in terms of the motivation,
41
00:03:52.385 --> 00:03:53.765
what we saw,
42
00:03:54.145 --> 00:03:56.965
and this was last summer when we really started
43
00:03:57.665 --> 00:03:59.605
to dig into this, is that
44
00:04:00.090 --> 00:04:03.790
a task based orchestration system was increasingly out of step
45
00:04:04.250 --> 00:04:09.070
with the way that data teams wanted to approach their data platforms overall
46
00:04:09.445 --> 00:04:11.625
and also the way that all the constituent
47
00:04:12.005 --> 00:04:12.505
tools
48
00:04:13.045 --> 00:04:17.465
in modern data, I'll avoid the term modern data stack, which were very kind of
49
00:04:18.090 --> 00:04:28.045
asset oriented or model oriented. So if you go to Erabyte, they'll talk about sources. Right? And they, like, model a distinct set of sources in their system or 5 Transimrally.
50
00:04:28.745 --> 00:04:29.805
If you go to dbt,
51
00:04:30.345 --> 00:04:34.045
they'll be modeling the system in terms of dbt models and not tasks.
52
00:04:34.570 --> 00:04:37.710
And what was interesting, and this kinda came out in this, you know,
53
00:04:38.090 --> 00:04:43.245
discussion around bundling and unbundling platforms, which happened in the spring. But, effectively,
54
00:04:43.545 --> 00:04:44.685
tons of the information
55
00:04:45.705 --> 00:04:51.485
that used to be in the task based orchestration layer were pushed down into these constituent tools.
56
00:04:52.319 --> 00:04:55.220
Whereas, for example, prior to say DBT,
57
00:04:56.080 --> 00:04:59.699
each table would have had a corresponding task in airflow,
58
00:05:00.395 --> 00:05:03.855
say. But now you're just invoking dbt cloud or dbt
59
00:05:04.395 --> 00:05:12.530
run. That entire DAG is represented by a single task in the orchestration layer. And, therefore, the orchestration layer is not nearly as useful
60
00:05:13.070 --> 00:05:18.075
in terms of providing a single place where you can have visibility into your entire data platform.
61
00:05:18.615 --> 00:05:20.635
So we saw that happening
62
00:05:21.095 --> 00:05:23.035
where task based orchestration
63
00:05:23.415 --> 00:05:26.635
no longer map to the constituent tools as much.
64
00:05:27.030 --> 00:05:32.010
And then second of all, just in terms of even absent those external vendors,
65
00:05:33.030 --> 00:05:37.935
the platforms are becoming more and more complicated, and people wanted a more declarative approach
66
00:05:38.475 --> 00:05:41.615
where they could manage more complexity in their head.
67
00:05:41.915 --> 00:05:46.590
And the kind of more declarative approach of of software defined assets really
68
00:05:47.210 --> 00:05:51.470
resonated with us and our early users. And that was kind of the motivation that happened there.
69
00:05:51.985 --> 00:06:03.860
And in terms of that early experience of releasing this capability, digging into that particular approach, and some of the ways that that maps to different teams' mental models. I'm wondering what were
70
00:06:04.240 --> 00:06:10.725
some of the notable outcomes as far as the development approach that teams had to building their
71
00:06:11.185 --> 00:06:16.165
data systems and data products and how that factored into their overall
72
00:06:16.639 --> 00:06:20.740
practices of how they thought about what the output of their work actually was.
73
00:06:21.280 --> 00:06:22.580
Yeah. That's a great question.
74
00:06:22.880 --> 00:06:29.155
So in terms of outcomes here, I think, first of all, the practitioners who are working in Python
75
00:06:29.855 --> 00:06:35.760
and orchestrating all these systems and building data assets in Python, they just became more productive
76
00:06:36.300 --> 00:06:38.960
using the system. They have to keep less stuff in their head.
77
00:06:39.419 --> 00:06:41.840
They no longer have to write 1 centralized
78
00:06:42.715 --> 00:06:45.375
DAG artifact or flow artifact manually.
79
00:06:45.835 --> 00:06:51.419
So if you come from airflow, there's typically this DAG construction file, which ends up being a sort of
80
00:06:51.879 --> 00:06:52.379
centralized,
81
00:06:52.759 --> 00:06:54.620
unknown dumping ground of code
82
00:06:55.159 --> 00:06:56.699
that gets more complex
83
00:06:57.240 --> 00:07:04.175
as your data platform gets more complex. And with software defined assets, you no longer have that centralized artifact. The complexity is distributed
84
00:07:04.875 --> 00:07:10.320
across the entire system in a good way, meaning that that centralized DAG is constructed for you.
85
00:07:10.699 --> 00:07:14.400
So it just makes it so that coding is faster, collaboration is easier,
86
00:07:14.860 --> 00:07:15.919
and it's an overall
87
00:07:16.300 --> 00:07:18.285
win. I think more profoundly,
88
00:07:19.385 --> 00:07:26.605
you know, what we found, and this is also based on some early feedback too, is that this allowed users who want to, say,
89
00:07:27.210 --> 00:07:27.710
develop
90
00:07:28.250 --> 00:07:29.790
their entire data platform
91
00:07:30.330 --> 00:07:33.070
across a bunch of different independent GitHub repos
92
00:07:33.530 --> 00:07:34.910
by independent teams.
93
00:07:35.215 --> 00:07:39.475
They can each individually code in their own world. Once deployed,
94
00:07:40.175 --> 00:07:44.275
they can actually interconnect all the assets that are built in those different constituent
95
00:07:44.710 --> 00:07:47.610
GitHub repos. So it's really been a way to
96
00:07:48.310 --> 00:07:48.810
unify
97
00:07:49.349 --> 00:07:55.185
the different practitioners who might be working quite independently, but are still building up to 1 cohesive data platform.
98
00:07:55.805 --> 00:07:58.465
And then that's kinda connected to the last bit,
99
00:07:59.245 --> 00:08:10.240
which is now they finally have a place to go to where they can understand how their data products interrelate. So they no longer have to go into different constituent tools and kind of, like, have it all in their heads. There's just, like, 1 spot
100
00:08:10.854 --> 00:08:12.634
where everything is defined. It's been
101
00:08:13.014 --> 00:08:34.175
pretty interesting to see how that's developed. Like, 1 anecdote I really like is that we had an early cloud customer who started to adopt software defined assets, and they kinda started with our out of the box integrations. So they use the ingest tools, they use some custom Python scripts, and they integrate that in the software to find assets, and then they also imported their DBT graph.
102
00:08:34.635 --> 00:08:38.170
And they saw this interconnected web of assets. But then
103
00:08:38.710 --> 00:08:40.970
they also use a feature store called Feast,
104
00:08:41.750 --> 00:08:50.125
and it kind of was in its own world because they hadn't imported that to software defined assets, and we didn't have an out of the box integration for that.
105
00:08:50.505 --> 00:08:53.805
So they actually built their own integration to bring Feast
106
00:08:54.610 --> 00:09:01.269
into software defined assets land. And I thought that was interesting. 1, it was just like the value of it compelled them to write that integration.
107
00:09:02.004 --> 00:09:08.105
And second of all, it actually interconnected the data engineering and ML sides of of the house into 1 cohesive
108
00:09:08.884 --> 00:09:09.785
lineage graph.
109
00:09:10.230 --> 00:09:15.130
And I think that's a lot of the promise here is that you no longer have to desilo your organization.
110
00:09:15.590 --> 00:09:20.365
Right? You can kind of conform everyone to this common standard and have your entire org,
111
00:09:20.824 --> 00:09:25.005
regardless of persona, kind of interconnected in this single fabric of assets.
112
00:09:25.785 --> 00:09:27.485
And to that point of
113
00:09:28.029 --> 00:09:29.170
building their own integration
114
00:09:29.550 --> 00:09:43.810
to be able to represent what was happening in Feast as a software defined asset, can you talk a bit more about what is actually involved in building that type of an integration and what it means to actually represent some of these external systems through this asset oriented
115
00:09:44.510 --> 00:09:45.010
abstraction?
116
00:09:45.630 --> 00:09:50.930
I mean, in the end, if you look at the Python code, that's a software defined asset. It's just a function.
117
00:09:52.035 --> 00:09:55.735
So it's not this huge lift to do an integration.
118
00:09:56.195 --> 00:09:57.815
It's just merely annotating
119
00:09:58.670 --> 00:10:06.130
your existing because all these systems generally have Python integrations. Right? So in the end, you're writing some Python code, which invokes
120
00:10:06.510 --> 00:10:08.210
the tool that you wanna invoke.
121
00:10:08.615 --> 00:10:10.715
The magic is sort of how we structure
122
00:10:11.095 --> 00:10:12.955
the software and structure the metadata.
123
00:10:13.415 --> 00:10:15.755
So it just means that you've written integration.
124
00:10:16.215 --> 00:10:25.105
It ends up being a Python function, which evokes something else, but then you can plug into our system to interconnect it to the other assets, and that's kind of the critical piece here.
125
00:10:25.505 --> 00:10:27.764
And as you went through that
126
00:10:28.144 --> 00:10:31.125
early phase of proving out the idea,
127
00:10:31.505 --> 00:10:33.685
testing it out with your early adopters,
128
00:10:34.030 --> 00:10:38.050
and then iterating on that with some of the subsequent releases. I know that you have
129
00:10:38.350 --> 00:10:41.810
stabilized the API, and I'm wondering what were some of the
130
00:10:42.275 --> 00:10:43.895
shifts in the interfaces
131
00:10:44.355 --> 00:10:50.215
and the abstractions and paradigms you wanted to expose for people to be able to work in this mode of the software defined asset,
132
00:10:50.760 --> 00:10:54.380
And what were some of the modifications that were necessary
133
00:10:54.840 --> 00:11:00.155
internally to DAXTER and the framework to be able to support that as a first class capability?
134
00:11:00.935 --> 00:11:05.755
So I'll start with the first 1, which is what are the changes we had to make to core DAXTER
135
00:11:06.110 --> 00:11:07.250
in order to do this?
136
00:11:07.870 --> 00:11:10.210
And the answer is that the core
137
00:11:10.990 --> 00:11:12.770
was relatively stable.
138
00:11:13.524 --> 00:11:14.024
Right?
139
00:11:14.404 --> 00:11:16.745
So software defined assets is just a layer
140
00:11:17.204 --> 00:11:18.584
on top of
141
00:11:19.125 --> 00:11:20.504
our existing core.
142
00:11:20.805 --> 00:11:25.850
Right? So at its core, a software defined asset refers to an op,
143
00:11:26.470 --> 00:11:31.690
which is our core unit of computation, kind of the the foundation of our task based orchestration layer.
144
00:11:32.225 --> 00:11:37.125
If you write an asset, it builds the centralized DAG artifact, the job,
145
00:11:37.745 --> 00:11:38.485
for you,
146
00:11:38.945 --> 00:11:43.040
and it also manages interactions at a fine grade level with our asset catalog,
147
00:11:43.420 --> 00:11:47.520
which also predated software defined assets. So it's actually this relatively
148
00:11:47.980 --> 00:11:49.435
thin layer of software
149
00:11:49.815 --> 00:11:51.355
on top of the existing capabilities
150
00:11:52.135 --> 00:11:55.515
in our core system. And that's important as well
151
00:11:55.980 --> 00:12:02.720
because our task based orchestrator is not going away, that core orchestration engine. Some tasks,
152
00:12:03.345 --> 00:12:08.565
not to double use the word, some activities in data platforms are still highly amenable
153
00:12:09.025 --> 00:12:14.820
to kind of more imperative programming models, and those are critical to support, and we will always support that.
154
00:12:15.200 --> 00:12:21.325
And in fact, the software defined asset layer needs that layer in order to execute well. You can kind of, like, think of it
155
00:12:21.704 --> 00:12:23.165
it's not a perfect analogy,
156
00:12:23.945 --> 00:12:26.950
but in that way, you know, Spark, for example,
157
00:12:29.510 --> 00:12:36.955
computations are encoded in SQL. And maybe in the, like, long term Spark, like, 80% of the computations will be in spark SQL,
158
00:12:37.335 --> 00:12:39.675
but there's still gonna be this critical 20%
159
00:12:40.295 --> 00:12:54.540
that are encoded in data frames. It's more imperative, but you need to do it in order to accomplish a lot of the activities you wanna do. We kind of view software defined assets in our task based system in a similar sort of way, where we anticipate over time more and more of the activity in the system
160
00:12:54.985 --> 00:13:13.279
will be encoded in terms of the software defined assets API, but there's still gonna be absolutely critical activities that you need to use this lower level for. So that all being said, software defined assets is a layer on top of our op layer, and that will forever and always be true. So most of the work
161
00:13:13.915 --> 00:13:15.215
was actually more about
162
00:13:15.595 --> 00:13:18.095
kind of conceptual work and getting that layer
163
00:13:18.635 --> 00:13:25.820
absolutely right, and then building tooling on top of it, which was by far the where the lion's share of the effort was.
164
00:13:26.280 --> 00:13:32.144
So that's 1 thing in terms of your question about changes to the interfaces and internals of the system.
165
00:13:33.084 --> 00:13:37.265
Your other question was, how do we respond to feedback and how the API
166
00:13:37.850 --> 00:13:38.350
shift?
167
00:13:38.890 --> 00:13:42.510
And I think that focused on maybe 3 different things.
168
00:13:43.130 --> 00:13:45.950
And 1 was a lot of the users
169
00:13:46.445 --> 00:13:47.105
who understood
170
00:13:48.365 --> 00:13:49.425
this very intuitively
171
00:13:49.964 --> 00:13:59.570
and wanted it were our modern data stack users that were heavy DBT users and wanted much more advanced orchestration capabilities on top of DBT.
172
00:14:00.430 --> 00:14:06.134
So tons of features there. In particular, I thought 1 was interesting, which speaks to what I was talking about before
173
00:14:06.514 --> 00:14:09.175
was, you know, we had this ability initially
174
00:14:09.540 --> 00:14:11.800
where you can, like, write in the same GitHub repo,
175
00:14:12.580 --> 00:14:14.360
be able to load dbt projects
176
00:14:14.820 --> 00:14:18.520
and orchestrate it right there. But what tons of teams wanted actually
177
00:14:19.045 --> 00:14:35.310
was the ability to have their analytics engineering team to actually have a more independent GitHub repo in their own development cycle, and then have the orchestrator actually consume the dbt manifest file, for example. And that required some changes on our end, but it spoke to that
178
00:14:35.715 --> 00:14:41.095
people are really thinking about how to the interfaces between teams in these systems
179
00:14:41.475 --> 00:14:50.759
and then how to make it so that, you know, yes, they can deploy and operate independently, but once deployed to prod, the orchestrator kind of stitches the entire thing together.
180
00:14:51.305 --> 00:14:52.925
So we got a lot of feedback
181
00:14:53.305 --> 00:14:53.805
around
182
00:14:54.345 --> 00:15:04.259
how our DBT integration worked and then how that kind of trend and then I also viewed it as feedback about how people wanna organize their teams, and that was super interesting.
183
00:15:04.720 --> 00:15:06.180
On a more tactical level,
184
00:15:06.625 --> 00:15:16.325
you know, we were originally gonna deemphasize our config system a little bit in the software defined assets layer, which is effectively the ability to parameterize computations without changing
185
00:15:17.100 --> 00:15:27.904
code. But super early on, people were like, wait. I want this. I miss this capability in the old system. So, like, we actually had to re frontload that quite a bit. And then lastly,
186
00:15:28.685 --> 00:15:31.345
you know, we knew it was gonna be a demand,
187
00:15:31.725 --> 00:15:34.785
but, you know, immediately, people wanted to focus on interoperability.
188
00:15:35.720 --> 00:15:38.380
So they have their existing task graphs.
189
00:15:39.079 --> 00:15:52.655
Some stuff are appropriate to model as task graphs. How do you wanna interleave those layers? So I think those were kind of the early themes and feedback which we responded to. But the core conceptual underpinnings of the project have remained quite constant from the beginning.
190
00:15:53.160 --> 00:15:58.940
As you have gone through this process of working through the software defined assets capability,
191
00:16:00.095 --> 00:16:00.595
updating
192
00:16:01.135 --> 00:16:05.154
the interfaces and APIs and some of the tooling and just general
193
00:16:05.774 --> 00:16:07.315
messaging around that functionality.
194
00:16:07.740 --> 00:16:28.850
In parallel, you've also been going through a journey of the early beta testing and early adoption of your cloud product, and I'm curious how that journey has paralleled the capabilities of software defined assets and some of the ways that those 2 areas of effort have played off of each other in terms of your own internal engineering work? Yeah. That's a good question.
195
00:16:29.230 --> 00:16:29.810
I mean,
196
00:16:30.190 --> 00:16:32.130
I think the easiest answer
197
00:16:32.555 --> 00:16:35.935
and why I'm so excited about the cloud product happening
198
00:16:36.634 --> 00:16:37.454
is that
199
00:16:38.154 --> 00:16:40.334
the ability to have a hosted service
200
00:16:41.529 --> 00:16:42.750
allows you to deploy
201
00:16:43.690 --> 00:16:44.670
new capabilities
202
00:16:45.209 --> 00:16:46.350
and bug fixes
203
00:16:47.050 --> 00:16:48.430
and new system infrastructure
204
00:16:49.365 --> 00:16:52.265
to your users in a completely permissionless way,
205
00:16:52.805 --> 00:17:01.770
and that allows you to iterate with your users far more quickly. You know, in a pure open source model, let's say you fix a bug or have a new feature in your UI,
206
00:17:02.390 --> 00:17:09.175
you push it up to PyPI, and then you're kinda like waiting for people to upgrade and like that's their and then you're subject to their own
207
00:17:09.655 --> 00:17:10.155
internal
208
00:17:10.615 --> 00:17:12.795
infrastructure release processes and whatnot.
209
00:17:13.575 --> 00:17:14.955
And with cloud,
210
00:17:15.335 --> 00:17:17.355
we can push whenever we want,
211
00:17:17.760 --> 00:17:23.299
get new capabilities into users hands extremely quickly and have a much tighter feedback loop with them.
212
00:17:23.600 --> 00:17:26.260
And that's been able to accelerate our product development
213
00:17:26.635 --> 00:17:29.215
dramatically, and I'm super excited to be able to do that at scale.
214
00:17:29.835 --> 00:17:35.934
So you've been going through this process of adding the software defined assets capability, cementing that API.
215
00:17:36.620 --> 00:17:43.679
Just prior to that, you went through the work of updating the core abstractions for DAXTER from these nomenclatures
216
00:17:43.980 --> 00:17:46.159
of solids and pipelines and
217
00:17:46.765 --> 00:17:48.385
Tobias, you're gonna bring those up?
218
00:17:49.164 --> 00:17:54.304
And that whole work and then, you know, streamlining that into this jobs ops graphs paradigm.
219
00:17:55.080 --> 00:18:00.940
So you've gone through a number of iterations. There's been a whole host of work. You've seen a lot of adoption.
220
00:18:01.559 --> 00:18:10.545
And so that leads us to what you mentioned earlier. On August 9th, you're having this DAXTER Day event where you're going to be committing to a 1
221
00:18:10.545 --> 00:18:14.720
stable version of DAXTER. You're going to be releasing cloud product as GA.
222
00:18:15.260 --> 00:18:21.760
I'm wondering what are the major lessons that you and your team have learned in the process of going through
223
00:18:22.365 --> 00:18:28.465
the initial development, the initial adoption cycles, figuring out what are the concepts that actually make sense in this space,
224
00:18:28.925 --> 00:18:29.425
exploring
225
00:18:29.805 --> 00:18:30.785
how you can
226
00:18:31.430 --> 00:18:36.010
keep up with the rapid pace of evolution in the data ecosystem writ large
227
00:18:36.310 --> 00:18:49.885
while staying true to the core ideas and abstractions that you're putting into DAXTER and just some of the commitments and what it is about where you are in your journey right now that makes you feel confident in committing to that stable version identifier.
228
00:18:50.590 --> 00:18:53.170
1 of the big takeaways here is don't be too cute
229
00:18:53.630 --> 00:18:54.770
with your name in.
230
00:18:55.150 --> 00:19:06.515
The name Solid was my fault. It was actually originally gonna be the name of the project itself, and I kinda like the name. And then I anyway, we don't need to belabor that 1. So don't be too cute. Name things in the most obvious way possible.
231
00:19:07.135 --> 00:19:08.434
You know, it was a critical
232
00:19:09.120 --> 00:19:12.020
thing for us to switch to a more
233
00:19:12.480 --> 00:19:13.700
natural nomenclature
234
00:19:14.559 --> 00:19:24.895
and fix some very core API problems with that original pipeline solid. There are other things called modes and presets and all sorts of stuff. And we wanted to simplify and consolidate that
235
00:19:25.360 --> 00:19:29.300
into a far more elegant ADBI. And that was a huge step forward for the system.
236
00:19:30.000 --> 00:19:33.380
And then assets were always gonna be a layer on top of that.
237
00:19:33.765 --> 00:19:35.065
So really the foundation
238
00:19:35.605 --> 00:19:39.304
of the 1 0 release and the big breaking change we needed to make
239
00:19:39.845 --> 00:19:48.090
was switching from to this op job graph model. And that is the stable core foundation that we're building everything on top of.
240
00:19:48.390 --> 00:19:51.610
The feedback on those APIs has been extremely positive.
241
00:19:52.345 --> 00:19:54.765
All of our major users have migrated everything.
242
00:19:55.465 --> 00:19:59.005
We're super confident in those APIs, that core Sable Foundation.
243
00:19:59.380 --> 00:20:02.360
And it was really that laid the foundation for this 1 over lease.
244
00:20:02.740 --> 00:20:04.520
And that doesn't mean we're gonna be stopping
245
00:20:04.900 --> 00:20:09.000
development by any sort of means, but any further changes are gonna be additive.
246
00:20:09.735 --> 00:20:13.595
And we will, like, commit to supporting these APIs for the long term.
247
00:20:13.895 --> 00:20:27.235
I guess, can you remind me the rather subcommos of the question? There was a bunch in there. The main driving point of the question was why you're confident with committing to this 1.0 version specifier, but also just some of the overall
248
00:20:28.015 --> 00:20:34.755
sort of lessons that you've learned as far as how to keep up with the rapid pace of change in the data ecosystem
249
00:20:35.470 --> 00:20:37.090
and what are the core
250
00:20:38.110 --> 00:20:46.465
ideas and lessons that you've been able to lean on that have stayed true regardless of the external turbulence that is a constant in this space.
251
00:20:47.005 --> 00:20:49.424
The word that you used, I think, is telling,
252
00:20:50.044 --> 00:20:51.184
which is turbulence.
253
00:20:51.804 --> 00:20:57.720
So I think there's 2 things going on here. There's, like, what are the fundamental things we are trying to accomplish in our jobs?
254
00:20:58.340 --> 00:21:01.480
And regardless of all the different noise in the ecosystem,
255
00:21:02.205 --> 00:21:03.825
that doesn't change that frequently.
256
00:21:04.125 --> 00:21:07.745
And the end of the day, we're writing code to produce assets and whatnot.
257
00:21:08.125 --> 00:21:10.705
There is tons of activity in the tooling ecosystem
258
00:21:11.470 --> 00:21:13.650
to optimize different parts of that workflow
259
00:21:14.030 --> 00:21:17.090
and make certain parts easier and improve ergonomics and whatnot.
260
00:21:17.630 --> 00:21:23.015
So, you know, with the 1.0 signifier is mostly around our core abstractions.
261
00:21:23.875 --> 00:21:26.375
You know? And then there's the entire integration layer
262
00:21:26.750 --> 00:21:30.130
and all the different libraries that both we and our community maintain.
263
00:21:30.590 --> 00:21:34.690
And that's kind of where you, you know, are dealing with the turbulence, so to speak.
264
00:21:35.230 --> 00:21:35.730
And
265
00:21:36.075 --> 00:21:42.255
as to new tools some come and go, the core notions in our framework, like, will remain the same.
266
00:21:42.795 --> 00:21:45.890
Along with the GA release of cloud,
267
00:21:46.429 --> 00:21:50.450
what are some of the new capabilities that you're planning on releasing
268
00:21:50.750 --> 00:21:51.809
as part of that?
269
00:21:52.110 --> 00:21:55.625
So there's a ton of new stuff that we're gonna be rolling out.
270
00:21:55.925 --> 00:21:56.425
But,
271
00:21:56.805 --> 00:22:00.185
you know, 1 thing I'm incredibly excited about
272
00:22:00.885 --> 00:22:02.345
in RGA release
273
00:22:02.725 --> 00:22:13.440
that's coming out. So first of all, for the DAXR 1 0, there's no huge changes coming out with DAXR 1 0. Right? The whole point is that there's a stable set of features that we are marking as mature,
274
00:22:13.885 --> 00:22:20.385
and you can be confident that things aren't gonna change in that stable core going forward. The Cloud GA release
275
00:22:20.845 --> 00:22:21.905
is a bit different.
276
00:22:22.285 --> 00:22:27.769
And first of all, what is the cloud product? And it's a managed orchestration service
277
00:22:28.149 --> 00:22:28.809
so that
278
00:22:29.269 --> 00:22:33.305
you can write your code and we host your control plane.
279
00:22:33.845 --> 00:22:36.025
So you can deploy Dagster effortlessly
280
00:22:36.485 --> 00:22:42.420
in an enterprise grade way, and it comes with features like, you know, role based asset control, authentication,
281
00:22:43.840 --> 00:22:46.740
and, you know, you can just focus on your business logic.
282
00:22:47.040 --> 00:22:53.404
A new capability that's coming out that I'm incredibly excited about, which I think solves the sort of,
283
00:22:53.784 --> 00:22:55.085
like, unsolved problem
284
00:22:55.544 --> 00:22:57.725
in data engineering and data management
285
00:22:58.710 --> 00:23:00.410
is what we call branch deployments.
286
00:23:01.110 --> 00:23:01.610
So,
287
00:23:01.910 --> 00:23:05.370
you know, there's this current problem in, I think, all the orchestrators,
288
00:23:06.155 --> 00:23:08.495
which is you can spend a bunch of effort
289
00:23:08.875 --> 00:23:11.295
structuring your code so that it's locally testable.
290
00:23:11.755 --> 00:23:13.935
And so you have a fast development life cycle.
291
00:23:14.260 --> 00:23:18.919
That's only in certain of the frameworks. Like, Airflow doesn't really support local development at all.
292
00:23:19.539 --> 00:23:22.740
And then what teams are left to do in those cases
293
00:23:23.245 --> 00:23:35.890
well, let me step back. So let's imagine you do do a bunch of work and you're, like, mocking out your data warehouses and doing all this work to try and make it so you have a nice local development flow on your laptop. There's a problem there, which is 1,
294
00:23:36.350 --> 00:23:54.780
there's a ton of stuff that still isn't tested if you aren't interacting with the services you're actually gonna interact with. And then if you try to interact with those services from your laptop, a lot of time your, you know, security policies prevent you from interacting with that. And then what people are left to do is still to do a bunch of testing in a staging environment.
295
00:23:55.559 --> 00:23:59.419
But those centralized staging environments are also a pain in the butt to manage.
296
00:23:59.945 --> 00:24:06.684
And often, if you have multiple developers deploying code to that staging environment, they're overriding each other's changes, and it just gets complicated
297
00:24:07.225 --> 00:24:07.965
and whatnot.
298
00:24:08.560 --> 00:24:16.580
So you either have these, like, clumsy to muse, clumsy to manage staging environments, or what people all too often do is just end up testing in production.
299
00:24:16.904 --> 00:24:20.445
They just push their code, and then they manually trigger
300
00:24:20.825 --> 00:24:25.164
a job or a DAG and just see if it seems to work. And it's this incredibly low,
301
00:24:25.520 --> 00:24:29.460
low productivity life cycle. What branch appointments allow you to do
302
00:24:29.840 --> 00:24:34.100
is that when using cloud, is that for every PR that you make,
303
00:24:34.565 --> 00:24:36.265
it actually creates a
304
00:24:36.645 --> 00:24:38.184
lightweight staging environment
305
00:24:38.645 --> 00:24:40.265
that's specific for that PR.
306
00:24:40.645 --> 00:24:43.625
So you write your code, you branch, you push,
307
00:24:44.120 --> 00:24:45.980
then our infrastructure takes over,
308
00:24:46.360 --> 00:24:47.500
deploys your container,
309
00:24:48.200 --> 00:24:52.059
and then actually spins up a lightweight environment where you can either
310
00:24:52.465 --> 00:24:59.445
automatically launch a job every on every push or manually do it in our UI. And it provides a sort of,
311
00:25:00.059 --> 00:25:01.279
like, very lightweight
312
00:25:01.820 --> 00:25:05.840
staging environment that serves almost as like an IDE because you can, like, click runs
313
00:25:06.140 --> 00:25:09.554
and see if they complete or not. It's a complete game changer
314
00:25:10.015 --> 00:25:12.115
in terms of the development life cycle,
315
00:25:12.495 --> 00:25:15.875
and it just fits so naturally into development workflow.
316
00:25:16.410 --> 00:25:18.590
And it's just an incredibly powerful capability.
317
00:25:19.610 --> 00:25:21.630
So I'm incredibly excited about that.
318
00:25:24.245 --> 00:25:26.905
It's time to make sense of today's data tooling ecosystem.
319
00:25:27.445 --> 00:25:29.065
Go to data engineering podcast.com/rudder
320
00:25:30.405 --> 00:25:35.720
to get a guide that will help you build a practical data stack for every phase of your company's journey to data maturity.
321
00:25:36.260 --> 00:25:40.440
The guide includes architectures and tactical advice to help you progress through 4 stages,
322
00:25:40.740 --> 00:25:44.675
starter, growth, machine learning, and real time. Go to dataengineeringpodcast.com/rudder
323
00:25:46.895 --> 00:25:49.475
today to drop the modern data stack and use
324
00:25:53.309 --> 00:25:55.169
And to your point of
325
00:25:55.789 --> 00:25:57.169
the staging experience
326
00:25:57.710 --> 00:25:58.929
being able to make
327
00:25:59.345 --> 00:26:01.365
data workflows and data orchestration
328
00:26:02.065 --> 00:26:04.645
something that is tenable for having a
329
00:26:05.025 --> 00:26:07.845
fast feedback loop for the development workflow.
330
00:26:08.649 --> 00:26:18.350
It's definitely great having that capability built into the orchestrator, but then there's also the question about what about all the pieces that that orchestrator is connecting to? How do you manage those branches and
331
00:26:18.765 --> 00:26:19.985
preventing accidentally
332
00:26:20.365 --> 00:26:20.865
polluting
333
00:26:21.165 --> 00:26:24.225
the actual production data or the production systems and just
334
00:26:24.525 --> 00:26:27.585
the overall space of developer tooling, CICD,
335
00:26:28.350 --> 00:26:36.610
just making sure that you have those fast feedback cycles and are able to validate your logic and your end to end workflow in the data space is
336
00:26:37.185 --> 00:26:40.565
still very painful. And I'm wondering what you see as
337
00:26:41.025 --> 00:26:57.565
some of the other sharp edges that folks are going to have to deal with now that they do have this capability of being able to easily branch their orchestration layer, just how to think about what the broader impact is on the rest of their systems with that capability? Robert Leonard It's a great question, Tobias,
338
00:26:58.345 --> 00:27:00.685
and, you know, we are able to use the orchestrator
339
00:27:01.065 --> 00:27:13.695
itself to manage all these different test environments. So let me give you a simple example. Snowflake actually has good support for this. They have a feature where you can clone a schema, which effectively makes a lightweight copy
340
00:27:14.075 --> 00:27:15.054
of your data
341
00:27:15.434 --> 00:27:19.695
in a copy on right way, but it does it safely. So if you do any sort of mutation,
342
00:27:20.130 --> 00:27:27.910
it does not affect the source production data. This lends itself very nicely to this branch deployment model because effectively, you make a clone schema
343
00:27:28.425 --> 00:27:29.965
for every branched environment.
344
00:27:30.665 --> 00:27:36.525
You can't actually pollute the production data, but you have access to it in order to test your flows.
345
00:27:37.130 --> 00:27:39.870
And then if you actually do any writes in that data warehouse,
346
00:27:40.250 --> 00:27:42.270
it still is safe. Right?
347
00:27:42.810 --> 00:27:44.910
What's awesome about branch deployments
348
00:27:45.290 --> 00:27:54.865
is that you can set it up so that you can automatically run a job whenever you push up new code. So the way it works is that you actually have a job in Daxter
349
00:27:55.325 --> 00:27:55.790
that
350
00:27:56.270 --> 00:27:57.890
clones the Snowflake schema.
351
00:27:58.430 --> 00:28:00.370
And every time you push up
352
00:28:00.670 --> 00:28:07.184
a new PR and or a new change within that PR, it reclones the schema every time, So you kinda start from fresh.
353
00:28:07.565 --> 00:28:12.784
So by marrying the ability to have underlying infrastructure kinda support these things
354
00:28:13.160 --> 00:28:19.500
and have the orchestrator be able to orchestrate the management of those test environments. It kind of all comes together in 1 nice package.
355
00:28:19.800 --> 00:28:21.740
Now the different vendors and tools
356
00:28:22.325 --> 00:28:23.784
support those type of things
357
00:28:24.404 --> 00:28:32.539
to different degrees. Right? So Snowflake has kind of the nice version. If you have, like, a full unmanaged data lake, you might actually have to kick off a job that copies a bunch of data.
358
00:28:32.840 --> 00:28:33.980
Right? Or
359
00:28:34.360 --> 00:28:36.539
write what we call custom IO manager
360
00:28:36.840 --> 00:28:42.125
that would know how to read from kind of the source tables and then write out to copies
361
00:28:42.665 --> 00:28:43.165
elsewhere.
362
00:28:43.465 --> 00:28:46.205
So depending on the tool support and whatnot,
363
00:28:46.740 --> 00:28:49.559
it depends on how much work you have to do. But the entire idea
364
00:28:50.019 --> 00:28:53.880
is that by having this capability built into the orchestration layer,
365
00:28:54.315 --> 00:28:57.534
the orchestration layer itself can manage all these test environments.
366
00:28:57.914 --> 00:29:00.255
And that's really where this magic comes together.
367
00:29:00.715 --> 00:29:02.014
For the data lake
368
00:29:02.380 --> 00:29:18.730
use case, also, there are things like Lake FS that allow you to have a sort of branch and merge style workflow for your data in s 3. I know Iceberg and Project Nessie are looking to be able to do that in the space of things like Hive Tables or tables in your, your, you know, Amazon
369
00:29:19.190 --> 00:29:20.810
Athena Lake or things like that.
370
00:29:21.190 --> 00:29:30.285
And now that there is more movement in some of these different vendors and in what you're providing with the branch deployments, and it is a conversation that is being had.
371
00:29:30.745 --> 00:29:33.405
I'm curious what you see as some of the
372
00:29:33.785 --> 00:29:35.005
potential next steps
373
00:29:35.470 --> 00:29:39.010
that we, as a community, can and should take to help
374
00:29:39.550 --> 00:29:43.035
keep that ball moving or keep that flywheel spinning and continue
375
00:29:43.495 --> 00:29:48.315
on improvements in being able to actually manage these complex and heavyweight
376
00:29:48.695 --> 00:29:49.195
workflows
377
00:29:50.060 --> 00:29:53.520
while being able to safely iterate quickly on
378
00:29:53.980 --> 00:29:56.160
updating logic, modifying schemas,
379
00:29:56.540 --> 00:30:02.865
figuring out how these data flows actually impact the business, and just the the end to end workflow of being able to
380
00:30:03.245 --> 00:30:04.304
manage these
381
00:30:04.605 --> 00:30:05.345
high value,
382
00:30:05.645 --> 00:30:07.910
high risk, you know, heavyweight tasks?
383
00:30:08.530 --> 00:30:13.990
This is 1 of the reasons why I'm so excited about an orchestrator finally supporting this type of thing
384
00:30:14.290 --> 00:30:17.695
is that I think it will provide a great incentive structure
385
00:30:18.155 --> 00:30:20.895
so that people are forced to make progress more quickly.
386
00:30:21.275 --> 00:30:30.250
You know? Because what what will happen will be something like this where people orchestrate in 2 different technologies, and 1 of them, like Snowflake, has a super easy or LakeFS
387
00:30:30.789 --> 00:30:38.375
has, like, a very straightforward way of doing this. And then they're trying to orchestrate it with a tool that doesn't. And then they'll immediately be like, wait.
388
00:30:38.835 --> 00:30:48.550
It'll like, they have this workflow ready to go, and it's just 1 tool that's preventing it. And then those users will exert pressure on their vendors and tools
389
00:30:48.850 --> 00:30:50.550
in order to support that better.
390
00:30:51.010 --> 00:30:56.085
So, yeah, I'm just super excited about having a place where there's an end to end workflow
391
00:30:56.385 --> 00:30:59.605
that kind of aligns with this trend that you're seeing
392
00:30:59.985 --> 00:31:00.725
so that,
393
00:31:01.190 --> 00:31:03.690
you know, it makes it clear what tools have
394
00:31:04.150 --> 00:31:05.370
fallen short here,
395
00:31:06.070 --> 00:31:14.335
so to speak. And I think it will provide a nice incentive structure for everyone to get their ducks in a row. Because right now, if you're trying to orchestrate all these different capabilities,
396
00:31:15.035 --> 00:31:23.100
right, like FFS and, you know, Snowflake's clone scheme are copying whatever and doing all their different bespoke branch and merge workflows.
397
00:31:23.800 --> 00:31:31.524
It's so much work to construct that end that no 1 does it. And therefore, people don't even know what they're missing yet. So to me,
398
00:31:31.985 --> 00:31:33.605
the way to really make progress
399
00:31:34.225 --> 00:31:47.635
on these type of issues is not to, like, gather a conference together and, like, convince every single vendor and, like, nag them to do this and this and this. It's just, like, with actual tooling, with actual users and actual incentives,
400
00:31:48.255 --> 00:31:50.115
just move everything much more quickly.
401
00:31:50.575 --> 00:31:52.275
I agree that having that capability
402
00:31:52.655 --> 00:31:53.155
of
403
00:31:53.659 --> 00:31:56.720
managing the full kind of test environment
404
00:31:57.179 --> 00:32:15.700
in the core of your data platform is definitely going to be a massive leap forward in terms of people's experience and even thinking about that as being a possibility because as you said, otherwise, it's this very heavyweight exercise that is full of toil of just saying, okay. How do I automate each of these pieces individually
405
00:32:16.000 --> 00:32:20.405
and be able to manage all the infrastructure and manage all of the and manage all of the logic and hopefully
406
00:32:21.025 --> 00:32:24.085
not shoot myself in both feet by accident in the process.
407
00:32:24.865 --> 00:32:25.765
Yeah. Totally.
408
00:32:26.610 --> 00:32:27.110
And
409
00:32:27.490 --> 00:32:36.525
and going back to this release of the 1 version of Dagster, the GA release of the cloud product with these new capabilities.
410
00:32:37.225 --> 00:32:48.190
As you said, you've got this DAXTER Day event that you're planning to make all of these announcements. I'm wondering what are some of the other things that you have planned as part of that event and what you're hoping
411
00:32:48.809 --> 00:32:55.385
the participants will take away from it. The event is, you know, short but sweet. Yeah. Or I think we're gonna,
412
00:32:56.404 --> 00:33:05.720
clock in, you know, 45 minutes ish or so. And so what we really wanna be is like a very information dense, you know, high single to noise ratio
413
00:33:06.580 --> 00:33:08.200
where you can come
414
00:33:08.955 --> 00:33:11.695
understand the entire value proposition of the platform
415
00:33:11.995 --> 00:33:19.110
end to end all the way from the framework to the cloud product, and then have a forum where you can connect to other community members and ask questions.
416
00:33:19.730 --> 00:33:30.145
And in general, be kind of the first to know what's going on with all this stuff. And, you know, it'll be available afterwards, of course, for to view. But we're really excited to say, you know, it really is this
417
00:33:30.525 --> 00:33:32.385
coming of age moment for the platform
418
00:33:32.845 --> 00:33:33.345
where,
419
00:33:33.700 --> 00:33:35.000
you know, we're powering
420
00:33:35.940 --> 00:33:37.720
with cloud and open source,
421
00:33:38.420 --> 00:33:47.295
like, some of the most sophisticated data and ML platforms in the world. People have bet their entire mission critical workflows and their entire data platforms on this technology.
422
00:33:47.995 --> 00:33:54.150
And this is more kind of a a coming of age party that communicates that underlying reality
423
00:33:54.850 --> 00:33:55.350
accurately
424
00:33:55.970 --> 00:34:01.414
and then opens up this capability to the entire world. So, you know, it's a big day for
425
00:34:01.715 --> 00:34:02.774
us. 1 of the interesting
426
00:34:03.075 --> 00:34:14.190
aspects of marketing something as stable with that 1.0 version is that it opens the doors for a lot of folks to even consider using it in the first place because there are
427
00:34:14.490 --> 00:34:14.990
organizations
428
00:34:15.290 --> 00:34:20.444
that have restrictions that say, we will never actually use something until it hits version 1 0,
429
00:34:20.664 --> 00:34:22.125
or there are people who
430
00:34:22.744 --> 00:34:23.484
aren't comfortable
431
00:34:24.220 --> 00:34:29.200
being the guinea pigs or testing things out with the possibility of breaking changes because
432
00:34:29.580 --> 00:34:33.840
they don't want to put in the time to actually stay up to date with things as they evolve.
433
00:34:34.204 --> 00:34:36.065
I'm curious what you anticipate
434
00:34:36.444 --> 00:34:36.944
as
435
00:34:37.805 --> 00:34:39.744
the overall impact on the
436
00:34:40.285 --> 00:34:41.585
size and composition
437
00:34:41.885 --> 00:34:43.265
of the community around
438
00:34:43.670 --> 00:34:45.609
now that you are hitting the stable milestone
439
00:34:45.910 --> 00:34:48.650
and some of the things that you as an organization
440
00:34:49.030 --> 00:34:52.890
and managers of the community are doing to prepare for
441
00:34:53.275 --> 00:34:56.575
these new categories of users that are likely to
442
00:34:56.875 --> 00:34:58.494
start kicking the tires on Dagster?
443
00:34:59.035 --> 00:35:02.494
Yeah. Like you said, there's a whole set of users out there who
444
00:35:02.990 --> 00:35:05.170
it's extremely important for them
445
00:35:05.790 --> 00:35:12.815
to have the company and the project signal to them that they're not gonna be broken and it's a stable foundation to be built on.
446
00:35:13.935 --> 00:35:17.474
And the reality is we've been operating like a 1.0 project
447
00:35:17.775 --> 00:35:18.275
for
448
00:35:18.815 --> 00:35:21.810
a long time now, both in terms of the reliability
449
00:35:22.349 --> 00:35:25.490
that we provide to our users as well as
450
00:35:26.030 --> 00:35:27.410
effectively the job.
451
00:35:28.025 --> 00:35:34.445
The core task based layer, for example, has been stable for a long, long time and we don't we don't break anyone there. So on an operational
452
00:35:35.440 --> 00:35:35.940
basis,
453
00:35:36.240 --> 00:35:37.059
day to day,
454
00:35:37.359 --> 00:35:39.460
not that much is gonna change for us,
455
00:35:39.839 --> 00:35:45.375
actually. We've been, like, operating like this for a while. You know, we have done renewed focus
456
00:35:46.234 --> 00:35:47.615
on our documentation.
457
00:35:48.155 --> 00:35:52.619
You know, we have a full time technical writing staff now. They're doing fantastic
458
00:35:53.000 --> 00:35:53.500
work
459
00:35:53.880 --> 00:35:56.380
to improve the docs. So we're preparing to scale
460
00:35:56.920 --> 00:36:00.059
to scale our product education and community management
461
00:36:00.359 --> 00:36:01.420
alongside this.
462
00:36:02.025 --> 00:36:03.645
Now that you do have this
463
00:36:04.185 --> 00:36:06.765
stable point, this marker of
464
00:36:07.705 --> 00:36:09.725
a next stage in your life cycle,
465
00:36:10.180 --> 00:36:11.560
what are some of the
466
00:36:11.860 --> 00:36:15.000
new and upcoming capabilities that you're starting to look to
467
00:36:15.620 --> 00:36:18.680
with this solid platform to be able to build from?
468
00:36:18.994 --> 00:36:23.815
We are gonna continue to invest, you know, in additional integrations
469
00:36:24.115 --> 00:36:29.280
and additional tooling on top of the software defined assets, you know, stuff in particular.
470
00:36:29.660 --> 00:36:33.520
And I think the, you know, you can think of the system as layered.
471
00:36:33.900 --> 00:36:35.040
Right? So
472
00:36:35.575 --> 00:36:37.915
when you have a core stable foundation,
473
00:36:38.615 --> 00:36:39.835
then your activities
474
00:36:40.215 --> 00:36:43.835
generally end up being focused at the tooling built on top of that.
475
00:36:44.140 --> 00:36:48.160
And that's really where a bunch of our focus is gonna be as well as on integrations.
476
00:36:48.859 --> 00:36:51.760
I think there is just a ton of work
477
00:36:52.184 --> 00:36:57.244
in terms of both integrating both at the software defined assets level, but also working with partners
478
00:36:57.545 --> 00:37:00.765
to make those tools work with branch deployments, for example.
479
00:37:01.470 --> 00:37:03.410
And I don't think we can underestimate
480
00:37:04.350 --> 00:37:11.965
how much both work that is and how much value there is in there. So I think you'll see us focus a ton
481
00:37:12.425 --> 00:37:12.925
on
482
00:37:13.225 --> 00:37:16.925
integrations work and value added to win over that stable core.
483
00:37:17.420 --> 00:37:18.640
In terms of other directions,
484
00:37:19.180 --> 00:37:19.839
you know,
485
00:37:20.140 --> 00:37:27.115
on the commercial side, you'll see us also working on supporting organizations as they scale across our platform,
486
00:37:27.494 --> 00:37:29.435
whether that means more advanced
487
00:37:29.895 --> 00:37:30.954
role based assets,
488
00:37:31.255 --> 00:37:32.234
RBAC capabilities,
489
00:37:33.289 --> 00:37:37.789
the ability to, you know, track lineage across your entire org
490
00:37:38.170 --> 00:37:39.390
as you cross deployments
491
00:37:39.690 --> 00:37:40.910
in the Dexter platform.
492
00:37:41.529 --> 00:37:42.029
But,
493
00:37:42.464 --> 00:37:46.165
you know, we are super focused on getting 1.0
494
00:37:46.545 --> 00:37:58.500
and the GA product out the door, and, you know, we'll be publishing more of a roadmap after that. 1 of our earlier episodes, we had a good conversation about your opinions and framework for deciding
495
00:37:59.244 --> 00:38:00.065
what capabilities
496
00:38:00.684 --> 00:38:05.825
belong in the open source framework capabilities of Dagster and which features
497
00:38:06.460 --> 00:38:08.960
belong in the commercial cloud offering.
498
00:38:09.260 --> 00:38:16.744
And I'm wondering how this stable marker of the platform and of the cloud system and its release into general availability
499
00:38:17.605 --> 00:38:19.224
is going to
500
00:38:19.525 --> 00:38:21.944
factor into your ongoing commitment
501
00:38:23.130 --> 00:38:30.350
to or modification of that philosophy and just some of the ways that you're continuing to manage that tension of which capabilities
502
00:38:30.835 --> 00:38:33.894
need to be paid for the sustainability of the organization
503
00:38:34.515 --> 00:38:35.414
and which capabilities
504
00:38:35.795 --> 00:38:39.095
need to be part of the open source because they are
505
00:38:39.530 --> 00:38:41.869
necessary for the continued viability
506
00:38:42.250 --> 00:38:43.950
of DAXTER as its own project.
507
00:38:44.570 --> 00:38:49.435
First of all, let's step back and talk about how it even approached this issue here.
508
00:38:49.975 --> 00:38:51.835
The goal of Dagster has become
509
00:38:52.135 --> 00:38:55.115
as broadly as adopted standard as possible
510
00:38:55.530 --> 00:38:58.030
for structuring your data platforms and data assets.
511
00:38:58.570 --> 00:39:02.430
And then, DAXTER Cloud is to build, you know, an awesome
512
00:39:02.730 --> 00:39:05.725
managed surface that is structured on top of that standard.
513
00:39:06.185 --> 00:39:11.325
And in terms of what goes into that standard, the open source versus what goes into the proprietary platform,
514
00:39:11.865 --> 00:39:14.045
I kinda divide the world into 3
515
00:39:14.850 --> 00:39:16.550
layers of complexity here.
516
00:39:17.010 --> 00:39:17.910
Application complexity,
517
00:39:18.290 --> 00:39:19.750
operational, and enterprise.
518
00:39:20.610 --> 00:39:29.765
So application complexity is about how developers work with the framework to structure their code in order to have, like, better API ergonomics,
519
00:39:30.225 --> 00:39:30.725
testability,
520
00:39:31.430 --> 00:39:33.050
all that stuff. And that's effectively
521
00:39:34.070 --> 00:39:34.730
the relationship
522
00:39:35.510 --> 00:39:36.010
between
523
00:39:36.550 --> 00:39:38.650
the engineer, developer, practitioner,
524
00:39:39.454 --> 00:39:41.795
and the code that runs in their process.
525
00:39:42.095 --> 00:39:45.395
Right? Like, literally, they are pip installing something,
526
00:39:46.255 --> 00:39:51.760
and they are running Python code that we have authored that calls into their code.
527
00:39:52.300 --> 00:39:53.599
So on a very
528
00:39:54.115 --> 00:39:59.735
practical level, that needs to be open source because you have code in the same stack trace, and there's no process boundary
529
00:40:00.355 --> 00:40:01.815
and all that stuff.
530
00:40:02.410 --> 00:40:04.349
And, also, we want
531
00:40:04.730 --> 00:40:09.390
that framework to be composable and embeddable and all sorts of interesting contexts
532
00:40:09.695 --> 00:40:14.595
because we can't predict how everything's going to be used. And that's like the fun and the joy of open source.
533
00:40:14.975 --> 00:40:21.140
So that's 1 layer, the application complexity and the open source framework that will forever and always be open source.
534
00:40:21.600 --> 00:40:26.180
Then you get into the land of like tooling built on top of that stuff.
535
00:40:26.605 --> 00:40:30.065
Right? And I think that's where this gets more subtle and interesting.
536
00:40:30.925 --> 00:40:33.345
So we want teams to be able
537
00:40:33.645 --> 00:40:34.145
to
538
00:40:34.859 --> 00:40:36.720
self host open source Dagster
539
00:40:37.180 --> 00:40:41.680
and run production workloads on it. And we want them to be able to run that on different
540
00:40:42.295 --> 00:40:42.795
infrastructure
541
00:40:43.255 --> 00:40:46.955
like ECS or Kubernetes or just an EC2 node or whatnot.
542
00:40:47.335 --> 00:40:52.750
We want people to be able to do that. We want them to able to self host that, and we will always support that.
543
00:40:53.050 --> 00:41:00.225
But we also have Daixo Cloud, which is a managed service which hosts the computations on their behalf. And the way that I kind of divide
544
00:41:01.005 --> 00:41:04.465
the divide the world here in terms of what should be proprietary
545
00:41:05.005 --> 00:41:06.305
and what should not be
546
00:41:06.650 --> 00:41:10.990
is actually a lot of it is in terms of execution efficiency of the organization.
547
00:41:11.930 --> 00:41:13.375
Meaning, it is much
548
00:41:13.695 --> 00:41:17.395
easier for us to develop advanced operational capabilities
549
00:41:17.935 --> 00:41:18.835
on a centralized
550
00:41:19.375 --> 00:41:19.875
hosted
551
00:41:20.175 --> 00:41:20.675
platform.
552
00:41:21.119 --> 00:41:23.940
We can deploy it whenever we want. We can fix bugs immediately
553
00:41:24.240 --> 00:41:26.900
and so on and so forth. Let me give you a very specific example.
554
00:41:27.359 --> 00:41:28.660
Like a database migration,
555
00:41:29.359 --> 00:41:30.099
it is,
556
00:41:30.605 --> 00:41:34.385
like, so much easier for every single stakeholder involved
557
00:41:35.164 --> 00:41:37.905
if we run 1 centralized database migration
558
00:41:38.285 --> 00:41:39.265
instead of forcing
559
00:41:39.565 --> 00:41:45.570
our, you know, thousands of open source users to run their own database migration against their own infrastructure.
560
00:41:46.190 --> 00:41:54.285
Doesn't mean we're never gonna implement anything that requires a database migration. It just is an example of if we can centrally manage something,
561
00:41:54.745 --> 00:42:01.230
we can move faster and we don't have infinite engineering resources. And then we can fix bugs on the user's behalf and whatnot.
562
00:42:01.770 --> 00:42:04.270
So I often think of this in terms of
563
00:42:04.585 --> 00:42:15.900
if there are operational capabilities where we can centrally host it and there are massive economies of scale for 1 organization to do it instead of distributing that work across thousands of organizations.
564
00:42:16.680 --> 00:42:21.099
We should really bias at minimum towards initial development being in the proprietary
565
00:42:21.800 --> 00:42:29.645
domain because we can move faster and provide better quality of service. I think the value exchange here, and this is something we're rolling out with GA, is that
566
00:42:29.960 --> 00:42:32.940
there should be a fair usage based pricing model
567
00:42:33.320 --> 00:42:36.540
where small players can participate as well as the largest enterprises.
568
00:42:37.445 --> 00:42:41.705
So just to give another example of, like, where economies of scale don't apply,
569
00:42:42.005 --> 00:42:49.350
you know, we have a Kubernetes executor. It's not like we're gonna withhold a bug fix from that and only apply that to
570
00:42:49.890 --> 00:42:51.350
the proprietary stack.
571
00:42:51.810 --> 00:43:05.170
First of all, it's just the wrong thing to do and goes against our values. But in the rubric that I just laid out, there's no economies of scale to not fixing that more broadly. Right? It just is better for everyone. And then the last layer of complexity
572
00:43:05.950 --> 00:43:07.410
is enterprise complexity.
573
00:43:07.710 --> 00:43:08.210
So
574
00:43:08.510 --> 00:43:10.130
security, auditing, etcetera,
575
00:43:10.704 --> 00:43:13.365
these are capabilities which are complicated to implement
576
00:43:13.905 --> 00:43:16.244
that we wanna be able to push changes instantaneously
577
00:43:16.625 --> 00:43:19.170
and fix any issues there. And that
578
00:43:19.470 --> 00:43:21.089
enterprises who want those capabilities
579
00:43:21.470 --> 00:43:23.809
typically want a paid relationship anyway.
580
00:43:24.349 --> 00:43:29.065
Like, they want to write a check because then there's someone on the hook when something goes wrong.
581
00:43:29.445 --> 00:43:32.905
So, again, that was kind of a long winded answer, but 3 layers,
582
00:43:33.365 --> 00:43:36.825
application complexity, that's really the relationship with the individual practitioner.
583
00:43:37.210 --> 00:43:39.390
Those capabilities will always be open source.
584
00:43:40.090 --> 00:43:41.710
Managing operational complexity,
585
00:43:42.010 --> 00:43:43.870
we wanna support production workloads,
586
00:43:44.250 --> 00:43:47.585
but, you know, for really complicated capabilities
587
00:43:48.605 --> 00:43:50.465
that take a lot of work to host
588
00:43:51.244 --> 00:43:56.660
and fix and all that. We wanna bias towards centrally hosting those. And then lastly, enterprise
589
00:43:57.119 --> 00:44:00.660
complexity, which is strictly in the province of the proprietary domain.
590
00:44:01.135 --> 00:44:05.875
Digging into that middle layer of the operational complexity where there is this
591
00:44:06.495 --> 00:44:10.990
decision to be made about whether it is proprietary or open source.
592
00:44:11.290 --> 00:44:12.430
You mentioned that
593
00:44:12.730 --> 00:44:19.935
at least the initial development for some of these capabilities belongs in the commercial offering, and I'm wondering what your
594
00:44:20.475 --> 00:44:24.655
decision structure or mechanisms are around being able to
595
00:44:25.355 --> 00:44:27.215
say, okay. We've built this internally,
596
00:44:27.860 --> 00:44:43.635
but this is actually something that is useful in the open source. And so being able to do that initial development, offering it as a paid capability for your cloud customers, and then eventually migrating that into the open source and just how that decision structure and how that actual
597
00:44:44.260 --> 00:44:49.000
management of the code and being able to make it something that can be migrated
598
00:44:49.300 --> 00:44:50.920
plays out in your mind.
599
00:44:51.275 --> 00:44:56.895
Yeah. We don't have a specific process in place yet. I guess what I'll say is that a lot of it just kind of happens
600
00:44:57.435 --> 00:44:57.935
because
601
00:44:58.600 --> 00:45:01.260
the proprietary service is really kind of a shell
602
00:45:01.800 --> 00:45:02.300
where
603
00:45:02.840 --> 00:45:06.780
work that's done on it naturally flows down into our core infrastructure.
604
00:45:07.400 --> 00:45:07.900
So
605
00:45:08.225 --> 00:45:10.725
that kind of just happens for free.
606
00:45:11.185 --> 00:45:14.565
And in terms of, like, an ongoing process where we
607
00:45:15.025 --> 00:45:18.470
develop things that are focused on the proprietary platform In
608
00:45:22.677 --> 00:45:23.177
your
609
00:45:27.545 --> 00:45:31.885
In your experience of rolling out the software defined assets capability
610
00:45:32.345 --> 00:45:40.560
and onboarding some of your early customers to the cloud platform? What are some of the most interesting or innovative or unexpected ways that you've seen those capabilities used?
611
00:45:41.020 --> 00:45:42.880
Couple examples come to mind.
612
00:45:43.335 --> 00:45:44.715
A bunch of our cloud
613
00:45:45.175 --> 00:45:45.675
users
614
00:45:46.055 --> 00:45:46.795
are actually
615
00:45:47.095 --> 00:45:50.234
energy companies that you probably have heard of, and
616
00:45:51.335 --> 00:45:52.234
1 of them
617
00:45:52.690 --> 00:45:53.190
has
618
00:45:53.650 --> 00:45:54.790
really integrated
619
00:45:55.970 --> 00:45:56.470
Dagster
620
00:45:56.849 --> 00:45:57.670
into their
621
00:45:57.970 --> 00:45:58.470
geologists
622
00:45:58.849 --> 00:45:59.349
workflow
623
00:45:59.730 --> 00:46:01.990
as they, you know, are doing analysis
624
00:46:02.675 --> 00:46:08.135
to the point where it's actually completely integrated with their own internal custom apps.
625
00:46:08.435 --> 00:46:12.420
So they have internal apps where they click a button and it actually kicks off
626
00:46:13.220 --> 00:46:14.599
Dagster Cloud runs.
627
00:46:15.299 --> 00:46:17.640
That metadata is then fed back into
628
00:46:17.940 --> 00:46:19.480
their own custom proprietary
629
00:46:19.780 --> 00:46:27.455
app. Kitellum is kind of like their custom operational tools and our tooling has almost like fused into 1 centralized system,
630
00:46:27.835 --> 00:46:34.010
which is super cool to see. You know, it's just like it demonstrates the value of, like, using open source standards,
631
00:46:34.390 --> 00:46:35.210
like GraphQL
632
00:46:35.510 --> 00:46:38.170
to model your web app because then someone can literally
633
00:46:38.565 --> 00:46:43.464
effectively build a lot of those capabilities in different contexts and integrate it in interesting ways.
634
00:46:43.925 --> 00:46:46.265
So that's been super interesting to see. Unexpected,
635
00:46:47.299 --> 00:46:49.319
this is 1 of my favorites. 1 of our
636
00:46:49.619 --> 00:46:53.880
customers is KIPP, the knowledge is power program, which is a charter school program.
637
00:46:54.339 --> 00:46:54.839
And
638
00:46:55.395 --> 00:47:00.215
they actually or a couple of their geographies, they have a fairly fragmented organization,
639
00:47:00.675 --> 00:47:03.335
are actually using DAXTER to, in effect,
640
00:47:04.210 --> 00:47:04.710
build
641
00:47:05.010 --> 00:47:07.990
report cards and progress reports for their students,
642
00:47:08.530 --> 00:47:09.990
which sounds wild,
643
00:47:10.290 --> 00:47:14.055
but it makes sense if you think about that. They actually use
644
00:47:14.755 --> 00:47:16.535
a bunch of different SaaS tools
645
00:47:16.994 --> 00:47:17.734
to track
646
00:47:18.355 --> 00:47:21.095
student activity, to record their work,
647
00:47:21.650 --> 00:47:23.589
to, you know, do attendance,
648
00:47:23.890 --> 00:47:29.270
all sorts of stuff. And then they actually built a data platform in order to assemble those tools
649
00:47:29.970 --> 00:47:30.470
and
650
00:47:30.935 --> 00:47:35.515
integrate the data and then produce a cohesive report that can be viewed by parents and students.
651
00:47:36.135 --> 00:47:36.635
So,
652
00:47:37.015 --> 00:47:42.400
you know, I love it because I can explain this to, like, my mother. It's like, okay. Imagine you have this report card.
653
00:47:42.780 --> 00:47:55.995
That is an affected data asset in the system, which is composed of other data assets, and it's been integrated and computed upon. And in the end, you know, there are data pipelines which produce these data assets for parents and teachers, and it's a very, like,
654
00:47:56.695 --> 00:47:58.955
graspable example that people can understand.
655
00:47:59.380 --> 00:48:01.640
And something I did not anticipate that
656
00:48:02.099 --> 00:48:09.914
DAXR would be used as an orchestration platform for student report cards, but here we are. It's all data. It's all information that has to go somewhere.
657
00:48:10.855 --> 00:48:11.515
That's right.
658
00:48:12.295 --> 00:48:14.795
In your experience of bringing
659
00:48:15.255 --> 00:48:15.755
DAXTER
660
00:48:16.460 --> 00:48:17.280
and elemental
661
00:48:17.660 --> 00:48:19.200
up to this point of
662
00:48:19.820 --> 00:48:20.640
having a
663
00:48:21.260 --> 00:48:21.760
stable,
664
00:48:22.140 --> 00:48:29.335
solid foundation to build from and marking it with this 1 release and the general availability of the cloud product
665
00:48:29.875 --> 00:48:39.130
and all of the new capabilities and features that you have planned subsequent to that. What what are some of the most interesting or unexpected or challenging lessons that you've learned in that process?
666
00:48:39.750 --> 00:48:42.090
It does speak to the 1.0 release,
667
00:48:42.715 --> 00:48:45.375
and it's something I understood intellectually before,
668
00:48:45.995 --> 00:48:50.415
but now really felt in my bones. It's just how important it is
669
00:48:51.000 --> 00:48:51.740
to communicate
670
00:48:52.040 --> 00:48:53.020
clear expectations
671
00:48:54.200 --> 00:48:54.700
around
672
00:48:55.400 --> 00:48:55.900
stability
673
00:48:56.200 --> 00:48:59.660
and future changes and make sure to do enough upfront design
674
00:49:00.175 --> 00:49:01.635
in order to do that effectively.
675
00:49:02.015 --> 00:49:08.995
I think early in the project we were a little fast and loose about kind of we want to be able to move quickly and whatnot
676
00:49:09.370 --> 00:49:10.670
but it's very different.
677
00:49:11.210 --> 00:49:14.510
You know, you can change your eyes quickly, and that's fine
678
00:49:14.810 --> 00:49:15.310
because
679
00:49:15.850 --> 00:49:19.895
you're not breaking people's code, but you just have to have a very different standard,
680
00:49:20.355 --> 00:49:22.455
you know, in terms of communicating
681
00:49:23.475 --> 00:49:28.630
that this is a stable foundation to build on, and that you have to do more upfront deliberate design,
682
00:49:29.010 --> 00:49:31.830
especially in these yeah. So I think that's been a lesson
683
00:49:32.130 --> 00:49:32.630
and
684
00:49:33.010 --> 00:49:39.405
have had to learn that. And, you know, we've learned from that and have changed our processes. And now we're singling this new phase.
685
00:49:39.785 --> 00:49:42.045
And the other thing, and I spoke to it earlier,
686
00:49:42.505 --> 00:49:44.285
is again, I knew it
687
00:49:44.960 --> 00:49:45.460
intellectually,
688
00:49:46.079 --> 00:49:50.420
but it just in the last 6 months, it has just become super, super real
689
00:49:50.720 --> 00:49:55.655
is that, you know, we're excited about cloud. Like, obviously, you know, we're excited to build a sustainable
690
00:49:56.195 --> 00:50:00.135
business model and make forward progress on that front so we become a sustainable organization.
691
00:50:00.770 --> 00:50:03.990
But cloud has also just increased
692
00:50:04.370 --> 00:50:06.070
our pace of product velocity
693
00:50:06.530 --> 00:50:09.190
because we have a tighter relationship with those customers.
694
00:50:10.035 --> 00:50:13.335
We can see the effects of our changes immediately.
695
00:50:13.875 --> 00:50:16.775
We can push them changes. They can provide feedback.
696
00:50:17.250 --> 00:50:22.390
In a day, we can fix a bug or add a feature, and it's just like the super exciting new phase of development.
697
00:50:23.010 --> 00:50:26.964
So those are kind of the 2 things that come to mind. So we've spent the
698
00:50:27.424 --> 00:50:35.045
past hour or so extolling all of the virtues of DAXTER and the new capabilities and benefits that you're providing, but what are the cases where
699
00:50:35.490 --> 00:50:40.310
and or a DAXTER cloud are actually the wrong choice and somebody is better suited with
700
00:50:40.610 --> 00:50:47.065
a different orchestration system or just writing a bunch of bash scripts or just using the built in orchestrator
701
00:50:47.445 --> 00:50:49.385
for their point solution tool?
702
00:50:49.845 --> 00:50:51.545
Yeah. So to the last thing
703
00:50:51.925 --> 00:50:52.585
is that
704
00:50:53.119 --> 00:50:57.220
if you are only using that 1 tool and as a built in orchestrator,
705
00:50:57.680 --> 00:51:04.395
you know, just use that 1 tool. I think that's, you know, fairly clear. And you only need like cron based scheduling, for example.
706
00:51:05.175 --> 00:51:05.675
But
707
00:51:06.055 --> 00:51:06.555
broadly,
708
00:51:06.855 --> 00:51:09.195
I do think that orchestration is a base capability
709
00:51:09.730 --> 00:51:16.390
that you should build in for your platform from day 1. And obviously, it's, like, self interested in me to say that, but I do believe it's true.
710
00:51:16.770 --> 00:51:17.910
It's kind of like
711
00:51:18.365 --> 00:51:18.865
telling
712
00:51:19.245 --> 00:51:19.745
someone
713
00:51:20.125 --> 00:51:21.345
if they're, like, writing
714
00:51:21.885 --> 00:51:23.905
Python or this isn't the best example,
715
00:51:24.445 --> 00:51:24.945
but,
716
00:51:25.299 --> 00:51:29.559
you know, let's say, like, someone's a Python program. It's like, well, only use classes
717
00:51:29.859 --> 00:51:35.155
if your program is gonna get big. You know? It's like, no. You should, like, structure your programming
718
00:51:35.935 --> 00:51:38.595
from day 1 to kind of you know,
719
00:51:39.055 --> 00:51:45.430
even small things become big later. So you might as well kinda make good engineering decisions all the way from front to back.
720
00:51:45.810 --> 00:51:46.630
In terms of
721
00:51:46.930 --> 00:51:48.630
within the orchestration domain,
722
00:51:49.285 --> 00:51:51.785
when Dagster and Dagster Cloud is the wrong
723
00:51:52.085 --> 00:51:52.585
choice,
724
00:51:53.125 --> 00:51:56.345
I think you're actually seeing a pretty interesting bifurcation
725
00:51:56.805 --> 00:51:58.185
in the orchestration domain
726
00:51:58.720 --> 00:52:00.980
where, you know, when people are evaluating
727
00:52:01.920 --> 00:52:07.859
orchestration from ground up these days, they're kind of generally choosing between 3 options, prefect,
728
00:52:08.665 --> 00:52:09.165
astronomer,
729
00:52:09.465 --> 00:52:10.285
and Dagster.
730
00:52:10.585 --> 00:52:13.484
And I think those 3 solutions are actually
731
00:52:13.945 --> 00:52:18.410
kind of going their separate ways, so to speak, with their most recent directions.
732
00:52:19.190 --> 00:52:21.690
So, for example, Prefect has their new Orion
733
00:52:22.150 --> 00:52:26.705
project, which is gonna be their 2.0 project. And I actually think Orion is quite cool.
734
00:52:27.725 --> 00:52:30.625
And it actually makes it a system much more like temporal,
735
00:52:31.325 --> 00:52:33.825
which is a microservices orchestration engine.
736
00:52:34.210 --> 00:52:36.069
So Temporal and Orion
737
00:52:36.769 --> 00:52:41.349
are a prefect, I should say. They're actually more general tools than DAXTER,
738
00:52:42.095 --> 00:52:45.954
and they're more operationally flexible, but they're much more imperative.
739
00:52:46.734 --> 00:52:49.954
So if you need to kind of build a state machine
740
00:52:50.670 --> 00:52:51.810
instead of a DAG,
741
00:52:52.270 --> 00:52:53.170
you should definitely
742
00:52:53.470 --> 00:52:54.770
choose those projects.
743
00:52:55.310 --> 00:53:01.655
And they're much less tailored towards the data platform use case. So they have no notion of data assets.
744
00:53:02.115 --> 00:53:12.490
You can kind of write almost like a Turing complete state machine using them. But as a result, you can't pre visualize what the computations are gonna be. There's very strict trade offs there. So if you want something more general
745
00:53:13.030 --> 00:53:13.930
and more imperative,
746
00:53:14.550 --> 00:53:18.570
and something with less operational constraints, and therefore more operationally complicated,
747
00:53:19.015 --> 00:53:20.474
you know, temporal
748
00:53:20.934 --> 00:53:26.555
prefix is kind of the way to go. Airflow is kind of your, you know, task based orchestration system,
749
00:53:27.015 --> 00:53:27.515
and
750
00:53:27.900 --> 00:53:31.119
that doesn't really think about the full developer workflow.
751
00:53:31.740 --> 00:53:36.640
And if you don't think that orchestration should have a fast developer feedback loop
752
00:53:37.215 --> 00:53:46.755
and that's not really what its purpose is, that you should have scripts and tools and then the orchestrator's only job is to order and schedule them in production and that's its only domain,
753
00:53:47.200 --> 00:53:49.380
then Airflow is a better solution.
754
00:53:50.080 --> 00:53:53.300
But for DAXTER, it's like, if you are building a data platform
755
00:53:53.760 --> 00:53:55.460
and need a control plane for it
756
00:53:56.005 --> 00:54:03.865
and you believe that you're bringing this you should bring kind of software engineering best practices to the data domain and you wanna orchestrate it to be the seat
757
00:54:04.220 --> 00:54:08.000
of your full end to end life cycle, then we're kind of the solution for you.
758
00:54:08.460 --> 00:54:27.690
We've touched on this a bit already as far as what you've got planned for the near to medium term future of Daxter and elemental. I'm just not sure if there is anything else that you wanted to bring up before we start to close out the show. Dan Oh, there's so many plans and future directions to go. You know, we can save that for the next show. We are, you know, as a company
759
00:54:28.070 --> 00:54:33.770
internally and as well for our external communication purposes, we are laser focused on
760
00:54:34.075 --> 00:54:34.575
DAGSTER
761
00:54:35.115 --> 00:54:35.615
Day
762
00:54:35.995 --> 00:54:40.335
and the 1 release and the, you know, launching cloud to the world. So
763
00:54:40.795 --> 00:54:41.935
those are our plans.
764
00:54:42.430 --> 00:54:51.005
Well, for anybody who wants to get in touch with you and follow along with the work that you're doing or find out more about DAX today, I'll have you add your preferred contact information to the show notes.
765
00:54:51.325 --> 00:54:54.144
And as the final question, what is
766
00:54:54.444 --> 00:54:59.984
your perspective on what you see as being the biggest gap in the tooling or technology that's available for data management today?
767
00:55:00.360 --> 00:55:04.300
And, Tobias, you always ask this of tool authors and vendors,
768
00:55:04.760 --> 00:55:07.100
and I feel like we're being
769
00:55:07.555 --> 00:55:14.055
sucked into a trap because, you know, it's gotta be what we're working on. Well, obviously, it's not because you're already solving that problem.
770
00:55:15.954 --> 00:55:18.550
Yeah. You know, to me, it's a couple things.
771
00:55:19.250 --> 00:55:22.950
1, I do think the lack of a cohesive end to end developer workflow
772
00:55:23.410 --> 00:55:27.275
when the context of data is still a huge problem, an unsolved problem.
773
00:55:27.575 --> 00:55:32.875
And I think some of the stuff that we're working on is gonna be a linchpin for that, but it's gonna be a ecosystem wide
774
00:55:33.255 --> 00:55:35.035
initiative to get that solved.
775
00:55:35.660 --> 00:55:36.400
The other
776
00:55:36.700 --> 00:55:38.240
thing that comes to mind,
777
00:55:38.700 --> 00:55:42.080
and this goes back to this kind of unbundling bundling conversation,
778
00:55:42.595 --> 00:55:51.015
And not to rehash that, but this general notion that we are asking too much of data teams that they have to assemble, like, 12 different tools
779
00:55:51.630 --> 00:55:59.170
in order to get basic end to end capabilities. And I saw a tweet on this since, like, if the data ecosystem was in charge of the car industry,
780
00:55:59.855 --> 00:56:01.555
there'd be a tire vendor,
781
00:56:01.935 --> 00:56:08.515
a steering wheel vendor, a seat vendor, a body vendor, and every single consumer would have to assemble the car by themselves.
782
00:56:09.880 --> 00:56:15.500
And that resonated with me. That's why in our platform, we are building in a bunch of capabilities.
783
00:56:16.275 --> 00:56:19.974
I call them like a base layer because we're not trying to replace all the products, but,
784
00:56:20.275 --> 00:56:23.255
you know, we have integrated linears, observability,
785
00:56:23.954 --> 00:56:24.454
orchestration
786
00:56:24.755 --> 00:56:28.799
where in a basic catalog. So that out of the box, you kind of have,
787
00:56:29.180 --> 00:56:37.105
like, these basic capabilities built in so that if you only need a simple version of those tools, you don't have to integrate a full another vendor. So
788
00:56:37.485 --> 00:56:40.145
to me, it's about the end to end developer workflow
789
00:56:40.445 --> 00:56:42.545
and also this notion of,
790
00:56:43.085 --> 00:56:43.585
like,
791
00:56:43.940 --> 00:56:45.560
not having to integrate
792
00:56:46.180 --> 00:56:48.680
a dozen vendors just to get basic capabilities
793
00:56:48.980 --> 00:56:49.960
into your platform.
794
00:56:50.580 --> 00:57:04.180
Well, thank you very much for taking the time today to join me and share some of the recent updates in DAXTER and the work that you're doing at Elementl and on DAXTER Cloud. It's definitely great to see this commitment to stable APIs
795
00:57:04.880 --> 00:57:09.940
and releasing your cloud product as generally available. So definitely excited to see
796
00:57:10.255 --> 00:57:22.000
that happen and what comes next. I appreciate all the time and energy that you and your team are putting into building such a quality product. So I thank you again for that, and hope you enjoy the rest of your day. Thanks so much, Tobias. Thanks for having me.
797
00:57:28.235 --> 00:57:41.400
Thank you for listening. Don't forget to check out our other shows, the Data Engineering Podcast, which covers the latest on modern data management, and the Machine Learning Podcast, which helps you go from idea to production with machine learning. Visit the site at pythonpodcast.com
798
00:57:42.339 --> 00:57:50.055
to subscribe to the show, sign up for the mailing list, and read the show notes. And if you have learned something or tried out a project from the show, then tell us about it. Email hostspythonpodcast.com
799
00:57:51.555 --> 00:57:57.420
with your story. And to help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers.