文字稿 🔗
WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 10/11/2025 21:34:17
Duration: 3118.179
Channels: 1
1
00:00:11.280 --> 00:00:15.460
Hello, and welcome to the Data Engineering podcast, the show about modern data management.
2
00:00:16.155 --> 00:00:24.735
Data teams everywhere face the same problem. They're forcing ML models, streaming data, and real time processing through orchestration tools built for simple ETL.
3
00:00:25.355 --> 00:00:29.295
The result, inflexible infrastructure that can't adapt to different workloads.
4
00:00:30.160 --> 00:00:32.820
That's why Cash App and Cisco rely on Prefect.
5
00:00:33.360 --> 00:00:42.020
Cash App's fraud detection team got what they needed, flexible compute options, isolated environments for custom packages, and seamless data exchange between workflows.
6
00:00:42.735 --> 00:00:44.434
Each model runs on the right infrastructure,
7
00:00:44.735 --> 00:00:47.635
whether that's high memory machines or distributed compute.
8
00:00:48.415 --> 00:00:52.675
Orchestration is the foundation that determines whether your data team ships or struggles.
9
00:00:53.375 --> 00:00:56.114
ETL, ML model training, AI engineering,
10
00:00:56.570 --> 00:01:00.670
streaming, Prefect runs it all from ingestion to activation in one platform.
11
00:01:01.530 --> 00:01:02.430
Whoop and 1Password
12
00:01:02.730 --> 00:01:04.830
also trust Prefect for their data operations.
13
00:01:05.530 --> 00:01:12.485
If these industry leaders use Prefect for critical workloads, see what it can do for you at dataengineeringpodcast.com/prefect.
14
00:01:12.865 --> 00:01:21.205
Are you tired of data migrations that drag on for months or even years? What if I told you there's a way to cut that timeline by up to a factor of six while guaranteeing accuracy?
15
00:01:22.145 --> 00:01:31.450
DataFold's migration agent is the only AI powered solution that doesn't just translate your code. It validates every single data point to ensure a perfect parity between your old and new systems.
16
00:01:32.229 --> 00:01:42.695
Whether you're moving from Oracle to Snowflake, migrating stored procedures to dbt, or handling complex multisystem migrations, they deliver production ready code with a guaranteed timeline and fixed price.
17
00:01:43.395 --> 00:01:48.774
Stop burning budget on endless consulting hours. Visit dataengineeringpodcast.com/datafold
18
00:01:48.914 --> 00:01:53.655
to book a demo and see how they turn months long migration nightmares into week long success stories.
19
00:01:54.270 --> 00:02:08.370
Your host is Tobias Macy, and today I'm welcoming back Nick Schrock to talk about building an AI analytical system that keeps data teams in the loop in the form of Compass. So, Nick, can you start by introducing yourself for people who haven't heard any of your past appearances?
20
00:02:09.144 --> 00:02:16.364
Yeah. Sure. And, thanks for having me, Tobias. It's always a pleasure being on. So, yeah, briefly, I'm Nick Schrock. I'm the CTO
Load full transcript
and founder
21
00:02:16.905 --> 00:02:19.325
of DAXR Labs, which is the company behind
22
00:02:19.784 --> 00:02:21.060
DAXR, which is a,
23
00:02:21.780 --> 00:02:22.599
open source
24
00:02:22.900 --> 00:02:24.360
data orchestration platform,
25
00:02:24.900 --> 00:02:32.200
and Daxter Plus, which is our commercial hosted product on top of that, and now kind of an additional product, which is called Compass,
26
00:02:32.500 --> 00:02:42.195
which I'm super excited to talk about. Yeah. Before that, kind of I cut my teeth at Facebook engineering, and the thing I was best known for was being one of the cocreators of GraphQL.
27
00:02:42.735 --> 00:02:45.315
So that's kind of my story.
28
00:02:45.935 --> 00:02:47.075
Founded Daxter
29
00:02:47.375 --> 00:02:48.655
in 2018,
30
00:02:48.655 --> 00:02:55.269
so a while ago now, and but kinda kinda really got the company off the ground 2019, hired my first employee then. And we've been
31
00:02:55.650 --> 00:03:04.230
working really hard for a long time and have an at scale open source project and a really healthy commercial business and looking forward to many more years
32
00:03:04.610 --> 00:03:05.269
of success.
33
00:03:06.305 --> 00:03:09.924
You've been running Dagster for almost as long as I've been running this podcast.
34
00:03:10.784 --> 00:03:11.444
It's true.
35
00:03:11.905 --> 00:03:26.620
Your podcast is actually one of the major ways I got up to speed on the domain. In particular, the episode you did about DataOps was Chris Berg. That was, like, kind of a real unlock for me. So I feel like you and I have kind of been on the journey together in some ways.
36
00:03:27.160 --> 00:03:30.460
Absolutely. And it's it's been a crazy ride over the past,
37
00:03:31.000 --> 00:03:33.020
what, eight years now. So Yeah.
38
00:03:33.480 --> 00:03:37.575
So I guess the next stop in that ride is Agentic Systems.
39
00:03:37.955 --> 00:03:46.935
And so because you're working in the technology space, you're obligated to build an Agentic system. So I'm wondering if you could just give a bit of an overview about
40
00:03:47.315 --> 00:03:49.815
what your thoughts are on the application of
41
00:03:50.340 --> 00:03:52.520
agentic systems to data analysis
42
00:03:53.140 --> 00:04:01.240
and some of the ways that you thought about the approach to Compass that keeps data teams in the loop without just leaving them on the sidelines
43
00:04:01.540 --> 00:04:04.600
and letting the AI run rampant over all of their hard work.
44
00:04:05.565 --> 00:04:07.185
Yeah. It's been a fascinating
45
00:04:07.805 --> 00:04:08.305
journey,
46
00:04:08.685 --> 00:04:09.185
actually.
47
00:04:10.045 --> 00:04:11.985
I think both me and
48
00:04:12.444 --> 00:04:16.785
Daxter Labs as a company has been fairly conservative when it's come to
49
00:04:17.460 --> 00:04:22.200
AI and agentic systems up until now. You know, last summer, I wrote this piece.
50
00:04:23.140 --> 00:04:24.820
That was a blog post about,
51
00:04:25.140 --> 00:04:29.480
what I called the rise of medium code and the properties of a software system
52
00:04:29.855 --> 00:04:35.075
that needs to happen or to be an amenable target for AI cogent. And it really focused
53
00:04:35.775 --> 00:04:37.315
on minimizing slop
54
00:04:37.695 --> 00:04:43.235
and having a technical blast radius so the AI can't do much damage to your system and
55
00:04:43.650 --> 00:04:46.310
all that. So I've always thought about it in those terms,
56
00:04:46.690 --> 00:04:47.670
but I was always
57
00:04:48.050 --> 00:05:02.805
a little skeptical about how good the agents could get. And it's really their the progress has really exceeded my expectations in the last year. And I think really a huge I didn't realize the time, but I think a huge release was this release in February
58
00:05:03.425 --> 00:05:03.925
where
59
00:05:04.385 --> 00:05:11.125
Anthropic released Sonnet three seven, I think it was, and Quad Code in the same release.
60
00:05:11.840 --> 00:05:14.099
And those two things was a simultaneous
61
00:05:14.400 --> 00:05:23.460
innovation at the model layer, but, also, I think maybe even more importantly, the application layer over that model layer. And that kind of moment
62
00:05:23.845 --> 00:05:28.985
and the right afterwards was a huge wake up call for me that was like, oh,
63
00:05:29.285 --> 00:05:30.025
these systems
64
00:05:30.485 --> 00:05:36.025
are super ready for prime time now if you apply the right tools and techniques.
65
00:05:36.700 --> 00:05:41.280
And then that really momentum has been building. Actually, you know, in in June,
66
00:05:41.900 --> 00:05:46.880
now this term kind of became part of the ether context engineering.
67
00:05:47.260 --> 00:05:49.840
Toby looked up Postapata, and then it was canonized
68
00:05:50.220 --> 00:05:51.120
by Carpathi.
69
00:05:51.615 --> 00:05:53.395
But it really described how,
70
00:05:53.775 --> 00:05:54.735
you know, you can
71
00:05:55.615 --> 00:06:00.914
you know, it's a kind of a rebrand of prompt engineering, but it it kinda describes how you can programmatically
72
00:06:01.854 --> 00:06:05.155
inject context in the right place in the right time to the right model.
73
00:06:05.509 --> 00:06:06.970
And that mentality
74
00:06:07.350 --> 00:06:11.850
really clicked with me in terms of, like, oh, you know, me,
75
00:06:12.229 --> 00:06:32.945
the lowly product and infrastructure engineer who doesn't know hired the higher order math necessary to kinda build a foundation model. Like, I can really participate in this in a super first class way. So that's kind of the context of my journey. At this point, I'm very, you know, AI pill, I would say, in terms of I believe that we are witnessing the simultaneous
76
00:06:33.245 --> 00:06:33.745
disruption
77
00:06:34.500 --> 00:06:36.680
of multiple layers of the stack simultaneously
78
00:06:37.139 --> 00:06:41.000
in a way that we've never experienced before. So AI is revolutionizing
79
00:06:41.379 --> 00:06:44.360
the way we build software, the way we structure infrastructure,
80
00:06:45.220 --> 00:06:47.800
the way our stakeholder relationships work,
81
00:06:48.185 --> 00:06:49.164
and also
82
00:06:49.785 --> 00:06:57.245
the the consumption layer. Right? ChatDTP has been, like, a dramatic change in the way that we interact with computer systems.
83
00:06:57.865 --> 00:07:02.604
And that has not really reached the enterprise at all either, which is a very interesting topic of discussion.
84
00:07:03.210 --> 00:07:05.710
So I think we're at this massive
85
00:07:06.330 --> 00:07:13.550
way. And it is you know, you you kind of alluded to it how, like, okay. You're in software, so you have to be thinking about agentic code.
86
00:07:13.930 --> 00:07:14.670
And, unfortunately,
87
00:07:15.370 --> 00:07:20.015
kind of that is true. And I think some people are more obnoxious about it. But it's kind of like
88
00:07:20.475 --> 00:07:24.095
ignoring agentic AI would be like ignoring the Internet
89
00:07:24.875 --> 00:07:29.295
in the nineties, you know, and not really thinking about how that's gonna impact your system.
90
00:07:29.669 --> 00:07:30.410
And, you know,
91
00:07:31.270 --> 00:07:43.050
it comp the Internet completely revolutionized all domains of computing. Right? Even the ones that weren't putatively you know? It kinda started with consumer, but then all our infrastructure changed too. So I think this is a similar
92
00:07:43.384 --> 00:07:46.125
a similar wave. So I think it's extremely
93
00:07:47.065 --> 00:07:47.565
exciting,
94
00:07:47.945 --> 00:07:50.604
and I'm just really enthusiastic about the future.
95
00:07:51.945 --> 00:07:53.645
And to help frame
96
00:07:53.945 --> 00:07:55.405
the rest of the conversation,
97
00:07:55.785 --> 00:07:57.085
can you describe
98
00:07:57.465 --> 00:08:02.190
the scope and purpose of Compass and some of the problems that you're trying to solve with it?
99
00:08:02.810 --> 00:08:05.310
Yeah. Without going into features or technologies,
100
00:08:05.690 --> 00:08:06.669
I think the
101
00:08:06.970 --> 00:08:11.789
the highest level the problem we are trying to solve in sort of human terms
102
00:08:12.285 --> 00:08:12.785
is
103
00:08:13.085 --> 00:08:14.225
to completely
104
00:08:14.605 --> 00:08:15.105
restructure
105
00:08:15.645 --> 00:08:16.385
the relationship
106
00:08:16.685 --> 00:08:19.585
between a data platform team and their stakeholders.
107
00:08:20.045 --> 00:08:22.225
Meaning that kind of right now,
108
00:08:22.605 --> 00:08:27.345
I think that data teams feel like they are cogs in a machine,
109
00:08:28.050 --> 00:08:29.669
that they are cost centers,
110
00:08:30.290 --> 00:08:40.550
that they are there to do a job. Business stakeholders ask for data. Business stakeholders ask for dashboards. But then you're kinda disconnected from those business business users because
111
00:08:41.065 --> 00:08:43.965
your work is intermediated by these tools,
112
00:08:44.585 --> 00:08:54.605
which are often not that pleasant to deal with. Like, you know, BI tools being an example. I often joke that BI category, it it feels like it was invented by Dostoevsky
113
00:08:54.950 --> 00:09:13.295
because, like, all BI tools are terrible, but they're all terrible in their own ways. And so what we really want to do is rather than have you know, you you mentioned the term self serve. Rather than think about it as complete self serve, we wanna redefine the relationship between the stakeholders so that the data team is
114
00:09:13.915 --> 00:09:24.894
collaborating with the business stakeholders in real time in a highly positive way where instead of be being viewed as a cost center, they are the face of the value. And so they're collaborate collaboratively working
115
00:09:25.250 --> 00:09:27.430
with their stakeholders, and they're empowering
116
00:09:28.529 --> 00:09:30.230
empowering way more of those stakeholders.
117
00:09:30.610 --> 00:09:34.470
Now that is the problem we're trying to solve. And in the end, that means
118
00:09:34.850 --> 00:09:35.910
much more accessibility
119
00:09:36.769 --> 00:09:40.709
to data, and you can leverage your data platform to do more things in the organization. They're bore thereby increasing the value. K. So that was like a
120
00:09:43.525 --> 00:09:54.025
value. K. So that was like a whole a lot of stuff that I just talked about. But we're redefining the stakeholder relationships such that the perceived and real value of a data platform is higher in the organization.
121
00:09:54.725 --> 00:09:59.120
Now how do we do that? You know, Compass looks fairly
122
00:09:59.500 --> 00:10:00.000
innocent
123
00:10:00.620 --> 00:10:01.680
at first blush.
124
00:10:02.220 --> 00:10:04.560
It is a Slack native
125
00:10:05.100 --> 00:10:05.600
experience
126
00:10:06.060 --> 00:10:10.080
where you can interact with your data in natural language.
127
00:10:10.555 --> 00:10:17.855
It's processed by AI, which sort of acts like a junior analyst, and it can interrogate your data warehouse and do interesting analyses.
128
00:10:18.475 --> 00:10:22.175
That is the user experience. But it ends up
129
00:10:22.569 --> 00:10:26.269
being fairly, I would say, transformative, dare I say, revolutionary,
130
00:10:26.810 --> 00:10:27.949
and it has been internally
131
00:10:28.730 --> 00:10:30.589
because you have the stakeholders
132
00:10:31.529 --> 00:10:36.750
interacting with both this agentic tool, the the the analyst. But then
133
00:10:37.265 --> 00:10:44.245
because there's a agent analyst, the the data team members that are in there in that Slack thread with the business stakeholders,
134
00:10:44.865 --> 00:10:46.245
they're no longer analysts.
135
00:10:46.625 --> 00:10:48.645
They act much more like data stewards.
136
00:10:48.990 --> 00:10:53.649
They're, like, guiding the user to do the correct analysis, and then they manage
137
00:10:54.350 --> 00:10:55.089
the context.
138
00:10:55.470 --> 00:11:00.450
And they manage the context store so they can govern the AI in a very scalable
139
00:11:01.070 --> 00:11:07.185
fashion. And then the business stakeholders, they never leave Slack. There's data vis right there. The stakeholders
140
00:11:08.125 --> 00:11:09.345
can make data requests.
141
00:11:09.805 --> 00:11:10.545
They can
142
00:11:10.925 --> 00:11:20.305
request context corrections. Often, the AI just figures out how to do that for them. So they plug in all sorts of workflows. They can schedule these analyses on a regularized basis.
143
00:11:20.700 --> 00:11:23.520
And in our demos, we really we almost brag.
144
00:11:23.900 --> 00:11:37.555
Right? That's like, you're not gonna see a web UI during this entire demo, and that is super deliberate. Because who wants to learn another web UI? You, like, get bounces web UI. You have to auth to it. You have to learn a completely new information architecture.
145
00:11:37.935 --> 00:11:50.510
Right? You have to learn completely new concepts, and it bounces you out of your collaborative zone. You can no longer go, like, at mention people, etcetera, etcetera. So, yeah, that's kind of the approach. It's a, you know, it's a Slack native natural
146
00:11:50.890 --> 00:11:54.670
natural language analytics experience that is both collaborative,
147
00:11:55.370 --> 00:11:59.790
governed, and natural language in AI driven by AI.
148
00:12:00.745 --> 00:12:09.004
So the common challenge when dealing with all of these agent based systems, as you already pointed out, is this challenge of context engineering
149
00:12:09.545 --> 00:12:11.324
or the alchemy of turning
150
00:12:11.704 --> 00:12:14.764
raw bits into useful information is
151
00:12:15.090 --> 00:12:22.630
the entire purpose of data engineering, to be very reductive about it. And so those two things are
152
00:12:23.010 --> 00:12:24.230
intention because
153
00:12:24.690 --> 00:12:27.030
the data engineer doesn't want to
154
00:12:27.410 --> 00:12:27.910
forego
155
00:12:28.495 --> 00:12:30.995
their purpose and hand it off to an AI
156
00:12:31.295 --> 00:12:38.515
and particularly if they have any pride in their work because they know that the AI isn't going to do an appropriate job of understanding
157
00:12:38.975 --> 00:12:46.200
the business needs, the business context, all of the hardwood knowledge that they've already encoded into the data assets that they're building. Whereas
158
00:12:46.660 --> 00:13:03.274
the consumers of that data, to your point, don't wanna have to deal with learning all the information architecture. They don't want to have to dig through all the docs or go through all the pipelines to really understand what it is that they're actually looking at. And so I'm wondering if you can just talk to some of the ways that you're thinking about that context management
159
00:13:03.575 --> 00:13:04.954
and the handoff between
160
00:13:05.334 --> 00:13:11.915
the data teams who are doing all the
hard work of bringing all this information together and hydrating it with that business context
161
00:13:12.380 --> 00:13:19.760
and the ways that the agentic analyst is able to actually retrieve and interact with that context to be able to understand
162
00:13:20.220 --> 00:13:25.120
how to map the probably very vaguely worded request from the stakeholders
163
00:13:25.725 --> 00:13:30.545
into a concrete plan of action and means of discovery and enumerating
164
00:13:31.165 --> 00:13:34.545
all of the information that's required to be able to fulfill that request.
165
00:13:35.325 --> 00:13:39.024
Right. So there's a lot there. Just trying to start the thesis apart here.
166
00:13:39.390 --> 00:13:41.010
So I guess the
167
00:13:41.550 --> 00:13:42.050
dispositionally
168
00:13:42.590 --> 00:13:44.930
and because I also think it's how the world
169
00:13:45.310 --> 00:13:52.370
works is that I think of the AI as a bicycle for your brain as opposed to a replacement
170
00:13:52.750 --> 00:13:56.475
for it. And in some ways, actually, almost in every way,
171
00:13:56.855 --> 00:13:59.514
these AIs make judgment and taste
172
00:14:00.214 --> 00:14:01.514
that much more leveraged.
173
00:14:01.975 --> 00:14:03.834
Because if you have good taste and judgment,
174
00:14:04.214 --> 00:14:07.834
you can get the AIs to do an extraordinary amount of work on your behalf,
175
00:14:08.450 --> 00:14:18.470
that's high quality. But if you don't have that, then it can you know, I call it a technical debt super spreader. It can copy bad patterns and it can go off the races and hallucinate
176
00:14:18.850 --> 00:14:23.795
lord knows what. So that's kind of my starting place is that we build tools
177
00:14:24.495 --> 00:14:28.915
that keeps humans in the loop, that are governed, and that accelerate
178
00:14:29.455 --> 00:14:30.755
the work and amplify
179
00:14:31.455 --> 00:14:38.529
the work of subject matter experts rather than sort of eviscerating it. So that's kinda that's kinda my starting.
180
00:14:38.990 --> 00:14:42.930
I, so where do you want me to take it next? So that's, like, kind of, like, high level philosophical.
181
00:14:43.550 --> 00:14:50.055
So I think the interesting bit is I'll admit to everybody listening, I've already witnessed the demo of Compass, so I already
182
00:14:52.055 --> 00:14:56.955
know a lot of the details of how this is operating. So these are somewhat leading questions and a little bit of inside baseball. But my understanding,
183
00:14:57.575 --> 00:15:00.714
the Compass utility relies on a repository
184
00:15:01.415 --> 00:15:04.475
of context artifacts for being able to
185
00:15:05.000 --> 00:15:25.475
understand how to map some of the semantics of these analytical requests into the actual data assets that are available with these data assets, at least in its current formulation, largely being restricted to a data warehouse environment for being able to create and execute SQL queries and, warehouse environment
for being able to create and execute SQL queries. And so I'm wondering if you can just talk to how you're thinking about the initialization
186
00:15:25.935 --> 00:15:27.394
of that context repository
187
00:15:28.495 --> 00:15:29.954
from that set of
188
00:15:30.420 --> 00:15:38.920
tables and data assets that already exist in a manner that isn't just a lot of extra busy work on the behalf of the data team, but provides all of the necessary
189
00:15:39.380 --> 00:15:45.960
information and guidance to the agentic system for being able to fulfill the requests of the business stakeholders.
190
00:15:46.795 --> 00:16:02.920
Right. Okay. That makes total sense. So I think there's, like there is an initial step where we bootstrap the contact store with as much information as possible. That takes the form of, you know, setup is super easy. All you do is you plop in your data warehouse creds, and you're good to go. Now what happens
191
00:16:03.460 --> 00:16:12.885
is that we for the tables that you allow us to see, we query the information schema. We get as much information as possible from there. We also
192
00:16:13.265 --> 00:16:16.644
sample the data, and then we programmatically generate
193
00:16:17.425 --> 00:16:22.165
context. I think where a lot of people go wrong with that type of step
194
00:16:22.625 --> 00:16:28.240
is that they aren't thoughtful enough about producing the precise context that your application
195
00:16:28.700 --> 00:16:29.200
expects.
196
00:16:29.580 --> 00:16:34.960
They just, like, dump raw metadata into some context window and, like, hope and pray
197
00:16:35.340 --> 00:16:40.480
that the agent figure stuff out. We're my very much of the belief that you need to very deliberately
198
00:16:41.095 --> 00:16:42.395
produce context programmatically
199
00:16:42.855 --> 00:16:50.075
in a way that's guided and specific to your application. That means there's a level of precision and control. Like, I firmly believe that increasingly,
200
00:16:50.455 --> 00:16:53.275
we're gonna move from data pipelines to context pipelines,
201
00:16:53.600 --> 00:17:16.385
meaning that context will be computed. It'll be computed from other context and other data, and that is data pipelining. Right? So that's one of the reasons we we kind of have an engineering approach to it. And the automatically generated data docs are kind of like an initial step to do that. The second critical piece that's in the context store is the oh, and actually, I'll start. All of this is managed with Git, and we think that's very important. Context
202
00:17:16.845 --> 00:17:17.345
occupies
203
00:17:17.725 --> 00:17:18.545
this fascinating
204
00:17:19.340 --> 00:17:19.840
space
205
00:17:20.140 --> 00:17:24.960
that's sort of in between code and data, meaning that context is computed
206
00:17:25.340 --> 00:17:28.000
just like data, but it also very directly
207
00:17:28.300 --> 00:17:28.800
determines
208
00:17:29.420 --> 00:17:36.155
system behavior just
like code does. This is why we've kind of set on this path of having programmatically
209
00:17:36.615 --> 00:17:37.835
generated context
210
00:17:38.215 --> 00:17:40.155
checked in to a Git repository.
211
00:17:40.615 --> 00:17:50.860
Because if it's checked in, you can track changes, you can revert things, you could write tooling over it to change it in place. It allows all sorts of flexibility
212
00:17:51.240 --> 00:18:09.545
and precision. Right? So imagine you're really developing this thing at scale. You have evals so you can evaluate the performance of the agent. Right? You can do, like, a bisect against the context just like code, and that's a super powerful dynamic. The second piece that we capture in Git are these manual context corrections
213
00:18:10.005 --> 00:18:14.745
that we get directly from the business stakeholders. So this is another critical piece
214
00:18:15.045 --> 00:18:18.585
of the context puzzle. The first one, again, being programmatic generation
215
00:18:19.045 --> 00:18:34.995
of context. And then the second piece is actually getting the information out of the brains of your stakeholders who actually know the domain and into some governed context store where the agents can utilize that to give correct results. Like, for example, at one of our early customers,
216
00:18:36.015 --> 00:18:37.795
they use the term core
217
00:18:38.255 --> 00:18:47.955
as a kind of special code word for a project that doesn't really mean core, and they flag this. You know? They're like, this has screwed up AIs before,
218
00:18:48.340 --> 00:19:01.720
and they totally hallucinate because, obviously, it has its own idea of what core means in the foundation layer. So we created this context correction that very specifically laid out, like, okay. Core actually means these, this and this, these context,
219
00:19:02.035 --> 00:19:13.095
etcetera, etcetera, and the system performed well. Now what's really magical about the way Compass works is that all of this is kind of captured either explicitly or in an ambient sense
220
00:19:13.395 --> 00:19:41.795
from the interactions that are happening in Slack. So, for example, if the business stakeholder is presented within Slack with a data visualization that looks wrong, like the demo that we give is, like, some sales rep has a 90% win rate, and that never happens. So the demo is, like, you look into it, you do some investigation, you figure out that the sales rep is actually a customer success manager, and then that automatically submits that context back to the contact store and says, like, hey. Don't count CSMs
221
00:19:42.140 --> 00:20:00.855
as sales reps. So and then that, it's then checked into the context store. Knowledge is captured, and then the data team can kind of take that and manage it very explicitly. And that's a very powerful model to boot strap. You know, generally, alternative and more heavyweight systems like semantic layers is to do this upfront
222
00:20:01.315 --> 00:20:34.395
process that is very burdensome and complicated, and you need to send the business stakeholder to a custom tool or a web app or something. And they never do that. So the knowledge stays captured in their head, which does no good to anyone except for them, maybe. But as we demonstrated, humans have limited context windows too. And what's good for the goose is good for the gander here. So it's important to get that out of your head and into a place where the the agents can take advantage of it. And it's really this lightweight interface within Slack that's really the magic on in in that part of the process.
223
00:20:34.775 --> 00:20:36.155
One of the other aspects
224
00:20:36.980 --> 00:20:45.000
of data teams who are responsible for the care and feeding of analytical systems, particularly when you're dealing with business intelligence,
225
00:20:45.380 --> 00:20:50.040
is that it's often very difficult to gain any real insight into
226
00:20:50.565 --> 00:20:51.065
how
227
00:20:51.365 --> 00:21:00.024
the stakeholders are interacting with those systems. You might be able to have some audit logs to see how frequently people are running certain queries or dashboards.
228
00:21:00.404 --> 00:21:09.050
But beyond that, you don't know why they're going to those dashboards, what they're doing with the information once they retrieve it. And I'm curious how you're thinking about the ways
229
00:21:09.430 --> 00:21:10.890
of bringing more visibility
230
00:21:11.510 --> 00:21:14.170
both to the data teams and at the organizational
231
00:21:14.550 --> 00:21:16.970
level of how the overall
232
00:21:17.270 --> 00:21:35.120
company is interacting with the data assets that you have and are creating and some of the ways that that can create some feedback loops to the data teams either to prune unused data assets or to understand what new data assets they need to generate to be able to fulfill the needs of the organization,
233
00:21:35.500 --> 00:21:38.560
and in particular, how using the conversational
234
00:21:38.940 --> 00:21:42.560
system of record for the company helps to provide some of that visibility?
235
00:21:43.740 --> 00:21:45.040
Oh, it's a great question.
236
00:21:45.340 --> 00:21:52.924
And I think I could talk for hours on this subject because it's not just what's currently happening, but I think the roadmap on this front
237
00:21:53.465 --> 00:21:56.445
is super, super bright. You know, we're still early stages
238
00:21:56.745 --> 00:22:01.130
here. So, you know, I guess, like, just the first thing that happens
239
00:22:01.910 --> 00:22:03.210
is that when
240
00:22:04.150 --> 00:22:06.730
the business stakeholder is in the same channel
241
00:22:07.030 --> 00:22:16.304
as the analyst, and the analyst can literally see what the business stakeholder wants, gets literally compiled to SQL in the data warehouse,
242
00:22:16.845 --> 00:22:21.345
that gap between business language and, like, your column and table names
communicates
243
00:22:21.725 --> 00:22:31.730
so much about what is actually happening and what people actually want. If you're with a traditional BI tool or exploratory data analysis tool, that that translation
244
00:22:32.190 --> 00:22:36.769
does not exist in a format that can be discernible by one of the analysts.
245
00:22:37.070 --> 00:22:53.065
So I think that is the dynamic. Just the social dynamics end up really producing a ton of insight. And we are just at the beginnings of being able to use, I like your term, the conversational system of record to drive more value out of that. The initial
246
00:22:53.365 --> 00:22:57.920
kind of, like, feature we have that I think demonstrates that is that we have the ability
247
00:22:58.380 --> 00:23:09.794
to create data requests in a ticketing system automatically based on what's been happening in a specific thread. So let me give you an example of how this works. So I was asking specifically
248
00:23:10.414 --> 00:23:10.914
Compass
249
00:23:11.535 --> 00:23:27.660
about product analytics, and I asked and this but it expands beyond that. So I was asking what I thought was a very simple question, which was, how many of our customers use declarative automation, which is one of our scheduling features? It turns out our warehouse didn't really have that particular feature
250
00:23:28.040 --> 00:23:37.315
explicitly modeled well. So what the AI did, and this was awesome and terrifying to watch, what it actually did is we have Gong
251
00:23:37.615 --> 00:23:38.115
transcripts,
252
00:23:38.575 --> 00:23:41.955
meaning Gong is a system that records and transcribes
253
00:23:42.255 --> 00:23:44.995
sales calls. So the AI decided
254
00:23:45.350 --> 00:23:48.330
to use Snowflake's features of AI based analyses
255
00:23:48.950 --> 00:23:53.210
to it couldn't find the exact information, but what it did do is it found
256
00:23:53.510 --> 00:24:00.005
all the customers that had mentioned declarative automation in one of their sales calls, which was obviously
257
00:24:00.385 --> 00:24:08.165
imprecise, but gave me a hint or at least a floor of how many people use it and interesting customers that use it. It was also terrifying
258
00:24:08.625 --> 00:24:51.160
because those capabilities are extremely expensive in Snowflake. But as a result of this experience, I was like, make a data request ticket so that we have this information first class in the data warehouse. And what the system does is it scoops up that entire conversation. And think about all the context you have. You have you have who asked for it. You have all this SQL that was generated that is navigating the warehouse and trying to figure out where where things are going on. You have follow-up questions. You might have a conversation. Maybe our analyst jumps in and says, like, oh, yeah. We don't have that because of this historical reason. It scoops all that up, synthesizes it, and creates a data request. Right? And that's an LLM assisted
process. So we're really building, you know, Databricks calls these compound AI applications.
259
00:24:51.620 --> 00:24:54.440
But what it means is kind of injecting where appropriate
260
00:24:54.900 --> 00:24:55.960
LLM augmentation
261
00:24:56.260 --> 00:24:56.919
and processing
262
00:24:57.274 --> 00:25:02.335
into parts of the workflow. And I think we're just at the beginnings of this. As we develop this product
263
00:25:02.715 --> 00:25:09.455
over time, we will be doing much leveraging the conversational system of record much more. You can imagine
264
00:25:09.755 --> 00:25:12.335
doing post hoc processing on threads
265
00:25:12.799 --> 00:25:14.179
that allow you to
266
00:25:14.480 --> 00:25:15.860
discern with more detail,
267
00:25:16.160 --> 00:25:17.220
like, what suggested
268
00:25:17.520 --> 00:25:19.380
context corrections, for example,
269
00:25:19.760 --> 00:25:21.059
scooping up all that information.
270
00:25:21.520 --> 00:25:25.700
You can imagine having observability and ins insights tools
271
00:25:26.054 --> 00:25:31.034
across all the conversations happening in all different channels across your business so you can understand,
272
00:25:31.575 --> 00:25:36.955
like, what's happening, what data is being requested the most frequently, what data is not being requested.
273
00:25:37.335 --> 00:25:40.520
You know, my vision is kinda like, it's almost you know how
274
00:25:40.980 --> 00:25:42.040
during COVID,
275
00:25:42.340 --> 00:25:49.880
right, Google, you could figure out when the COVID was spreading with Google Trends, and it was ahead of any reporting
276
00:25:50.420 --> 00:25:59.635
because people would start asking about, you know, oh, I'm losing my I I I can no longer taste something. You could kind of see that go through the country and lead
277
00:26:00.015 --> 00:26:19.870
be a leading indicator of the true metrics that get reported. And I think I'm imagining a future where at a company, you can really get a sense of, like, what people care about in aggregate. If you if the analytical queries are shifting in the business, that's actually a good insight in a large organization about what people care about, what they're worried about, etcetera, etcetera. So I think there's a huge
278
00:26:20.330 --> 00:26:21.149
space here
279
00:26:21.529 --> 00:26:34.425
to get broad based intelligence from this conversational system of record and how the context is being accessed to. So I think it's, like, a a very perceptive question, and I'm like, I think, like, there's just a huge amount of greenfield around there.
280
00:26:35.285 --> 00:26:42.649
It's interesting too. I was actually, just earlier today having a conversation with somebody who's building an agentic
281
00:26:43.110 --> 00:26:43.610
coding
282
00:26:44.070 --> 00:26:45.210
platform for
283
00:26:45.590 --> 00:26:48.889
doing software engineering in an autonomous fashion, and
284
00:26:49.269 --> 00:26:58.105
it brought up the whole idea of Conway's Law about how the structure of the software is defined by the communication patterns of the organization.
285
00:26:58.645 --> 00:27:05.065
And once you start introducing these agentic systems, that changes the communication patterns to also incorporate those LLMs,
286
00:27:05.445 --> 00:27:06.505
which by necessity
287
00:27:07.125 --> 00:27:22.265
modifies the structures of the software that gets created. And I'm wondering how you see that analogy play out in the context of these agentic analytical systems and the role that it plays in terms of the design and orchestration
288
00:27:22.565 --> 00:27:29.225
of the data assets that you're building and the ways that people are interacting with those data systems. But because we have these LLMs
289
00:27:29.685 --> 00:27:39.070
in play, it is no longer human to human interaction or human to deterministic machine interaction. The LLM then plays a role in that communication system
290
00:27:39.370 --> 00:27:42.190
and modifies the ways that people are interacting with it.
291
00:27:43.130 --> 00:28:13.340
That is really interesting. The Conway's law analogy, I hadn't thought about, but it makes total sense. Because one of the things that I think is happening in the AI era is that I think nearly every stakeholder relationship is going to be reimagined for this era. And I think part of that is because the new consumption layers facilitate new team organizations because of the Conway's law. So just as an example that's not in data platforms, what I mean by that is that there's a wire there's, like, a all software engineering is going away boomlet.
292
00:28:13.720 --> 00:28:18.495
Right? That was kind of a big conversation, which I thought was complete load of, you know, whatever.
293
00:28:18.935 --> 00:28:20.715
It's a family show. Right?
294
00:28:21.415 --> 00:28:21.915
Tobias.
295
00:28:22.535 --> 00:28:27.915
The, but, you know, PMs vibe coding does not mean that software engineers are going away. However,
296
00:28:28.615 --> 00:28:29.355
the ability
297
00:28:30.220 --> 00:28:31.840
of PMs to prototype
298
00:28:32.380 --> 00:28:36.000
and build things in the native system of the engineers
299
00:28:36.460 --> 00:28:46.815
fundamentally and completely transforms their stakeholder relationship. And I think this is partially kind of one of these Conway law esque effect. And that's also what's happening in Compass between the business stakeholder
300
00:28:47.195 --> 00:28:57.695
and the analyst where the business stakeholder can now kind of do Vibe analytics in their own way and communicate directly in the native medium that the
301
00:28:58.370 --> 00:28:58.870
analyst
302
00:28:59.490 --> 00:29:02.790
can understand. So and this Conway of Law effect
303
00:29:03.330 --> 00:29:04.310
is I am
304
00:29:04.610 --> 00:29:05.510
I am incredibly
305
00:29:06.050 --> 00:29:06.550
bullish
306
00:29:06.930 --> 00:29:07.990
on this
307
00:29:08.450 --> 00:29:09.590
UI interaction
308
00:29:10.210 -->
00:29:11.510
of multiplayer
309
00:29:11.970 --> 00:29:13.110
agentic chat
310
00:29:13.725 --> 00:29:19.424
in b two b context. So, you know, single player agentic chat, like ChatDVT
311
00:29:19.725 --> 00:29:20.785
and its competitors,
312
00:29:21.165 --> 00:29:21.985
have completely
313
00:29:22.365 --> 00:29:32.679
remade consumer software and are in the process of doing so. And I think that this multiplayer collaborative chat is gonna do the is the same order of magnitude change in the enterprise.
314
00:29:32.980 --> 00:29:41.320
You know, we're seeing it right now in Compass because, you know, if you stack that on top of the data platform, you effectively don't need reporting functionality
315
00:29:41.940 --> 00:29:50.684
across all of your vertical SaaS apps. It's just in this one spot, which is super, super exciting. And, you know, and I think this, like, agentic chat is what does it because
316
00:29:50.985 --> 00:29:59.940
you're bringing in people. You can bring in the random stakeholders. And then instead of, like, the agent, I think a lot of people's mental model of the agent is, like, someone's alone
317
00:30:00.320 --> 00:30:08.019
and talking to the agent in an enterprise context. That doesn't make sense. What the Slack modality does is that the agent is a participant
318
00:30:08.395 --> 00:30:09.215
in a collective
319
00:30:09.515 --> 00:30:11.535
conversation that incorporates workflows,
320
00:30:11.995 --> 00:30:31.280
and that is a super powerful dynamic that also kind of changes the communication structures here. So there's a lot of I think people have the wrong mental model of this. There's, like there's also a boom lit about, like, oh, there's gonna be, like, one person startups that are billion dollar companies. And, like, I don't really think that's true
321
00:30:31.660 --> 00:30:38.000
either because I just I don't imagine a world where, like, one human is talking to, like, n agents
322
00:30:38.684 --> 00:30:41.184
and building a company like that.
323
00:30:41.485 --> 00:31:00.750
I think it more of, like, there's there's there could be fewer people, more hyper empowered people, but it's always gonna be hybrid where there's lots of humans and lots of agents and this the the humans are sort of up leveling their work. At least that's the way maybe it's just the way the I want the world to work, but I is I I think it is the way the world will work.
324
00:31:01.210 --> 00:31:02.190
I think it's also
325
00:31:02.730 --> 00:31:08.830
indicative of just the overall tendency for people to take a proof of concept and extrapolate
326
00:31:09.130 --> 00:31:26.760
to a larger scale that is not true. I mean, as with anything in software and technology, it's, oh, I built this system in a weekend, so therefore, I can build an entire production company by the end of the week. But the the the factors of scale are something that nobody ever properly accounts for where you're dealing with
327
00:31:27.060 --> 00:31:29.080
exponential complexity but logarithmic
328
00:31:29.380 --> 00:31:37.720
capability. And so you're you're going to diverge sooner rather than later in terms of what you could actually feasibly maintain. And so I think similarly
329
00:31:38.044 --> 00:32:03.260
with that idea of the one person company where I just have 50 different agents, it's like your your head's going to explode trying to keep up with them. And, eventually, the agents you're going to hit the law of diminishing return where the inaccuracies of the agents are going to start compounding, and it's going to drive your multibillion dollar company into the ground before it ever takes off. And I think it's it's also indicative of the hype cycle that came out with the initial release of chat GPT about saying, oh, well,
330
00:32:03.560 --> 00:32:11.775
AGI is now just around the corner. We're going to have it by the end of next year, and now it's in the end of two years or five years or it keeps getting pushed back.
331
00:32:12.875 --> 00:32:16.175
Yeah. And what is even AGI? You know? Like,
332
00:32:16.635 --> 00:32:19.215
it's a very difficult thing to define.
333
00:32:19.515 --> 00:32:20.990
You know, I use the term
334
00:32:21.290 --> 00:32:26.590
technical debt super spreader and things like that. I actually think that's a specific instantiation
335
00:32:27.050 --> 00:32:33.870
of a more general trend that's gonna be across multidomains. Because, like, I think with AI, we're going to be entering a complexity crisis
336
00:32:34.215 --> 00:32:40.794
effectively. Like, the ability of agentic systems and humans empowered by agentic systems to produce complexity,
337
00:32:41.174 --> 00:32:46.315
junk content, interrelated cons concepts that you fundamentally don't understand
338
00:32:46.695 --> 00:33:09.045
is very, very high. So I think that the ability to manage and model complexity will become only more and more leveraged. You know? And that's that's what I think about when I'm doing agentic engineering is really compartmentalizing complexity in a way where the agents can contribute the right things at the right time. But, yeah, we are it's going to be a very
339
00:33:09.424 --> 00:33:12.645
complicated world with all these agents running around.
340
00:33:13.120 --> 00:33:23.059
And now bringing us back around to Compass and these agentic systems for AI and the role of the data infrastructure and the data teams in that landscape,
341
00:33:23.600 --> 00:33:26.260
what are some of the ways that the
342
00:33:26.715 --> 00:33:28.975
requirements of the data infrastructure change
343
00:33:30.075 --> 00:33:32.975
when you have to support these agentic systems,
344
00:33:33.355 --> 00:33:40.895
and what are some of the aspects that can remain the same and the agent is able to just use systems as they exist today?
345
00:33:41.690 --> 00:33:43.070
You know, it's such a broad
346
00:33:43.450 --> 00:33:43.950
question.
347
00:33:44.410 --> 00:33:46.590
And, you know, the
agentic systems
348
00:33:46.970 --> 00:33:47.790
are so
349
00:33:48.170 --> 00:33:53.950
new, and not that many people have deployed them at real scale, that I think it's actually very difficult
350
00:33:54.565 --> 00:33:55.225
to understand
351
00:33:55.765 --> 00:34:19.290
at this time how exactly it's going to impact everything. You know? I think lots of people there lots of people have these like, a lot of people are like, oh, there's gonna be more unstructured data. I don't even know if that's true, for example, because, actually, for these AI systems to operate over and do super leverage thing, you actually want tons of structure and tons of metadata, tons of context. You know, I think that real time,
352
00:34:19.590 --> 00:34:20.570
more complex
353
00:34:21.190 --> 00:34:21.690
workflows
354
00:34:22.475 --> 00:34:24.335
are going to be incredibly
355
00:34:24.795 --> 00:34:25.295
important.
356
00:34:25.915 --> 00:34:28.735
I'm quite bullish on systems like Temporal,
357
00:34:29.115 --> 00:34:29.775
for example,
358
00:34:30.155 --> 00:34:31.215
to manage
359
00:34:31.595 --> 00:34:40.180
the agentic workflows and the complicated agentic workflows that go on because the ability to pause and resume compute
360
00:34:40.640 --> 00:34:46.980
will be very important. Because, like, one of the interesting things happening is that agentic workflows are so high latency.
361
00:34:47.360 --> 00:34:49.300
Right? Users are now trained
362
00:34:50.525 --> 00:34:58.465
to think for a computer to think for minutes on end on your behalf, which is very different from, say, the web era where, like, every millisecond
363
00:34:58.765 --> 00:35:00.385
counted. And utilizing
364
00:35:00.845 --> 00:35:02.625
computational resources efficiently
365
00:35:02.925 --> 00:35:14.420
in those contexts, I think, is actually quite challenging. There's any number of things I could spout off about, but I think I think anyone who gives you definitive answers about how all this stuff is gonna impact infrastructure
366
00:35:15.120 --> 00:35:20.845
doesn't really know what they're talking about. Because, like I said, every layer in the stack is getting disrupted,
367
00:35:21.145 --> 00:35:24.125
and both the consumption layer is changing.
368
00:35:24.585 --> 00:35:39.290
That implies changes to the compute that's actually running, but also the AI is impacting the way these infrastructure things are built. So there's multiple dimensions of variability right now, so I think it's very difficult to project beyond pure print conjecture what's gonna be changing.
369
00:35:39.750 --> 00:35:41.610
One of the other aspects of
370
00:35:41.910 --> 00:35:44.570
bringing an agent into the equation is obviously
371
00:35:44.950 --> 00:35:48.375
cost because LLMs have very unpredictable
372
00:35:48.914 --> 00:35:49.734
cost patterns.
373
00:35:50.115 --> 00:35:50.775
And so
374
00:35:51.155 --> 00:36:05.790
you don't want to route every request through the LLM and especially
having it do a huge body of work if it's something that you already have a stored data asset for. And I'm wondering if how you're thinking about some of the methods around
375
00:36:06.090 --> 00:36:15.845
taking some of the common questions and interaction patterns with that agent and being able to then either cache them for quick retrieval
376
00:36:16.465 --> 00:36:24.005
or materialize them into a more durable asset so it's not something that gets recomputed every time or just some of the ways to enforce
377
00:36:24.545 --> 00:36:26.165
the interaction patterns
378
00:36:26.465 --> 00:36:27.525
of the
379
00:36:27.930 --> 00:36:40.990
stakeholders to say, you don't have to ask the LM this question every time. You can go here for it, or it's going to deliver this to you without you having to take any action into some of the means of mitigation of unbounded cost.
380
00:36:41.805 --> 00:36:45.025
Yeah. No. It's a you know, I felt this very personally
381
00:36:45.325 --> 00:36:50.865
because I went from zero to, like, a 100 on agentic coding this summer,
382
00:36:51.484 --> 00:36:56.385
and I hadn't signed up for the Claude Max plan. I just used our corporate account,
383
00:36:56.829 --> 00:37:05.970
which doesn't have that sort of high usage limit. And in my first two weeks of Claude code usage, I cost the company $3,000.
384
00:37:06.349 --> 00:37:10.005
So we're able to get it under control, but, you know, you can consume
385
00:37:10.385 --> 00:37:19.125
a lot of cost doing that. I kind of like I don't even wanna think about, like, how much natural gas was burned to produce those, you know, 10,000 lines of code or whatever.
386
00:37:20.065 --> 00:37:34.380
So the cost you know, I mentioned a complexity crisis before. I also think there's a cost crisis coming. And I think the first answer here is that earlier in this episode, I mentioned that I think context pipelines are the kind of the new data pipelines.
387
00:37:34.680 --> 00:37:38.380
And that is gonna one, is that you wanna be precise
388
00:37:38.680 --> 00:37:39.180
about
389
00:37:39.575 --> 00:37:41.515
when and how you recalculate
390
00:37:41.974 --> 00:37:42.474
context.
391
00:37:42.775 --> 00:37:46.634
And that means it's a data pipelining problem, like doing event based computation,
392
00:37:47.174 --> 00:37:54.315
crafting the computation in a very specific way, and then producing it in a highly tailored way so it's perfect for your application.
393
00:37:54.615 --> 00:37:55.115
So
394
00:37:55.460 --> 00:38:09.640
writing data pipelines that become context pipelines and then matching that with context engineering, meaning taking those produced artifacts and feeding it to the right model at the right time, the combination of those two techniques, I think, are going to be essential
395
00:38:09.940 --> 00:38:10.760
for controlling
396
00:38:11.355 --> 00:38:26.815
costs. You know, because, like, the larger the context window, the more expensive the compute is. Like, prefill is quadratic with respect of context window size, and it just determines a ton about model performance. But I think that the cost crisis coming is real.
397
00:38:27.330 --> 00:38:29.830
I think the chickens will come home to roost
398
00:38:30.610 --> 00:38:32.550
for a lot of these firms
399
00:38:32.850 --> 00:38:33.590
who aren't
400
00:38:33.890 --> 00:38:37.510
passing through enough of the compute cost to their customers,
401
00:38:37.970 --> 00:38:48.165
and their customers will have a rude awakening and churn. And I think, you know, some of the coding startups are encountering that challenge right now. So, yeah, I think there's gonna be a huge amount of techniques,
402
00:38:48.785 --> 00:38:50.645
and those techniques will stay
403
00:38:51.510 --> 00:38:53.450
extremely relevant even as the models
404
00:38:53.910 --> 00:38:57.610
get better and even as they get cheaper
405
00:38:57.990 --> 00:39:00.810
too. Because some of this context management,
406
00:39:01.110 --> 00:39:05.370
I view it as kind of almost like big o notation or algorithmic complexity,
407
00:39:05.875 --> 00:39:10.855
Meaning that no matter how good Moore's law, an o of n squared sort algorithm
408
00:39:11.235 --> 00:39:13.895
can only go so far because no matter
409
00:39:14.195 --> 00:39:27.859
how fast the processor is. And I think the same thing is gonna be true with context engineering. You know? Like, we're even seeing this now. Like, we're getting to a million and, like, even enormously large context windows, but they have enormous amounts of diminishing
410
00:39:28.480 --> 00:39:33.955
return. And even it can be a negative thing if you pollute the context with contradictory
411
00:39:34.495 --> 00:39:40.915
information. You know, this is famously called, like, context poisoning or context rot and all this stuff. So I think, like,
412
00:39:41.215 --> 00:39:42.195
context engineering
413
00:39:42.735 --> 00:39:46.800
is gonna be more and more expansive. I think that is gonna be
414
00:39:47.200 --> 00:39:56.340
a common theme to control cost. And then beyond that, having more control over fine tuning. And, I think there's a whole undiscovered country in terms of democratizing
415
00:39:57.040 --> 00:40:12.154
fine tuning and then having the model providers have built in capabilities so you can do fine tuning over their closed source models. But it is early days. But we are going to burn a lot of money and energy along the way, and it's gonna be becoming increasingly more important to control it.
416
00:40:12.454 --> 00:40:16.875
As you have been going through this journey of building Compass,
417
00:40:17.319 --> 00:40:19.980
testing it out, getting it in front of some early adopters.
418
00:40:20.280 --> 00:40:28.700
What are some of the most interesting or innovative or unexpected ways that you've seen teams apply these agentic capabilities on top of their existing data investments?
419
00:40:29.735 --> 00:40:31.355
That is a good question.
420
00:40:31.735 --> 00:40:57.705
You know, I think the thing that has really struck out at me is that, you know, the only in our funnel you know, we're still on the order of dozens of users, and we're you know, this is week of October 6, and we're opening the floodgates a little bit. You know, we have hundreds of people on the wait list. I think the thing that has struck out at me is that effectively, once people connect their data warehouse to the system, we have a 100% retention on the platform,
421
00:40:58.085 --> 00:41:00.025
which is crazy. So
422
00:41:00.325 --> 00:41:04.825
people start using it, and they the usage is intense, and they get
423
00:41:05.125 --> 00:41:07.065
tons and tons of stakeholders
424
00:41:07.525 --> 00:41:09.619
in the system. Right? And, internally,
425
00:41:10.079 --> 00:41:27.575
you know, we actually purchased some datasets that we're gonna make public that effectively are the moral equivalent of, like, the pitch book data. So companies and their fundraising histories and their revenue numbers and all this sort of stuff. And then a people database, which is kinda like the LinkedIn dataset. And just Compass
426
00:41:28.115 --> 00:41:29.335
plus those things
427
00:41:29.795 --> 00:41:31.414
make it, like, the best
428
00:41:31.795 --> 00:41:32.295
prospecting,
429
00:41:32.755 --> 00:41:42.110
meaning salespeople finding customers that will be open to purchasing the product. Prospecting and recruiting tool, it's, like, more powerful than LinkedIn sales navigator.
430
00:41:42.410 --> 00:41:52.270
It's crazy. You know, SQL is so powerful. And natural language on top of SQL, doubly so. So we've already seen every single ops role use this tool
431
00:41:52.695 --> 00:41:53.195
effectively
432
00:41:53.895 --> 00:41:54.395
across
433
00:41:54.855 --> 00:41:55.355
recruiting,
434
00:41:55.735 --> 00:41:56.235
across
435
00:41:56.775 --> 00:41:57.275
HR,
436
00:41:57.575 --> 00:41:58.075
across
437
00:41:58.615 --> 00:41:59.115
FinOps,
438
00:41:59.655 --> 00:42:04.955
RevOps, sales ops. There's lots of ops these days. Product ops queries,
439
00:42:05.500 --> 00:42:08.320
doing these sort of things on our own data platform,
440
00:42:08.700 --> 00:42:13.760
in fact. So the breadth of use cases has been pretty awesome.
441
00:42:14.060 --> 00:42:14.560
And,
442
00:42:14.860 --> 00:42:16.460
yeah, you know, it's been great. I
443
00:42:17.180 --> 00:42:39.950
the a lot of our early product market fit are actually investment firms. So and they use it for interesting stuff. We thought they would use it for kind of trying to find new companies to invest in, but they have a sales pipeline just like a but their sales are investing in something. So they kind of know what stage they're looking at their company in. They kind of have a pretty formalized pipeline. And the
444
00:42:40.350 --> 00:42:53.795
generally, there's one investment ops person who kind of manages that, and they have to field requests from the partners, which is often very time sensitive and stressful. But they've actually gotten their partners, the people who run the firms, to use this tool directly
445
00:42:54.174 --> 00:43:08.259
and which has both been efficient, but also an incredible stress reducer, which is really literally why on our marketing site, we can have a pull quote that says, quote, unquote, Compass saved my life, which is always something you wanna hear as a founder.
446
00:43:08.640 --> 00:43:11.700
But the reason why that, person said that
447
00:43:12.000 --> 00:43:23.785
is because we not just saved her time, but enormous amounts of stress dealing with kind of time sensitive time sensitive requests from very important people. So I think this investor use case has been pretty interesting to see.
448
00:43:24.325 --> 00:43:26.825
In your work of building the system
449
00:43:27.205 --> 00:43:27.705
and
450
00:43:28.085 --> 00:43:28.585
understanding
451
00:43:28.965 --> 00:43:29.465
the
452
00:43:29.820 --> 00:43:31.840
capabilities and use cases and limitations
453
00:43:32.220 --> 00:43:34.080
of an agentic analytics
454
00:43:34.460 --> 00:43:36.480
platform and how to tie it into
455
00:43:36.940 --> 00:43:43.280
existing data infrastructure and data assets, what are some of the most interesting or unexpected or challenging lessons that you learned in the process?
456
00:43:44.394 --> 00:43:47.694
I mean, it's still early days. It's amazing how
457
00:43:47.994 --> 00:43:50.255
once you once you go from
458
00:43:50.555 --> 00:44:03.010
one person on the go to market team being able to interact with the data warehouse to 80% of your team being able to interact with the data warehouse, you really start to see how many gaps there are both in understanding
459
00:44:03.630 --> 00:44:10.210
and your data model, but also, like, these gaps and understanding of, like, what people actually care about. So
460
00:44:10.510 --> 00:44:12.130
I think that has been
461
00:44:12.555 --> 00:44:14.895
super interesting to have rollout
462
00:44:15.355 --> 00:44:16.494
in real time.
463
00:44:17.355 --> 00:44:23.295
And what are the situations where you would advise against going down the agentic path for
464
00:44:23.595 --> 00:44:24.095
these
465
00:44:24.474 --> 00:44:24.974
exploratory
466
00:44:25.515 --> 00:44:27.454
or analytical use cases?
467
00:44:28.770 --> 00:44:38.230
Yeah. So the you know, we don't call it a BI tool. We call it exploratory data analysis because it's actually a very distinct use case. The for
468
00:44:38.530 --> 00:44:39.990
BI tools often
469
00:44:40.370 --> 00:44:40.870
drive
470
00:44:41.250 --> 00:44:41.750
absolutely
471
00:44:42.050 --> 00:44:42.950
mission critical
472
00:44:43.275 --> 00:44:43.775
things,
473
00:44:44.635 --> 00:44:46.654
like revenue reporting that is subject
474
00:44:47.275 --> 00:44:48.575
to regulatory scrutiny,
475
00:44:49.035 --> 00:44:50.654
or comp decisions,
476
00:44:51.115 --> 00:44:51.615
or
477
00:44:51.994 -->
00:44:52.494
pricing
478
00:44:52.795 --> 00:44:53.295
decisions.
479
00:44:54.075 --> 00:44:55.295
And Compass is explicitly
480
00:44:55.674 --> 00:44:59.310
not designed for that use case. It is for exploratory,
481
00:45:00.089 --> 00:45:07.069
rapidly rapid directionally correct data analysis, which is a very different use case. So we don't purport no desire to be a replacement
482
00:45:07.450 --> 00:45:19.825
for those those core BI assets. We think that should be managed by the BI tools. It's Kind of one of our principles here is that we want to designers call this truth in materials. We don't want to pretend like it's not an LOM. We don't want to pretend
483
00:45:20.204 --> 00:45:40.805
that it's a 100% accurate or bulletproof. It's not its purpose. Right? We want it to be rapidly correct or directionally correct and eventually correct. And by eventual correctness, I mean that the context store gets added to, and then the the queries get more and more accurate over time to some kind of asymptotic level. So, you know, there are domains where
484
00:45:41.345 --> 00:45:45.105
absolute precision in all cases is absolutely required. That is not,
485
00:45:45.585 --> 00:45:47.045
it is for facilitating,
486
00:45:47.345 --> 00:45:49.685
as I said, directionally correct rapid analyses.
487
00:45:50.465 --> 00:45:52.405
And as you continue to
488
00:45:52.839 --> 00:45:55.819
invest in and iterate on this agentic
489
00:45:56.359 --> 00:45:56.859
exploratory
490
00:45:57.160 --> 00:46:05.660
analytics use case, what are some of the things you have planned for the near to medium term or any particular projects or problem areas or capabilities that you're excited to explore?
491
00:46:06.654 --> 00:46:12.755
Yeah. So one thing I'm super interested in, I think for obvious reasons, is deep integration
492
00:46:13.214 --> 00:46:16.355
between Compass and Dagster plus and Dagster.
493
00:46:16.734 --> 00:46:23.700
You know? And this comes in many, many different forms, both using data pipelines to produce context and manage context,
494
00:46:24.079 --> 00:46:28.660
integrating the context store with our operational system of record, and then also
495
00:46:28.960 --> 00:46:32.900
using this tool. You know, we have this ability to create data requests,
496
00:46:33.365 --> 00:46:39.065
which can be, like, very detailed, and then using that as a basis of AI agentic authoring workflows,
497
00:46:39.365 --> 00:46:44.825
which we actually have kind of working already and is very, very effective. So I'm very excited
498
00:46:45.780 --> 00:46:57.960
for that dimension, kind of integrating Compass even more first class into data platforms. I'm very excited to work on our at a, a more enterprise SKU of Compass. I think these kind of organizational observability
499
00:46:58.260 --> 00:46:58.760
features
500
00:46:59.305 --> 00:47:19.920
will be part of that, as well as sort of on prem versions, which will have its own challenges, but will really unlock usage in a ton of places that will deliver a ton of value and we feel will be very successful in terms of being a healthy business. Yeah. And then just kind of you know, this the way this is set up, you know, we can attack all sorts of interesting use cases
501
00:47:20.300 --> 00:47:24.320
one by one by one. You know, we view just in the initial stages,
502
00:47:24.860 --> 00:47:26.080
right, every dashboard
503
00:47:26.540 --> 00:47:27.280
in every
504
00:47:27.825 --> 00:47:30.964
vertical SaaS app is our opportunity in effect.
505
00:47:31.265 --> 00:47:38.565
And that's very exciting to see. And I'm excited to so much of the information and knowledge work that happens still,
506
00:47:39.585 --> 00:47:41.045
it's so much drudgery.
507
00:47:41.585 --> 00:47:42.085
Manually
508
00:47:42.780 --> 00:47:55.360
fielding a request to add such and such to this Salesforce dashboard and then, you know, hooking this and that up. And I think people are a little too pessimistic about, like, AI taking all of our jobs.
509
00:47:55.725 --> 00:48:13.780
I don't think that will happen. I think people will move up the stack and have to deal with much less drudgery. And that's kind of the way I approach this and what I seek to do as participating and helping with this product. I think the future is bright. You know, it it kind of always comes up. I'm maybe anticipating a question you might ask, but,
510
00:48:14.100 --> 00:48:21.240
should my kids study software engineering? Is software engineering gonna have a future? And blah blah blah. And I couldn't be more bullish
511
00:48:21.655 --> 00:48:30.235
about the future of software engineering. It's just gonna change the definition of what software engineering is. But the the core foundations of learning how computation
512
00:48:30.695 --> 00:48:31.915
works, learning
513
00:48:32.215 --> 00:48:35.835
how to think about this stuff from first principles will only become more leveraged.
514
00:48:36.570 --> 00:48:43.470
Are there any other aspects of this space of agentic analytics, the work that you're doing on Compass, the
515
00:48:43.850 --> 00:48:44.350
leveraging
516
00:48:44.730 --> 00:48:45.470
of existing
517
00:48:46.330 --> 00:48:50.110
data infrastructure and data assets into this more
518
00:48:50.410 --> 00:48:51.150
AI driven
519
00:48:51.675 --> 00:48:55.615
interaction pattern that we didn't discuss yet that you'd like to cover before we close out the show?
520
00:48:56.155 --> 00:49:01.215
No. I think we've we've done a pretty good we've covered a lot of ground, so I think we'll leave it here.
521
00:49:01.515 --> 00:49:15.660
Alright. Well, for anybody who wants to get in touch with you and follow along along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest gap of the tooling or technology for data management today.
522
00:49:16.040 -->
00:49:16.859
It's always
523
00:49:17.365 --> 00:49:20.985
extremely unfair when you ask vendors this because we're morally obligated
524
00:49:21.685 --> 00:49:25.945
to talk our own book. But I am super interested
525
00:49:26.485 --> 00:49:30.025
in I'm, like, obsessed with this context engineering
526
00:49:30.325 --> 00:49:42.020
notion. And I think it's, like, gonna be a defining discipline for the next ten years. I think it's super, super early days. I actually think about a lot because my other kind of passion project right now is figuring out how to deploy
527
00:49:42.400 --> 00:49:49.835
AI and agentic authoring in real and large software systems. And I am very interested in the problem of keeping
528
00:49:50.235 --> 00:49:52.175
this sounds simple, but I think it's a big problem.
529
00:49:52.715 --> 00:49:55.215
Keeping markdown files checked into a project
530
00:49:55.515 --> 00:50:17.405
up to date with the underlying code. Because I think this is a big problem. Because I think of these markdown files, generally, that are computed by agents, they're just to me, they're just token caches. Right? That LLM has, like, evaluated a bunch of tokens in the code base and then materialized that knowledge in more condensed form. Right? And I think that's actually gonna happen recursively in large software projects. But keeping it up to date, it's actually
531
00:50:17.705 --> 00:50:24.365
another instance of a data pipelining problem because you can't recompute it every time because it ends up being too expensive. So how can you do that intelligently
532
00:50:24.745 --> 00:50:30.359
and keep it up to date? I think it's just one pillar of what is gonna be needed to do AI accelerated
533
00:50:30.740 --> 00:50:36.839
software engineering at scale. That's my term, by the way. I despise the term vibe coding and hope we don't talk about it here.
534
00:50:37.155 --> 00:50:42.055
Yeah. An alternative term that I heard recently is AI native engineering.
535
00:50:42.675 --> 00:50:52.615
That's pretty good. I will take it. I will take it. Agentic engineering is pretty good too, but I don't know. Agentic is like one of these words now, which I, like, only use as a last resort.
536
00:50:54.140 --> 00:50:54.640
Absolutely.
537
00:50:54.940 --> 00:51:08.855
Well, thank you very much for taking the time today to join me and share the work that you've been doing on your agentic analytics system and, the experiences that you've learned there. So I appreciate that, and I hope you enjoy the rest of your day. Alright.
538
00:51:09.155 --> 00:51:11.095
Thanks, Tobias. Thanks for having me.
539
00:51:18.195 --> 00:51:22.490
Thank you for listening, and don't forget to check out our other shows. Podcast.net
540
00:51:22.490 --> 00:51:31.470
covers the Python language, its community, and the innovative ways it is being used, and the AI Engineering Podcast is your guide to the fast moving world of building AI systems.
541
00:51:32.170 --> 00:51:39.425
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a project from the show, then tell us about
542
00:51:39.725 --> 00:51:42.365
it. Email hosts@dataengineeringpodcast.com
543
00:51:42.365 --> 00:51:48.545
with your story. Just to help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers.