TRANSKRIBEERI 🔗
WEBVTT
00:00:00.020 --> 00:00:04.179
Dashboards are a hack for the human visual cortex.
00:00:04.500 --> 00:00:08.480
That's how my guest today puts it. They pipe
00:00:08.480 --> 00:00:12.779
a lot of information into our brains fast so
00:00:12.779 --> 00:00:16.199
we can spot the patterns. But more and more,
00:00:16.359 --> 00:00:19.760
the thing reading your company's data isn't a
00:00:19.760 --> 00:00:24.199
person. It's an AI agent. And agents are already
00:00:24.199 --> 00:00:27.899
great at writing SQL. It compiles and it runs.
00:00:28.370 --> 00:00:31.969
almost every time. The hard part is whether the
00:00:31.969 --> 00:00:35.369
agent understood the question and the business
00:00:35.369 --> 00:00:39.429
behind it. In the benchmarks he's seen, an agent
00:00:39.429 --> 00:00:42.869
turned loose on raw enterprise data gets about
00:00:42.869 --> 00:00:46.729
30% of its answers semantically right. Give
00:00:46.729 --> 00:00:50.289
it context and a semantic layer, and it's north
00:00:50.289 --> 00:00:53.829
of 90. I'm Brian Teller from Teller's Tech, and
00:00:53.829 --> 00:00:55.800
this is Ship It Weekly.
00:01:12.300 --> 00:01:13.579
Welcome back to Ship
00:01:13.579 --> 00:01:16.769
It Weekly where I filter the noise and focus
00:01:16.769 --> 00:01:19.109
on what matters when you are the one running
00:01:19.109 --> 00:01:22.890
infrastructure and owning reliability. Most weeks,
00:01:22.930 --> 00:01:27.590
it's a quick DevOps, SRE, platform, cloud, and
00:01:27.590 --> 00:01:31.250
security news recap. In between those, I do conversation
00:01:31.250 --> 00:01:34.549
episodes with people building and operating these
00:01:34.549 --> 00:01:38.090
systems. Ship It Weekly turns one this Friday.
00:01:38.370 --> 00:01:41.310
To celebrate through mid-December, you are getting
00:01:41.310 --> 00:01:44.730
two conversations a week. Mondays and Wednesdays,
00:01:44.730 --> 00:01:48.129
with the news every Friday. Then, we're back
00:01:48.129 --> 00:01:51.790
to our usual rhythm. Today, I'm joined by Tristan
00:01:51.790 --> 00:01:56.109
Handy, co-founder of dbt Labs, and now president
00:01:56.109 --> 00:01:59.750
of Fivetran and dbt. If you've worked anywhere
00:01:59.750 --> 00:02:03.750
near a data team, you've probably run into dbt.
00:02:03.930 --> 00:02:07.090
His team likes to call it Terraform for data.
00:02:07.290 --> 00:02:10.729
We talk about what changes when AI agents become
00:02:10.729 --> 00:02:14.430
the main consumers of your data, why the hard
00:02:14.430 --> 00:02:18.169
part isn't text to SQL, what a context layer
00:02:18.169 --> 00:02:22.550
is, and why you need one, and why he thinks the
00:02:22.550 --> 00:02:25.849
most interesting work is operating on the system
00:02:25.849 --> 00:02:30.110
instead of inside it. He also gets into what
00:02:30.110 --> 00:02:34.449
dbt got wrong selling to enterprises, and the
00:02:34.449 --> 00:02:39.069
agent loop his team built to ship dbt 2. All right,
00:02:39.150 --> 00:02:40.397
let's jump in.
00:02:45.045 --> 00:02:48.550
Tristan, thank you for joining me. Thanks for having me. So starting off, I'm
00:02:48.550 --> 00:02:51.229
curious. We've spent years building data platforms
00:02:51.229 --> 00:02:54.689
around people looking at dashboards. What changes
00:02:54.689 --> 00:02:57.270
when an AI agent becomes the thing consuming
00:02:57.270 --> 00:03:00.669
that data? Oh, gosh. Yeah, we're just diving
00:03:00.669 --> 00:03:04.780
right in, aren't we? Maybe the first and most
00:03:04.780 --> 00:03:09.240
kind of obvious thing is that dashboards aren't
00:03:09.240 --> 00:03:14.729
the product anymore. Dashboards are a hack for
00:03:14.729 --> 00:03:18.490
the human visual cortex. They are a way to pipe
00:03:18.490 --> 00:03:20.669
lots of information into our brains very quickly
00:03:20.669 --> 00:03:25.909
and allow us to notice patterns. And that makes
00:03:25.909 --> 00:03:29.909
a ton of sense when our brains are the ones that
00:03:29.909 --> 00:03:31.770
are looking for the patterns and applying the
00:03:31.770 --> 00:03:37.310
intelligence. But it is becoming true that agents
00:03:37.310 --> 00:03:42.439
are replacing us as the primary form of intelligence
00:03:42.439 --> 00:03:45.199
that is looking at our data and trying to find
00:03:45.199 --> 00:03:47.539
patterns and the meaning of those patterns. And
00:03:47.539 --> 00:03:53.740
they don't really want to consume data via images.
00:03:54.060 --> 00:03:58.719
And so dashboards go away. Dashboards will stay,
00:03:58.780 --> 00:04:01.580
I think, as a translation layer so that if agents
00:04:01.580 --> 00:04:03.939
notice something interesting, the next step will
00:04:03.939 --> 00:04:07.580
be, help explain to me what that thing is that
00:04:07.580 --> 00:04:10.939
you noticed. I think often a dashboard will be
00:04:10.939 --> 00:04:14.840
a component of that answer. But so the asset
00:04:14.840 --> 00:04:21.660
shifts lower in the stack and agents will consume
00:04:21.660 --> 00:04:26.139
some combination of your gold layer, which is
00:04:26.139 --> 00:04:30.160
essentially like well-modeled data in tables
00:04:30.160 --> 00:04:33.500
in your data lake or data warehouse or whatever.
00:04:34.360 --> 00:04:39.379
There will be a semantic layer which will actually
00:04:39.379 --> 00:04:42.660
capture, here is how our organization calculates
00:04:42.660 --> 00:04:45.660
metrics. We can talk about why that's important,
00:04:45.980 --> 00:04:50.240
but the organizational design of your metrics
00:04:50.240 --> 00:04:52.699
is not primarily a technical artifact. It's primarily
00:04:52.699 --> 00:04:55.420
a decision-making artifact. It's a reflection
00:04:55.420 --> 00:04:57.699
of how your organization wants to see itself.
00:04:58.120 --> 00:05:02.680
And then a context store, so that your agent
00:05:02.680 --> 00:05:07.199
has the ability to understand how to navigate
00:05:07.199 --> 00:05:10.000
all the
Load full transcript
data that it has available to it and
00:05:10.000 --> 00:05:12.480
make meaning out of it. So I think that those
00:05:12.480 --> 00:05:16.579
are the primary three places that agents will
00:05:16.579 --> 00:05:19.310
interface with your data stack. And when you
00:05:19.310 --> 00:05:22.470
say gold, you're talking like a medallion
00:05:22.560 --> 00:05:25.339
setup? Okay. Just so our listeners that may not
00:05:25.339 --> 00:05:27.939
understand the gold, bronze, silver, yeah. Totally.
00:05:31.139 --> 00:05:35.279
I understand that the world of the data practitioner
00:05:35.279 --> 00:05:37.060
is a little bit different than the world of the
00:05:37.060 --> 00:05:41.759
SRE or DevOps practitioner. So I'm happy to go
00:05:41.759 --> 00:05:46.360
as deep or not deep as relevant. But yeah, assume
00:05:46.360 --> 00:05:49.800
that all of your data comes into your data lake
00:05:49.800 --> 00:05:55.180
in a pretty raw form. And oftentimes, it actually
00:05:55.180 --> 00:05:59.889
takes many stacked... functions, transformations
00:05:59.889 --> 00:06:02.949
in a DAG in order to actually get to a place
00:06:02.949 --> 00:06:06.569
where it represents some meaningful unit of business
00:06:06.569 --> 00:06:09.269
activity. And that's typically what you call
00:06:09.269 --> 00:06:13.029
your gold layer. And for this data, are we looking
00:06:13.029 --> 00:06:15.910
at this differently for an agent than we would
00:06:15.910 --> 00:06:18.089
a person like a practitioner that's actually
00:06:18.089 --> 00:06:20.610
analyzing this data? Are agents looking at it
00:06:20.610 --> 00:06:23.069
the same way? Do we have to reason about it differently?
00:06:23.209 --> 00:06:31.230
Do we have to structure it differently? Yeah, okay, that's a great question. I, I think that the core
00:06:31.230 --> 00:06:36.670
way that you structure data can be very similar.
00:06:37.050 --> 00:06:40.730
It's, you know, whenever it was the 90s or something
00:06:40.730 --> 00:06:45.230
when Ralph Kimball wrote the first book on, you
00:06:45.230 --> 00:06:48.550
know, how do you structure relational data for
00:06:48.550 --> 00:06:54.540
analytics. That approach... And there's several
00:06:54.540 --> 00:06:56.480
competing approaches, but whatever. The approach
00:06:56.480 --> 00:06:57.819
that you've taken to data modeling historically,
00:06:58.000 --> 00:07:04.699
I think, can persist pretty well. From an agent's
00:07:04.699 --> 00:07:10.040
perspective, really, they don't really care that
00:07:10.040 --> 00:07:12.199
much about exactly how you structure your data
00:07:12.199 --> 00:07:16.759
as long as they can understand how you have structured
00:07:16.759 --> 00:07:20.819
it. The constraints on how you structure your...
00:07:21.110 --> 00:07:23.810
data are much more imposed by the query engine.
00:07:24.310 --> 00:07:28.750
So everybody's really very focused these days
00:07:28.750 --> 00:07:35.550
on optimizing your spend on tokens. In my world,
00:07:35.569 --> 00:07:41.310
for several years, the runaway cost was the compute
00:07:41.310 --> 00:07:45.689
on top of your data lake. A very blunt instrument
00:07:45.689 --> 00:07:49.610
and scans can be very expensive. So your data
00:07:49.610 --> 00:07:51.860
modeling is primarily done to make sure that
00:07:51.860 --> 00:07:53.779
you're not spending an arm and a leg in your
00:07:53.779 --> 00:07:57.399
data lake compute costs. Makes sense. Okay,
00:07:57.480 --> 00:08:00.800
so in SRE, we can tell you whether a request
00:08:00.800 --> 00:08:03.980
succeeded, but an agent can get a successful
00:08:03.980 --> 00:08:07.360
response and still make the wrong decision. What
00:08:07.360 --> 00:08:10.459
does reliability look like at the data layer?
00:08:10.860 --> 00:08:14.740
Wait, is your question about reliability in the
00:08:14.740 --> 00:08:18.180
context of agents, or is it just data generally?
00:08:18.639 --> 00:08:21.519
on the agent side specifically, if you can answer
00:08:21.519 --> 00:08:24.959
that. Yeah, because it's a somewhat hard question
00:08:24.959 --> 00:08:28.360
to answer when humans are in the loop too. That's
00:08:28.360 --> 00:08:34.759
fair. But there are benchmarks when it comes
00:08:34.759 --> 00:08:38.639
to evaluating agents and their accuracy for data
00:08:38.639 --> 00:08:41.919
-related tasks. I think that the one that comes
00:08:41.919 --> 00:08:44.940
up most often is called Spider. I think there's
00:08:44.940 --> 00:08:51.259
one, Spider 2. Benchmarks like this attempt to model
00:08:51.259 --> 00:08:53.919
the complexity of real world enterprise data,
00:08:54.139 --> 00:08:58.799
which has, it's very messy. It has a tremendous
00:08:58.799 --> 00:09:02.960
number of different entities modeled in it. Join
00:09:02.960 --> 00:09:04.779
pathways are not clear, et cetera, et cetera.
00:09:05.259 --> 00:09:14.549
And you can then ask questions of this data set.
00:09:15.230 --> 00:09:17.669
and use an agent to go get you the answers to
00:09:17.669 --> 00:09:22.169
these questions. And you can experiment with
00:09:22.169 --> 00:09:26.519
how you ask the questions, what tools you give
00:09:26.519 --> 00:09:29.279
the agents, what kind of context you give them,
00:09:29.379 --> 00:09:34.580
and measure the difference in the results on
00:09:34.580 --> 00:09:38.679
the benchmark. There are pretty stark differences
00:09:38.679 --> 00:09:46.809
in the performance you will see. If I could summarize
00:09:46.809 --> 00:09:52.210
the kind of general industry wisdom on these
00:09:52.210 --> 00:09:56.590
evals today, it is that agents are very good
00:09:56.590 --> 00:10:00.809
at writing SQL. They can write SQL that compiles
00:10:00.809 --> 00:10:02.610
and executes against your database with almost
00:10:02.610 --> 00:10:06.490
100% accuracy. The problem today is not text
00:10:06.490 --> 00:10:12.990
to SQL. The challenge today is, did the agent...
00:10:13.230 --> 00:10:15.049
fully understand the
question that you are asking?
00:10:15.190 --> 00:10:19.169
Do they understand the context of the data? And
00:10:19.169 --> 00:10:22.190
do they actually get you the semantically correct
00:10:22.190 --> 00:10:27.090
answer to your question? And that one is much,
00:10:27.169 --> 00:10:34.000
much harder. The most recent benchmarks that
00:10:34.000 --> 00:10:38.019
I've seen on this are that if you just turn an
00:10:38.019 --> 00:10:42.169
agent loose on enterprise data with the Spider 2
00:10:42.169 --> 00:10:45.799
benchmark, you're at something like 30% accuracy
00:10:45.799 --> 00:10:52.860
for this semantic correctness. But if you supply
00:10:52.860 --> 00:10:59.059
it with context and a semantic layer, then you
00:10:59.059 --> 00:11:02.080
will be able to get that accuracy north of 90%.
00:11:02.080 --> 00:11:09.340
So really, the world of data today is all about
00:11:09.340 --> 00:11:13.720
how do we construct systems that take the knowledge
00:11:13.720 --> 00:11:17.419
that humans either have in their brains or know
00:11:17.419 --> 00:11:19.659
how to get access to via whatever mechanisms
00:11:19.659 --> 00:11:24.000
and put that at the hands of agents so that they
00:11:24.000 --> 00:11:27.379
can come up with correct, trustworthy answers.
00:11:27.919 --> 00:11:31.460
Yeah, no, it makes 100% sense. I guess kind
00:11:31.460 --> 00:11:33.960
of pulling on that thread, now talking about
00:11:33.960 --> 00:11:36.179
like the governance side of it, if an agent can
00:11:36.179 --> 00:11:38.879
discover data and act on it, where do we put
00:11:38.879 --> 00:11:41.950
the boundary between useful autonomy? and access
00:11:41.950 --> 00:11:44.110
that you'll end up regretting. Say more about
00:11:44.110 --> 00:11:47.490
that. Where's the regret coming from? Well, I'm
00:11:47.490 --> 00:11:50.009
more thinking the difference between reading
00:11:50.009 --> 00:11:52.529
data and taking action. And that taking action
00:11:52.529 --> 00:11:58.529
step can obviously devolve into some crazy modifications
00:11:58.529 --> 00:12:03.809
of your application, your infrastructure. So
00:12:03.809 --> 00:12:08.659
where do we draw that line? I think that it is
00:12:08.659 --> 00:12:12.389
very interesting to hear you ask that. And the
00:12:12.389 --> 00:12:16.710
reason that I scratched my head at first is that
00:12:16.710 --> 00:12:20.889
I don't think that's generally a question that
00:12:20.889 --> 00:12:23.789
data practitioners are wrestling with yet. I
00:12:23.789 --> 00:12:26.049
think it's one where they will absolutely have
00:12:26.049 --> 00:12:32.649
to, but the kind of organizational history of
00:12:32.649 --> 00:12:38.750
data is that it is used not as a part of the...
00:12:39.159 --> 00:12:41.779
operational flow of the business, but as a part
00:12:41.779 --> 00:12:44.500
of the ability of the business to introspect
00:12:44.500 --> 00:12:46.820
itself. And so the result is always prepared
00:12:46.820 --> 00:12:51.940
for some human brain to look at and potentially
00:12:51.940 --> 00:12:56.580
take action on. But I completely believe that...
00:12:56.580 --> 00:13:01.379
So what data people and data technologists have
00:13:01.379 --> 00:13:05.080
been doing for 30 years now is essentially building
00:13:05.080 --> 00:13:08.549
these highly refined nervous systems whereby
00:13:08.549 --> 00:13:13.070
data is collected and refined and eventually
00:13:13.070 --> 00:13:16.950
presented. And I think that those nervous systems
00:13:16.950 --> 00:13:21.580
are going to be quickly hooked up to action taking
00:13:21.580 --> 00:13:24.360
and agents are what's going to do that. And so
00:13:24.360 --> 00:13:26.879
we will quickly have to wrestle with the question
00:13:26.879 --> 00:13:30.580
of like, okay, how do we assess the seriousness
00:13:30.580 --> 00:13:33.600
of a bad outcome here? And how do we put the
00:13:33.600 --> 00:13:35.059
right checks in place and all of that stuff?
00:13:35.120 --> 00:13:39.720
But that's largely not where the industry is
00:13:39.720 --> 00:13:44.159
at yet. What about even on the PII side of that
00:13:44.159 --> 00:13:48.639
data or incorrect assumptions from the data that
00:13:48.639 --> 00:13:51.379
the agent may be making? Is that a concern from
00:13:51.379 --> 00:13:57.000
the data pipeline side? So I'm not a security
00:13:57.000 --> 00:14:01.080
expert, but what I would suggest is that there
00:14:01.080 --> 00:14:05.100
was, in the very early days of the... AI data
00:14:05.100 --> 00:14:07.320
conversation. There was a lot of concern with,
00:14:07.419 --> 00:14:11.840
well, can I let my model, the models have access
00:14:11.840 --> 00:14:17.220
to customers' PII? And I think that we've largely
00:14:17.220 --> 00:14:20.980
moved past that conversation because it's such
00:14:20.980 --> 00:14:23.799
an obviously important commercial thing. And
00:14:23.799 --> 00:14:28.940
so at this point, from my perspective, letting
00:14:28.940 --> 00:14:34.750
AWS store customer's data in S3 versus letting
00:14:34.750 --> 00:14:37.509
them process it using bedrock. It's like kind
00:14:37.509 --> 00:14:39.809
of all the same. AWS has a mechanism for doing
00:14:39.809 --> 00:14:42.210
that. We already trust our cloud providers, et
00:14:42.210 --> 00:14:45.429
cetera. So I don't think that's a big thing absent
00:14:45.429 --> 00:14:51.740
the fact that certainly it is possible to design
00:14:51.740 --> 00:14:54.759
agents in a, you know, we're all too aware of
00:14:54.759 --> 00:14:57.820
this now in the post-hugging face world. It's
00:14:57.820 --> 00:15:01.669
possible to design agents that can do inappropriate
00:15:01.669 --> 00:15:04.070
things with data. And so you just have to make
00:15:04.070 --> 00:15:06.990
sure you're staying really far away from anything
00:15:06.990 --> 00:15:10.269
like that. I think that was half of your question.
00:15:10.350 --> 00:15:13.289
What was the other half? Oh, incorrect results
00:15:13.289 --> 00:15:16.929
of decisions. Okay, this is an interesting one
00:15:16.929 --> 00:15:19.570
because different organizations think about this
00:15:19.570 --> 00:15:25.330
differently. There are organizations today where,
00:15:25.659 --> 00:15:31.490
I don't know, I might call it run and gun. Where
00:15:31.490 --> 00:15:35.320
it's just like, this is the answer and I'm just
00:15:35.320 --> 00:15:37.820
going to by default trust it. And we generally
00:15:37.820 --> 00:15:40.440
have pretty high accuracy rates. So like, let's
00:15:40.440 --> 00:15:42.919
go. And you know, maybe you'll get some decisions
00:15:42.919 --> 00:15:48.220
wrong in that modality, but I don't know. Maybe
00:15:48.220 --> 00:15:52.279
it's appropriate if velocity is the thing that
00:15:52.279 --> 00:15:54.120
you're optimizing for. Because you certainly
00:15:54.120 --> 00:16:00.429
can move real damn fast that way. In an enterprise
00:16:00.429 --> 00:16:04.990
context, typically you move more slowly and that
00:16:04.990 --> 00:16:08.889
final check is done in the same way that when
00:16:08.950 --> 00:16:11.690
you build a software factory today, you have
00:16:11.690 --> 00:16:14.129
this decision to make who merges the pull requests.
00:16:14.389 --> 00:16:16.909
And some people say the human does and some people
00:16:16.909 --> 00:16:18.850
say that the agent does. And so you get to make
00:16:18.850 --> 00:16:22.210
that decision. Okay, so let's move on to analytics
00:16:22.210 --> 00:16:25.470
engineering. So dbt helped make data transformation
00:16:25.470 --> 00:16:28.500
look more like software engineering. As agents
00:16:28.500 --> 00:16:31.960
write more SQL and consume more data, how does
00:16:31.960 --> 00:16:36.220
the analytics engineer's job change? My general
00:16:36.379 --> 00:16:43.879
guidance here is really not about analytics engineers
00:16:43.879 --> 00:16:48.980
or SREs or software engineers. It is all of these
00:16:48.980 --> 00:16:57.250
very technology knowledge work fields are going
00:16:57.250 --> 00:17:02.129
to be systematized in a way that they haven't
00:17:02.129 --> 00:17:10.009
been before. And you as a professional will either
00:17:10.009 --> 00:17:14.829
be operating inside the system or you'll be operating
00:17:14.829 --> 00:17:20.660
on the system. And from my perspective, the more
00:17:20.660 --> 00:17:24.359
interesting work to do is to be operating on
00:17:24.359 --> 00:17:27.930
the system, to be designing the loops the evals,
00:17:27.930 --> 00:17:31.609
et cetera, as opposed to having tickets assigned
00:17:31.609 --> 00:17:38.250
to you. I think that this is true of all of the
00:17:38.250 --> 00:17:40.849
different types of jobs that I just mentioned
00:17:40.849 --> 00:17:43.630
and many more. And so for analytics engineers,
00:17:43.829 --> 00:17:49.230
that really looks like not, how do I really cleverly
00:17:49.230 --> 00:17:52.990
model this data to solve this kind of thorny
00:17:52.990 --> 00:18:00.109
quantitative problem? It turns out that, you know, Astra 6 is going
00:18:00.109 --> 00:18:02.750
to be able to solve that problem as well as you
00:18:02.750 --> 00:18:07.430
can, if not better. And instead, you should be
00:18:07.430 --> 00:18:11.049
thinking about the system holistically and saying,
00:18:11.150 --> 00:18:14.930
what evals do I have? What evals do I need? How
00:18:14.930 --> 00:18:20.230
do I see the things that are unseen, like the
00:18:20.230 --> 00:18:22.789
latent business needs in the organization that
00:18:22.789 --> 00:18:24.650
are not being reflected in the loop that we have
00:18:24.650 --> 00:18:29.740
today, etc. And that type of work is going to
00:18:29.740 --> 00:18:34.180
be important for a long time to come. Yeah, so
00:18:34.180 --> 00:18:37.400
it's moving up a layer, essentially. 100%. One
00:18:37.400 --> 00:18:41.019
of our values that I now authored 10 years ago
00:18:41.019 --> 00:18:43.859
is we believe in moving up the stack. And I think
00:18:43.859 --> 00:18:46.220
this is a big moving up the stack moment. We've
00:18:46.220 --> 00:18:49.980
all been very identified with our own set of
00:18:49.980 --> 00:18:52.460
technical skills because we spent a long time
00:18:52.460 --> 00:18:57.140
building those technical skills. And it turns out that the technical
00:18:57.140 --> 00:18:58.960
skills are not going to be the differentiators
00:18:58.960 --> 00:19:02.180
for us moving forwards. Yeah, I think you're
00:19:02.180 --> 00:19:05.730
right. When you were building dbt, what made
00:19:05.730 --> 00:19:08.349
practitioners trust it enough to put it into their
00:19:08.349 --> 00:19:11.849
production workflow? How did you make that happen?
00:19:11.950 --> 00:19:14.750
How did you build that trust with the community?
00:19:15.130 --> 00:19:20.900
That's a good question. I think that there were
00:19:20.900 --> 00:19:26.420
a couple different factors. One is that it is,
00:19:27.019 --> 00:19:30.140
dbt is a lot like Terraform. We've often thought
00:19:30.140 --> 00:19:33.480
about it as Terraform for data. And Terraform
00:19:33.480 --> 00:19:37.140
is not really in the production request response
00:19:37.140 --> 00:19:42.079
flow. And so while it's certainly a critical
00:19:42.079 --> 00:19:45.480
piece of infrastructure, it doesn't have the
00:19:45.480 --> 00:19:49.740
requirements on it that something in the request
00:19:49.740 --> 00:19:59.400
response flow has. It had so much, it was very
00:19:59.400 --> 00:20:04.440
unique and did not have a ton of substitutes.
00:20:04.500 --> 00:20:08.519
And so our early adopters felt like they were
00:20:08.519 --> 00:20:12.579
willing to do the work to look at the code, engage
00:20:12.740 --> 00:20:17.420
with the community and say, you know, like literally
00:20:17.420 --> 00:20:19.559
our first, you know, reasonably sized company
00:20:19.559 --> 00:20:22.019
that started using dbt showed up in our Slack
00:20:22.019 --> 00:20:24.660
community and said, you guys don't have any tests.
00:20:25.470 --> 00:20:28.970
is this a real thing or is this just some side
00:20:28.970 --> 00:20:33.210
project? And, you know, with dbt at that point,
00:20:33.230 --> 00:20:36.049
it existed for like all of six weeks or something.
00:20:36.410 --> 00:20:39.410
And yeah, we didn't, we hadn't built a test suite
00:20:39.410 --> 00:20:42.150
yet. And so we were like, Oh no, no, no, no.
00:20:42.289 --> 00:20:44.789
If, if you'll use it, we'll, we promise to build
00:20:44.789 --> 00:20:47.670
tests. So like that kind of pressure over time
00:20:47.670 --> 00:20:51.349
eventually sharpens you. That's fair. Do you
00:20:51.349 --> 00:20:53.369
think that there's something that developer-first
00:20:53.369 --> 00:20:55.960
companies get wrong when they start selling
00:20:55.960 --> 00:20:59.339
to enterprises? Oh my God. That was a painful process.
00:20:59.339 --> 00:21:01.279
I'll answer a slightly different version of the
00:21:01.279 --> 00:21:03.559
question, which is, what, what do I think we got
00:21:03.559 --> 00:21:12.329
wrong? Um, I, I think that I was too willing to
00:21:12.329 --> 00:21:19.450
say yes to take the revenue. And, um, I think that
00:21:19.450 --> 00:21:26.039
in part that is because I didn't have some grand
00:21:26.039 --> 00:21:28.920
vision of what our commercial product was going
00:21:28.920 --> 00:21:33.420
to become. I had a pretty clear vision of the
00:21:33.420 --> 00:21:37.619
role for our open source product, but the commercial
00:21:37.619 --> 00:21:41.640
product was really just, let's see what things
00:21:41.640 --> 00:21:47.799
people are willing to pay for. And in the early
00:21:47.799 --> 00:21:52.319
days, we experienced some real Fortune 500 pull.
00:21:54.299 --> 00:21:57.079
We essentially, as we built up our engineering
00:21:57.079 --> 00:21:59.579
team, because, you know, back in 2020, you had
00:21:59.579 --> 00:22:02.819
to actually hire engineers to write code. And
00:22:02.819 --> 00:22:05.740
we built up our engineering team and we built
00:22:05.740 --> 00:22:08.559
enterprise features. And for like, I don't know,
00:22:08.619 --> 00:22:14.240
two years, we weren't able to make meaningful
00:22:14.240 --> 00:22:18.460
forward progress on the product vision because
00:22:18.460 --> 00:22:20.779
we were just so focused on building enterprise
00:22:20.779 --> 00:22:28.089
crap, enabling multi-cloud deployment, role
00:22:28.220 --> 00:22:31.119
-based access control, all of these different
00:22:31.119 --> 00:22:37.720
things that you have to build, but you want to
00:22:37.750 --> 00:22:40.210
be real mindful of when you sequence the work
00:22:40.210 --> 00:22:44.269
in. How did you decide what stayed open source
00:22:44.269 --> 00:22:47.349
versus became that commercial product? We ended
00:22:47.349 --> 00:22:51.890
up, after maybe two years, we kind of solidified
00:22:51.890 --> 00:22:56.039
an idea of
of that dividing line. The answer
00:22:56.039 --> 00:22:59.400
effectively became state and user interface.
00:23:00.099 --> 00:23:05.240
And so if this was a feature that required a
00:23:05.309 --> 00:23:09.970
database, then it was better for a cloud product.
00:23:10.430 --> 00:23:13.190
I mean, our installed product, it operated purely
00:23:13.190 --> 00:23:16.750
on top of files. It still does operate on top
00:23:16.750 --> 00:23:23.380
of files. There's no database persistence then
00:23:23.380 --> 00:23:26.819
if you wanted a GUI associated with it, and there
00:23:26.819 --> 00:23:29.420
ended up being a pretty meaningful group of users
00:23:29.420 --> 00:23:35.539
who, because of where data practitioners' careers
00:23:35.539 --> 00:23:38.839
kind of come from, the GUI was a useful bridge
00:23:38.839 --> 00:23:43.839
for them. Has the Fivetran combination changed
00:23:43.839 --> 00:23:46.000
how you think about practitioner choice at all?
00:23:46.279 --> 00:23:48.640
Like between open source and commercial? Yeah.
00:23:50.509 --> 00:23:55.430
I think that the... The merger that we have is
00:23:55.430 --> 00:23:58.890
a real journey getting through this. It's a year
00:23:58.890 --> 00:24:03.150
-long process. We closed the merger about five
00:24:03.150 --> 00:24:09.430
months ago at this point. There is a lot that
00:24:09.430 --> 00:24:13.410
it changes about our world. It doesn't change
00:24:13.410 --> 00:24:18.289
what we are delivering to our open source community.
00:24:18.630 --> 00:24:23.680
In fact, it enables us to... be more open because
00:24:23.680 --> 00:24:27.059
we have more options to monetize across a broader
00:24:27.059 --> 00:24:34.000
set of platform features. But I think that we
00:24:34.000 --> 00:24:42.349
are now very focused on innovating in a way that
00:24:42.349 --> 00:24:47.329
looks more like Lego bricks, as opposed to historically,
00:24:47.630 --> 00:24:53.599
we thought about really like a... And it's very similar to a Confluent or
00:24:53.599 --> 00:24:56.779
a Mongo or whatever. You think about like, here's
00:24:56.779 --> 00:24:59.660
our community offering and here's our enterprise
00:24:59.660 --> 00:25:02.259
offering, and you get to make a choice. Which one
00:25:02.259 --> 00:25:05.220
of those do you want? And the way that we are
00:25:05.220 --> 00:25:09.119
constructing the world now is that we have, as
00:25:09.119 --> 00:25:10.900
a combined company, we have multiple different
00:25:10.900 --> 00:25:13.900
open source offerings. dbt is our biggest open
00:25:13.900 --> 00:25:17.880
source offering. And then there are modules that
00:25:17.880 --> 00:25:20.619
you can adopt a la carte. They don't ask you to move
00:25:20.619 --> 00:25:22.859
into a different version of the software. They
00:25:22.859 --> 00:25:26.819
plug into, uh, the, what you're already using. And
00:25:26.819 --> 00:25:29.119
rather than adopting some kind of enterprise
00:25:29.119 --> 00:25:33.099
version,
you say like, I want, I want dbt state
00:25:33.099 --> 00:25:36.380
or I want lake compute or whatever, and you can
00:25:36.380 --> 00:25:38.940
uh, on a consumption basis, pay for that specific
00:25:38.940 --> 00:25:42.180
thing. Okay, yeah, that makes sense. I was, in my
00:25:42.180 --> 00:25:44.200
mind, I'm thinking about it, going back to that
00:25:44.200 --> 00:25:47.920
Terraform example, HashiCorp, IBM, you know, acquisition
00:25:47.920 --> 00:25:50.220
and how that kind of changed licensing, and yeah.
00:25:50.220 --> 00:25:56.079
So, yeah. Yeah, pretty different. Yeah, no, no, it's
00:25:56.079 --> 00:25:58.880
uh, we've, we've only become more permissive with
00:25:58.880 --> 00:26:03.450
licenses, uh, since, since the merger. And, um, we
00:26:03.450 --> 00:26:06.930
and Fivetran are super different than HashiCorp
00:26:06.930 --> 00:26:12.589
and IBM in that both of, both companies, uh, kind
00:26:12.589 --> 00:26:14.990
of grew up together and are directly complementary
00:26:15.779 --> 00:26:20.160
pieces of technology. And so they together form
00:26:20.160 --> 00:26:25.299
a very nicely packaged platform offering. Like
00:26:25.299 --> 00:26:28.019
symbiotic relationship more than, yeah, gotcha.
00:26:28.670 --> 00:26:33.130
Okay, so if I run a platform or data team and
00:26:33.130 --> 00:26:35.930
people are asking me to connect agents to our
00:26:35.930 --> 00:26:38.630
enterprise data, what is the first thing you'd
00:26:38.630 --> 00:26:42.569
have me do? Oh my goodness. Well, the first thing
00:26:42.569 --> 00:26:47.150
is actually, it's like eat your spinach. All
00:26:47.700 --> 00:26:50.319
of the things that were good ideas in the past
00:26:50.319 --> 00:26:56.359
are still good ideas. If you have a well-governed
00:26:56.359 --> 00:27:01.839
data lake and the data in it is well-modeled
00:27:01.839 --> 00:27:06.480
and you have a semantic layer, and these are
00:27:06.480 --> 00:27:08.599
all things that have been useful for a decade,
00:27:08.960 --> 00:27:14.559
then you're already a long way down the path.
00:27:16.170 --> 00:27:21.279
What you still need to do is you need to figure
00:27:21.279 --> 00:27:24.599
out an answer to the context layer, which is
00:27:24.599 --> 00:27:28.599
a legitimately new thing. Now, it's not rocket
00:27:28.599 --> 00:27:33.960
science. I mean, these are the kinds of things
00:27:33.960 --> 00:27:38.329
that, I mean, like I remember 10 years ago, it
00:27:38.329 --> 00:27:41.690
was very trendy to build your own data tooling.
00:27:42.049 --> 00:27:47.119
Uber built their own orchestrator. Airbnb built
00:27:47.119 --> 00:27:50.619
their own orchestrator. And this is like a thing
00:27:50.619 --> 00:27:53.160
that people did. Airbnb built their own data
00:27:53.160 --> 00:27:55.480
visualization tool. Nobody builds that kind of
00:27:55.480 --> 00:27:56.660
stuff anymore because they're like, whatever,
00:27:56.819 --> 00:27:59.200
there's good enough off the shelf stuff.
But
00:27:59.200 --> 00:28:03.079
so like these same types of companies are currently
00:28:03.079 --> 00:28:05.059
building their own context layers. And so you
00:28:05.059 --> 00:28:08.980
can do that. There are starting to be context
00:28:08.980 --> 00:28:12.380
layers that you can buy. We have a context layer.
00:28:12.480 --> 00:28:16.299
It's very new. It's like, I don't know, a couple
00:28:16.299 --> 00:28:18.200
dozen companies using it. We use it internally.
00:28:18.339 --> 00:28:20.960
It's freaking awesome. The difference between
00:28:20.960 --> 00:28:25.579
not having a context layer and having a context
00:28:25.579 --> 00:28:28.240
layer is the difference between a data agent
00:28:28.240 --> 00:28:30.180
that you want to come back to and ask more questions
00:28:30.180 --> 00:28:34.609
of and the one that you don't. And then you need
00:28:34.609 --> 00:28:37.529
to implement evals to make sure that as people
00:28:37.529 --> 00:28:41.130
on the data team implement changes throughout
00:28:41.130 --> 00:28:43.930
the entire stack, that the accuracy of answers
00:28:43.930 --> 00:28:47.890
goes up and not down. And then you probably are
00:28:47.890 --> 00:28:51.130
thinking about real time because you're going
00:28:51.130 --> 00:28:54.710
to want to start having agents act on behalf
00:28:54.710 --> 00:28:57.549
of the intelligence that they get and not just
00:28:57.549 --> 00:29:02.910
serve it up to you as a human. And real time
00:29:02.910 --> 00:29:06.289
is important for action taking. So there's this
00:29:06.289 --> 00:29:12.369
whole set of things that you will do. But really,
00:29:12.529 --> 00:29:15.359
step one is. Make sure your house is in order
00:29:15.359 --> 00:29:20.420
from the best practices. And two is context layer.
00:29:20.619 --> 00:29:25.160
And then you need to get adoption across your
00:29:25.160 --> 00:29:29.140
company. And this is very different than the
00:29:29.140 --> 00:29:32.579
adoption of a BI tool. People know what a BI
00:29:32.579 --> 00:29:35.940
tool is. Most people have lived their entire
00:29:35.940 --> 00:29:39.420
careers in a context where there's always been
00:29:39.420 --> 00:29:41.940
some type of way to get a dashboard for something.
00:29:42.589 --> 00:29:46.690
But this is a new type of modality. And you don't
00:29:46.690 --> 00:29:48.829
just want somebody to go through a training and
00:29:48.829 --> 00:29:50.930
say like, okay, cool, I know how to make a dashboard.
00:29:52.029 --> 00:29:54.869
What people who have deployed these things in
00:29:54.869 --> 00:29:59.470
production find is that there is a real depth
00:29:59.470 --> 00:30:03.710
of engagement that occurs with analytics agents
00:30:03.710 --> 00:30:08.029
where you can experience over the course of several
00:30:08.029 --> 00:30:11.170
months of adoption, you can experience Query
00:30:11.170 --> 00:30:14.630
volumes go up by 50x just because there were
00:30:14.630 --> 00:30:16.930
all these latent questions that people throughout
00:30:16.930 --> 00:30:19.190
your organization had that the friction was just
00:30:19.190 --> 00:30:22.670
too high to get them answered previously. Yeah,
00:30:22.710 --> 00:30:25.910
now that makes sense. Okay, so wrapping up, we
00:30:25.910 --> 00:30:29.490
were talking before we started recording. I'm
00:30:29.490 --> 00:30:31.750
just curious, what have you been building lately
00:30:31.750 --> 00:30:38.140
with AI tools? Oh gosh. So I have for the past
00:30:38.140 --> 00:30:41.109
month ever since, personal agents have become
00:30:41.109 --> 00:30:47.690
the focus of things in the Valley. It's either
00:30:47.690 --> 00:30:51.430
personal agents like Muse or Jev, where everybody's
00:30:51.430 --> 00:30:56.130
talking about one of the two. And I don't know
00:30:56.130 --> 00:31:00.289
if you are like me at all, but I really just,
00:31:00.369 --> 00:31:04.849
after two decades of consumer tech, I just really
00:31:04.849 --> 00:31:07.490
don't believe that... My personal incentives
00:31:07.490 --> 00:31:09.829
are aligned with any of the companies that are
00:31:09.829 --> 00:31:13.509
building personal agents today. And so I've been
00:31:13.509 --> 00:31:19.289
building my own personal agent. And it's actually
00:31:19.289 --> 00:31:22.589
not like, it's not rocket science. It's a bunch
00:31:22.589 --> 00:31:25.630
of integrations and it's a bunch of, you can
00:31:25.630 --> 00:31:27.309
build really clear permissions into it, blah,
00:31:27.369 --> 00:31:29.930
blah, blah. But the thing that we were talking
00:31:29.930 --> 00:31:33.210
about before hopping on is I recently decided
00:31:33.210 --> 00:31:37.309
to get a little more serious about the development
00:31:37.309 --> 00:31:39.990
workflow and have refactored everything from
00:31:39.990 --> 00:31:43.660
a, like, Claude Code, sit at my terminal and ask
00:31:43.660 --> 00:31:46.640
it to do stuff to a software factory where I
00:31:46.640 --> 00:31:49.019
file tickets and comment on pull requests and
00:31:49.019 --> 00:31:52.460
this kind of thing. And that transition has been
00:31:52.460 --> 00:31:58.119
super, super fun for me because it's just much
00:31:58.279 --> 00:32:01.940
more effective. And I think that most people
00:32:01.940 --> 00:32:06.029
are still experiencing the world of agentic development
00:32:06.029 --> 00:32:11.569
in a Claude Code terminal. And I know that that
00:32:11.569 --> 00:32:14.329
only really started working well a year ago,
00:32:14.450 --> 00:32:17.910
a little less than a year ago. But I don't think
00:32:17.910 --> 00:32:20.049
that's the primary modality that we should be
00:32:20.049 --> 00:32:22.549
interacting with these things anymore. So do
00:32:22.549 --> 00:32:24.650
you think building these software factories,
00:32:24.890 --> 00:32:29.009
do you think that this will have an impact on
00:32:29.009 --> 00:32:33.569
dbt open source? the way you run an organization,
00:32:33.809 -->
00:32:36.630
or are you thinking about this just as like small
00:32:36.630 --> 00:32:40.130
side project efficiencies? It's a good question.
00:32:40.210 --> 00:32:47.369
I think that most software maintenance will end
00:32:47.369 --> 00:32:51.190
up needing to be turned into a software factory.
00:32:51.650 --> 00:32:55.529
And so that absolutely will. I mean, here's one
00:32:55.529 --> 00:32:57.970
of the ways that we implemented this internally.
00:33:01.079 --> 00:33:04.220
We've just gone through a big upgrade from dbt 1
00:33:04.220 --> 00:33:08.240
to dbt 2. We do not increment major versions very
00:33:08.240 --> 00:33:12.180
often. That's been five years and coming. And
00:33:12.180 --> 00:33:15.519
because of the way that dbt's product surface
00:33:15.519 --> 00:33:18.119
area looks, there's a non-trivial migration
00:33:18.119 --> 00:33:25.160
effort dbt 2 has much tighter language constructs,
00:33:25.160 --> 00:33:28.880
and you have to validate users' codes more effectively.
00:33:30.940 --> 00:33:33.900
We care a lot about conformance. How do you make
00:33:33.900 --> 00:33:36.079
sure that dbt 2 supports all the functionality
00:33:36.079 --> 00:33:40.740
of dbt 1? And we have a metric that we track,
00:33:40.880 --> 00:33:45.519
which is our conformance metric. And we just
00:33:45.519 --> 00:33:48.000
weren't making progress fast enough. We have
00:33:48.000 --> 00:33:50.099
a team of something like 20 people that were
00:33:50.099 --> 00:33:53.059
just focused on conformance all day, every day.
00:33:53.500 --> 00:33:58.960
And a year ago, they made some early wins and
00:33:58.960 --> 00:34:01.130
then it kind of flattened off a little bit. And
00:34:01.130 --> 00:34:04.769
so we kind of took almost like a two-month pause
00:34:04.769 --> 00:34:07.750
and we implemented something called Conforman.
00:34:08.510 --> 00:34:15.639
And it was stressful because I'm like, hey, everybody,
00:34:15.780 --> 00:34:17.320
I want the line to keep going up. And they're
00:34:17.320 --> 00:34:20.300
like, no, no, no, we're going to pause and then
00:34:20.300 --> 00:34:24.500
it's going to go up fast. And Conforman is essentially
00:34:24.500 --> 00:34:30.119
a loop that identifies conformance defects. It
00:34:30.119 --> 00:34:33.400
writes tests for them. It writes the code to fix
00:34:33.400 --> 00:34:37.679
them and validate them, etc. And we drove up conformance
00:34:37.679 --> 00:34:41.900
from, I can't remember, something like 45% in January
00:34:41.900 --> 00:34:46.480
to now 93%. And, you know, every successive percent
00:34:46.480 --> 00:34:49.420
gets harder. You have to fight for it harder. So
00:34:49.420 --> 00:34:53.840
it's been a dramatic, dramatic... Uh, I don't think
00:34:53.840 --> 00:34:56.300
we would have been able to ship V2 without it.
00:34:56.300 --> 00:34:59.400
That's awesome. Where can people find more out
00:34:59.400 --> 00:35:03.960
about dbt, Fivetran, and yourself? dbt is getdbt.com.
00:35:03.960 --> 00:35:09.000
Fivetran is fivetran.com. And I, these
00:35:09.000 --> 00:35:14.340
days, primarily, I've hidden from most forms
00:35:14.340 --> 00:35:17.360
of social media, but I still have a podcast at
00:35:17.360 --> 00:35:20.000
the Analytics Engineering Roundup and the Analytics
00:35:20.000 --> 00:35:22.679
Engineering Podcast. Cool. I'll link all of those
00:35:22.679 --> 00:35:24.860
in the show notes. Tristan, thank you so much
00:35:24.860 --> 00:35:26.159
for coming on. Really, really appreciate your
00:35:26.159 --> 00:35:29.300
time. Thanks, it's been fun. That was my conversation
00:35:29.300 --> 00:35:34.380
with Tristan Handy from Fivetran and dbt. The
00:35:34.380 --> 00:35:36.980
biggest takeaway for me is that the hard part
00:35:36.980 --> 00:35:40.199
of pointing agents at your data isn't the SQL.
00:35:40.780 --> 00:35:44.440
Context and a semantic layer are the difference
00:35:44.440 --> 00:35:49.639
between about 30% right and north of 90. So
00:35:49.639 --> 00:35:53.820
his first step for a platform or data team is
00:35:53.820 --> 00:35:58.070
kind of boring. Eat your spinach. A well-governed
00:35:58.070 --> 00:36:01.829
data lake, well-modeled data, and a semantic
00:36:01.829 --> 00:36:05.969
layer are still the foundation. Then add the
00:36:05.969 --> 00:36:09.329
piece that's genuinely new, a context layer,
00:36:09.550 --> 00:36:14.150
plus evals. So accuracy goes up and not down
00:36:14.150 --> 00:36:17.949
as people change the stack. I also liked his
00:36:17.949 --> 00:36:22.269
take on careers. As more of this work gets systematized,
00:36:22.389 --> 00:36:25.280
you are either working inside the system, or
00:36:25.280 --> 00:36:29.639
on it. The interesting work is on the system, designing
00:36:29.639 --> 00:36:32.940
the loops and the evals, and spotting the needs
00:36:32.940 --> 00:36:36.909
nobody has written down yet. He calls that moving
00:36:36.909 --> 00:36:41.670
up the stack. And his team lives it. Their dbt 2
00:36:41.670 --> 00:36:45.670
conformance work had stalled. So they paused for
00:36:45.670 --> 00:36:49.349
two months and built an agent loop that finds
00:36:49.349 --> 00:36:53.010
the gaps, writes the tests, and fixes the code.
00:36:53.610 --> 00:36:59.010
It went from something like 45% to 93%. He doesn't
00:36:59.010 --> 00:37:01.369
think they could have shipped version 2 without
00:37:01.369 --> 00:37:07.190
it. You can find dbt at getdbt.com, Fivetran
00:37:07.190 --> 00:37:11.710
at fivetran.com, and Tristan's Analytics Engineering
00:37:11.710 --> 00:37:16.070
Roundup newsletter and podcast all linked in
00:37:16.070 --> 00:37:19.469
the show notes. Follow or subscribe to Ship It
00:37:19.469 --> 00:37:22.320
Weekly wherever you listen, and find previous
00:37:22.320 --> 00:37:26.500
episodes at shipitweekly.fm. Thanks for listening,
00:37:26.559 --> 00:37:28.500
and I'll see you later this week.