Listen now | Recording of AURA - Chris Gambill & Shane Gibson on 10th July 2026

WEBVTT
Kind: captions
Language: en
00:00:00.080 --> 00:00:02.869
Hey Chris, welcome to the second Aura
00:00:02.879 --> 00:00:05.030
session. Hey, thanks for uh joining me.
00:00:05.040 --> 00:00:07.590
It's going to be fun for uh for a few
00:00:07.600 --> 00:00:09.589
months at least until we see how we go.
00:00:09.599 --> 00:00:12.390
So uh how you going? What's happening?
00:00:12.400 --> 00:00:15.589
>> No, too much. Too much. Went to the data
00:00:15.599 --> 00:00:18.390
brick summit then u then immediately
00:00:18.400 --> 00:00:21.429
followed by a family trip just a cruise
00:00:21.439 --> 00:00:23.670
to Alaska and driving up the west coast
00:00:23.680 --> 00:00:27.349
in the US. And so lots of uh lots of
00:00:27.359 --> 00:00:31.109
fun, all all kinds of exciting things.
00:00:31.119 --> 00:00:33.270
>> How many people were at the uh data
00:00:33.280 --> 00:00:34.790
brick summit? How many they have those
00:00:34.800 --> 00:00:35.910
things these days?
00:00:35.920 --> 00:00:39.510
>> Physically over 30,000 people,
00:00:39.520 --> 00:00:43.350
>> which was insane. I mean, it was like
00:00:43.360 --> 00:00:44.869
like I keep on saying this, but it was
00:00:44.879 --> 00:00:47.430
like a rock concert. Like the keynotes
00:00:47.440 --> 00:00:50.069
like there's lights strobing and like
00:00:50.079 --> 00:00:51.430
there's a little bit of mist in the
00:00:51.440 --> 00:00:55.270
room. Like they definitely try to, you
00:00:55.280 --> 00:00:57.270
know, get y'all hyped up for sure.
00:00:57.280 --> 00:01:00.069
>> And then how many people on the cruise?
00:01:00.079 --> 00:01:03.670
>> There were probably there were 2,411
00:01:03.680 --> 00:01:04.950
people on the cruise. So
00:01:04.960 --> 00:01:06.630
>> So you you're going to come down with a
00:01:06.640 --> 00:01:08.230
cold, right? [laughter] There's no way
00:01:08.240 --> 00:01:10.070
you've been in that mass of people
00:01:10.080 --> 00:01:11.510
without catching.
00:01:11.520 --> 00:01:13.190
>> No, I had a little bit of sniffles for
00:01:13.200 --> 00:01:14.870
sure. And I wasn't sure if it was like
00:01:14.880 --> 00:01:16.950
all the different climates that I was in
00:01:16.960 --> 00:01:19.109
or what over the past couple weeks, but
00:01:19.119 --> 00:01:19.910
yeah.
00:01:19.920 --> 00:01:21.510
>> Excellent. Excellent. Hey, just a shout
00:01:21.520 --> 00:01:23.590
out to UA who's there from Helsinki.
00:01:23.600 --> 00:01:26.469
Jessica, I see you come on. And Andreas
00:01:26.479 --> 00:01:29.190
uh asking how we going. Nice to be here.
00:01:29.200 --> 00:01:31.510
Yeah, I'm going hot. I'm in the UK.
00:01:31.520 --> 00:01:34.230
We're in a heat wave. It's 35 degrees
00:01:34.240 --> 00:01:36.069
Celsius. Proper temperature measurement,
00:01:36.079 --> 00:01:38.710
not that Fahrenheit stuff. So, uh yeah.
00:01:38.720 --> 00:01:41.109
Yeah. End of a Friday and I'm I'm
00:01:41.119 --> 00:01:42.870
sweltering, but that's okay. I got I got
00:01:42.880 --> 00:01:44.469
my mate Chris on. We're going to talk
00:01:44.479 --> 00:01:46.149
about some data stuff. So, if anybody's
00:01:46.159 --> 00:01:47.429
got any questions, just put it in the
00:01:47.439 --> 00:01:49.510
chat. Otherwise, as it said in the
00:01:49.520 --> 00:01:50.950
promo, we're just going to talk [ __ ]
00:01:50.960 --> 00:01:53.350
about data. Um, yeah,
00:01:53.360 --> 00:01:55.030
>> that's right. We'll just randomly say
00:01:55.040 --> 00:01:56.389
things. It'll be [laughter]
00:01:56.399 --> 00:01:57.429
>> it'll be fine.
00:01:57.439 --> 00:01:58.789
>> Like we normally do when we catch up,
00:01:58.799 --> 00:02:02.310
right? Just recording it for now. So,
00:02:02.320 --> 00:02:03.109
>> go ahead. Go ahead.
00:02:03.119 --> 00:02:04.870
>> No, no, you you go, man. I was just
00:02:04.880 --> 00:02:06.789
going to say uh like there's so many
00:02:06.799 --> 00:02:09.109
things going on like just kind of going
00:02:09.119 --> 00:02:10.949
back to like the data bricks conference
00:02:10.959 --> 00:02:14.150
like the word of apparently the word of
00:02:14.160 --> 00:02:17.270
all of the kind of summits and those
00:02:17.280 --> 00:02:21.270
those big conferences was like context
00:02:21.280 --> 00:02:22.309
right
00:02:22.319 --> 00:02:24.550
>> ontology and context were the big words
00:02:24.560 --> 00:02:25.589
like
00:02:25.599 --> 00:02:27.510
>> because the fabric conference was the
00:02:27.520 --> 00:02:29.990
same you know the Microsoft build
00:02:30.000 --> 00:02:32.630
snowflake the big word was context and
00:02:32.640 --> 00:02:34.869
ontology and data bricks was they're
00:02:34.879 --> 00:02:36.229
they're all going to be the masters of
00:02:36.239 --> 00:02:38.550
ontology like it was crazy.
00:02:38.560 --> 00:02:41.030
>> Yep. Uh there's lots of uh lots of
00:02:41.040 --> 00:02:43.509
commentary about ontology and the uh
00:02:43.519 --> 00:02:45.830
buzzwashing of it by the data vendors.
00:02:45.840 --> 00:02:48.949
Uh and you know we we hated it in our
00:02:48.959 --> 00:02:51.350
domain when vendors took words that we'd
00:02:51.360 --> 00:02:53.990
spent years defining and working with
00:02:54.000 --> 00:02:56.070
and now that they're stealing those
00:02:56.080 --> 00:02:58.070
terms from the uh library sciences.
00:02:58.080 --> 00:03:00.790
It'll be interesting to see how on true
00:03:00.800 --> 00:03:03.990
to ontologies their stuff really is or
00:03:04.000 --> 00:03:06.869
uh whether it's not. So uh yeah I mean
00:03:06.879 --> 00:03:10.229
what did you did you feel like it was
00:03:10.239 --> 00:03:12.390
all it's not even PowerPoint now what is
00:03:12.400 --> 00:03:14.070
it you know are they Google slides are
00:03:14.080 --> 00:03:16.630
they vibecoded presentation apps now was
00:03:16.640 --> 00:03:19.750
it all PowerPoint wear or was it a bit
00:03:19.760 --> 00:03:21.589
more close to reality
00:03:21.599 --> 00:03:23.830
>> it was as far as like all the
00:03:23.840 --> 00:03:25.589
presentations and stuff go is that what
00:03:25.599 --> 00:03:26.790
you're asking
00:03:26.800 --> 00:03:29.589
>> it so
00:03:29.599 --> 00:03:32.309
uh it was a lot of vibecoded
00:03:32.319 --> 00:03:34.949
presentations right it was that they
00:03:34.959 --> 00:03:37.110
used, you could see during the
00:03:37.120 --> 00:03:39.509
presentation like the, you know, the
00:03:39.519 --> 00:03:42.390
actual [clears throat] explorer line was
00:03:42.400 --> 00:03:44.149
claude.ai and that was their
00:03:44.159 --> 00:03:46.309
presentation, right? So, it was it was a
00:03:46.319 --> 00:03:48.149
lot of that. Not that they weren't the
00:03:48.159 --> 00:03:49.509
people that presented weren't very
00:03:49.519 --> 00:03:51.190
intelligent about what they were talking
00:03:51.200 --> 00:03:55.589
about, but definitely AI augmented on uh
00:03:55.599 --> 00:04:00.070
their presentation themselves.
00:04:00.080 --> 00:04:01.670
Ed's here. Hey, Ed.
00:04:01.680 --> 00:04:05.110
>> Yeah. And and we got Yua asking a hard
00:04:05.120 --> 00:04:07.270
question. Uh he's going to be on at the
00:04:07.280 --> 00:04:08.309
end of the month, so he can pay him
00:04:08.319 --> 00:04:09.990
back, but he's uh he's asking you
00:04:10.000 --> 00:04:11.670
specifically, which I'm I'm glad about
00:04:11.680 --> 00:04:12.789
because I don't want to have to answer
00:04:12.799 --> 00:04:15.190
this one. Uh explain like a 5-year-old
00:04:15.200 --> 00:04:17.830
version of Genieontology.
00:04:17.840 --> 00:04:20.069
>> Like I always have the hard time with
00:04:20.079 --> 00:04:22.230
the like I'm a 5-year-old thing. just
00:04:22.240 --> 00:04:24.070
explain it like he's a data practitioner
00:04:24.080 --> 00:04:24.469
to
00:04:24.479 --> 00:04:27.990
>> my Yeah, because you you know um Okay,
00:04:28.000 --> 00:04:30.629
so Geneontology for those of you that
00:04:30.639 --> 00:04:32.230
didn't haven't gotten a chance to go out
00:04:32.240 --> 00:04:35.270
and look at look at it or be see the
00:04:35.280 --> 00:04:37.430
conference virtually or be there.
00:04:37.440 --> 00:04:40.390
Basically, Geneontology is going to take
00:04:40.400 --> 00:04:43.189
the repository of all your PDFs and all
00:04:43.199 --> 00:04:45.510
your word docs and basically read all of
00:04:45.520 --> 00:04:49.830
that in to determine what the right
00:04:49.840 --> 00:04:52.230
calculations, what the right source of
00:04:52.240 --> 00:04:54.790
your data is based on the question
00:04:54.800 --> 00:04:59.350
that's being as asked. So, if a you know
00:04:59.360 --> 00:05:02.710
CEO goes in and says, okay, what was my
00:05:02.720 --> 00:05:06.550
ARR for the last, you know, 12 months?
00:05:06.560 --> 00:05:09.430
Then it's going to know it's going to go
00:05:09.440 --> 00:05:11.909
through basically crawl all those
00:05:11.919 --> 00:05:14.070
documents and understand what the right
00:05:14.080 --> 00:05:15.990
tables are and what the right
00:05:16.000 --> 00:05:17.990
calculations and everything that should
00:05:18.000 --> 00:05:19.990
be made in order to give an accurate
00:05:20.000 --> 00:05:21.909
answer in a consistent manner. That's
00:05:21.919 --> 00:05:24.390
the theory, right? That's the theory of
00:05:24.400 --> 00:05:27.350
how it'll work. I
00:05:27.360 --> 00:05:30.390
personally think that there's a lot more
00:05:30.400 --> 00:05:33.110
work to do on the business side of that
00:05:33.120 --> 00:05:35.670
to determine what all those pieces are.
00:05:35.680 --> 00:05:38.870
Right? So it that's that's my thing is
00:05:38.880 --> 00:05:41.830
that yes, that solves a problem, but it
00:05:41.840 --> 00:05:44.070
doesn't solve the problem, right?
00:05:44.080 --> 00:05:45.909
>> Yeah. And Ed asked a question about
00:05:45.919 --> 00:05:47.590
describe, you know, explain as a
00:05:47.600 --> 00:05:49.430
5-year-old onetology is. So I'm I'm
00:05:49.440 --> 00:05:50.629
going to bail out of that one. I'm going
00:05:50.639 --> 00:05:53.029
to say go read Jessica Talisman's uh
00:05:53.039 --> 00:05:55.510
content because she works in that domain
00:05:55.520 --> 00:05:58.230
and she's describing it um in ways that
00:05:58.240 --> 00:05:59.029
I can't.
00:05:59.039 --> 00:06:01.510
>> Yeah. But I will I will argue that uh
00:06:01.520 --> 00:06:03.749
from what I've read the the data bricks
00:06:03.759 --> 00:06:07.029
genie stuff may be a genie but it it
00:06:07.039 --> 00:06:11.270
ain't hor's ass of an ontology. Um and
00:06:11.280 --> 00:06:12.710
you know I I think because I'm doing a
00:06:12.720 --> 00:06:13.990
lot of work in the space from a data
00:06:14.000 --> 00:06:15.830
point of view and I think about it as BI
00:06:15.840 --> 00:06:19.670
semantic layers or metric layers uh and
00:06:19.680 --> 00:06:22.390
um context right and so the way I
00:06:22.400 --> 00:06:24.150
describe it is if you hear people
00:06:24.160 --> 00:06:27.510
talking about metrics joins uh
00:06:27.520 --> 00:06:30.870
definitions of words um and that a last
00:06:30.880 --> 00:06:33.990
mile tool BI tool or an AI agent or your
00:06:34.000 --> 00:06:36.550
LLM is using it uh and it's typically
00:06:36.560 --> 00:06:38.469
tacked on the end of your final
00:06:38.479 --> 00:06:40.629
consumption layer. of your data. If you
00:06:40.639 --> 00:06:42.150
run a layered data architecture, which
00:06:42.160 --> 00:06:43.510
you should, I'm not going to use the
00:06:43.520 --> 00:06:46.070
colors. Um, then that's a BI semantic
00:06:46.080 --> 00:06:47.749
layer, right? We're creating semantic
00:06:47.759 --> 00:06:51.430
meaning uh and hints basically for those
00:06:51.440 --> 00:06:53.270
LLMs to query that data.
00:06:53.280 --> 00:06:53.590
>> Yeah.
00:06:53.600 --> 00:06:56.070
>> If I talk about context, what I'm saying
00:06:56.080 --> 00:06:57.909
is there's a whole lot of other stuff we
00:06:57.919 --> 00:07:00.150
need, right? We need other information
00:07:00.160 --> 00:07:01.990
for other parts of our data work. If
00:07:02.000 --> 00:07:03.430
we're going to go and get it to create
00:07:03.440 --> 00:07:05.510
our ETLs for us, if we're going to get
00:07:05.520 --> 00:07:07.589
it to create data quality or trust
00:07:07.599 --> 00:07:08.790
rules, right? there's a whole lot of
00:07:08.800 --> 00:07:11.270
data work uh that we can get the LLMs
00:07:11.280 --> 00:07:12.870
the geni stuff to do now and it needs
00:07:12.880 --> 00:07:16.550
more context um and so that's really
00:07:16.560 --> 00:07:18.150
really valuable right but it's more than
00:07:18.160 --> 00:07:21.749
just a BI semantic layer that look like
00:07:21.759 --> 00:07:23.029
you know if I think about data bricks
00:07:23.039 --> 00:07:24.469
and I think about unity catalog and I
00:07:24.479 --> 00:07:26.230
think about their integrated platform
00:07:26.240 --> 00:07:27.270
that's where they're going to go right
00:07:27.280 --> 00:07:28.390
they're going to go to what I call the
00:07:28.400 --> 00:07:29.749
control plane and I call it a control
00:07:29.759 --> 00:07:31.189
plane because I think it sits above
00:07:31.199 --> 00:07:33.430
everything else uh sorry context plane
00:07:33.440 --> 00:07:34.790
because I think of it like a control
00:07:34.800 --> 00:07:36.790
plane like a security plane you know
00:07:36.800 --> 00:07:39.830
control plane has security, auditing,
00:07:39.840 --> 00:07:41.430
logging, all those kind of things that
00:07:41.440 --> 00:07:44.230
we do. Um, but the thing that interested
00:07:44.240 --> 00:07:46.790
me was they I think you know vendors
00:07:46.800 --> 00:07:48.150
like to do big numbers and they talked
00:07:48.160 --> 00:07:52.629
about 4.5 million ontology snippets.
00:07:52.639 --> 00:07:52.950
>> Yeah.
00:07:52.960 --> 00:07:55.670
>> Right. Which I kind of read briefly and
00:07:55.680 --> 00:07:57.589
I kind of thought it sounded like well
00:07:57.599 --> 00:07:58.790
they've got some examples from
00:07:58.800 --> 00:08:00.710
industries of terms and they're going to
00:08:00.720 --> 00:08:03.510
go use them. And you know, I've been
00:08:03.520 --> 00:08:05.189
around this game for a while and we've
00:08:05.199 --> 00:08:08.950
done multiple waves of trying to create
00:08:08.960 --> 00:08:11.189
uh cookbooks for industry, right? In
00:08:11.199 --> 00:08:13.670
fact, actually, I was just watching um
00:08:13.680 --> 00:08:15.510
Mother Duck come out with their recipes
00:08:15.520 --> 00:08:17.670
now or their whatever they called
00:08:17.680 --> 00:08:18.070
>> and
00:08:18.080 --> 00:08:19.589
>> I think I have a few of those books,
00:08:19.599 --> 00:08:22.150
too. Like data bricks cookbook and all
00:08:22.160 --> 00:08:23.270
these. Yes. Yes.
00:08:23.280 --> 00:08:24.869
>> Yeah. And so what there was like when I
00:08:24.879 --> 00:08:26.950
was at Oracle, there was like uh out of
00:08:26.960 --> 00:08:29.430
the box cookbooks for receivables, you
00:08:29.440 --> 00:08:31.430
know, you kind of just drop it in.
00:08:31.440 --> 00:08:33.750
>> Um and what happens is it makes sense,
00:08:33.760 --> 00:08:35.430
right? That 80% of what we're doing is
00:08:35.440 --> 00:08:37.029
reusable. We've done it before, right?
00:08:37.039 --> 00:08:38.709
You go and hit Stripe.
00:08:38.719 --> 00:08:40.469
>> Why do we have to reinvent how we join
00:08:40.479 --> 00:08:43.029
Stripe? 20% of it's unique, right? You
00:08:43.039 --> 00:08:45.110
actually go and how people measure a
00:08:45.120 --> 00:08:46.949
subscription in Stripe and it ain't just
00:08:46.959 --> 00:08:49.990
off one table. Um but I've never seen it
00:08:50.000 --> 00:08:51.590
been commercially viable, right? I've
00:08:51.600 --> 00:08:53.350
never seen people actually adopt it. So
00:08:53.360 --> 00:08:56.070
is that your take that the 4.5 million
00:08:56.080 --> 00:08:58.790
geneontology snippets and I may have got
00:08:58.800 --> 00:09:00.949
the terms wrong but I don't think I have
00:09:00.959 --> 00:09:03.110
>> was like throw at it and there's this
00:09:03.120 --> 00:09:07.190
big wisdom knowledge information data
00:09:07.200 --> 00:09:09.030
>> in fact in fact like what they showed up
00:09:09.040 --> 00:09:11.030
on the screen was like a knowledge graph
00:09:11.040 --> 00:09:12.790
kind of thing that when they were
00:09:12.800 --> 00:09:14.310
talking about it right where it would
00:09:14.320 --> 00:09:16.470
basically a question would be asked and
00:09:16.480 --> 00:09:18.949
you would turn it was like turning red
00:09:18.959 --> 00:09:21.030
based on okay this was the original
00:09:21.040 --> 00:09:23.269
origin point of the question and then it
00:09:23.279 --> 00:09:25.110
kind of branched out looking at all the
00:09:25.120 --> 00:09:27.030
different pieces that it needed to look
00:09:27.040 --> 00:09:30.310
at in order to resolve what the question
00:09:30.320 --> 00:09:33.350
was. But yeah, it very much the way that
00:09:33.360 --> 00:09:35.670
you explained it. Absolutely.
00:09:35.680 --> 00:09:36.949
>> Yeah. So, we got a question from
00:09:36.959 --> 00:09:39.430
Andreas. Uh, Cortex versus Genie.
00:09:39.440 --> 00:09:40.710
That'll be an interesting one. And he's
00:09:40.720 --> 00:09:44.070
also asked us Snowflake versus uh uh
00:09:44.080 --> 00:09:45.430
Data Bricks. Who will be the winner? I'm
00:09:45.440 --> 00:09:46.470
not sure that's a question of who will
00:09:46.480 --> 00:09:48.630
be the winner in the conference stake uh
00:09:48.640 --> 00:09:51.750
or the uh data data platform stake. What
00:09:51.760 --> 00:09:53.350
do you think? Cortex versus Genie. I
00:09:53.360 --> 00:09:54.949
mean, have you played much with Coco?
00:09:54.959 --> 00:09:58.470
>> It I've very limited. I have very
00:09:58.480 --> 00:10:00.790
limited exposure to Snowflake these days
00:10:00.800 --> 00:10:02.710
because so much of my practice is
00:10:02.720 --> 00:10:04.389
working with companies that are using
00:10:04.399 --> 00:10:08.389
data bricks or fabric. Um, but I have
00:10:08.399 --> 00:10:11.590
some experience with Cortex. I like
00:10:11.600 --> 00:10:14.470
Genie better. Genie feels more natural
00:10:14.480 --> 00:10:18.389
to me. But again, to be fair, 90% of my
00:10:18.399 --> 00:10:20.150
time is in data bricks as opposed to
00:10:20.160 --> 00:10:23.350
another platform. So that's my take. I
00:10:23.360 --> 00:10:25.509
don't know chain if you get more chance
00:10:25.519 --> 00:10:28.710
to be in both. I know that you're not as
00:10:28.720 --> 00:10:30.550
maybe as much in data bricks I think as
00:10:30.560 --> 00:10:31.269
as I am.
00:10:31.279 --> 00:10:33.990
>> N cut me open. I believe uh Google cloud
00:10:34.000 --> 00:10:36.470
platform these days. Um I mean we build
00:10:36.480 --> 00:10:39.670
everything we we do on that platform. Um
00:10:39.680 --> 00:10:41.990
I coach teams obviously and and one of
00:10:42.000 --> 00:10:44.470
the teams I was coaching uh using Coco.
00:10:44.480 --> 00:10:46.389
Um so I kind of observe what they're
00:10:46.399 --> 00:10:49.350
doing rather than doing it. Um, and I
00:10:49.360 --> 00:10:51.829
kind of like the way it had lots of
00:10:51.839 --> 00:10:53.750
moving parts that you needed, right? So,
00:10:53.760 --> 00:10:55.910
you could give it uh trusted queries.
00:10:55.920 --> 00:10:57.509
You could give it business questions.
00:10:57.519 --> 00:10:58.470
You can give it a whole lot of
00:10:58.480 --> 00:11:01.750
reinforcement uh context to help it get
00:11:01.760 --> 00:11:04.230
more accurate. Um, but yeah, never used
00:11:04.240 --> 00:11:07.910
it in anger. I like my favorite newish
00:11:07.920 --> 00:11:10.389
thing that's in Genie uh that you could
00:11:10.399 --> 00:11:12.550
do is like if you build your own harness
00:11:12.560 --> 00:11:14.949
or your build your own skill like you
00:11:14.959 --> 00:11:17.269
can load the repo with your skill and
00:11:17.279 --> 00:11:20.069
actually point Genie to that skill
00:11:20.079 --> 00:11:23.350
either as a individual skill for you or
00:11:23.360 --> 00:11:25.509
as a workplace skill for everybody
00:11:25.519 --> 00:11:27.829
that's in the workplace which is I think
00:11:27.839 --> 00:11:29.829
is unique to data bricks. I'm not sure
00:11:29.839 --> 00:11:32.150
that Cortex is doing something like that
00:11:32.160 --> 00:11:33.030
or not.
00:11:33.040 --> 00:11:35.030
>> Not yet. I I don't know. Right. But you
00:11:35.040 --> 00:11:37.269
know, I think we see how valuable skills
00:11:37.279 --> 00:11:38.870
are in Claude and we're going to see all
00:11:38.880 --> 00:11:41.269
the vendors, you know, copy that pattern
00:11:41.279 --> 00:11:43.590
or or implement that value depending on
00:11:43.600 --> 00:11:45.269
how you look at it. It's kind of a race,
00:11:45.279 --> 00:11:47.430
right? So, um, it's going to be
00:11:47.440 --> 00:11:49.350
interesting though. Do you reckon do you
00:11:49.360 --> 00:11:50.870
reckon skills are going to become
00:11:50.880 --> 00:11:53.430
valuable? And what I mean, not to use,
00:11:53.440 --> 00:11:55.990
but to build and publish like do you
00:11:56.000 --> 00:11:57.990
think there's going to become
00:11:58.000 --> 00:12:01.030
proprietary skills that get paywalled?
00:12:01.040 --> 00:12:04.389
it I think that's so there is a ton of
00:12:04.399 --> 00:12:06.629
work right now in the open spa you know
00:12:06.639 --> 00:12:10.710
in the open uh space that people are
00:12:10.720 --> 00:12:13.030
working on both harnesses and skills and
00:12:13.040 --> 00:12:15.990
I think that because people are doing it
00:12:16.000 --> 00:12:19.030
in this open space that I don't know if
00:12:19.040 --> 00:12:20.949
people are going to come out with a
00:12:20.959 --> 00:12:24.389
monetization method of monetizing skills
00:12:24.399 --> 00:12:27.190
or not I'm sure that somebody eventually
00:12:27.200 --> 00:12:29.590
will will be like I have this very
00:12:29.600 --> 00:12:31.269
specific
00:12:31.279 --> 00:12:34.230
very like top-of-the-line skill and if
00:12:34.240 --> 00:12:35.670
you use it, everything's going to be
00:12:35.680 --> 00:12:36.949
perfect, right? That's going to be the
00:12:36.959 --> 00:12:38.870
sales pitch. Um,
00:12:38.880 --> 00:12:40.470
>> one skill to rule them all.
00:12:40.480 --> 00:12:42.949
>> One one skill to rule them all. But like
00:12:42.959 --> 00:12:46.470
um in in a couple weeks I'm doing
00:12:46.480 --> 00:12:49.670
another podcast. Um,
00:12:49.680 --> 00:12:53.190
and we're going to talk about like meta
00:12:53.200 --> 00:12:56.069
harnesses and skills and stuff and super
00:12:56.079 --> 00:12:59.110
excited about that. But um it's going to
00:12:59.120 --> 00:13:03.269
be with Scott Haynes and um
00:13:03.279 --> 00:13:05.670
oh my gosh, I forgot her name. She's
00:13:05.680 --> 00:13:09.274
going to kill me to be Oh, and uh
00:13:09.284 --> 00:13:11.910
[clears throat] Melanie Warick. Um she
00:13:11.920 --> 00:13:16.389
is uh the dev rail for Temporal. But um
00:13:16.399 --> 00:13:18.310
but we're going to do a live se live
00:13:18.320 --> 00:13:20.470
stream on meta harnesses and skills and
00:13:20.480 --> 00:13:22.230
I'm super excited about that. But all
00:13:22.240 --> 00:13:25.110
that to say, like there's companies out
00:13:25.120 --> 00:13:28.150
there now that are working on
00:13:28.160 --> 00:13:32.629
creating some meta harnesses that will
00:13:32.639 --> 00:13:35.190
do some of that work and they're trying
00:13:35.200 --> 00:13:36.710
to get to the point where they're
00:13:36.720 --> 00:13:38.949
monetizing it. So there is some work in
00:13:38.959 --> 00:13:40.389
that area now. So I think that
00:13:40.399 --> 00:13:41.990
eventually we'll get to that point. I
00:13:42.000 --> 00:13:43.590
don't think that we're there
00:13:43.600 --> 00:13:45.190
>> yet.
00:13:45.200 --> 00:13:47.750
>> Yeah. I think um it's interesting. I was
00:13:47.760 --> 00:13:49.750
I made a comment I think today in the
00:13:49.760 --> 00:13:51.910
practical data community in the PDC that
00:13:51.920 --> 00:13:54.470
uh I found somebody said you know am I
00:13:54.480 --> 00:13:56.470
using uh open source or offtheshelf
00:13:56.480 --> 00:13:58.550
harnesses um there's a bunch of them out
00:13:58.560 --> 00:14:02.150
there now and I replied with no I'm not
00:14:02.160 --> 00:14:03.990
I'm rolling my own and then I sat there
00:14:04.000 --> 00:14:06.230
going you know I'm kind of like a data
00:14:06.240 --> 00:14:08.150
engineer building my whole data platform
00:14:08.160 --> 00:14:09.670
not because I should just because I
00:14:09.680 --> 00:14:12.230
could. Um, but what I found whenever I
00:14:12.240 --> 00:14:14.949
grab a a harness,
00:14:14.959 --> 00:14:17.670
it doesn't do everything I want, right?
00:14:17.680 --> 00:14:20.069
And so, yeah, it applies if I want to
00:14:20.079 --> 00:14:21.829
manage my email, but then I can't get it
00:14:21.839 --> 00:14:25.990
to manage uh metrics or manage uh
00:14:26.000 --> 00:14:28.629
writing my ETL or manage uh testing my
00:14:28.639 --> 00:14:31.590
data for me. So, um and I and I run a
00:14:31.600 --> 00:14:34.230
personal harness, a personal OS that I
00:14:34.240 --> 00:14:36.230
use to try and automate some of my life.
00:14:36.240 --> 00:14:38.710
and we built and I'm building one for
00:14:38.720 --> 00:14:40.710
our product right which I use every day
00:14:40.720 --> 00:14:43.110
and for me I've broken it down to uh
00:14:43.120 --> 00:14:45.350
agile data practitioner which is the
00:14:45.360 --> 00:14:47.269
data work bit and the information
00:14:47.279 --> 00:14:48.949
product builder right kind of at the
00:14:48.959 --> 00:14:50.870
moment and they have sub agents within
00:14:50.880 --> 00:14:53.670
them um so yeah I think we're seeing
00:14:53.680 --> 00:14:56.069
everybody I talk to who's in the space
00:14:56.079 --> 00:14:58.389
now and kind of playing with it they're
00:14:58.399 --> 00:15:00.150
all building out their own harnesses and
00:15:00.160 --> 00:15:01.670
I know lots of vendors you know
00:15:01.680 --> 00:15:03.430
consulting companies are doing it make
00:15:03.440 --> 00:15:05.430
themselves faster and then obviously the
00:15:05.440 --> 00:15:08.069
big vend ers will want you to use their
00:15:08.079 --> 00:15:10.470
harness because it should be tailored
00:15:10.480 --> 00:15:12.310
for the way their products work and make
00:15:12.320 --> 00:15:14.629
you more efficient. So, I yeah, I I
00:15:14.639 --> 00:15:16.310
don't see how we're going to monetize
00:15:16.320 --> 00:15:18.949
them, but I see how the people with the
00:15:18.959 --> 00:15:21.829
best harnesses will get customers to be
00:15:21.839 --> 00:15:23.430
bound to their platforms more because
00:15:23.440 --> 00:15:25.269
it'll be too hard to leave.
00:15:25.279 --> 00:15:28.470
>> Yeah. Yeah. That that makes a lot of
00:15:28.480 --> 00:15:29.750
sense to me.
00:15:29.760 --> 00:15:31.590
>> Uh we got a question there from Andre.
00:15:31.600 --> 00:15:33.509
So, uh, let's let's ask the and excuse
00:15:33.519 --> 00:15:35.829
me if I ever get anybody's name wrong,
00:15:35.839 --> 00:15:38.389
which I will. So, that's just a given.
00:15:38.399 --> 00:15:39.990
Uh, otherwise, we'll make the guess.
00:15:40.000 --> 00:15:42.629
I'll make my my friends say it. Um, so
00:15:42.639 --> 00:15:44.790
first question is, should they jump into
00:15:44.800 --> 00:15:47.350
data bricks? So, you know, you work
00:15:47.360 --> 00:15:49.749
primarily in data bricks. I had Tim on
00:15:49.759 --> 00:15:52.710
before and he's done a bit in Snowflake.
00:15:52.720 --> 00:15:55.590
Uh, I'm Google Cloud through and through
00:15:55.600 --> 00:15:58.150
these days. You know, if you're a
00:15:58.160 --> 00:15:59.749
consultant, what do you recommend? Do
00:15:59.759 --> 00:16:02.470
you recommend trying to be across all of
00:16:02.480 --> 00:16:04.230
them? Like apart from fabric, right?
00:16:04.240 --> 00:16:06.710
Because that just sucks. But [laughter]
00:16:06.720 --> 00:16:09.749
>> fabric fabric has its place in the in
00:16:09.759 --> 00:16:12.710
the right size of business and but yeah,
00:16:12.720 --> 00:16:15.509
I mean overall um [laughter]
00:16:15.519 --> 00:16:19.910
but so little bit of background I did
00:16:19.920 --> 00:16:22.629
spend like three years in Snowflake like
00:16:22.639 --> 00:16:24.230
only Snowflake before I even knew
00:16:24.240 --> 00:16:27.910
anything about data bricks. Um, so my
00:16:27.920 --> 00:16:30.310
first venture into those platforms was
00:16:30.320 --> 00:16:32.150
Snowflake and then I migrated to data
00:16:32.160 --> 00:16:35.350
bricks and kind of fell in love. I think
00:16:35.360 --> 00:16:38.629
that a and I coach people that their
00:16:38.639 --> 00:16:41.110
businesses are using both in some cases.
00:16:41.120 --> 00:16:42.710
I have a couple of directors that I
00:16:42.720 --> 00:16:46.949
coach and their business uses both. And
00:16:46.959 --> 00:16:50.389
um I think that Snowflake is fantastic
00:16:50.399 --> 00:16:52.870
from the analytics standpoint, but if
00:16:52.880 --> 00:16:55.670
you're going to be doing any AI ML type
00:16:55.680 --> 00:16:57.910
work, I think that data bricks is the
00:16:57.920 --> 00:17:01.189
place to be, right? Um that's that's
00:17:01.199 --> 00:17:03.189
kind of where their bread and butter is.
00:17:03.199 --> 00:17:05.189
That's where they grew up from is that
00:17:05.199 --> 00:17:10.549
ML place. And so they have a platform
00:17:10.559 --> 00:17:14.870
that is very tuned into those types of
00:17:14.880 --> 00:17:18.230
like huge volumes of data,
00:17:18.240 --> 00:17:22.549
a lot of ML type work. Um they're
00:17:22.559 --> 00:17:24.710
they've done all this work with Genie
00:17:24.720 --> 00:17:27.189
and working to perfect it. They have all
00:17:27.199 --> 00:17:29.669
these um like Spark Lake flow
00:17:29.679 --> 00:17:31.270
declarative pipelines which the name of
00:17:31.280 --> 00:17:32.870
that whole thing changes every six
00:17:32.880 --> 00:17:35.270
months because at one point it was DT,
00:17:35.280 --> 00:17:37.430
right? it they have that that really
00:17:37.440 --> 00:17:40.230
does a fantastic job of a lot of trans
00:17:40.240 --> 00:17:42.710
transformations and even like data
00:17:42.720 --> 00:17:44.870
quality rules to where you don't really
00:17:44.880 --> 00:17:46.950
need like a dbt on top of data bricks
00:17:46.960 --> 00:17:50.150
right so I think my preference is data
00:17:50.160 --> 00:17:54.470
bricks um for those workloads and then
00:17:54.480 --> 00:17:56.549
you could even still have snowflake for
00:17:56.559 --> 00:17:58.150
some of your other workloads I do also
00:17:58.160 --> 00:18:00.710
also see I have one one client that has
00:18:00.720 --> 00:18:04.310
fabric for uh their analytics side of
00:18:04.320 --> 00:18:06.789
things and and they went into data bicks
00:18:06.799 --> 00:18:10.710
for some of their more advanced ML stuff
00:18:10.720 --> 00:18:12.549
uh to create their data lake. And the
00:18:12.559 --> 00:18:15.270
great thing about that is that fabric
00:18:15.280 --> 00:18:19.350
and data bricks have um now recently
00:18:19.360 --> 00:18:22.950
have the one lake uh kind of zero copy
00:18:22.960 --> 00:18:25.110
between the two and so that's nice.
00:18:25.120 --> 00:18:27.590
>> So when you say they use fabric for
00:18:27.600 --> 00:18:29.430
analytics, you mean they use PowerBI on
00:18:29.440 --> 00:18:31.590
top of data bricks? Yeah. Well, I just
00:18:31.600 --> 00:18:33.350
wanted to check that. Yeah, they well
00:18:33.360 --> 00:18:34.710
not only that I like they're they're
00:18:34.720 --> 00:18:36.470
loading data into one lake. Like they're
00:18:36.480 --> 00:18:38.070
you loading like their business data
00:18:38.080 --> 00:18:40.390
into one lake and then they're loading
00:18:40.400 --> 00:18:43.029
all of their engineering workloads and
00:18:43.039 --> 00:18:45.669
like CAD drawings and stuff into
00:18:45.679 --> 00:18:47.990
>> data bricks and then vectorizing it from
00:18:48.000 --> 00:18:49.909
there and going that route.
00:18:49.919 --> 00:18:52.070
>> Cool. I'd probably I'm going to answer
00:18:52.080 --> 00:18:53.270
it different ways, right? So I'm going
00:18:53.280 --> 00:18:56.070
to say say um I'm assuming it's a
00:18:56.080 --> 00:18:57.350
consulting question, right? as a
00:18:57.360 --> 00:18:59.430
consultant uh rather than for an
00:18:59.440 --> 00:19:01.029
organization or a person in an
00:19:01.039 --> 00:19:03.270
organization looking for the next job.
00:19:03.280 --> 00:19:05.590
uh and and I look at it from the five E
00:19:05.600 --> 00:19:08.549
right so if what you're getting paid for
00:19:08.559 --> 00:19:11.990
is effort uh then I'd probably pick a
00:19:12.000 --> 00:19:13.110
platform
00:19:13.120 --> 00:19:15.270
>> and become good on at doing effort
00:19:15.280 --> 00:19:17.350
becoming visible right so that your
00:19:17.360 --> 00:19:20.390
network knows if you need uh somebody to
00:19:20.400 --> 00:19:22.630
do data bricks you know it's Chris if
00:19:22.640 --> 00:19:24.310
you need somebody to do GCP it's Shane
00:19:24.320 --> 00:19:25.430
if you need somebody for snowflake it's
00:19:25.440 --> 00:19:26.549
somebody else right so you're going to
00:19:26.559 --> 00:19:28.390
say if you want somebody to come in and
00:19:28.400 --> 00:19:30.549
do some effort do some hours uh these
00:19:30.559 --> 00:19:31.909
are the platforms that I know really
00:19:31.919 --> 00:19:33.510
really
00:19:33.520 --> 00:19:35.430
From there, I'd say if you're going to
00:19:35.440 --> 00:19:37.669
sell expertise, then you probably want
00:19:37.679 --> 00:19:41.350
to pick one core one and a little edge
00:19:41.360 --> 00:19:43.510
ones, right? I'd always say you probably
00:19:43.520 --> 00:19:44.870
want to pick some of the new stuff
00:19:44.880 --> 00:19:47.830
coming out that's quite small um to keep
00:19:47.840 --> 00:19:49.190
your hand in and give you a little bit
00:19:49.200 --> 00:19:51.110
of a flavor. So, you know, I'm an expert
00:19:51.120 --> 00:19:53.029
in data bricks, but I also know a little
00:19:53.039 --> 00:19:54.630
bit about, you know, about these other
00:19:54.640 --> 00:19:56.310
things so that you are slightly
00:19:56.320 --> 00:19:58.070
different from every other data bricks
00:19:58.080 --> 00:20:00.549
um expert, right? Yeah. But the key is
00:20:00.559 --> 00:20:02.310
you are an expert in data bricks. You
00:20:02.320 --> 00:20:04.549
probably are certified. You you live and
00:20:04.559 --> 00:20:06.630
breathe that and that kind of stuff.
00:20:06.640 --> 00:20:10.390
>> If you're paid for experience then uh I
00:20:10.400 --> 00:20:11.830
believe you need to suck out the
00:20:11.840 --> 00:20:14.710
patterns of the platforms uh and be good
00:20:14.720 --> 00:20:17.190
at least at two or three of them.
00:20:17.200 --> 00:20:18.630
>> So you got to understand how all the
00:20:18.640 --> 00:20:20.070
moving parts especially with you know
00:20:20.080 --> 00:20:21.830
the AI world. So where's the you know
00:20:21.840 --> 00:20:24.310
the context live? How are they building
00:20:24.320 --> 00:20:26.070
agents? Uh how are they doing the
00:20:26.080 --> 00:20:27.669
transformations? All those kind of
00:20:27.679 --> 00:20:30.789
things. how storage work and then you
00:20:30.799 --> 00:20:32.390
should pretty much jump across the
00:20:32.400 --> 00:20:34.549
platforms over time. Yeah. Because you
00:20:34.559 --> 00:20:36.310
want to say is I I've got the
00:20:36.320 --> 00:20:37.669
experience, I've got the patterns, I can
00:20:37.679 --> 00:20:40.549
work on anything. Um if you're paid for
00:20:40.559 --> 00:20:43.029
efficiency, then again you go back to a
00:20:43.039 --> 00:20:44.789
single platform, right? And you're
00:20:44.799 --> 00:20:46.470
focused on reducing costs on that
00:20:46.480 --> 00:20:48.950
platform or doing the data work uh
00:20:48.960 --> 00:20:51.190
faster and cheaper than anybody else. So
00:20:51.200 --> 00:20:54.230
not ours but a fixed outcome price. So
00:20:54.240 --> 00:20:57.270
that's my view is like it depends. Hey,
00:20:57.280 --> 00:20:59.110
I would agree with that. Like from a W2
00:20:59.120 --> 00:21:01.350
standpoint, like if your focus is to be
00:21:01.360 --> 00:21:06.630
on a W2 um [clears throat] salary, then
00:21:06.640 --> 00:21:09.430
I would start with one and then branch
00:21:09.440 --> 00:21:11.110
out from there because over time you're
00:21:11.120 --> 00:21:12.470
going to have to bounce from one to the
00:21:12.480 --> 00:21:14.630
other, right? That's just as that's the
00:21:14.640 --> 00:21:17.590
reality of a W2 employee. um as a
00:21:17.600 --> 00:21:21.590
consultant like as a consulting agency
00:21:21.600 --> 00:21:25.830
like I focus on data bricks right with a
00:21:25.840 --> 00:21:29.190
side of fabric right [laughter]
00:21:29.200 --> 00:21:32.870
is kind of how I define it um and
00:21:32.880 --> 00:21:35.270
because like for for instance I'm both
00:21:35.280 --> 00:21:39.029
to your point like as a data bricks
00:21:39.039 --> 00:21:42.310
consultant I am certifi I'm a you know
00:21:42.320 --> 00:21:44.950
certified professional data engineer I
00:21:44.960 --> 00:21:47.350
am a data bricks partner
00:21:47.360 --> 00:21:49.830
I'm also a Microsoft partner partially
00:21:49.840 --> 00:21:51.590
because there's Azure data bricks right
00:21:51.600 --> 00:21:54.549
and that's a that's a thing so but and
00:21:54.559 --> 00:21:56.870
because of that I have that side of
00:21:56.880 --> 00:22:00.870
fabric but but yes 100% I agree with
00:22:00.880 --> 00:22:02.870
everything that you were going over over
00:22:02.880 --> 00:22:03.190
there
00:22:03.200 --> 00:22:06.310
>> and and for this non-Americans of us W2
00:22:06.320 --> 00:22:07.669
means you're an employee is that
00:22:07.679 --> 00:22:10.710
>> yes W2 is a FD right full-time
00:22:10.720 --> 00:22:12.870
>> okay all right excellent right all right
00:22:12.880 --> 00:22:15.029
got a comment or question from you so
00:22:15.039 --> 00:22:16.149
I'll just read it
00:22:16.159 --> 00:22:18.230
Uh, in the harness skills discussion, I
00:22:18.240 --> 00:22:19.510
think we have a bit of an expertise
00:22:19.520 --> 00:22:21.830
bias, an expert bias. Of course, us
00:22:21.840 --> 00:22:23.750
techies will build our own from scratch.
00:22:23.760 --> 00:22:25.430
We've always done that for all tech. The
00:22:25.440 --> 00:22:26.630
average consumer will want to get
00:22:26.640 --> 00:22:28.230
something off the shelf with very little
00:22:28.240 --> 00:22:30.549
config or customization. Yep. I I agree.
00:22:30.559 --> 00:22:31.430
What do you think?
00:22:31.440 --> 00:22:35.430
>> Yeah, 100%. We've always I like that it
00:22:35.440 --> 00:22:38.630
us techies have always uh I I always
00:22:38.640 --> 00:22:41.270
refer to it as us nerds, right? Us nerds
00:22:41.280 --> 00:22:43.510
have always just done it ourselves,
00:22:43.520 --> 00:22:45.669
right? were the ones that when we were
00:22:45.679 --> 00:22:47.990
sitting in front of Windows 95, we went
00:22:48.000 --> 00:22:49.830
through and basically reprogrammed
00:22:49.840 --> 00:22:52.230
everything to match all the sounds that
00:22:52.240 --> 00:22:54.630
we want to our sounds to sound like and
00:22:54.640 --> 00:22:56.870
like my opening and closing of my
00:22:56.880 --> 00:22:59.750
computer was like uh
00:22:59.760 --> 00:23:03.190
was a BonJovi song back in 95 and so you
00:23:03.200 --> 00:23:05.430
know I did all that right so but yeah to
00:23:05.440 --> 00:23:07.510
that point we'll make our own thing and
00:23:07.520 --> 00:23:10.149
then yeah I I could I could see that
00:23:10.159 --> 00:23:13.990
where from a monetization standpoint
00:23:14.000 --> 00:23:16.070
There there would be use cases outside
00:23:16.080 --> 00:23:17.750
of the techies for sure.
00:23:17.760 --> 00:23:19.110
>> Yeah. Yeah. I mean I think
00:23:19.120 --> 00:23:20.789
>> finance would probably be a really great
00:23:20.799 --> 00:23:23.270
one. Finance, healthcare. Um
00:23:23.280 --> 00:23:26.070
>> yeah, I think um we're in the we're in
00:23:26.080 --> 00:23:27.830
the beginning stages of of the
00:23:27.840 --> 00:23:29.510
technology and the wave, right? I mean
00:23:29.520 --> 00:23:31.830
there's a whole lot of unknowns. So, you
00:23:31.840 --> 00:23:34.390
know, everybody in who's a tech head in
00:23:34.400 --> 00:23:35.990
the space right now are probably groing
00:23:36.000 --> 00:23:38.549
a Mac Mini or two on their desktop, you
00:23:38.559 --> 00:23:40.070
know, to actually play around with it
00:23:40.080 --> 00:23:42.230
or, you know, some Raspberry Pies or
00:23:42.240 --> 00:23:44.470
some little little things to kind of
00:23:44.480 --> 00:23:46.230
isolate, you know, and sandbox what
00:23:46.240 --> 00:23:48.870
they're doing and see what happens. Um,
00:23:48.880 --> 00:23:50.149
it's going to take a while, I think,
00:23:50.159 --> 00:23:52.070
till we get to mainstream. And the
00:23:52.080 --> 00:23:53.830
reason I say that is if you look under
00:23:53.840 --> 00:23:55.270
the covers, if you try and do this
00:23:55.280 --> 00:23:57.029
stuff, there's just so many moving
00:23:57.039 --> 00:23:59.350
parts. there's so many many to many
00:23:59.360 --> 00:24:01.350
relationships and every week something
00:24:01.360 --> 00:24:04.549
changes you know so in Claude I'd built
00:24:04.559 --> 00:24:07.190
my own review process so whenever I did
00:24:07.200 --> 00:24:09.110
something I then sent it off to to do a
00:24:09.120 --> 00:24:12.230
review um but then a whole lot of new
00:24:12.240 --> 00:24:15.269
native Claude skills came out and so I
00:24:15.279 --> 00:24:16.870
just replaced it with that and I'm
00:24:16.880 --> 00:24:19.029
getting a much better response now and
00:24:19.039 --> 00:24:21.190
that was effectively free right but I
00:24:21.200 --> 00:24:24.149
had to know to switch it um I had to
00:24:24.159 --> 00:24:27.669
know it changed um you There's there's
00:24:27.679 --> 00:24:29.190
one that I've just been playing with at
00:24:29.200 --> 00:24:32.149
the moment, which is uh when you ask it
00:24:32.159 --> 00:24:33.909
to do something, if it's something kind
00:24:33.919 --> 00:24:35.190
of important, you're not just chatting
00:24:35.200 --> 00:24:38.390
with it. You you tell it to spawn an
00:24:38.400 --> 00:24:40.630
agent to do the task and then you tell
00:24:40.640 --> 00:24:42.549
it after that spawn another agent to
00:24:42.559 --> 00:24:45.830
review the task. Uh and that twoshot
00:24:45.840 --> 00:24:47.909
behavior tends to give you a better
00:24:47.919 --> 00:24:49.669
outcome. I mean, you burn more tokens,
00:24:49.679 --> 00:24:51.510
kill more butterflies, but the quality
00:24:51.520 --> 00:24:54.230
of the work coming back is better. So I
00:24:54.240 --> 00:24:55.830
think the technology is changing, the
00:24:55.840 --> 00:24:57.430
patterns are changing, all the moving
00:24:57.440 --> 00:24:59.029
parts are changing on such a regular
00:24:59.039 --> 00:25:01.430
basis that you know it might be a while
00:25:01.440 --> 00:25:03.029
before we get to consumer grade, but who
00:25:03.039 --> 00:25:03.909
knows?
00:25:03.919 --> 00:25:06.549
>> Well, and I I do want to make a comment
00:25:06.559 --> 00:25:08.230
about what you said there too, right?
00:25:08.240 --> 00:25:11.669
Because yes, like the sub agent concept
00:25:11.679 --> 00:25:14.310
of spawning sub agents and having like a
00:25:14.320 --> 00:25:16.950
orchestrator agent basically that's
00:25:16.960 --> 00:25:19.190
coaching each of those agents on okay
00:25:19.200 --> 00:25:20.870
and pushing back feedback to each of
00:25:20.880 --> 00:25:23.990
them as they report back is a fantastic
00:25:24.000 --> 00:25:26.149
pattern right it you get from a quality
00:25:26.159 --> 00:25:28.549
standpoint you get more work I wouldn't
00:25:28.559 --> 00:25:30.390
go out and use fabric 5 because you'll
00:25:30.400 --> 00:25:32.310
use up all your consumption credits in
00:25:32.320 --> 00:25:35.990
about two minutes but um but also like
00:25:36.000 --> 00:25:38.230
we have omni now and if If you haven't
00:25:38.240 --> 00:25:40.549
used OmniGen, it's really cool because
00:25:40.559 --> 00:25:44.070
you can wire in your Claude, you could
00:25:44.080 --> 00:25:47.990
wire in codeex, you could wire in um
00:25:48.000 --> 00:25:51.110
pretty much any other agent and
00:25:51.120 --> 00:25:53.190
basically have them debate on what the
00:25:53.200 --> 00:25:55.669
right path is and then spin up sub
00:25:55.679 --> 00:25:59.190
agents to actually do the work. And
00:25:59.200 --> 00:26:01.590
like you're using all of the LMS at
00:26:01.600 --> 00:26:05.269
once. And I've I've figured out that
00:26:05.279 --> 00:26:08.149
when Omni realizes that you're out of
00:26:08.159 --> 00:26:10.310
tokens with one, it will take the
00:26:10.320 --> 00:26:11.830
remaining workload and move it to the
00:26:11.840 --> 00:26:13.269
other one automatically.
00:26:13.279 --> 00:26:14.390
>> Really cool.
00:26:14.400 --> 00:26:15.830
>> I'm going to I'm going to go point my
00:26:15.840 --> 00:26:17.430
harness at that when I got I've got some
00:26:17.440 --> 00:26:19.830
more Fable. They they reset the Fable
00:26:19.840 --> 00:26:21.750
credit last night. So, it's like, yeah,
00:26:21.760 --> 00:26:23.510
I was done. I had the red line and now
00:26:23.520 --> 00:26:24.789
I'm back into it. So, I'm going to go
00:26:24.799 --> 00:26:26.470
maybe point it at that new harness and
00:26:26.480 --> 00:26:29.110
see if it does it. Um
00:26:29.120 --> 00:26:30.310
yeah, the cost thing is going to be
00:26:30.320 --> 00:26:31.990
interesting, right? Uh I mean the other
00:26:32.000 --> 00:26:33.269
one that I read the other day and I
00:26:33.279 --> 00:26:35.350
haven't tried it is you let's say you're
00:26:35.360 --> 00:26:38.870
in Claude code uh you basically get it
00:26:38.880 --> 00:26:41.110
to write what you want then you open up
00:26:41.120 --> 00:26:43.590
another session and you say to Claude
00:26:43.600 --> 00:26:46.070
that chat GPT wrote this code could it
00:26:46.080 --> 00:26:48.390
review it and somehow it changes its
00:26:48.400 --> 00:26:50.230
mindset [laughter]
00:26:50.240 --> 00:26:52.870
that it has a mind right so um I think
00:26:52.880 --> 00:26:55.990
it's about removing the memory as much
00:26:56.000 --> 00:26:57.830
as you can when the review cycle comes
00:26:57.840 --> 00:27:00.149
in um so it's not just patting itself on
00:27:00.159 --> 00:27:02.230
the back Uh yeah, I wrote that. Uh we're
00:27:02.240 --> 00:27:03.590
all good.
00:27:03.600 --> 00:27:06.549
>> Yeah, exactly. Uh
00:27:06.559 --> 00:27:09.430
yeah, it that again that's why I like
00:27:09.440 --> 00:27:11.830
playing with Omni Jet because it will
00:27:11.840 --> 00:27:13.830
automatically just take that context and
00:27:13.840 --> 00:27:17.990
back and it specifically has a debate um
00:27:18.000 --> 00:27:21.190
what are they called function basically
00:27:21.200 --> 00:27:22.950
that you could tell it to have them
00:27:22.960 --> 00:27:24.390
debate which is really cool.
00:27:24.400 --> 00:27:26.390
>> Oh like a red blue red blue team and
00:27:26.400 --> 00:27:29.190
then a judge. Yeah. Do you find that um
00:27:29.200 --> 00:27:31.190
you now have multiple sessions running
00:27:31.200 --> 00:27:33.269
at any one time and you're context
00:27:33.279 --> 00:27:36.070
switching as a human? Yeah,
00:27:36.080 --> 00:27:38.310
>> I I would hate to see some for somebody
00:27:38.320 --> 00:27:39.990
to look at my desktop literally right
00:27:40.000 --> 00:27:43.269
now because I have Claude open, I have
00:27:43.279 --> 00:27:47.510
Codeex open, I have Omni open, I have
00:27:47.520 --> 00:27:49.269
Gemini open, and I have data bricks
00:27:49.279 --> 00:27:52.310
open, right? I mean, it's
00:27:52.320 --> 00:27:54.230
it just kind of bouncing back and forth
00:27:54.240 --> 00:27:56.789
between them, kind of figuring out what
00:27:56.799 --> 00:27:59.190
the best answer is that I could get,
00:27:59.200 --> 00:27:59.909
right?
00:27:59.919 --> 00:28:02.710
>> Do you find that those tools make you
00:28:02.720 --> 00:28:04.710
more efficient or effective?
00:28:04.720 --> 00:28:07.669
>> It I think it depends. Sometimes it's a
00:28:07.679 --> 00:28:09.990
distraction, right? Sometimes I think
00:28:10.000 --> 00:28:12.789
it's a distraction. I also have come
00:28:12.799 --> 00:28:17.750
into teams where they were so heavily
00:28:17.760 --> 00:28:20.549
into AI. Um, and I think that we're
00:28:20.559 --> 00:28:22.389
starting I'm starting to see companies
00:28:22.399 --> 00:28:24.710
kind of backpedal on this, but they were
00:28:24.720 --> 00:28:27.269
so heavily into AI that they were doing
00:28:27.279 --> 00:28:31.750
everything in AI, even the simple stuff.
00:28:31.760 --> 00:28:35.190
But the simple stuff you could do
00:28:35.200 --> 00:28:38.549
quicker, right? then AI will think about
00:28:38.559 --> 00:28:41.750
it and return an answer. Especially if
00:28:41.760 --> 00:28:43.110
you [snorts] have people that are just
00:28:43.120 --> 00:28:45.750
defaulting to like medium or high for
00:28:45.760 --> 00:28:47.750
everything and and it's going to think
00:28:47.760 --> 00:28:49.909
and think and think until gives an
00:28:49.919 --> 00:28:52.230
answer. So I think sometimes it's a
00:28:52.240 --> 00:28:55.510
distraction. I think for other times
00:28:55.520 --> 00:28:57.909
like there's been times when I've asked
00:28:57.919 --> 00:29:01.909
a question and it gets me and and I've
00:29:01.919 --> 00:29:03.510
said we've talked about this before it
00:29:03.520 --> 00:29:06.070
gets you like 80 90% of the way there
00:29:06.080 --> 00:29:08.070
and you're just tweaking some stuff
00:29:08.080 --> 00:29:10.630
after that in those cases especially for
00:29:10.640 --> 00:29:12.630
the bigger more complex things I think
00:29:12.640 --> 00:29:15.510
that it I know that it speeds up my work
00:29:15.520 --> 00:29:17.750
in addition like you like I have a
00:29:17.760 --> 00:29:20.310
harness that I use for some of my like
00:29:20.320 --> 00:29:24.149
architecture audits and I just point the
00:29:24.159 --> 00:29:27.510
architecture to that uh to the LLM and
00:29:27.520 --> 00:29:29.909
basically it cranks out and evaluates it
00:29:29.919 --> 00:29:31.669
for me. And I still have to go through
00:29:31.679 --> 00:29:34.470
and like pick through the responses and
00:29:34.480 --> 00:29:36.630
just kind of and still evaluate the
00:29:36.640 --> 00:29:39.190
platform, but
00:29:39.200 --> 00:29:41.750
from a wording standpoint, from a kind
00:29:41.760 --> 00:29:43.990
of this is the biggest bang for your
00:29:44.000 --> 00:29:45.830
buck standpoint, it does a pretty good
00:29:45.840 --> 00:29:47.190
job.
00:29:47.200 --> 00:29:49.350
>> Yeah. Um, I think we're going to see a
00:29:49.360 --> 00:29:50.950
lot more routing agents come through
00:29:50.960 --> 00:29:53.029
where, you know, you ask a question and
00:29:53.039 --> 00:29:54.389
the first thing it's going to do is
00:29:54.399 --> 00:29:55.909
figure out where the cheapest which
00:29:55.919 --> 00:29:57.510
which model, you know, whether it's a
00:29:57.520 --> 00:30:00.230
vendor model or a model within a vendor
00:30:00.240 --> 00:30:02.389
uh is the cheapest to get you um there.
00:30:02.399 --> 00:30:03.909
I mean, lots of people are talking at
00:30:03.919 --> 00:30:05.350
the moment about running the heavy
00:30:05.360 --> 00:30:06.870
especially fable running the heavy
00:30:06.880 --> 00:30:08.630
models for planning and then running the
00:30:08.640 --> 00:30:11.110
cheaper models for execution. But, you
00:30:11.120 --> 00:30:13.669
know, the thing I find is I'm I'm vibe
00:30:13.679 --> 00:30:15.990
coding and I'm vibe hardnessing. And
00:30:16.000 --> 00:30:17.750
what I mean by that is I don't have an
00:30:17.760 --> 00:30:20.230
email framework, right? Actually, I've
00:30:20.240 --> 00:30:21.510
got Fable running in the background
00:30:21.520 --> 00:30:22.789
right now, burning some tokens, building
00:30:22.799 --> 00:30:25.269
me one to see what would happen. Um, but
00:30:25.279 --> 00:30:28.310
I make changes and I go that probably
00:30:28.320 --> 00:30:30.070
helped me uh yeah, gave me a better
00:30:30.080 --> 00:30:31.750
response in the future or it doesn't.
00:30:31.760 --> 00:30:33.510
Uh, versus, you know, if you read some
00:30:33.520 --> 00:30:35.430
of the anthropic papers, they're
00:30:35.440 --> 00:30:36.870
actually measuring well, they say they
00:30:36.880 --> 00:30:39.750
are they're measuring uh those changes.
00:30:39.760 --> 00:30:41.830
So, uh, yeah, we're we're in the early
00:30:41.840 --> 00:30:43.669
days for that one.
00:30:43.679 --> 00:30:45.430
>> We've got a few questions. So
00:30:45.440 --> 00:30:47.190
>> all right, you you take the next two
00:30:47.200 --> 00:30:49.669
which I think are both. Yeah.
00:30:49.679 --> 00:30:51.909
>> Andreas kind of going back to Andreas,
00:30:51.919 --> 00:30:55.029
he was asking if besides data bricks,
00:30:55.039 --> 00:30:57.909
what other tools do you guys rely on for
00:30:57.919 --> 00:31:03.110
ETL process or ELT processes? Um
00:31:03.120 --> 00:31:05.909
I guess I'll go first
00:31:05.919 --> 00:31:08.310
and primarily I'm in data bricks, right?
00:31:08.320 --> 00:31:11.590
I mean, again, that's kind of the focus
00:31:11.600 --> 00:31:17.110
of what I do, but um
00:31:17.120 --> 00:31:18.710
obviously I've done some that some
00:31:18.720 --> 00:31:22.950
companies have ADF, right? Uh and I've
00:31:22.960 --> 00:31:24.710
been in a couple of orgs where they were
00:31:24.720 --> 00:31:28.710
using Informatica. Um
00:31:28.720 --> 00:31:30.996
>> I thought you going to say ECIS. Yep.
00:31:31.006 --> 00:31:32.630
[laughter]
00:31:32.640 --> 00:31:35.029
>> I'd much rather SSIS than in than like
00:31:35.039 --> 00:31:38.310
Power Center. Um so so there there are
00:31:38.320 --> 00:31:41.190
those there are those uh there's
00:31:41.200 --> 00:31:42.870
definitely like usually the companies
00:31:42.880 --> 00:31:46.789
that are on ADF also still have like CIS
00:31:46.799 --> 00:31:48.870
packages out SSIS packages out there
00:31:48.880 --> 00:31:50.789
that they're still working to migrate
00:31:50.799 --> 00:31:53.590
but they're terrified to touch because
00:31:53.600 --> 00:31:56.549
somebody 15 years ago created that SSIS
00:31:56.559 --> 00:31:59.350
package and they don't want to touch it.
00:31:59.360 --> 00:32:01.269
>> It's usually running some finance end of
00:32:01.279 --> 00:32:03.350
month process too that they're relying
00:32:03.360 --> 00:32:04.470
on.
00:32:04.480 --> 00:32:07.110
>> Yeah. Um, yeah, we effectively built our
00:32:07.120 --> 00:32:10.389
own ETL tool on our platform. So, uh,
00:32:10.399 --> 00:32:13.029
yeah, that's what I use every day. Uh,
00:32:13.039 --> 00:32:14.389
yeah, I think if you looked at the
00:32:14.399 --> 00:32:16.630
market, I think DBT is still the number
00:32:16.640 --> 00:32:19.909
one transformation tool out there at the
00:32:19.919 --> 00:32:22.710
moment. Uh, I think with what Snowflake
00:32:22.720 --> 00:32:24.310
are doing and forking them in and with
00:32:24.320 --> 00:32:25.669
the changes of them being bought by
00:32:25.679 --> 00:32:28.310
Fiverr, when the Fiverr tax turns up, I
00:32:28.320 --> 00:32:31.430
think that will change. And did you see
00:32:31.440 --> 00:32:33.190
that they changed their pricing model
00:32:33.200 --> 00:32:34.789
for DBT? Oh,
00:32:34.799 --> 00:32:36.549
>> there's a surprise. Sorry, that's a fine
00:32:36.559 --> 00:32:38.870
joke. I haven't caught up on the five.
00:32:38.880 --> 00:32:40.070
We knew that was coming. Yeah,
00:32:40.080 --> 00:32:41.509
>> we knew that was coming when five.
00:32:41.519 --> 00:32:42.950
>> Do you know what the change is now?
00:32:42.960 --> 00:32:45.909
>> I I know that it's No, I've not I saw an
00:32:45.919 --> 00:32:48.070
article come out and I've I know that it
00:32:48.080 --> 00:32:50.149
increased the price structure, but I
00:32:50.159 --> 00:32:51.029
don't remember exactly.
00:32:51.039 --> 00:32:52.549
>> All right, we need to we need to write a
00:32:52.559 --> 00:32:53.750
question for ourselves so we can
00:32:53.760 --> 00:32:55.830
research that for next month and get in
00:32:55.840 --> 00:32:58.230
there. So um so I think what what we're
00:32:58.240 --> 00:32:59.669
seeing is I think we're seeing the core
00:32:59.679 --> 00:33:02.789
platforms you know GCP's got data form
00:33:02.799 --> 00:33:05.110
uh data bricks uh what have they got for
00:33:05.120 --> 00:33:07.430
the actual transformation ecod stuff is
00:33:07.440 --> 00:33:08.950
that the DT stuff isn't it
00:33:08.960 --> 00:33:11.509
>> DT mostly yeah the expectations within
00:33:11.519 --> 00:33:12.389
DT
00:33:12.399 --> 00:33:15.590
>> yeah and then snowflake fork dbt but
00:33:15.600 --> 00:33:17.509
they also bought SQL mesh didn't they
00:33:17.519 --> 00:33:18.070
>> y
00:33:18.080 --> 00:33:19.990
>> yeah so they kind of blending a few of
00:33:20.000 --> 00:33:23.909
those um I'm assuming fabric's still ADF
00:33:23.919 --> 00:33:25.909
right they haven't rebranded something
00:33:25.919 --> 00:33:27.669
else. And okay, so that's kind of where
00:33:27.679 --> 00:33:29.909
they're still sitting. Um, and then
00:33:29.919 --> 00:33:31.350
we've got the Oh,
00:33:31.360 --> 00:33:33.190
>> yeah. I was going to say they have
00:33:33.200 --> 00:33:35.190
Synapse, but it's really ADF within
00:33:35.200 --> 00:33:36.070
Synapse, right?
00:33:36.080 --> 00:33:38.549
>> Synapse is a database, right?
00:33:38.559 --> 00:33:41.909
>> Right. It's ADF within within fabric.
00:33:41.919 --> 00:33:42.230
So,
00:33:42.240 --> 00:33:44.870
>> yeah. Yeah. Uh, and then we've got, you
00:33:44.880 --> 00:33:47.110
know, and and these are all my opinions,
00:33:47.120 --> 00:33:49.269
but I have them. We got the legacy, you
00:33:49.279 --> 00:33:51.269
know, Oracle dis data integrator,
00:33:51.279 --> 00:33:52.630
whatever, ODI is still around.
00:33:52.640 --> 00:33:54.950
Informatica still lots of legy legacy
00:33:54.960 --> 00:33:56.950
stuff around um you know there's still
00:33:56.960 --> 00:33:58.310
lots of ETL tools that have been around
00:33:58.320 --> 00:34:01.509
for a long long time. And then we had
00:34:01.519 --> 00:34:04.149
the the wave before the modern data
00:34:04.159 --> 00:34:07.430
stack wave. So the wearcapes
00:34:07.440 --> 00:34:10.310
um they'll hate me saying it but who's
00:34:10.320 --> 00:34:13.589
the guys that were in the UK that do ETL
00:34:13.599 --> 00:34:16.310
and have cloudified it now? I can't
00:34:16.320 --> 00:34:18.069
remember what they were called. They're
00:34:18.079 --> 00:34:21.109
still around. Um and then we had like
00:34:21.119 --> 00:34:24.230
the new breed come out um you know to
00:34:24.240 --> 00:34:27.030
try and they were kind of either at
00:34:27.040 --> 00:34:28.710
right at the end of the mon data stack
00:34:28.720 --> 00:34:31.750
or straight after it trying to you know
00:34:31.760 --> 00:34:33.510
take it because effectively what happens
00:34:33.520 --> 00:34:36.310
is it's it's like a cake right
00:34:36.320 --> 00:34:38.389
>> everybody they kind of go down don't
00:34:38.399 --> 00:34:39.829
they? So you start up at the top being
00:34:39.839 --> 00:34:41.349
the cool new thing, right? And then you
00:34:41.359 --> 00:34:42.790
get a bit bigger and you go down to the
00:34:42.800 --> 00:34:43.990
next layer and you get a few more
00:34:44.000 --> 00:34:46.149
customers and you move into you go away
00:34:46.159 --> 00:34:48.629
from being 10,000 a month a year to you
00:34:48.639 --> 00:34:50.230
know half a million enterprise and then
00:34:50.240 --> 00:34:52.470
you go down bit bigger and then you get
00:34:52.480 --> 00:34:54.710
bought right by somebody and you sit on
00:34:54.720 --> 00:34:57.190
the on the bottom of the layer cake and
00:34:57.200 --> 00:34:59.910
>> go moldy. Um, so
00:34:59.920 --> 00:35:03.349
>> it's also like Dagster and Fiverr and
00:35:03.359 --> 00:35:05.430
like data bricks has like flow connect
00:35:05.440 --> 00:35:07.589
from a ELT standpoint, right? Just
00:35:07.599 --> 00:35:10.550
getting stuff out of source into it.
00:35:10.560 --> 00:35:11.109
>> But yeah.
00:35:11.119 --> 00:35:12.310
>> Yeah.
00:35:12.320 --> 00:35:15.910
>> Yep. All right. Uh, let's jump over to
00:35:15.920 --> 00:35:17.670
Mitchell. We'll come back and D to the
00:35:17.680 --> 00:35:18.790
other couple questions, but I'll just
00:35:18.800 --> 00:35:20.550
jump to Mitchell. Uh, so have you guys
00:35:20.560 --> 00:35:23.030
heard of AI cleanup consultants? Have
00:35:23.040 --> 00:35:24.550
you been approached for that in the
00:35:24.560 --> 00:35:26.230
context of data engineering projects?
00:35:26.240 --> 00:35:28.870
And are you familiar with demand?
00:35:28.880 --> 00:35:32.710
H wonder that like you're thinking he's
00:35:32.720 --> 00:35:34.950
thinking somebody's vibe coded some [ __ ]
00:35:34.960 --> 00:35:37.349
and now you've got to go in and make it
00:35:37.359 --> 00:35:39.510
run or make it accurate,
00:35:39.520 --> 00:35:42.230
>> make it secure and accurate. Like I
00:35:42.240 --> 00:35:43.910
always had in the back of my head that
00:35:43.920 --> 00:35:47.349
eventually like there would be this
00:35:47.359 --> 00:35:49.990
concept of we need somebody to come in
00:35:50.000 --> 00:35:51.750
and clean up what we've done because
00:35:51.760 --> 00:35:54.230
we've made a mess of it. um
00:35:54.240 --> 00:35:55.829
>> AI janitor,
00:35:55.839 --> 00:35:58.470
>> but but I've not I've not felt any of
00:35:58.480 --> 00:36:00.390
that demand at this point.
00:36:00.400 --> 00:36:01.750
>> But if you were going to do it, would
00:36:01.760 --> 00:36:05.510
you use a GI tool to clean up the GI
00:36:05.520 --> 00:36:07.030
tool?
00:36:07.040 --> 00:36:09.430
>> I don't know. It like like it it would
00:36:09.440 --> 00:36:11.589
be I think that would be one of those
00:36:11.599 --> 00:36:14.310
things where you've built a very custom
00:36:14.320 --> 00:36:17.190
harness that explicitly does that,
00:36:17.200 --> 00:36:20.790
right? But but man,
00:36:20.800 --> 00:36:21.510
>> if you're going to build it,
00:36:21.520 --> 00:36:22.870
>> it's like the snake eating itself,
00:36:22.880 --> 00:36:23.430
right?
00:36:23.440 --> 00:36:24.630
>> Yeah. But if you're going to build it,
00:36:24.640 --> 00:36:27.109
would you use it to scan the vibe coded
00:36:27.119 --> 00:36:30.069
crap code and reverse engineer it into
00:36:30.079 --> 00:36:33.109
something useful, or would you take the
00:36:33.119 --> 00:36:36.150
intent of that code and its outcome,
00:36:36.160 --> 00:36:38.630
point it at the beginning, and get it to
00:36:38.640 --> 00:36:40.870
rebuild again from scratch, ignoring the
00:36:40.880 --> 00:36:42.470
code in the middle?
00:36:42.480 --> 00:36:43.670
>> Which way would you go?
00:36:43.680 --> 00:36:45.670
>> I like the former. I I think I would
00:36:45.680 --> 00:36:47.829
give it the initial kind of use case and
00:36:47.839 --> 00:36:51.109
expected outcomes, have it read the
00:36:51.119 --> 00:36:53.510
code, figure out what it's done and what
00:36:53.520 --> 00:36:55.349
the gaps were between that and the
00:36:55.359 --> 00:36:58.390
original vision, and then basically
00:36:58.400 --> 00:37:01.589
start from scratch because
00:37:01.599 --> 00:37:03.589
I think
00:37:03.599 --> 00:37:07.430
I think that fixing it probably takes as
00:37:07.440 --> 00:37:10.710
much time as actually just building it.
00:37:10.720 --> 00:37:12.470
>> Yeah. See, I wouldn't let it see the
00:37:12.480 --> 00:37:14.230
messy middle because I think it would
00:37:14.240 --> 00:37:15.349
influence the models.
00:37:15.359 --> 00:37:16.870
>> Influence the model probably. Yeah.
00:37:16.880 --> 00:37:19.109
>> Well, not the model, the the GI tooling,
00:37:19.119 --> 00:37:20.550
right? I think once it's decent, it
00:37:20.560 --> 00:37:21.990
can't unsee it. I think it it'll
00:37:22.000 --> 00:37:23.430
naturally go in no matter how much you
00:37:23.440 --> 00:37:25.270
tell it and try and rewrite that
00:37:25.280 --> 00:37:28.870
spaghetti. So, um but maybe it won't. I
00:37:28.880 --> 00:37:30.870
don't know. Something to try. It'll be
00:37:30.880 --> 00:37:32.710
interesting to see uh what kind of
00:37:32.720 --> 00:37:33.750
markets out there for that. I think
00:37:33.760 --> 00:37:35.750
there's a market. I'm not sure how you
00:37:35.760 --> 00:37:38.230
how you do it. Uh I'll probably follow
00:37:38.240 --> 00:37:40.550
the big five, right? And all their proof
00:37:40.560 --> 00:37:42.950
of concepts. Um that might be the way to
00:37:42.960 --> 00:37:44.950
go. All right. Uh, next question from
00:37:44.960 --> 00:37:47.030
him. Uh, where are you guys seeing the
00:37:47.040 --> 00:37:49.030
most demand for ML adjacent data
00:37:49.040 --> 00:37:51.510
engineers where you'd be happy with ML
00:37:51.520 --> 00:37:54.150
engineering overlap? What platforms and
00:37:54.160 --> 00:37:57.190
what tools do you just suggest? Oh,
00:37:57.200 --> 00:37:59.030
machine learning stuff. That's probably
00:37:59.040 --> 00:38:00.790
you probably play in that space a lot
00:38:00.800 --> 00:38:04.630
more than I do. I mean, I hate to sound
00:38:04.640 --> 00:38:06.069
like a broken record, but if you're
00:38:06.079 --> 00:38:08.069
going to be in ML, I think that data
00:38:08.079 --> 00:38:11.589
bricks is is where you need to
00:38:11.599 --> 00:38:13.109
specialize in right now. I mean
00:38:13.119 --> 00:38:14.950
obviously you need to understand the
00:38:14.960 --> 00:38:18.230
fundamentals of just data science and
00:38:18.240 --> 00:38:23.270
machine learning. Um I think that
00:38:23.280 --> 00:38:25.109
I think that some of those ML engineers
00:38:25.119 --> 00:38:27.589
will start to venture into AI
00:38:27.599 --> 00:38:29.510
engineering, right? I think that there's
00:38:29.520 --> 00:38:31.270
a natural progression there for some of
00:38:31.280 --> 00:38:34.069
that work. Um
00:38:34.079 --> 00:38:35.670
so let me look at this question again.
00:38:35.680 --> 00:38:36.870
Most demand where are you seeing the
00:38:36.880 --> 00:38:39.750
most demand for ML adjacent engineer ML
00:38:39.760 --> 00:38:42.870
adjacent data engineers?
00:38:42.880 --> 00:38:44.390
H
00:38:44.400 --> 00:38:46.470
>> yeah maybe maybe give us a bit more
00:38:46.480 --> 00:38:48.710
detail about what you mean by ML Jason
00:38:48.720 --> 00:38:50.470
but
00:38:50.480 --> 00:38:53.589
we saw you know a while ago data science
00:38:53.599 --> 00:38:54.950
become hot right
00:38:54.960 --> 00:38:56.630
>> and lots of people going through and
00:38:56.640 --> 00:39:00.069
learning to to do the science of data
00:39:00.079 --> 00:39:01.430
and then as soon as they went into an
00:39:01.440 --> 00:39:03.190
organization they became data engineers
00:39:03.200 --> 00:39:04.950
because 80% of the time was getting the
00:39:04.960 --> 00:39:07.270
data that didn't exist. We see lots of
00:39:07.280 --> 00:39:09.910
people getting trained as data engineers
00:39:09.920 --> 00:39:12.150
but or or analysts but and becoming
00:39:12.160 --> 00:39:14.390
analytics engineers right and kind of
00:39:14.400 --> 00:39:18.470
doing the transformationy stuff. Um what
00:39:18.480 --> 00:39:19.750
are you seeing though from a machine
00:39:19.760 --> 00:39:22.310
learning right cuz in my head it's big
00:39:22.320 --> 00:39:25.990
fan companies and people who are experts
00:39:26.000 --> 00:39:28.550
in the science of machine learning and
00:39:28.560 --> 00:39:31.270
that probably is about 5% of everybody
00:39:31.280 --> 00:39:33.510
and now everybody else is just a not
00:39:33.520 --> 00:39:35.430
just but everybody else is just a data
00:39:35.440 --> 00:39:37.109
practitioner right
00:39:37.119 --> 00:39:40.150
>> yeah and even so adjacent was a poor
00:39:40.160 --> 00:39:43.270
choice of words yeah um where are you
00:39:43.280 --> 00:39:47.430
feed models so honestly the So when you
00:39:47.440 --> 00:39:50.150
go out and look at job postings,
00:39:50.160 --> 00:39:51.829
you're going to see a lot of those just
00:39:51.839 --> 00:39:53.750
listed as data engineers. Like you're
00:39:53.760 --> 00:39:55.670
not going to see ML data engineer.
00:39:55.680 --> 00:39:57.589
You're not going to see necessarily ML
00:39:57.599 --> 00:39:59.190
engineer. You're going to see data
00:39:59.200 --> 00:40:01.030
engineer as the title or senior data
00:40:01.040 --> 00:40:02.710
engineer or whatever as the title. And
00:40:02.720 --> 00:40:04.069
then when you look at the description,
00:40:04.079 --> 00:40:06.390
you're going to be like, oh, this is a
00:40:06.400 --> 00:40:08.150
machine learning organization.
00:40:08.160 --> 00:40:09.750
>> Ah, okay. So he's talking about an
00:40:09.760 --> 00:40:11.270
engineer that feeds a machine learning
00:40:11.280 --> 00:40:13.589
engineer and they're kind of the posy.
00:40:13.599 --> 00:40:17.190
Um, yeah, I I listened to a a
00:40:17.200 --> 00:40:19.349
presentation by a mate of mine, Sha Mur,
00:40:19.359 --> 00:40:21.109
ages ago. Well, no, actually, I didn't.
00:40:21.119 --> 00:40:22.870
That's a lie. Uh, he told me a story,
00:40:22.880 --> 00:40:24.710
right? Uh, he was in a presentation,
00:40:24.720 --> 00:40:25.829
that's right, by somebody from one of
00:40:25.839 --> 00:40:28.550
the fan companies. Um, and the guy was
00:40:28.560 --> 00:40:30.310
talking, you know, about what they did.
00:40:30.320 --> 00:40:32.950
And Sean, he's a Kiwi, so he asked the
00:40:32.960 --> 00:40:35.430
blunt questions. He said, "So, how do
00:40:35.440 --> 00:40:36.950
you get your data?" And the guy looks at
00:40:36.960 --> 00:40:38.550
him and he goes, "What do you mean?"
00:40:38.560 --> 00:40:40.390
He's like, "Well, when you need to feed
00:40:40.400 --> 00:40:41.750
data for your model, how do you do it?"
00:40:41.760 --> 00:40:43.829
He goes, "Well, simple. I just hop on
00:40:43.839 --> 00:40:45.910
Slack, do a request, and then a couple
00:40:45.920 --> 00:40:47.753
minutes later the data turns up.
00:40:47.763 --> 00:40:50.150
[laughter] And he's like,
00:40:50.160 --> 00:40:52.230
"Hey," he goes, "Well, we've got, you
00:40:52.240 --> 00:40:54.310
know, 5,000." I made that number up, but
00:40:54.320 --> 00:40:56.630
you know, 5,000 data engineers for
00:40:56.640 --> 00:40:57.910
machine learning and all they do is sit
00:40:57.920 --> 00:40:59.109
there waiting for our requests and they
00:40:59.119 --> 00:41:01.270
munch the data and give it to us. I
00:41:01.280 --> 00:41:02.710
don't think that's normal in most
00:41:02.720 --> 00:41:05.109
organizations, and I'm with you. I think
00:41:05.119 --> 00:41:06.870
>> most organizations, unless you're one of
00:41:06.880 --> 00:41:09.829
the big things, is uh hybrid, hybrid
00:41:09.839 --> 00:41:11.670
skills, hybrid roles. We're seeing a lot
00:41:11.680 --> 00:41:13.349
more of that. So that that's be my
00:41:13.359 --> 00:41:14.069
answer.
00:41:14.079 --> 00:41:16.950
>> Yeah. Yeah. I I would agree.
00:41:16.960 --> 00:41:17.349
>> Cool.
00:41:17.359 --> 00:41:19.270
>> Excellent. All right, let's pop back up.
00:41:19.280 --> 00:41:21.670
Uh so and see does a lot of working with
00:41:21.680 --> 00:41:23.349
teams in Latin America who are looking
00:41:23.359 --> 00:41:25.190
to overhaul their data infrastructure.
00:41:25.200 --> 00:41:28.150
Common theme uh is the teams eaggan to
00:41:28.160 --> 00:41:30.790
adopt modern data technologies. Um and
00:41:30.800 --> 00:41:32.150
they need the right roadmap to bridge
00:41:32.160 --> 00:41:33.990
the gap. What do you suggest to connect
00:41:34.000 --> 00:41:35.430
with them and help them adopt the best
00:41:35.440 --> 00:41:38.470
use cases? Ooh, so this is back, you
00:41:38.480 --> 00:41:40.309
know, in in the past, they would have
00:41:40.319 --> 00:41:42.950
just bought the 25 most popular modern
00:41:42.960 --> 00:41:45.190
data stack tools, uh, and then spent
00:41:45.200 --> 00:41:47.349
half their time integrating it, uh,
00:41:47.359 --> 00:41:48.550
hoping that, you know, the vendors
00:41:48.560 --> 00:41:50.470
didn't go under, uh, and then doing the
00:41:50.480 --> 00:41:52.470
work. What would you do now, though?
00:41:52.480 --> 00:41:54.790
Like, I think you'd go platform now.
00:41:54.800 --> 00:41:57.349
Like, you just pick a base platform and
00:41:57.359 --> 00:41:59.510
then fill the gaps, wouldn't you?
00:41:59.520 --> 00:42:01.910
>> Yeah. Yeah. In fact, like there's so
00:42:01.920 --> 00:42:05.109
much I mean, think about it. Snowflake,
00:42:05.119 --> 00:42:07.349
Snowflake or data bricks. Now you could
00:42:07.359 --> 00:42:09.109
just go out and write Python scripts,
00:42:09.119 --> 00:42:11.109
right? The the reality is that you don't
00:42:11.119 --> 00:42:13.829
need this data tool sprawl that you see
00:42:13.839 --> 00:42:16.069
that we used to see all the time of 25
00:42:16.079 --> 00:42:18.870
different tools to get everything done.
00:42:18.880 --> 00:42:20.309
Instead, you're hiring for the skill
00:42:20.319 --> 00:42:22.470
set, right? You're hiring human people
00:42:22.480 --> 00:42:24.309
or
00:42:24.319 --> 00:42:28.710
AI augmented people that will do the
00:42:28.720 --> 00:42:30.550
actual programming that goes along with
00:42:30.560 --> 00:42:34.710
it. um instead of hiring people that
00:42:34.720 --> 00:42:37.430
understand how to use like low code no
00:42:37.440 --> 00:42:40.470
code no code tools that
00:42:40.480 --> 00:42:43.829
are piece parted together because like
00:42:43.839 --> 00:42:46.230
that's where C data started right uh
00:42:46.240 --> 00:42:47.589
that's where a lot of C data started
00:42:47.599 --> 00:42:49.430
because they made these custom
00:42:49.440 --> 00:42:51.750
connectors to all these different
00:42:51.760 --> 00:42:55.109
platforms that natively SSIS or other
00:42:55.119 --> 00:42:58.710
tools weren't able to connect to. Um but
00:42:58.720 --> 00:43:00.870
that's that yeah that's my thing is pick
00:43:00.880 --> 00:43:02.550
a tool pick a platform whether it's
00:43:02.560 --> 00:43:05.430
snowflake or data bricks or you know
00:43:05.440 --> 00:43:10.309
fabric or GCP um and and learn it and go
00:43:10.319 --> 00:43:13.510
with it right um and as far as not
00:43:13.520 --> 00:43:16.230
knowing what the right road map is to
00:43:16.240 --> 00:43:18.790
bridge that gap
00:43:18.800 --> 00:43:20.470
like that's part of that's part of what
00:43:20.480 --> 00:43:23.990
I do right that's I come into you know I
00:43:24.000 --> 00:43:26.390
have a potential engagement here next
00:43:26.400 --> 00:43:29.109
month where they're looking to, you
00:43:29.119 --> 00:43:31.910
know, upskill their team from Snowflake
00:43:31.920 --> 00:43:34.069
into data bricks and they need somebody
00:43:34.079 --> 00:43:36.150
to come in and do some training and
00:43:36.160 --> 00:43:37.750
bridge that gap between where they're at
00:43:37.760 --> 00:43:39.030
and where they need to be for data
00:43:39.040 --> 00:43:43.430
bricks. Um, but that is taking time to
00:43:43.440 --> 00:43:45.829
evaluate the current team, taking time
00:43:45.839 --> 00:43:48.630
to evaluate exactly where they want to
00:43:48.640 --> 00:43:51.750
be within that data brick stack and then
00:43:51.760 --> 00:43:54.150
creating a custom road map for that team
00:43:54.160 --> 00:43:56.069
to get them there. That way you're
00:43:56.079 --> 00:43:57.589
trimming out the fat that they don't
00:43:57.599 --> 00:43:59.430
need because there's a lot of overlap in
00:43:59.440 --> 00:44:02.550
concept between the two and you're
00:44:02.560 --> 00:44:04.870
actually targeting and doing targeting
00:44:04.880 --> 00:44:07.349
training for that group.
00:44:07.359 --> 00:44:07.990
>> That's mine.
00:44:08.000 --> 00:44:09.190
>> Yeah. Yeah. Yeah, I think when I used to
00:44:09.200 --> 00:44:10.950
do data blueprints when I was consulting
00:44:10.960 --> 00:44:14.390
doing that, uh I I think about it as a
00:44:14.400 --> 00:44:16.470
bunch of Lego blocks and what I', you
00:44:16.480 --> 00:44:17.990
know, I'd had the Lego blocks that
00:44:18.000 --> 00:44:19.270
normally go together,
00:44:19.280 --> 00:44:21.109
>> you know, the the capabilities you need
00:44:21.119 --> 00:44:24.150
and then I would do the work to keep
00:44:24.160 --> 00:44:25.750
myself up to date with the
00:44:25.760 --> 00:44:27.670
interoperability of the Lego blocks. You
00:44:27.680 --> 00:44:29.430
know, what if you're kind of on this
00:44:29.440 --> 00:44:30.630
then you're going to use these ones
00:44:30.640 --> 00:44:32.230
because they're more natively integrated
00:44:32.240 --> 00:44:35.109
than those ones. But then it would all
00:44:35.119 --> 00:44:37.270
be about the organization. You know, the
00:44:37.280 --> 00:44:39.030
the stack or the road map or the
00:44:39.040 --> 00:44:41.349
blueprint for an organization with a
00:44:41.359 --> 00:44:43.829
threeperson data team versus one with
00:44:43.839 --> 00:44:46.390
five 15 person data teams is very very
00:44:46.400 --> 00:44:48.870
different. So I think you got to fit and
00:44:48.880 --> 00:44:51.109
then my view now is because we know that
00:44:51.119 --> 00:44:52.550
data bricks and snowflake in those
00:44:52.560 --> 00:44:55.349
platforms right like they they won't
00:44:55.359 --> 00:44:57.670
survive in the form they are right now.
00:44:57.680 --> 00:44:59.270
Yeah. Nobody ever does right. Are they
00:44:59.280 --> 00:45:01.750
ever going to get bigger? They or
00:45:01.760 --> 00:45:03.990
they're going to get bought or they're
00:45:04.000 --> 00:45:06.150
going to die, which is probably unlikely
00:45:06.160 --> 00:45:09.270
now. Um, but what I would now start
00:45:09.280 --> 00:45:11.589
focusing on is holding the context of
00:45:11.599 --> 00:45:13.589
all my work outside of the tooling that
00:45:13.599 --> 00:45:16.630
I use. So, I would find ways of storing
00:45:16.640 --> 00:45:18.950
the business logic um completely
00:45:18.960 --> 00:45:21.430
separately and then using it to hydrate
00:45:21.440 --> 00:45:24.230
whatever tool I'm using. Um, so I store
00:45:24.240 --> 00:45:26.230
it over here and then I write really
00:45:26.240 --> 00:45:28.309
smart stuff that then hydrates into that
00:45:28.319 --> 00:45:29.990
tool. Now, you're not going to be able
00:45:30.000 --> 00:45:32.230
to natively go from one ETL tool to
00:45:32.240 --> 00:45:34.390
another in a drop of a hat. But the
00:45:34.400 --> 00:45:36.390
trans the ability to move between them
00:45:36.400 --> 00:45:38.069
now with especially with the LLM helping
00:45:38.079 --> 00:45:40.790
us is uh is a lot faster than it used to
00:45:40.800 --> 00:45:43.190
be. So for me, it's that that context is
00:45:43.200 --> 00:45:45.510
the the key thing that I would keep
00:45:45.520 --> 00:45:47.589
separate from anything that I use these
00:45:47.599 --> 00:45:48.710
days.
00:45:48.720 --> 00:45:50.790
>> And it I want to talk to real quickly
00:45:50.800 --> 00:45:54.390
that transition, right? So [snorts] I
00:45:54.400 --> 00:45:55.750
have these examples because I do
00:45:55.760 --> 00:45:57.750
coaching. So I have a guy that I coach
00:45:57.760 --> 00:46:00.550
in Poland and his company migrated from
00:46:00.560 --> 00:46:03.349
Snowflake to data bricks. And the cool
00:46:03.359 --> 00:46:05.670
thing about that today that we didn't
00:46:05.680 --> 00:46:09.670
have 10 5 10 years ago, right, is that a
00:46:09.680 --> 00:46:11.670
lot of their workload in Snowflake was
00:46:11.680 --> 00:46:14.950
written in Python anyways. And so it was
00:46:14.960 --> 00:46:16.870
already in git. It was already version
00:46:16.880 --> 00:46:19.190
controlled. They had to revise their
00:46:19.200 --> 00:46:23.430
scripting a bit to fit data bricks, but
00:46:23.440 --> 00:46:26.230
that was a relatively lighter lift than
00:46:26.240 --> 00:46:28.470
it would have been 10 years ago, right?
00:46:28.480 --> 00:46:31.030
That's it's the nice thing about being
00:46:31.040 --> 00:46:32.630
able to do that scripting in those
00:46:32.640 --> 00:46:35.510
systems and and having git behind all of
00:46:35.520 --> 00:46:37.589
it that we just there was a time in our
00:46:37.599 --> 00:46:39.430
world that we didn't have all that.
00:46:39.440 --> 00:46:41.670
>> Yeah. So, keep keep that context
00:46:41.680 --> 00:46:43.670
separate, keep it version controlled,
00:46:43.680 --> 00:46:45.670
and then, you know, it's effectively a
00:46:45.680 --> 00:46:47.829
language change. It's still not free. I
00:46:47.839 --> 00:46:49.829
still see organizations changing
00:46:49.839 --> 00:46:52.150
platforms for the wrong reasons.
00:46:52.160 --> 00:46:53.510
>> Yeah.
00:46:53.520 --> 00:46:55.109
>> Yeah. And thinking it's going to solve
00:46:55.119 --> 00:46:57.430
all their problems without And the
00:46:57.440 --> 00:47:01.510
reality is that there are some bad
00:47:01.520 --> 00:47:03.270
patterns. There are some anti-atterns
00:47:03.280 --> 00:47:05.190
that they're using now that are really
00:47:05.200 --> 00:47:07.829
driving up their cloud costs and then
00:47:07.839 --> 00:47:09.589
they migrate to this new tool because
00:47:09.599 --> 00:47:11.589
there was some promise that it was going
00:47:11.599 --> 00:47:14.150
to reduce their overall cloud costs. And
00:47:14.160 --> 00:47:16.230
then as things continue to scale, they
00:47:16.240 --> 00:47:17.750
still the same seem the same problem
00:47:17.760 --> 00:47:19.750
because they did just lift and shift
00:47:19.760 --> 00:47:21.190
into that new form.
00:47:21.200 --> 00:47:22.710
>> That would probably save their costs in
00:47:22.720 --> 00:47:23.990
the first year, right? Because they got
00:47:24.000 --> 00:47:25.910
the sniff sniff snort offering from the
00:47:25.920 --> 00:47:27.750
vendor, right? Where you free credits or
00:47:27.760 --> 00:47:29.270
free compute for the first year,
00:47:29.280 --> 00:47:30.870
>> first hits free, right? Yeah.
00:47:30.880 --> 00:47:31.990
>> Google do that, don't they? They give
00:47:32.000 --> 00:47:33.670
you the first year free. So you can
00:47:33.680 --> 00:47:36.069
really get it into IV pumped into into
00:47:36.079 --> 00:47:37.990
the yarm. And the problem with that is
00:47:38.000 --> 00:47:39.670
it just causes anti patents because
00:47:39.680 --> 00:47:41.430
people don't need to care about the
00:47:41.440 --> 00:47:43.990
costs, right? It's just lazy. Like I put
00:47:44.000 --> 00:47:47.349
my hand up. I'm Opus 4.8 max all the
00:47:47.359 --> 00:47:49.109
way, baby. Until I get close to running
00:47:49.119 --> 00:47:51.349
out of credit, out of uh token limits
00:47:51.359 --> 00:47:54.550
and then I then I care. Um so, you know,
00:47:54.560 --> 00:47:56.309
you open up the IV, you got to you got
00:47:56.319 --> 00:47:57.990
to take it all. Um there's a question
00:47:58.000 --> 00:48:00.870
here from Andreas. Uh any good advice uh
00:48:00.880 --> 00:48:02.309
around the hiring process for an
00:48:02.319 --> 00:48:04.950
aspiring data engineer um and projects
00:48:04.960 --> 00:48:07.670
and that? Well, my my advice is uh go
00:48:07.680 --> 00:48:10.550
and get Chris to mentor you. He's bloody
00:48:10.560 --> 00:48:13.030
good at it. Um, but that's how you make
00:48:13.040 --> 00:48:15.670
your living, Chris. So, you know,
00:48:15.680 --> 00:48:16.309
>> if [laughter]
00:48:16.319 --> 00:48:17.430
you feel up to it, give us some
00:48:17.440 --> 00:48:18.630
freebies.
00:48:18.640 --> 00:48:21.670
>> Yeah. No, I mean I I um I'll tell you
00:48:21.680 --> 00:48:24.710
that like I do it
00:48:24.720 --> 00:48:27.990
looking at the time. So,
00:48:28.000 --> 00:48:29.829
I'll tell you a little bit of a story
00:48:29.839 --> 00:48:30.950
and then I'm going to get into the
00:48:30.960 --> 00:48:34.390
answer there. But at first, I did free
00:48:34.400 --> 00:48:37.589
mentoring and I found the first like
00:48:37.599 --> 00:48:39.190
three people that I mentored like I
00:48:39.200 --> 00:48:40.950
there was no cost. I was just doing it
00:48:40.960 --> 00:48:42.230
because I wanted to do it and wanted to
00:48:42.240 --> 00:48:44.710
give back and it was it was awesome,
00:48:44.720 --> 00:48:48.790
right? It it felt good. But about three
00:48:48.800 --> 00:48:51.030
weeks in with anybody, I would find that
00:48:51.040 --> 00:48:52.549
there was no follow-through by the
00:48:52.559 --> 00:48:54.950
person that I was mentoring, right? And
00:48:54.960 --> 00:48:57.910
um my wife, who is a mental health
00:48:57.920 --> 00:49:00.870
practitioner, she has her own uh her own
00:49:00.880 --> 00:49:03.030
practice where she does uh mental health
00:49:03.040 --> 00:49:05.750
therapy, but she was like, "It's because
00:49:05.760 --> 00:49:07.589
they don't have any skin in the game,
00:49:07.599 --> 00:49:10.390
right?" And so I found that when I start
00:49:10.400 --> 00:49:11.829
charging people, that's that's why I
00:49:11.839 --> 00:49:13.510
charge people as much as anything else
00:49:13.520 --> 00:49:16.230
because one, it's part of how I make my
00:49:16.240 --> 00:49:20.470
living, but now, but two, people are
00:49:20.480 --> 00:49:22.309
dedicated when they have skin in the
00:49:22.319 --> 00:49:24.549
game, for a lack of a better term. Um,
00:49:24.559 --> 00:49:26.950
okay. So, getting into the actual answer
00:49:26.960 --> 00:49:29.750
to your question. So, this is the way I
00:49:29.760 --> 00:49:31.510
structure my coaching program, right? I
00:49:31.520 --> 00:49:33.910
have people send me their resume. They
00:49:33.920 --> 00:49:35.990
identify five or ten jobs that they look
00:49:36.000 --> 00:49:37.750
at those job postings and they're like,
00:49:37.760 --> 00:49:40.230
"Yes, this is exactly the job that I
00:49:40.240 --> 00:49:42.150
want to get into." I have them
00:49:42.160 --> 00:49:44.069
specifically identify an industry that
00:49:44.079 --> 00:49:46.470
they want to target. And I have them do
00:49:46.480 --> 00:49:48.870
a self assessment. From there, I take
00:49:48.880 --> 00:49:52.150
all of that information and I'll go out
00:49:52.160 --> 00:49:54.870
and research that industry and find like
00:49:54.880 --> 00:49:58.230
three to five business related questions
00:49:58.240 --> 00:50:00.230
that can be answered with data, right?
00:50:00.240 --> 00:50:02.390
Four that are specific to that industry.
00:50:02.400 --> 00:50:05.030
And then I go out find some public data
00:50:05.040 --> 00:50:07.910
sources if I can that relate to
00:50:07.920 --> 00:50:09.430
answering that business question and
00:50:09.440 --> 00:50:11.270
then I'll build a full end to-end
00:50:11.280 --> 00:50:14.150
project around that business question
00:50:14.160 --> 00:50:16.790
and [clears throat] and those data sets.
00:50:16.800 --> 00:50:18.390
And that way when you're building your
00:50:18.400 --> 00:50:20.870
portfolio, they're not this generic
00:50:20.880 --> 00:50:23.109
Titanic or COVID data set, right? Which
00:50:23.119 --> 00:50:24.870
you if for those of you that follow me,
00:50:24.880 --> 00:50:27.829
you have heard me say if I see those on
00:50:27.839 --> 00:50:29.430
a resume, I'm probably just tossing it
00:50:29.440 --> 00:50:32.870
out, right? Um, but
00:50:32.880 --> 00:50:35.030
they're specific to the business. When
00:50:35.040 --> 00:50:36.069
somebody [clears throat] looks at that
00:50:36.079 --> 00:50:37.670
resume, then they look at that resume
00:50:37.680 --> 00:50:40.309
and they say, "Oh, this is an actual
00:50:40.319 --> 00:50:42.069
issue that we're having. I'm going to go
00:50:42.079 --> 00:50:44.549
out and actually look at this repo."
00:50:44.559 --> 00:50:46.230
Right? Otherwise, they're a hiring
00:50:46.240 --> 00:50:47.270
manager is not going to look at that
00:50:47.280 --> 00:50:49.030
repo. If they look at it, they see some
00:50:49.040 --> 00:50:50.309
projects, they're like, "Cool. Okay,
00:50:50.319 --> 00:50:51.430
they've done some projects and they're
00:50:51.440 --> 00:50:53.510
going to move on." But if they look at
00:50:53.520 --> 00:50:55.670
it and see, oh, that's something
00:50:55.680 --> 00:50:57.510
literally we're working on now. They're
00:50:57.520 --> 00:50:58.790
going to go and take a look and see what
00:50:58.800 --> 00:51:00.549
you've done, see what types of decisions
00:51:00.559 --> 00:51:04.150
you made to do those things. Um, and
00:51:04.160 --> 00:51:05.910
you're going to get an interview at that
00:51:05.920 --> 00:51:07.750
point, right? Unless they go out and
00:51:07.760 --> 00:51:09.109
look at your repo and you have a bunch
00:51:09.119 --> 00:51:11.510
of exposed secrets, right? Then they're
00:51:11.520 --> 00:51:13.190
probably going to call somebody else and
00:51:13.200 --> 00:51:14.710
have you reported.
00:51:14.720 --> 00:51:16.309
Um,
00:51:16.319 --> 00:51:18.470
but that's that's the big thing is build
00:51:18.480 --> 00:51:21.190
build build like three projects that are
00:51:21.200 --> 00:51:22.870
very specific to the industry that you
00:51:22.880 --> 00:51:25.510
want to get into that are very that are
00:51:25.520 --> 00:51:27.670
related to the tech stack that you
00:51:27.680 --> 00:51:29.670
identify in those five to 10 job
00:51:29.680 --> 00:51:33.190
postings and do the work. Be sure that
00:51:33.200 --> 00:51:35.190
you document all of your decision points
00:51:35.200 --> 00:51:36.630
along the way because that's what's
00:51:36.640 --> 00:51:38.309
going to tell somebody that a real
00:51:38.319 --> 00:51:42.790
person did it, not just AI. And then
00:51:42.800 --> 00:51:44.230
when you're putting together your
00:51:44.240 --> 00:51:45.910
resume, you know, you've got your
00:51:45.920 --> 00:51:47.349
professional summary, you have your
00:51:47.359 --> 00:51:49.430
skills, and then you have your projects,
00:51:49.440 --> 00:51:53.589
right, with the actual like impactful
00:51:53.599 --> 00:51:56.230
pieces of those projects similarly to
00:51:56.240 --> 00:51:58.950
how you would actually bullet out your
00:51:58.960 --> 00:52:01.990
uh actual work experience. That's
00:52:02.000 --> 00:52:03.349
>> good good point there about because
00:52:03.359 --> 00:52:06.390
everybody can now just use LLM to code,
00:52:06.400 --> 00:52:09.349
right? Um so the you know making the
00:52:09.359 --> 00:52:10.710
decision points it looks like it's a
00:52:10.720 --> 00:52:12.870
human. Uh the other thing I'll add is
00:52:12.880 --> 00:52:16.069
yeah so I mentor uh university students
00:52:16.079 --> 00:52:18.549
um every now and again just when they
00:52:18.559 --> 00:52:21.270
need to do a project. Uh so I finished
00:52:21.280 --> 00:52:23.349
mentoring um one who was doing a masters
00:52:23.359 --> 00:52:26.630
in data science and you know I got him
00:52:26.640 --> 00:52:28.630
to do a portfolio right to do a couple
00:52:28.640 --> 00:52:31.270
of projects and have a portfolio website
00:52:31.280 --> 00:52:34.150
and the key there was it wasn't about
00:52:34.160 --> 00:52:37.109
the code right it it was about the
00:52:37.119 --> 00:52:38.710
ability to have a clear conversation
00:52:38.720 --> 00:52:41.190
with a potential employer later and so
00:52:41.200 --> 00:52:43.829
what he found was uh you know gave him
00:52:43.839 --> 00:52:45.270
it wasn't Titanic right it was things
00:52:45.280 --> 00:52:47.190
that he was interested in um because he
00:52:47.200 --> 00:52:48.390
didn't really know which industry he was
00:52:48.400 --> 00:52:49.990
going to go into So, you know, it was
00:52:50.000 --> 00:52:51.430
problems that were interesting to him to
00:52:51.440 --> 00:52:53.270
solve. Uh, and he solved them and he
00:52:53.280 --> 00:52:54.950
wrote blogs about it. And, you know,
00:52:54.960 --> 00:52:56.470
that was unnatural to him, but that was
00:52:56.480 --> 00:52:58.390
part of the forcing function is every
00:52:58.400 --> 00:53:00.549
week he had to write write the blog. Did
00:53:00.559 --> 00:53:02.790
two things. Uh, forced him to do a bit
00:53:02.800 --> 00:53:04.150
of work each week on the on the
00:53:04.160 --> 00:53:06.710
portfolio. Um, but also made him think
00:53:06.720 --> 00:53:09.190
about it. And then what he found is when
00:53:09.200 --> 00:53:10.870
he was in interviews and he was asked a
00:53:10.880 --> 00:53:13.349
question, he had already thought about
00:53:13.359 --> 00:53:15.030
it to a degree because he had written a
00:53:15.040 --> 00:53:17.030
blog about it. And so he could come back
00:53:17.040 --> 00:53:19.510
with clarity of conversation. Oh, well,
00:53:19.520 --> 00:53:21.190
I did this and then I did this and then
00:53:21.200 --> 00:53:23.109
I did this and then he gets nailed to
00:53:23.119 --> 00:53:24.790
the hard questions, right? But the
00:53:24.800 --> 00:53:26.390
opening sentence showed that he'd done
00:53:26.400 --> 00:53:27.430
it before and he knew what he was
00:53:27.440 --> 00:53:29.750
talking about. And so again, I think you
00:53:29.760 --> 00:53:32.470
know that point you made, it's the doing
00:53:32.480 --> 00:53:35.030
is as important as the showing.
00:53:35.040 --> 00:53:37.670
>> Yeah. And to that point, like that's the
00:53:37.680 --> 00:53:39.430
big thing that I do like in my one-on-
00:53:39.440 --> 00:53:42.069
ones with people is we talk through what
00:53:42.079 --> 00:53:44.950
those decision points are. Um because
00:53:44.960 --> 00:53:47.750
then you walk into an interview and not
00:53:47.760 --> 00:53:49.270
only have you thought through it, but
00:53:49.280 --> 00:53:51.510
when you talk about it, you physically
00:53:51.520 --> 00:53:54.710
experienced those issues. Like you you
00:53:54.720 --> 00:53:56.710
are talking about it the way that you
00:53:56.720 --> 00:53:58.950
would talk about it as if you've really
00:53:58.960 --> 00:54:00.710
experienced it in a real work
00:54:00.720 --> 00:54:05.190
environment because you basically did.
00:54:05.200 --> 00:54:08.390
>> Yeah. Yeah. I agree. So yeah, do it and
00:54:08.400 --> 00:54:11.510
then you can talk about it. Uh you know
00:54:11.520 --> 00:54:13.190
>> Yeah. go in and get Claude to give you a
00:54:13.200 --> 00:54:16.032
summary and then you know [ __ ]
00:54:16.042 --> 00:54:16.470
[laughter]
00:54:16.480 --> 00:54:18.470
>> That's Yeah. Yeah.
00:54:18.480 --> 00:54:20.069
>> Be careful.
00:54:20.079 --> 00:54:21.670
>> All right. Well, we're pretty much end
00:54:21.680 --> 00:54:23.190
of time. What do you reckon? Uh if you
00:54:23.200 --> 00:54:25.670
were going to get somebody to do one
00:54:25.680 --> 00:54:27.750
thing over the next month before we come
00:54:27.760 --> 00:54:30.069
back. Um
00:54:30.079 --> 00:54:31.109
oh, hold on. We got one more question
00:54:31.119 --> 00:54:32.549
there. What are things you enjoy and
00:54:32.559 --> 00:54:34.309
hate when it comes to building and
00:54:34.319 --> 00:54:37.270
maintaining pipelines?
00:54:37.280 --> 00:54:39.349
Uh don't apologize for asking questions,
00:54:39.359 --> 00:54:41.829
man. It's called Ask Us Anything.
00:54:41.839 --> 00:54:43.829
>> Weird as this is why I'm finding people
00:54:43.839 --> 00:54:44.549
out there.
00:54:44.559 --> 00:54:47.589
>> Yeah, we love questions. Uh,
00:54:47.599 --> 00:54:49.990
>> okay. So, what what's the one thing you
00:54:50.000 --> 00:54:51.430
hate the most when it comes to building
00:54:51.440 --> 00:54:52.870
and maintaining pipelines?
00:54:52.880 --> 00:54:55.910
>> Maintaining pipelines, [laughter]
00:54:55.920 --> 00:54:57.670
the second half of that question. No, it
00:54:57.680 --> 00:54:59.750
like
00:54:59.760 --> 00:55:01.670
the big I think the biggest thing that I
00:55:01.680 --> 00:55:05.910
hate one is
00:55:05.920 --> 00:55:08.790
because the maintenance piece, you know
00:55:08.800 --> 00:55:09.910
that things are going to change
00:55:09.920 --> 00:55:11.829
eventually, right? So you try to build
00:55:11.839 --> 00:55:15.109
in stop gaps, right? You try to build
00:55:15.119 --> 00:55:17.109
try to create your pipelines so that
00:55:17.119 --> 00:55:19.589
they're resilient against schema drift
00:55:19.599 --> 00:55:21.750
and you don't have, you know, so that
00:55:21.760 --> 00:55:23.510
you could continue to have continuity in
00:55:23.520 --> 00:55:25.670
your business reporting. So you put
00:55:25.680 --> 00:55:28.630
things in place to when things change
00:55:28.640 --> 00:55:31.510
that it still feeds in the data that is
00:55:31.520 --> 00:55:38.069
critical but it quarantines the rest. Um
00:55:38.079 --> 00:55:41.589
I actually love that piece like it. So
00:55:41.599 --> 00:55:44.069
building my building pipelines
00:55:44.079 --> 00:55:45.510
the biggest thing I hate about that is
00:55:45.520 --> 00:55:48.549
when I have to do it in Informatica. Um,
00:55:48.559 --> 00:55:49.990
for those of you that have not used
00:55:50.000 --> 00:55:52.630
Informatica, I encourage you to go out
00:55:52.640 --> 00:55:55.270
and get like a free license to ICS and
00:55:55.280 --> 00:55:58.870
and check it out. But, um, but it is my
00:55:58.880 --> 00:56:00.710
opinion is that it's a very disjointed
00:56:00.720 --> 00:56:03.670
platform and it's it's hard to [snorts]
00:56:03.680 --> 00:56:06.630
to navigate. But, um, so I I love the
00:56:06.640 --> 00:56:08.309
actual building pipelines when I can
00:56:08.319 --> 00:56:10.870
actually build the pipeline, when I can
00:56:10.880 --> 00:56:12.390
actually
00:56:12.400 --> 00:56:14.470
control a lot of those pieces and build
00:56:14.480 --> 00:56:17.109
in that resiliency. Um, the thing that I
00:56:17.119 --> 00:56:20.230
hate about maintaining the pipelines is
00:56:20.240 --> 00:56:22.309
you can't anticipate everything that's
00:56:22.319 --> 00:56:25.510
ever going to happen, right? And there's
00:56:25.520 --> 00:56:28.950
going to be a 3:00 a.m. pager call for
00:56:28.960 --> 00:56:31.270
somebody that something blew up in a
00:56:31.280 --> 00:56:33.670
pipeline that you built. Whether it's a
00:56:33.680 --> 00:56:35.750
week from when you built it or three
00:56:35.760 --> 00:56:37.910
years from when you built it, at some
00:56:37.920 --> 00:56:39.670
point something's going to come through
00:56:39.680 --> 00:56:42.470
that that breaks it. The cool thing for
00:56:42.480 --> 00:56:44.470
those of you that haven't looked at it,
00:56:44.480 --> 00:56:45.990
um I'm starting to sound like a
00:56:46.000 --> 00:56:48.309
salesperson for data bricks, but but the
00:56:48.319 --> 00:56:50.069
cool thing is if if you haven't gotten a
00:56:50.079 --> 00:56:52.390
chance to look at what it is yet, uh the
00:56:52.400 --> 00:56:56.230
Genie Zero Ops looks like it will be a
00:56:56.240 --> 00:56:58.789
uh a great tool to help to mitigate some
00:56:58.799 --> 00:57:01.749
of that work we have to do to um
00:57:01.759 --> 00:57:02.390
[clears throat]
00:57:02.400 --> 00:57:04.710
to maintain the pipelines when things
00:57:04.720 --> 00:57:07.670
break. Um, but it's not out yet. And in
00:57:07.680 --> 00:57:08.950
the meantime, we still have to wake up
00:57:08.960 --> 00:57:10.630
at 3:00 a.m. And
00:57:10.640 --> 00:57:11.990
>> so that was one of those five coded
00:57:12.000 --> 00:57:14.390
PowerPoint ones, right? Um, so I think
00:57:14.400 --> 00:57:15.670
next next month, I'm just going to say
00:57:15.680 --> 00:57:17.109
it so I can get it back out of the
00:57:17.119 --> 00:57:19.430
transcript is, uh, I really want to go
00:57:19.440 --> 00:57:21.589
into dead letter queueing, failures of
00:57:21.599 --> 00:57:23.270
pipelines, right? Like is that actually
00:57:23.280 --> 00:57:24.950
a good process and what are the ups and
00:57:24.960 --> 00:57:28.150
downsides of it? Um, I think the other
00:57:28.160 --> 00:57:30.870
thing is how real are these self-healing
00:57:30.880 --> 00:57:32.470
agents going to be, right? this idea
00:57:32.480 --> 00:57:34.230
that there can be a crash, it goes to a
00:57:34.240 --> 00:57:36.150
jury ticket, an AI agent picks it up, it
00:57:36.160 --> 00:57:38.230
fix the codes and redeploy it. I would
00:57:38.240 --> 00:57:40.630
really love to know how how much reality
00:57:40.640 --> 00:57:43.430
that's going to be. Um, for me, my
00:57:43.440 --> 00:57:45.349
co-founder is the one that has to deal
00:57:45.359 --> 00:57:47.190
with the failures. So, he basically
00:57:47.200 --> 00:57:49.829
architects the system so he never gets
00:57:49.839 --> 00:57:53.670
paged. Um, and I hate writing writing
00:57:53.680 --> 00:57:55.750
code. So, we've architected the system
00:57:55.760 --> 00:57:57.349
so I don't have to write code because I
00:57:57.359 --> 00:57:59.829
can't. Uh the bit that I struggle with
00:57:59.839 --> 00:58:02.710
the most is apart from is what I'm
00:58:02.720 --> 00:58:04.150
writing valuable and and I'm sorting
00:58:04.160 --> 00:58:05.430
that you know that one's pretty much
00:58:05.440 --> 00:58:08.630
sorted now. Uh what I struggle with is I
00:58:08.640 --> 00:58:11.910
know it works but is the answer right?
00:58:11.920 --> 00:58:12.390
>> Right.
00:58:12.400 --> 00:58:14.789
>> Yeah. So like I know it it loaded. I
00:58:14.799 --> 00:58:17.270
know that it went all the way through
00:58:17.280 --> 00:58:20.069
but how do I know that that number that
00:58:20.079 --> 00:58:23.190
metric actually matches? And this is a
00:58:23.200 --> 00:58:24.950
problem for us because we normally can't
00:58:24.960 --> 00:58:27.750
see this whole system. And I would argue
00:58:27.760 --> 00:58:30.069
that the perfect resolution to that is
00:58:30.079 --> 00:58:32.870
actually your information product canvas
00:58:32.880 --> 00:58:36.230
uh course that you do on Maven um to do
00:58:36.240 --> 00:58:39.270
the project gathering and requirements.
00:58:39.280 --> 00:58:41.030
>> So that's good for solving the value
00:58:41.040 --> 00:58:42.549
problem, right? Am I building what a
00:58:42.559 --> 00:58:45.510
value? But it doesn't solve my is that
00:58:45.520 --> 00:58:48.150
number I'm surfacing actually right?
00:58:48.160 --> 00:58:50.309
Like I know it's the code's right, but
00:58:50.319 --> 00:58:52.150
how do I know that that actually is the
00:58:52.160 --> 00:58:54.230
number of customers when I can't see the
00:58:54.240 --> 00:58:56.870
source system that it comes from? Uh
00:58:56.880 --> 00:58:58.470
that one keeps you up at night. Yeah,
00:58:58.480 --> 00:58:59.910
>> that that that I could see where that
00:58:59.920 --> 00:59:02.789
keeps keeps you up at night. And the
00:59:02.799 --> 00:59:04.870
project requirement gathering is is like
00:59:04.880 --> 00:59:06.870
a big piece of
00:59:06.880 --> 00:59:08.309
>> making sure that that's right down the
00:59:08.319 --> 00:59:09.829
road. If you don't do that right to
00:59:09.839 --> 00:59:11.670
begin with, you know, the numbers wrong.
00:59:11.680 --> 00:59:13.670
>> Oh, actually UA's answered it. He's
00:59:13.680 --> 00:59:14.950
saying make sure you, you know, if
00:59:14.960 --> 00:59:16.470
you're the person on call, make sure you
00:59:16.480 --> 00:59:18.630
work in a time zone that's 12 hours
00:59:18.640 --> 00:59:20.470
different from [laughter]
00:59:20.480 --> 00:59:23.030
>> Nice. I love that one.
00:59:23.040 --> 00:59:24.950
>> Doesn't sleep. So, [laughter]
00:59:24.960 --> 00:59:27.510
>> that's awesome. Or hire a team that's,
00:59:27.520 --> 00:59:28.870
you know, halfway across the country.
00:59:28.880 --> 00:59:30.630
That's, you know, the world.
00:59:30.640 --> 00:59:32.710
>> Excellent. All right. Well, uh, we're
00:59:32.720 --> 00:59:34.710
done. That was an hour. Um, call out to
00:59:34.720 --> 00:59:37.589
anybody watching this. Uh, we'll
00:59:37.599 --> 00:59:39.270
announce the next, uh, the times for the
00:59:39.280 --> 00:59:41.829
next ones. Um, they'll be in Substack.
00:59:41.839 --> 00:59:44.069
If you go into the article uh
00:59:44.079 --> 00:59:46.390
announcement in there, write a question
00:59:46.400 --> 00:59:48.710
uh onto that article, Chris and I will
00:59:48.720 --> 00:59:51.589
read it. U we'll go and have a look um
00:59:51.599 --> 00:59:54.549
and then um we'll we'll go from there.
00:59:54.559 --> 00:59:57.750
Um yeah, partic yeah check sums are fine
00:59:57.760 --> 00:59:59.589
that the numbers coming in are correct
00:59:59.599 --> 01:00:01.750
for me. Like I know that the data we got
01:00:01.760 --> 01:00:04.630
is the data we received but then we do a
01:00:04.640 --> 01:00:06.710
whole lot of gymnastics to make the data
01:00:06.720 --> 01:00:08.470
answer the business reality. And how do
01:00:08.480 --> 01:00:11.670
I know that that definition of uh active
01:00:11.680 --> 01:00:13.750
subscriptions for marketing customers in
01:00:13.760 --> 01:00:15.829
the Latin South American region is
01:00:15.839 --> 01:00:17.349
actually the right answer because that's
01:00:17.359 --> 01:00:19.430
not in the source status. So uh yeah,
01:00:19.440 --> 01:00:21.109
maybe a chat for another day, but uh
01:00:21.119 --> 01:00:23.430
yeah, if you if you're watching it, uh
01:00:23.440 --> 01:00:26.069
ask us some questions uh offline. We'll
01:00:26.079 --> 01:00:29.030
try and answer them. Uh ask them live
01:00:29.040 --> 01:00:31.349
and uh as you'll see, you know, you ask
01:00:31.359 --> 01:00:32.950
them, we'll answer them.
01:00:32.960 --> 01:00:34.390
>> Yeah. Don't guarantee the quality
01:00:34.400 --> 01:00:35.910
answers, but I thought they were pretty
01:00:35.920 --> 01:00:37.990
good. So uh excellent. We'll uh we'll
01:00:38.000 --> 01:00:40.309
see you next month. Sounds good. Thanks,
01:00:40.319 --> 01:00:43.599
guys. Thanks everybody.