I'd Rather Have Hallucinations Than a Perfect Data Model - Joe Reis

WEBVTT
1
00:00:00.100 --> 00:00:10.060
Welcome to the Data Strategy Gurus podcast. In this show, we bring together the brightest minds in the world of data strategy, data management, artificial intelligence, and disruptive technologies.
2
00:00:10.320 --> 00:00:16.820
Thought leaders and experts share their insights, knowledge, and experience on how to stay ahead of the game in an ever-evolving data landscape.
3
00:00:17.140 --> 00:00:23.500
Whether you're a data professional, a business leader, or simply someone who is passionate about the power of data, this podcast is for you.
4
00:00:23.800 --> 00:00:35.520
So sit back, relax, and join us on a journey to explore the world of data, analytics, artificial intelligence, tech, and beyond. [upbeat music] Hi, and welcome to the Data Strategy Gurus podcast.
5
00:00:35.680 --> 00:00:44.220
Today, we have Joe Rice as a guest. Joe isn't just a data strategist. He co-wrote the book that's become a roadmap for data engineers.
6
00:00:44.400 --> 00:00:57.080
The fundamentals of data engineering takes you as a reader through the entire life cycle, from ingestion to serving data, while also teaching you how to ignore the noise and focus on what scales.
7
00:00:57.560 --> 00:01:10.940
And today, we bring that clarity, uh, to our conversation. Joe, uh, welcome. Yeah, thanks for having me on. Joe, your book bundles the, the full data engineering life cycle into a clear, actionable framework.
8
00:01:11.280 --> 00:01:26.280
Uh, what, what inspired you most that comprehensive life cycle based approach? Um, [sighs] I, I would say the life cycle, like, when we wrote the book, the life cycle was... It wasn't the first thought we had, actually.
9
00:01:26.400 --> 00:01:33.840
I think we were trying to figure out what data engineering was from first principles, and it was actually a really hard thing to figure out. Um,
10
00:01:34.780 --> 00:01:44.480
I think to take a step back, what, what inspired the book in the first place was just, I think, the lack of really good materials on, like, what data engineering was or there is now. So if...
11
00:01:44.620 --> 00:01:50.320
Back in the day, and you can still see this, but, you know, in the 2010s if you were to ask, "What is data engineering?"
12
00:01:50.360 --> 00:02:00.600
You'd typically get articles about how data engineering is using Hadoop and Pig or Spark or some tool set. And I felt
13
00:02:02.220 --> 00:02:14.140
that those were more common than first principles thinking, and I felt those were actually really terrible ways of describing, uh, a, a field, especially a very growing field. And so the, the goal was really to provide,
14
00:02:15.160 --> 00:02:24.880
you know, data engineers, whether they're established data engineers or, you know, new people in the field, uh, a, a, a roadmap from first principles of what data engineering,
15
00:02:25.920 --> 00:02:36.120
you know, is and how to think about it and how to practice it. And so the, the life cycle really came about, I think, as we're trying to think about w- well, what, what is data engineering, right?
16
00:02:36.160 --> 00:02:45.880
If you're described from, from, uh, from the ground up and, and I think we settled on life cycle management as, as being a pretty key thing. Um, it's very rarely, you know,
17
00:02:47.080 --> 00:02:54.040
just managing one aspect or, or another of that life cycle, whether that's, uh, ingestion or transformations or storage or serving.
18
00:02:54.100 --> 00:03:07.860
Certainly maybe at bigger companies it could be, but, um, I think having a comprehensive understanding of the life cycle and the undercurrents as well, which I think are almost more important, uh, that really establishes a baseline for data engineering, and that's how we ended up, uh, thinking about it and writing about it.
19
00:03:08.800 --> 00:03:16.160
Yeah, I'm, I'm thinking it through. I mean, I, I have the gray hairs as well. I started off as an ETL developer, did some software engineering. Yeah.
20
00:03:16.180 --> 00:03:25.080
So I hear data and engineering coming more and more together, and that's what I saw as well, applying- Mm-hmm... the principles of, of software engineering to data in, in such a way.
21
00:03:25.460 --> 00:03:33.200
But then on the other hand, I saw that a lot of these engineers are missing the, the data principles. I mean, doing- Yeah... ETL data modeling, all that stuff.
22
00:03:33.860 --> 00:03:45.380
Uh, does that help by, by presenting that in a certain way as a, as the life cycle that you focus on certain aspects and say, "Okay, start with the basics and then build up, and this is how you get to a full production
23
00:03:46.280 --> 00:03:55.360
life cycle data engineering solution or part of a full solution"? Data engineering is just a part of a, of a full solution, but how, how- Yeah... do you do that?
24
00:03:56.340 --> 00:04:07.220
I mean, I think it, it, there's certain foundational steps you have to take to master any field. For example, if you're doing mathematics, right, you're not gonna jump straight into, you know, differential equations.
25
00:04:07.700 --> 00:04:19.360
You'll probably, uh, understand [laughs] you know, different aspects of pre-algebra. Um, you know, and that'll set you up to calculus, which will then set you up, right, um, to do differential equations.
26
00:04:19.420 --> 00:04:24.040
So there's, I think there's a certain sequence of learning that is beneficial to a lot of people.
27
00:04:24.960 --> 00:04:37.100
But historically, right, I mean, this happens all the time, it, there has not, you know, until the book, been a good rubric for learning data engineering. So the, the, the suggestions were, well, go read Kimball's book,
28
00:04:38.240 --> 00:04:48.640
right? Right. Go read, uh- Yeah... Designing Data-Intensive Applications, which is a great book, but I don't think it really applies that much, uh, especially as tools get more abstracted, right?
29
00:04:48.680 --> 00:04:52.440
There's not much to design. You're more just managing systems that are built by somebody else.
30
00:04:53.420 --> 00:05:02.400
And, and so it's really, you know, I think a matter of, like, be- being able to comprehend, you know, sequentially, you know, how data arrives, what you do with it, and so forth, right?
31
00:05:02.440 --> 00:05:08.380
But these, the, this is a, these are, um, this is a, a sequence. So it doesn't...
32
00:05:08.720 --> 00:05:17.040
Data engineering doesn't really happen in the reverse where you're serving reports, but then you have to go get the data to serve the report. Like, it doesn't work that way. Uh, the order of operations is very...
33
00:05:17.820 --> 00:05:20.600
It's, it's, it's pretty, it's pretty clear how this works, right?
34
00:05:20.680 --> 00:05:29.020
You so, um [laughs]- Yeah, I know, but I've, I've seen so many report developers, they say, "I'm, I'm ready," and then you start finding duplicates and, and things that are not correct- Yeah...
35
00:05:29.060 --> 00:05:40.260
and aggregates that don't, don't, don't tie together. So there I feel there- Yeah... is still a lot of, of, uh, missing feeling for data or data analysis. I recall on a, on a job- Yeah...
36
00:05:40.260 --> 00:05:51.160
interview at McKinsey where I was applying for data modeling, uh, job, where they say, "Okay, here's the data set. Look in Excel. How do you look at it?" Uh, what do you mean with how do I look at it? I filter it.
37
00:05:51.340 --> 00:05:59.760
[laughs] I do a pivot table. I try to understand the data, and I was kind of- Interesting... this, this is it, how you, you, you understand the principles and the process.
38
00:05:59.820 --> 00:06:13.340
And that was for me kind of eye-opener of having that, that good, those skills and being able to, to go through and truly understand- Mm-hmm... if, if the data is correct in such a way. That's interesting.
39
00:06:13.600 --> 00:06:23.368
S- Was it, was the interview process then, like For you to evaluate what the data set meant by looking at the data set in, in, uh, spreadsheet form? Or what, what were they trying to a- assess?
40
00:06:23.468 --> 00:06:30.088
No, I think I was expecting, like you say, mostly, uh, explain Kimball in a certain way, and then you have- Okay... a good data model.
41
00:06:30.248 --> 00:06:40.508
But they were trying to look at how do you break down a problem, and this is typically in a- Oh... in a, in a McKinsey interview, how do you break down a problem? How do you- Yeah... uh, approach it and that.
42
00:06:40.628 --> 00:06:51.148
And that was more important, having that type of soft skill, than purely the data modeling. Of course- Yeah, absolutely... that it's my, my, my, my business as well. But yeah, I see there- Yeah...
43
00:06:51.388 --> 00:07:02.028
is something else what, with a lot of people, uh, is missing, especially where you say, "Learn Hadoop, learn Pig." And I see that very often, that people ask me, "Ah, you don't know anything about cloud.
44
00:07:02.108 --> 00:07:09.708
You're a traditional school kind of guy." I say, "I don't have a problem with the technology. I know how to approach it and implement it." Mm-hmm.
45
00:07:09.728 --> 00:07:20.207
And that's, that's more important than just knowing every in and out at the beginning. I mean, at the end, if you're into technology, I go deep, I learn all the ins and outs, and I know- Yeah...
46
00:07:20.228 --> 00:07:29.188
perfectly how to optimize it. That's, that is the feeling I get on, on, on data engineering. Well, uh, but, you know, especially i- i- I think it's easier to talk about the tech, right?
47
00:07:29.248 --> 00:07:36.848
And I think it's a convenient thing because, you know, there's no shortage of technologies, and it, it's fun, right? It, it's, it's, uh- Mm-hmm...
48
00:07:37.308 --> 00:07:41.588
you know, I was watching that movie Fast and Furious with my, my kid the other night, and, you know, they're, they're, uh,
49
00:07:42.628 --> 00:07:47.128
you know, I mean, the, you know, in the movie, uh, I don't know if you remember it, it's kind of an old one, but, uh, it, it's fun, right?
50
00:07:47.188 --> 00:07:55.568
They can walk up to any race car and just look at the engine and just kind of assess what's going on, and nerd out on, um, you know, the NOS and all the other stuff in the car, and I mean, that's, that's, that's fun, right?
51
00:07:55.668 --> 00:08:04.428
I love nerding out. I was actually writing a, an application all last night. Um, and it's addictive for sure, but then you do have to step back, and I g- and I, and I like what McKinsey did, actually.
52
00:08:04.468 --> 00:08:09.508
That's actually really smart, where it's just how do you break down a problem? How do you think about assessing a problem?
53
00:08:09.568 --> 00:08:20.048
That's really what we tried to accomplish with the book, and that's also what I convey with people, um, y- you know, e- even in the, uh, Coursera course I did, um, with, uh, Deep Learning AI.
54
00:08:20.688 --> 00:08:30.708
It, it really, the goal was to teach you how to think like a data engineer, so the first part of the course was, um, breaking down problems, right? Like, how do you talk to stakeholders? What do they want?
55
00:08:30.768 --> 00:08:38.008
How do you know they want that, right? So it's, it's more assessing the landscape first, and then applying the, uh, the technology to it.
56
00:08:38.028 --> 00:08:46.908
But the reverse is definitely a convenient thing for people to do, and it's easy, especially when you have the word engineer in your title. The thing you wanna do is, well, you wanna go engineer stuff.
57
00:08:46.948 --> 00:08:57.628
You wanna go clank on code and, you know, and then, um, especially now with, uh, the, you know, the, the suite of awesome AI tools out there, I mean, making applications is dead simple.
58
00:08:57.968 --> 00:09:10.088
Um, you know, I use it all the time. Like, I have Cloud Code running a lot. It's awesome, right? But it, it, but I also understand the problem I'm trying to solve, [laughs] so- Exactly... uh, at least I think I do.
59
00:09:10.148 --> 00:09:11.288
I don't know. Maybe I don't, actually.
60
00:09:11.368 --> 00:09:18.908
Um, you know, but it, it's, it's, it is one of these things where that I think the, the problem-solving skills and breaking down the problem is one of these things where it just isn't done enough.
61
00:09:19.508 --> 00:09:25.488
Uh, in this new book I'm writing on data modeling, I'm actually writing a whole chapter right now on organizational considerations. Um- Mm...
62
00:09:25.548 --> 00:09:35.248
you know, how to, you know, u- use things like Conway's Law to understand, okay, so the organization, what i- w- what does it look like? What do the systems most likely look like?
63
00:09:35.428 --> 00:09:41.848
And also within different departments, there might be different org structures within departments. How do you assess that?
64
00:09:41.928 --> 00:09:51.768
'Cause that's gonna give you the framework on how to think about, okay, if I'm gonna design a data model, how should I approach this, right? It's not just applying blindly applying Kimball to every single...
65
00:09:52.288 --> 00:10:01.908
Sometimes that is, like, the dumbest thing to do, period. [laughs] So- Exactly... you, you need to understand, like, what... You need to know what game you're playing and have the situational awareness.
66
00:10:02.348 --> 00:10:13.668
That's very lacking, though. Yeah, you have multiple ways of, of, of solving that. So in your book, you're al- also distilling complexity into, uh, simplicity. It's, it's, it's very tough.
67
00:10:13.808 --> 00:10:23.248
That's, that's how you address it. How do you approach that when, when writing the book? You mentioned already a part of it, uh, applies to data modeling and, and soft skills.
68
00:10:23.888 --> 00:10:31.488
Um, but yeah, it's, it's such a broad, uh, field to, to get that to people. I mean, I'll give you some context, right?
69
00:10:31.608 --> 00:10:49.208
So when we, uh, were talking with O'Reilly originally about the idea for Fundamentals of Data Engineering, uh, you know, the acquisitions editor, she told us that this was actually, you know, she actually tried to s- tell us not to do it, 'cause she said it was a very ambitious project, especially for two first-time authors to define a field from first principles, right?
70
00:10:49.228 --> 00:11:00.268
And I d- I can't disagree with her at all. Um, somehow we got it over the, uh, you know, over the fence, but it was, um, but it was, it was a hard one to write, as you, as you pointed out, right?
71
00:11:00.308 --> 00:11:11.788
Uh, to simplify things is actually very difficult. Um, and it looks easy, right? Uh, and I think one of the criticisms of the book is that it's... Everyone's like, "Well, I kind of already knew all this."
72
00:11:11.868 --> 00:11:20.948
And it's, yeah, maybe you did, maybe you didn't. I actually think that most people don't, um, because it's a lot to know. Um, you know, I've met enough data engineers, and I don't think that,
73
00:11:22.068 --> 00:11:26.008
you know, it's, it's possible for everyone to know everyth- or someone to know everything. I don't know, maybe that's...
74
00:11:26.028 --> 00:11:32.468
But, but to s- but I think to, to take principles and simplify them in a way that's easily digestible is very, very difficult.
75
00:11:32.868 --> 00:11:43.788
So one of the things I did is I actually, um, you know, had my, uh, wife read, um, you know, the first part of the book at least, 'cause I figured if she can understand it, and she's not from technology at all, right?
76
00:11:43.848 --> 00:11:54.188
So if she could understand what was in the book, um, and people like her could, you know, at least follow along, then I'm like, "Okay, we succeeded." Uh, it's not meant to be...
77
00:11:54.248 --> 00:11:58.108
I think all too often when people write, they, they write to show off to their friends- Mm...
78
00:11:58.128 --> 00:12:05.328
and show off how smart they are, but I think the way I show off is actually breaking down problems in, in, in, in ways that are extremely, uh, easy to understand.
79
00:12:05.368 --> 00:12:13.448
I think that's actually more of a flex than, "Let's see how complicated I can make my writing in, in a way that only no, you know, nobody can really understand what I wrote."
80
00:12:13.508 --> 00:12:23.248
I mean, to me, that's a huge disservice to the reader, right? They're, they're investing their time and money into your book. Like, take them along on a journey that's gonna make them wanna keep turning the pages.
81
00:12:24.036 --> 00:12:36.896
You know? Yeah. Um, otherwise, like, you, you wasted everybody's time. It's easy to do, though. Yeah, exactly. So your book also helps readers cut through the marketing hype when evaluating tools.
82
00:12:37.116 --> 00:12:48.076
Uh, in today's rapidly evolving ecosystem, what are some of the, of the buzzwords or the, the hype monsters that keep you up at night, and, and how do you recommend leaders, uh, avoiding these buzzwords?
83
00:12:49.496 --> 00:12:56.916
It's interesting. Buzzwords don't really keep me up at night anymore. I think that, you know, and, and I learned this from Bill Inman, right? We was, I was hanging out with him, um,
84
00:12:58.296 --> 00:13:03.736
uh, I think it was earlier this year, actually. Well, well, we hang out with him a lot, but we, uh- [laughs] You know, in one of the conversations,
85
00:13:05.036 --> 00:13:15.516
we were talking about just the difference of, like, fundamentalists and technologists, right? Yeah. And so I kind of straddle between both. Um, but the technologists, you know, there's always gonna be something new.
86
00:13:15.556 --> 00:13:19.376
There's always gonna be the next buzzword, the next tool, the next this, the next that.
87
00:13:20.076 --> 00:13:25.276
Certainly cool, especially right now, there's all kinds of crazy stuff happening, and, and I think it's, it's a fun time to be in the business.
88
00:13:25.336 --> 00:13:37.416
Like, I wrote, you know, on my blog a, a bit ago, like, I'm having more fun now than I have in a while because the reset button's been hit on the entire industry, and now's the time you can rediscover how to do things, and I think that's awesome.
89
00:13:37.576 --> 00:13:44.896
So much fun. But there are fundamentals that are pretty much gravity, right? And so- Mm-hmm...
90
00:13:45.336 --> 00:13:53.916
the things that don't change or are not gonna change as much are things like the data engineering life cycle, right, and integrating data from various sources, like data warehousing.
91
00:13:53.956 --> 00:14:05.496
That was the, the heart and soul of that is integration and transformation of data. That is still, that does not go away, ever. Um, you know, operational systems, uh, the relational model, for example, right?
92
00:14:05.536 --> 00:14:16.796
That's gravity. That essentially deduplicates, you know, data and makes it clean, right? That's, um... So there's certain fundamental things that just don't disappear, right? Security, for example, right?
93
00:14:16.836 --> 00:14:24.336
There's no situation in the world where you'd say, "Oh, well, I don't need to think about security anymore. That's, like, that's, that's out of fashion." [laughs] Um,
94
00:14:25.396 --> 00:14:34.536
so I think if you have a grasp of the fundamentals, right, and when you hear all these buzzwords, you're kind of going, it's like, where does this fit in? Sometimes it's an entirely new thing, too, right?
95
00:14:34.616 --> 00:14:46.816
So, so I'd say when, when, uh, mobile, mobile phones, you know, became popular with, you know, with the iPhone, that, that fundamentally changed a lot of things in terms of how we operate as, as, as a society and a civilization.
96
00:14:46.876 --> 00:14:54.876
I mean, you know, I often note that the universal posture of somebody is, like, you know, if you... Anywhere you go in the world, there's somebody hunched over their phone like this, so your neck is, like, bent.
97
00:14:54.876 --> 00:15:07.276
[laughs] And, um, you know, but that's, that is the posture these days, right? But so the, so you gotta understand the fundamentals, but, you know, the new technologies will come and, and that's just, you know...
98
00:15:07.336 --> 00:15:15.816
So as Mar- Marshall McLuhan said, the medium is the message, right? And so you always have to be aware of, like, where do you balance the fundamentals against, you know, inflection points in technology.
99
00:15:15.876 --> 00:15:19.596
The mobile phone didn't necessarily change the fundamentals of what humans wanna do.
100
00:15:20.236 --> 00:15:28.396
Y- you're on the phone because you wanna communicate and because you're, you, you, you like the dopamine rush of notifications, of this, that, or the other thing, right?
101
00:15:28.476 --> 00:15:38.876
But it just made, made it so all those things can just happen much more quickly and easily for you, for better or for worse, to be frank. Um, but- Yeah... you know, so that's how I view, like, AI.
102
00:15:38.936 --> 00:15:48.676
AI is, to me, is nothing more than a tool. My background's actually in machine learning and all that kind of stuff, so it's like that's, um, the world I came from, and, uh, and I guess data science.
103
00:15:48.796 --> 00:15:58.976
But so to me, it- it's, this is just another tool. It helps you do a lot of things, uh, quicker, but, you know, things like generative AI, it's awesome. I use it all the time.
104
00:15:59.036 --> 00:16:05.756
It can al- also help you do dumb things very quickly now, so you just... That's where you have to know the fundamentals probably even more.
105
00:16:05.796 --> 00:16:13.856
Like, the other day, I was reading, um, uh, rereading, uh, uh, parts of Clean Code, uh, by Robert Martin and Bob Martin, if you remember that book. Mm-hmm.
106
00:16:14.156 --> 00:16:24.936
And I think that, books like that are actually more relevant than ever. You know, it's, uh, good coding practices. If you have AIs writing code, like, how do you even know if it's any good? I don't know. Yeah.
107
00:16:25.256 --> 00:16:25.936
I, I see that.
108
00:16:25.996 --> 00:16:35.736
I'm, I'm using Cloud Code all day as well, and, and struggling a lot of the time with, with them to, uh, to keep on track, and then you think, "How is this possible that, that they're making so many mistakes?"
109
00:16:35.776 --> 00:16:43.236
So sometimes I feel like I'm- Yeah... I'm helping debug the, the, the whole, uh, large language model or training the model in a certain way. But like- Mm-hmm...
110
00:16:43.256 --> 00:16:52.276
like you say, if you don't have those principles, I love to, to hear these, uh, these, uh, YouTube videos where they say, "Ha, I built this application in two, uh, hours on a Sunday afternoon."
111
00:16:52.976 --> 00:16:58.436
Well, I don't think, uh, that that will happen if you want to scale it at a, at a productional, uh, level. Yeah.
112
00:16:58.476 --> 00:17:07.336
You still have to apply a lot of the, the real engineering practices to get a good application which is stable and scalable, uh, in, in, in such a way. Yeah, for sure.
113
00:17:07.936 --> 00:17:14.356
But last night, you know, I, I, I mean, I had Cloud Code create, um... So I'm working on a course for data modeling right now, right?
114
00:17:14.396 --> 00:17:23.056
And so I, I thought, "Oh, wouldn't it be fun to create a whole application where people can learn data modeling," right, with different lessons. And I just made...
115
00:17:23.676 --> 00:17:30.496
So in one, um, you know, I, like, in ChatGPT, I made, uh, you know, I prompted it saying, "Okay, so, like, let's, let's think this through. What would this contain?"
116
00:17:31.136 --> 00:17:41.516
When I had the course, uh, at least that, that, uh, the app ready, um, you know, the idea of it fleshed out, I was like, "Okay, make a, uh, a cloud data, uh, data MD file, and let's put it in Cloud Code, and let's build it."
117
00:17:42.516 --> 00:17:50.296
Right? Just to prototype it and see if this is even possible, see what this would look like, and that was cool, right? Within an hour, I had a working app, right, in TypeScript and, and Node.
118
00:17:50.356 --> 00:17:56.736
And the cool thing with this one, I didn't need a database 'cause a lot of the lessons are just self-contained. They're in files with the data, so I'm like, "That's pretty cool."
119
00:17:56.976 --> 00:18:07.016
Um, so you know, for, for me, the power of that was, you know, I'm gonna have to probably go back through and refactor the whole thing to make it production ready, but at least I have something I can try out. Yeah.
120
00:18:07.096 --> 00:18:15.736
Right? And that's, that's, that's, that's super powerful. Now you can just see, okay, so, like, is this a good idea? Like, what would you wanna do for an app? And I, nowadays my mo- my motto is show, don't tell.
121
00:18:16.436 --> 00:18:24.336
Back in the day, you'd probably come to me with, like, a Figma drawing or something. Those are still obviously useful, but now make the, make the app, make the demo.
122
00:18:24.536 --> 00:18:33.796
But also, to your point, right, like, you gotta be, like, you better know what this code does if you're gonna put it in production, or if you don't, I guess, like, you know, mess around and find out, as the saying goes.
123
00:18:34.856 --> 00:18:37.316
But Yeah, y- yeah.
124
00:18:37.496 --> 00:18:50.296
Now talking about these, these applications that we build in a few hours, you emphasize as well that data govern- well, governance and security shouldn't be a bolt-on, but instead embedded across that complete data life cycle.
125
00:18:50.836 --> 00:19:01.616
Mm-hmm. Could you unpack a bit what, what really m- does that mean for teams designing pipelines? Yeah. So I mean, it, it was interesting 'cause we, as we were writing the book,
126
00:19:02.816 --> 00:19:12.636
we actually didn't have the idea, the notion of the undercurrents at first. It was just a life cycle. But then you start asking yourself the very obvious question, and a very hard one actually,
127
00:19:13.936 --> 00:19:26.096
you know, okay, security, that's, that's a thing, right? Um, where does that go in the life cycle? You couldn't just put it in storage or some, some section of it. That would be reckless.
128
00:19:26.156 --> 00:19:32.996
It's like, "Oh, we only do security when, in the transformation layer, otherwise everything's o- fair game," right? No, of course not. Same with governance.
129
00:19:33.036 --> 00:19:35.956
Same with, you know, ar- ar- you know, architecture, orchestration, et cetera.
130
00:19:36.036 --> 00:19:46.376
So these are things that undercut the entire, uh, you know, cross-section of, of data engineering and, and, and frankly, through the entire data life cycle, not just data engineering.
131
00:19:46.416 --> 00:19:55.956
That's a subset of a broader data life cycle. When you're thinking about software applications and, uh, you know, ML and AI, it's like there's no situation where you wouldn't have security,
132
00:19:56.996 --> 00:20:05.336
um, unless you're an absolute idiot. And governance, that's the other one that I, I think is overlooked. And, and I think governance is also extremely misunderstood.
133
00:20:06.436 --> 00:20:09.336
Um, you know, it's almost a four-letter word in a lot of ways.
134
00:20:09.576 --> 00:20:19.116
Uh, if you mention governance, especially to, I think, a lot of modern data practitioners, they sort of run the other direction, uh, 'cause, 'cause it's like the fun police just showed up and you don't wanna talk to these guys.
135
00:20:19.156 --> 00:20:29.796
These guys suck. Um, so [laughs] um, you know, and, uh, that, that, that's, that's the perception of it, but all this stuff is necessary, right? So for this app, right, that I was building, it's like there's no way...
136
00:20:29.836 --> 00:20:34.776
I mean, for this version, I would probably, 'cause there's no, there's no login or anything obviously, 'cause it's a prototype.
137
00:20:34.816 --> 00:20:42.236
Like, this is something I, the, the V1, I'd probably restrict the functionality and put that on the web, right? We can just learn the basic principles, and it's a sandbox.
138
00:20:42.276 --> 00:20:52.636
But like, yeah, I'm not gonna put this openly out with like, you know, just no security on credentials or anything. So my take is like, don't have credentials, just put the sandbox out and just try it out.
139
00:20:52.776 --> 00:21:03.276
I don't really care. It's, it's pretty... But if you, if you just, if you didn't know that, if you weren't aware of that, you'd, you know, you'd say, "Oh yeah, just log in, and don't use SSL or anything.
140
00:21:03.736 --> 00:21:14.516
Just go for it, and don't [laughs] store things securely. Just store them in plain text files." Like, but you'd be, you'd be surprised. Crazy stuff like that happens all the time. Um, you know, so. Yeah.
141
00:21:14.916 --> 00:21:19.916
You don't have to tell me. I mean, uh, that's, that's something- I don't [laughs]... very often, and it's, it's so easy.
142
00:21:20.056 --> 00:21:25.896
I mean, I started off back in the days with SQL Server where, uh, everything was open, and then they started break, well,
143
00:21:27.076 --> 00:21:33.116
containing it, and then, well, finally as of some version, you had no access rights and you had to open it up.
144
00:21:33.236 --> 00:21:39.676
But in the end, you see, uh, very often you still had the admin rights and [laughs] had access to, uh, to all the systems and tables.
145
00:21:39.756 --> 00:21:46.316
So yeah, depends a bit on, on, uh, on the type of organization where you are, but you have to be- Absolutely...
146
00:21:46.356 --> 00:21:59.116
uh, very aware of, of these, these pitfalls and, and the dangers, uh, if you're developing that and, and making people aware of, hey, uh, we don't have our s- uh, governance or security in, uh, correctly in place.
147
00:22:00.476 --> 00:22:05.196
Well, and you mentioned the type of company that you're at too, or the type of organization. I think that's a big piece of it.
148
00:22:05.416 --> 00:22:14.596
Some companies, you know, either because they want to do the right thing or more to the point, because they have to do the right thing 'cause of regulations or, or whatever. Mm-hmm.
149
00:22:14.616 --> 00:22:25.536
Like, um, you know, it, it, it can, your mileage might vary. I guess mileage is a US word. Um, uh, your kilometers might vary. Uh, and so, um,
150
00:22:26.816 --> 00:22:31.176
the, 'cause, 'cause yeah, some- especially if you're in the US, right, companies...
151
00:22:32.016 --> 00:22:40.976
And I was talking about this the other day with some other European, um, data people, and we were talking about just the cultural differences. Um, you know, in Europe things, you know, I think are...
152
00:22:41.676 --> 00:22:50.256
There's more time, I think, and more general consideration to doing things more thoughtfully because you have more time. Like, you can't get fired really for most companies.
153
00:22:50.356 --> 00:22:59.956
Uh, you know, so it's like you have more time to put into doing things than the US. And again, I'm broadly generalizing, but I think this is kind of also true. Um,
154
00:23:00.976 --> 00:23:07.936
but you know, especially if you're at a startup, it's just move fast, break things, um, you know, take shortcuts 'cause your, that's your incentive.
155
00:23:07.996 --> 00:23:18.736
So I think it depends on what the incentive is for you to focus on things like security. 'Cause I've seen startups where that is, you know, you, you're gonna talk, you're gonna, um,
156
00:23:20.336 --> 00:23:31.556
you'll do it, you know, uh, I think you're gonna do the bare minimum you need to, uh, just because, you know. And even that, right, uh, the incentives might not be there to do much of anything.
157
00:23:32.536 --> 00:23:39.176
Um, when you see this with startups, I mean, that's why I think a lot of companies are very, you know, especially enterprises are, are concerned about working with startups is because
158
00:23:40.636 --> 00:23:44.196
there's, there are shortcuts that are taken. Yeah. That's how it is.
159
00:23:44.216 --> 00:23:56.916
[laughs] So, so, so Jewel- And, uh- Next, next to your book and, and, and you being very busy, you also have a newsletter offering long form deep dives, uh, into the, really the topics that matter.
160
00:23:57.296 --> 00:24:12.776
Where do you notice gaps in, in the current dialogue around data engineering and, uh, AI? Are there any topics you feel deserve more air time? Oh, so I, I have two Substacks, um, two, two newsletters.
161
00:24:12.836 --> 00:24:24.496
One's the, uh, practical data modeling where I more write the, uh, drafts of my book, um, I'm coming out with a data modeling book, and then there's my personal one, um, my other Substack.
162
00:24:24.716 --> 00:24:35.636
Uh, and I think with, with, with the personal one, that's where I write a lot of, uh, it's, they're more observations really, right? Um- Mm-hmm...
163
00:24:35.676 --> 00:24:44.816
you know, so the technical stuff more goes into my newsletter or it goes into a course. Like, I don't typically write that kind of stuff just 'cause I'm- I really, I don't know.
164
00:24:44.996 --> 00:24:49.476
I'm, I'm so focused on it normally when I'm working on my other stuff, I just, if I'm on a rant, I'm on a rant.
165
00:24:49.796 --> 00:24:58.276
And so a lot of the stuff on my newsletter are kind of unhinged, um, you know, off-the-cuff rants or just observations I see in the industry. I think that, you know, it's...
166
00:25:01.496 --> 00:25:15.456
The thing I like to do is say the quiet part out loud because I can. Um, I know a lot of people work at jo- you know, jobs or have positions where they are muzzled to a large degree. They can't say things
167
00:25:16.576 --> 00:25:23.856
because they'll get fired, or they'll be reprimanded, or they'll be looked down on, or something. People will just hate them and egg their house or something, I don't know. But,
168
00:25:25.316 --> 00:25:33.196
you know, for, for me, I feel I have a platform where I can say a, a lot of things, and so I have no problem calling out what I see as a lot of, um,
169
00:25:34.096 --> 00:25:44.916
you know, I would say e- excessive behavior or, um, you know, discussions where the industry gets ahead of its skis, but this happens all the time in hype cycles. Yeah.
170
00:25:45.036 --> 00:25:50.076
Um, at the same time, I think balancing that against just practical advice on how to navigate the, the world, right?
171
00:25:50.136 --> 00:25:56.216
I think that the, what the world is really, the industry is really missing is, I think, practical advice on, on how to,
172
00:25:57.416 --> 00:26:09.196
um, you know, just see and, and, and navigate through, you know, what's increasingly a lot of, a lot more noise. Uh, I mean, this la- this latest AI hype cycle, for example, there's, um, a ton of noise, you know?
173
00:26:09.276 --> 00:26:21.656
It seems to be, you know, uh, I think it's interesting as we speak now because it feels like we could be looking at a correction, uh, I think pretty soon in terms of, uh, generative AI and, and I think the expectations of it, which is definitely healthy.
174
00:26:21.816 --> 00:26:26.576
I mean, you know, I think that, that's good. I, I, I love generative AI. I think it's, it's great.
175
00:26:26.636 --> 00:26:37.236
I love machine learning and, and it's, it's everywhere, and I don't see the utility of generative AI dying down anytime soon. At least I hope not. I, I would hate to see a lot of these tools go away. That would suck.
176
00:26:37.596 --> 00:26:48.296
Um, you know, so I-- Uh, but at the same time, it, it's, you know, when I hear things like large language models are basically the step to AGI, I'm like, maybe it is, maybe it isn't.
177
00:26:48.756 --> 00:27:01.396
But there's a whole narrative that, you know, the singularity's almost here, artificial super intelligence will be here any moment, and then, you know, we will have, um, endless prosperity and dystopia all at the same time.
178
00:27:01.676 --> 00:27:15.216
Um, and then, of course, there's the next hy- hype bible, which I think is gonna be e- make AI look absolutely timid, is, is robotics. Um, you know, and, and again, I'm bullish on robotics. I think that is awesome.
179
00:27:15.476 --> 00:27:26.716
I-- When we all grew up watching sci-fi, I'm guessing you did. I, I know I did. Yeah. And reading sci-fi and the, the notion of having a robot society, I think that is going to happen. But, you know, that's...
180
00:27:26.756 --> 00:27:33.276
You're gonna see the jump from AI to robotics and humanoids, um, real soon, right? So I think that's the next one.
181
00:27:33.336 --> 00:27:44.616
But then all of a sudden, everybody who was a crypto turned AI expert overnight magically will now become a robotics expert seemingly in the next 24 hours. So that's gonna happen.
182
00:27:44.916 --> 00:27:55.496
Everyone will be like, "Oh, yeah, like I, huge expert in robotics." Um, just like they were when they went from crypto to AI a couple years ago. Um, so, so that's gonna happen, right?
183
00:27:55.616 --> 00:28:01.176
But I think just, that's the stuff I like to write about is just I think- Mm-hmm... just, you know, 'cause I mean, you and I are old enough.
184
00:28:01.216 --> 00:28:04.956
We've been through hype cycles before and, um, you know, I think that it's fun to watch.
185
00:28:05.016 --> 00:28:16.276
I mean, my, the fa- my favorite thing to do, my favorite subjects to read about are bubbles and market manias and crashes 'cause it-- what's fascinating to me is just the, the, um, just...
186
00:28:17.736 --> 00:28:29.856
For, for as much technical innovation as, as, as humanity's seen and our civilization has achieved, it's often on the back of hype cycles, um, and, and, and bubbles, and I think bubbles are good in a lot of ways.
187
00:28:29.896 --> 00:28:40.456
I'm not saying they're bad. They're necessary. Um, but it's just fascinating that we can't seem to, uh... Well, it's back to the dopamine rush that I was talking about earlier with, with mobile. Mm-hmm.
188
00:28:40.556 --> 00:28:53.796
It's the same thing with, with, uh, hype cycles, like we want the new thing. Um, you know, and, and because we want the new things, it's often easy to overlook the fundamentals because they're not sexy. No.
189
00:28:53.796 --> 00:29:05.996
[laughs] It's like wearing a hair shirt. [laughs] So... Talking about AI and, and, and what is, what is coming, let's, let's talk about AI readiness beyond the hype.
190
00:29:06.276 --> 00:29:16.336
Uh, what, what does it take for an organization, uh, for, for its data to truly be AI ready, as, as we call it? What, what-- how do you see that?
191
00:29:16.476 --> 00:29:22.516
I mean, we can't have data without AI, no AI without data, so it's, it's that kind of loop.
192
00:29:22.916 --> 00:29:35.556
But if we say make your data really AI ready, uh, what do you think is, is, is most important to get your data in good shape to be AI ready? It's interesting 'cause I think back in the
193
00:29:36.676 --> 00:29:51.636
M- uh, I, I call it the classical ML days, right? Mm-hmm. So pre, pre-generative AI bubble. Um, back then, it was really a matter of making sure that, you know, 'cause you're typically talking about tabular data, right?
194
00:29:51.696 --> 00:30:00.796
Rows and columns. Yeah. Maybe semi-structured data. But that was data upon which you could just, you know, build a matrix off of it, which you always do anyway. Um, but
195
00:30:01.796 --> 00:30:09.116
that, um, you know, that data really needed to h- you know, the values needed to, to be accurate-ish, right? Yeah.
196
00:30:09.136 --> 00:30:18.856
So they needed to at least represent something where if you're to, uh, you know, uh, convert those into numerical form, um, you can get some signal out of it, right? So,
197
00:30:20.256 --> 00:30:24.956
you know, and, and what that's often meant is, like, there's a lot of crappy data out there, as you know.
198
00:30:25.216 --> 00:30:36.156
I mean, the, the whole industry exists because, uh, there's a lot of creation of data and a lot of maintenance that needs to occur. Um, and so the data in most corporations is an absolute horror show, right?
199
00:30:36.256 --> 00:30:45.956
Uh, values don't match, lots of duplicates, lots of crappy data. Don't even need to get into that. This is just table stakes. It's been there for decades. That's what keeps people like us employed. Um-
200
00:30:47.120 --> 00:31:01.840
But now I think with, with AI, gener- it's a bit different, right? You're talking about, um, other data. It's not tabular anymore. It's tabular plus, uh, you know, unstructured data, so text, images, video, audio, right?
201
00:31:01.920 --> 00:31:13.480
So this is, this is the, the new landscape that, like, the, the data world hasn't been thinking of. So predominantly text, for example, we're talking about text. Um, you know, i- is, uh,
202
00:31:14.900 --> 00:31:16.820
what's in the text, [laughs] right?
203
00:31:17.000 --> 00:31:25.940
If we're going to, you know, suck all this into a large language model or just a language model, it doesn't matter, matter the size, like, you know, c- how, how much signal do we get off of this?
204
00:31:26.160 --> 00:31:38.460
Um, you know, what, uh, h- if we're going to combine this with one of the, um, you know, the, the large frontier models, uh, provided by the big providers- Yeah... um, how all of these operate together, right?
205
00:31:38.480 --> 00:31:47.360
Do, do the, does the data, are we able to get signal out of the data that we have, or, or are the models we're using too generalized for our specific situation?
206
00:31:47.420 --> 00:31:55.660
So what I'm often seeing now is it's more about, it's not necessarily is your data clean or not? I, I'm always under the assumption your data sucks. It's, it's terrible, right?
207
00:31:55.900 --> 00:32:04.100
So but how much signal can you get out of this terrible data, um, that, that really applies to your situation? I think it's more the question I am looking at now because- Yeah...
208
00:32:04.120 --> 00:32:14.940
well, there was that, that recent study from MIT, um, about how generative AI is failing in industries, and what it pointed out was, you know, for simple tasks, stuff like ChatGPT, it's widely used, and it's awesome.
209
00:32:15.200 --> 00:32:23.140
It kills it. It's, it's amazing. But the problem is these models don't have enough memory and enough context of your company, right?
210
00:32:23.200 --> 00:32:27.440
So I think that's where we need to figure out, okay, so how does the data that we have provide the context?
211
00:32:28.120 --> 00:32:39.680
Um, you know, how, context in terms of the s- you know, the, the, um, uh, describing your business in a way that, um, is understandable both to you and the AI. That's one form, form of context.
212
00:32:39.720 --> 00:32:52.140
The other part is a context window, which is a separate t- type of context, but, like, how much, um, h- how big of a context window are you talking about? Um, like how much information in, in our body of, say, text, uh,
213
00:32:53.760 --> 00:32:58.480
would even be useful if we were to ask questions of it, right? Yeah. So I think these are the types of questions we're still trying to figure out.
214
00:32:58.760 --> 00:33:07.300
Um, uh, and also now, you know, semantic or, you know, semantic correctness and all this other stuff, right? Like, how does this map back to the organization?
215
00:33:07.360 --> 00:33:16.660
Do, are the, are the concepts of the organization embedded within, it's a loaded word, um, are they contained within the, the data that we have, whether it's the tabular or the text and so forth?
216
00:33:16.700 --> 00:33:24.100
So these are the, you know, that's where you're seeing the rise of knowledge graphs and semantic layers now. Um, you know, so these are the types of things I've been thinking about with data.
217
00:33:24.160 --> 00:33:33.580
So it's not just tables and all this stuff. It's definitely that, but it's a lot more. A lot, lot more. Then you, then couple on that, images, audio, video, all this other stuff.
218
00:33:33.960 --> 00:33:42.800
I mean, thought experiment, when you walk around a city, or anywhere really, look at the number of cameras around you. Oh, yeah. What's, what's, what are we doing with that data?
219
00:33:43.180 --> 00:33:52.840
You're gonna start seeing that data being blended in, but, you know, what is the data? Like, what, what are the governance or what's the governance around using video, especially in a public setting?
220
00:33:52.880 --> 00:34:02.380
Like, these are the questions that we're gonna be thinking about. Um, but again, this poses whole new challenges for data engineering, 'cause up to now, it's been providing data for mostly analytical use cases.
221
00:34:02.440 --> 00:34:12.860
Now we're talking AI use cases of which we're still trying to figure out what those are. We're, I think we're just at the very, very beginning of everything, to be frank. Like, we're, we're still in the chatbot phase.
222
00:34:12.920 --> 00:34:23.480
Everyone's talking about agents now. I still have no idea what the hell we're talking about there. Um, but that's the buzzword. Um, and then you throw in robotics on top of that. So yeah, it's gonna be fascinating.
223
00:34:23.680 --> 00:34:31.200
Like, I'm... This is what keeps me... If you want, if you wanna know what g- makes me excited, it's this stuff. It's like now... 'Cause I've been, I was bored for ages with data. Mm-hmm.
224
00:34:31.240 --> 00:34:39.800
I was like, "Okay, we're gonna talk about tables. Great. Like, I'm gonna go skateboarding or something, 'cause I'm, [laughs] I just don't want... " [laughs] "Just, I'm just, I'm just so bored." Um- I get that feeling.
225
00:34:39.820 --> 00:34:46.640
Yeah, yeah, yeah. [laughs] But you know what it makes us, it's the same thing for decades. We talk about the same thing. Like, oh, we need to get more value out of our database. I'm like, great.
226
00:34:47.280 --> 00:34:53.880
We've, we- we've been trying this for ages. Maybe it requires a different way of thinking about it. But then, like I said, the reset button's been hit on the industry.
227
00:34:54.280 --> 00:35:02.820
The fundamentals still apply, but it gives us a new chance to, to try new things out. This is, uh, to me, this is like, this is cool. This is fun, so. Yeah.
228
00:35:03.000 --> 00:35:19.280
I, I've, I've learned a bit about doing the traditional stuff, the data warehousing, uh, dimensional modeling, where everything had to be, uh, pixel perfect, and, uh, if there was, uh, no match with your dimension value, you had to put that in an error, uh, pane or something like that to keep track of, of what is missing.
229
00:35:19.980 --> 00:35:28.260
But on one project, we, we, we come to a client, and they had no clue of their overview of sales and marketing. Ooh.
230
00:35:28.280 --> 00:35:39.260
[laughs] And they were already happy with some extracts and getting a view on 80% of what is happening and their cost breakdown of the car construction, and this was a kind of eye-opener for me.
231
00:35:39.400 --> 00:35:46.640
It's more important to have any signal and that direction and that view than just purely having everything perfect. Yeah.
232
00:35:46.660 --> 00:35:54.200
Like you say, th- I think that's, that's where the large language models come in and, and will help us on, in driving that. We still need to be...
233
00:35:54.240 --> 00:36:00.400
Well, they need to become trustworthy, that they don't keep on hallucinating- Yeah... and that you know when they're hallucinating or not.
234
00:36:00.680 --> 00:36:07.240
And when that happens, I think their, then their value will, uh, will show, uh, finally. Oh, absolutely.
235
00:36:07.320 --> 00:36:18.560
And the thing, the thing that has, that the industries or the data industry suffered from is adoption and feedback loops from users, right? Yeah. So typically it's been, oh, we'll make the perfect data model
236
00:36:19.700 --> 00:36:25.060
for reports that nobody's ever gonna look at. [laughs] So it's like, "What, what are you doing?"
237
00:36:25.340 --> 00:36:34.340
Um, I'd rather have somebody use a large language model and things end up being wrong, because then they're gonna come back to me and say, "This doesn't make sense. Let's figure it out." But it shows that they care.
238
00:36:35.340 --> 00:36:44.080
Yeah. Right? So the other day in my Discord group, um, you know, I, I run a, a practical data modeling or practical data community, but somebody was asking, um,
239
00:36:44.980 --> 00:36:52.720
you know, about a data project, project they were hired to do, and in this case it was setting up a, you know, they, they're charged with setting up analytics and reporting, pretty standard fare.
240
00:36:53.956 --> 00:37:03.576
And so this person asked, "Do I spend six months setting up, you know, DBT and tables?" And I said, "No, you need to spend the next two days making an Excel report and trying to answer the questions people want.
241
00:37:04.176 --> 00:37:13.676
And if you can put that in front of them, then you got quick wins, and then they trust you, and then they'll wanna work with you. If not, you're, you're the guy that showed up on day one, and then six months later
242
00:37:14.896 --> 00:37:23.196
you might have something for them, but you're not interacting with them." I was like, "Just do the low fidelity thing. I... You know, make a spreadsheet, show it to them. Um, refresh it."
243
00:37:23.676 --> 00:37:30.956
You know, Google Sheets, for example, right? I... And, and so I, I think I'm, I'm a fan of just doing things the low fidelity way. I mean, shit, write it on a piece of paper and go show it to them.
244
00:37:31.256 --> 00:37:34.136
Um, but, like, that interaction is what's really missing.
245
00:37:34.176 --> 00:37:45.436
But again, the large language models and things that provide quick feedback, like, that's, that to me is a net positive if it can be harnessed in a way where, um, everyone is learning and making it better, right?
246
00:37:45.516 --> 00:37:55.516
I see no downside to that. I, I prefer hallucinations actually over the, the picture perfect data model that nobody has ever used. That's a complete waste of time. That's a tragedy.
247
00:37:55.856 --> 00:38:03.996
That's a waste of time t- the person making the model too, right? 'Cause, like, what is the point of it? You have other things to do. Yeah. Maybe, I don't know.
248
00:38:04.036 --> 00:38:08.416
Maybe you're, maybe that's, maybe that's your only thing in life. I have no idea. Um- Well- It's hard for me to say.
249
00:38:08.416 --> 00:38:19.196
[laughs] I've been building enterprise data models, and then the question came up in my mind for who's gonna use it and, and how can we- Right... make it usable and, and, and valuable for the, for, for the business.
250
00:38:19.456 --> 00:38:26.676
So that's, that's the communication part where they always- Right... struck me and, and frustrated me at the same time. So [laughs] this- Mm-hmm...
251
00:38:26.736 --> 00:38:37.276
this is where the large language models come in and, and help us in, in going faster on that. So do you think that AI is changing the, the data life cycle in itself or just the tools within it?
252
00:38:37.456 --> 00:38:46.116
And, and how should t- data teams think differently about architecture in the age of LLMs? Yeah, I think it's changing.
253
00:38:46.896 --> 00:38:58.216
I don't-- I think the fundamental life cycle doesn't change, but what h- what does change is you're dealing with more types of data arriving at different velocities and speeds, um, for more use cases, right?
254
00:38:58.296 --> 00:39:08.156
So I think the variety has just grown as well as the velocity and the, the volume, right? So the, the three V's of, of big data. Um, and I, I do th- so there's a couple things.
255
00:39:08.216 --> 00:39:17.056
Obviously, the different types of data that, that's... We've talked about that. I also think that, um, you know, what you're seeing right now is, is real time is becoming
256
00:39:17.996 --> 00:39:26.656
much easier to use, so I called this the live data stack in, in the book a few years ago. And the, the prediction was, and I think it's actually coming true, is
257
00:39:27.596 --> 00:39:41.736
we had the modern data stack, and that was, that simplified the access of, you know, what were once very expensive, very complicated OLAP tools to people, you know, who could simply put in a credit card and away you go.
258
00:39:41.776 --> 00:39:45.456
You give it your data, and that's it. Snowflake, for example, right? It built from the ground up from the cloud.
259
00:39:45.996 --> 00:39:56.776
I think Redshift was the first attempt at this, actually, where, I think it was November of 2012 they announced Redshift. It was, what? 25 cents per, um, hour for a data warehousing process unit, I think.
260
00:39:57.296 --> 00:40:02.416
If you think about that, though, buying a data warehouse back in the day before Redshift, what was it?
261
00:40:02.476 --> 00:40:08.616
You'd have to sit there with a vendor and negotiate and go to these, like, stupid golf tournaments and dinners and all this stuff, and it, you know...
262
00:40:09.096 --> 00:40:16.036
And at the end of the day, strong arm you into signing this contract where you could never leave, or you'll pay a lot of penalties, and, like, that's gone. I don't know anybody doing that right now.
263
00:40:16.076 --> 00:40:24.336
But, I mean, the, the, these, there are some companies still signing these contracts, but, like, that, you, you have the option of not doing that. You have the option of going to the cloud and d- you know.
264
00:40:24.576 --> 00:40:34.496
Or now even, like, DuckDB, you can just get pretty badass data warehouse on your laptop for free, actually. And so you have a lot more options. But that was the world then.
265
00:40:34.576 --> 00:40:42.156
So what the da- modern data stack did is it simplified access to all these analytical tools that were once expensive. The same thing is happening right now with real time.
266
00:40:42.796 --> 00:40:51.836
Real time, I think, is undergoing a very similar change where tools are becoming, real time tools are becoming simpler, um, easier to deploy, uh, cost-effective, and so forth.
267
00:40:51.896 --> 00:41:04.696
So I think what you're gonna see is real time becoming more front and center, to the point where, um, you know, I sometimes joke 'cause it's not necessarily a good, uh, one-to-one comparison, but, like, I think batch might go the way of the dial-up modem at some point,
268
00:41:05.816 --> 00:41:06.116
so.
269
00:41:07.256 --> 00:41:20.736
Yeah, that's, that's the same feeling I see already for a few years, but it was always complex, uh, setting up these, these, uh, real time data flows and, and streaming systems where I had the feeling this is where we need to go- Yeah...
270
00:41:20.796 --> 00:41:32.476
in, in, in that direction. Not that you need that real time data, but it's just how, uh, systems evolve and help you get faster value out of, off, off your data. Absolutely.
271
00:41:32.856 --> 00:41:38.756
But then if you're talking about agents, right? So this is where it gets interesting. So a, a fun thought experiment I've been asking people to do is ask- Mm-hmm...
272
00:41:38.956 --> 00:41:52.076
ask your favorite AI or AIs, um, if, if you, the AI, were to design data models and infrastructure for agents, what would it look like? Right? This, it, it- Right...
273
00:41:52.116 --> 00:42:01.056
so the answers I've gotten back from my AIs, uh, and I've asked other people to do this, and it's actually pretty similar, is, is real time knowledge graphs and semantic layers, right?
274
00:42:01.176 --> 00:42:06.836
That, uh, that afford different agents the, the ability to discover- Mm-hmm... different data and update that in real time.
275
00:42:06.896 --> 00:42:13.196
And, and that's a connective tissue, I think, that is also coming up, but you need the streaming parts to do this, and you need the graphs.
276
00:42:13.236 --> 00:42:23.856
And I think this is where it's all coming together, um, where you're talking about sort of a post-human, um, data infrastructure and, and systems. And so people ask me what do I mean by AI native infrastructure.
277
00:42:23.896 --> 00:42:27.916
This is exactly what I'm talking about. This is not the infrastructure that humans are gonna be using.
278
00:42:28.056 --> 00:42:36.456
This is the infrastructure that AI is gonna be using, that we happen to be maybe overseeing, but I think it can probably do a good enough job overseeing it itself, actually. Yeah.
279
00:42:36.496 --> 00:42:44.276
And so the, to, to me the question is, what are humans gonna be doing? Um, you know, we'll still have data f- you know, and systems designed for humans, which we've had.
280
00:42:44.316 --> 00:43:00.152
This, these are the systems we, we've, we built for ourselves. But if you think about what's next, it, it, it, it could, it's likely fundamentally different Joe, we're up to our rapid fire round. Okay.
281
00:43:00.152 --> 00:43:14.132
[laughs] Few, uh, few one or the others. So what do you think? AI-ready data or business-ready data? Oh, man. I think on my last point, I think there's a bit of nuance here, right?
282
00:43:14.892 --> 00:43:23.272
Um, where if you, if you kind of extrapolate out 10 years from now, what i- what, what is a business? Um, I don't know.
283
00:43:24.172 --> 00:43:31.812
Uh, if, if the supposition is that AI runs more and more of the business operations, I, I've been, I've been thinking a- about this a lot, where-- This is not a rapid fire answer, by the way.
284
00:43:32.332 --> 00:43:41.112
Um, but what is, what is the nature of an organization when AI is running everything? I, I think fundamentally a, a lot of things change, uh, structurally for companies. Yeah.
285
00:43:41.152 --> 00:43:52.752
So you have AI-ready businesses, I think, or businesses ready for AI. I don't know. Yeah, but that's, that's a big question. Maybe I'm overthinking it. I don't know. [laughs] So next one.
286
00:43:53.592 --> 00:44:08.472
Governance first or innovation first? [laughs] Uh, um, I'm a fan of innovation. I would say innovation first.
287
00:44:08.652 --> 00:44:13.872
I, I think governance is one of these things where, to me, it's, it's, uh, um... I don't know.
288
00:44:13.892 --> 00:44:22.672
But it-- to me, it's almost a false dichotomy because you, you kind of need the governance to be, uh, to make sure your innovation lasts longer than a few seconds. Um, [laughs] so...
289
00:44:23.152 --> 00:44:32.352
But if it's just governance, then that's no fun either, right? I mean, what, what-- If you're, if you're governance only, then what do you have? I think you have the Soviet Union, um, [laughs] so, with no innovation.
290
00:44:32.652 --> 00:44:44.212
Um, so. Okay. That'll trigger probably a few people in your, in your listenership, but whatever. Yeah. Um, so [laughs] The modern data stack or the modern data scrap?
291
00:44:46.672 --> 00:44:55.412
Uh, I mean, I'm a fan of what the modern data stack had, had done. I, I think it definitely helped, uh, move a lot of things forward for the industry. Um, as I wrote, I, I feel like it's, uh...
292
00:44:56.392 --> 00:45:05.652
We can put, we can put the term to rest. So, um, uh, what was the latter term? The modern data s- Scrap... s-scrap?
293
00:45:06.191 --> 00:45:12.532
I, I think it can go-- I, I wouldn't-- I, I think it-- I, I wouldn't say it needs to go to the scrap heap of history, 'cause I think that's something more, um,
294
00:45:13.452 --> 00:45:18.612
uh, like throwing, uh, you know, statues of dictators into the, uh, you know, into the scrap heap of history.
295
00:45:18.712 --> 00:45:30.732
But I, I said, I think, uh, maybe we build a monument to it and say, you know, "The, the modern data stack did what it did," Um, you know, "Bless its heart. Time to move on." So... [laughs] So next one.
296
00:45:31.332 --> 00:45:46.232
Skepticism or optimism in AI? Oh, man, I'm, I'm probably right smack dab in the middle of that. Um, I think it-- For me, it depends on what, which, which, which timeframe you're talking about.
297
00:45:46.292 --> 00:45:52.952
I think in the immediate term, I'm always skeptical. Uh, long term, I'm always optimistic. Um, but it's never really the reverse.
298
00:45:53.012 --> 00:46:01.632
I'm never optimistic short term, um, just 'cause I've-- I think, uh, just by my nature, I've studied too many hype bubbles to, um, to be optimistic.
299
00:46:01.732 --> 00:46:08.932
I, I, I usually-- If I'm optimistic, it's in the sense where I'm really greedy and I try and profit off of the bubble. Um, but, but long term, I'm always optimistic.
300
00:46:08.992 --> 00:46:19.632
I fi-- I feel like technology is, is what pushes civilization forward, and, and innovation. So long term, I'm always optimistic. Okay. Except for, like, nuclear weapons and stuff. I don't think that's a good idea.
301
00:46:19.812 --> 00:46:28.072
But we-- That was an innovation that, you know [laughs] um, yeah. [laughs] So... But who knows?
302
00:46:28.112 --> 00:46:38.672
I mean, the, you know, if, if you-- I mean, I've had, I've had, uh, you know, AI doomers on my podcast before, and, and they, you know, they had very convincing arguments the other direction, where there's nothing to really be optimistic about, so I don't know.
303
00:46:39.132 --> 00:46:49.112
Maybe this time- Right... is different, I guess. So, but... Okay. Teaching or building? I think it's hard to teach unless you, you've built.
304
00:46:49.972 --> 00:47:05.632
Um, but I feel like to be a really good builder, to understand your craft, it does help to teach. And so, um, yeah. And in fact, I'm working on, uh, you know, my, uh, i- an article I might write, uh, really soon is about
305
00:47:06.532 --> 00:47:15.352
this dichotomy of building and teaching. To me, they're sort of inseparable, but I, I a- I also, um, like to teach. But, um, I think you, you should do both.
306
00:47:16.452 --> 00:47:23.612
You should build-- I think if you're teaching and you, and, and you don't build, you probably don't know what you're doing, or you, you, you're, you know, you're just regurgitating things.
307
00:47:24.392 --> 00:47:34.772
Um, but if-- I, I think at a certain point, if you've been building and, and, you know, teaching is a way of giving back, and I also think it de- definitely shows whether or not you actually do understand what you're talking about.
308
00:47:35.832 --> 00:47:45.092
Yeah. Teaching's hard. [inhales] Completely get it. Yeah. So the last one, clarity or complexity? Hmm.
309
00:47:50.532 --> 00:48:05.092
To me, I, I think you only get clarity once you understand the complexity. But complexity is easy. Complexity, I think, happens first. Mm-hmm. Um, 'cause you don't know-- you don't have clarity on things. Yeah.
310
00:48:05.192 --> 00:48:15.252
But at some point, you, if you truly understand what you're doing, um, and you truly understand the domain that you're in, uh, clarity arises. But I don't think the opposite ever happens. Maybe I could be wrong.
311
00:48:15.492 --> 00:48:29.372
I don't know. It's a, it's a complex question that I, um, am unclear about. So, um... [laughs] Okay. Well, we-- when we compare through the complexity, [laughs] we'll get back in touch and, and discuss it further. Yeah.
312
00:48:29.392 --> 00:48:36.352
But I, I find it striking on, on how you think about these things as well, and that makes me think and, and trigger my thinking as well.
313
00:48:36.512 --> 00:48:43.972
So, uh, I think you, you get an absolute point about the complexities just happens, and then you need to create the, the, the clarity.
314
00:48:44.412 --> 00:48:49.852
If I look at the mechanisms that are out there in the world, this is how I see it's, it's happening as well.
315
00:48:49.992 --> 00:48:58.752
We're explorers, discoverers, and trying to understand the world around us, and I think that ties in where you say first complexity and then clarity comes in.
316
00:48:59.740 --> 00:49:06.020
Yeah, I'm trying to think of where the opposite would happen, where you have, you have instant clarity, but then it becomes complex. Um, I, I don't...
317
00:49:06.180 --> 00:49:15.419
I, I think if you have instant clarity, either you haven't tried hard enough to understand the situation, or the situation is, is dead simple, um, in which case there's nothing to, um, to think about.
318
00:49:15.540 --> 00:49:19.660
But I'm trying to think of that, where that would fit. I don't know. Again, I could be overthinking it.
319
00:49:19.720 --> 00:49:29.300
I do that a lot, but you'll, you'll notice my, my, my answer's always very nuanced 'cause I, I think it's, it, it's always the, it depends on what you're talking about. [laughs] So.
320
00:49:29.340 --> 00:49:44.100
So Joe, writing a widely used book, advising across industries, and, uh, publishing long form, uh, you're wearing many hats. What, what drives you personally in, in this mission? Um, I mean, I, I...
321
00:49:47.320 --> 00:49:59.420
There's a few things really. I, I feel I'm at the stage in my career where I feel like I really wanna give back to, you know, um, you know, the current generation as well as the next generation of, of, uh, practitioners.
322
00:50:00.380 --> 00:50:08.680
Um, I, I feel that, you know, I was blessed by having a lot of great mentors, and I, you know, would love to, um, you know, give back to the world in that way.
323
00:50:09.500 --> 00:50:16.860
I'm also very driven by pursuing what's next in the industry, right? And so that, that's what a lot of what I've been working on too.
324
00:50:16.920 --> 00:50:25.260
I think by writing about these things, whether it's modeling or engineering, it helps me clarify my own opinions on, on where I would wanna take the industry.
325
00:50:26.020 --> 00:50:33.600
Um, you know, I'd like to, you know, I, I'm thankful I'm in, I'm in a position where I can help drive that direction now of the industry, and I feel that that's,
326
00:50:34.520 --> 00:50:42.600
um, you know, also what drives me is just what, what's next, right? But the thing I've been giving talks about lately is everybody has this opportunity now.
327
00:50:42.640 --> 00:50:49.700
The reset button has been hit for everybody in almost every industry, not just tech, but, you know, marketing and a bunch of others. And
328
00:50:50.760 --> 00:51:00.840
this is a time when you, when you can really rise to the occasion and, and, you know, create new paths, not just for yourself, but for a lot of other people. Um, and this doesn't happen very often.
329
00:51:01.160 --> 00:51:10.500
And so I think that's, you know, that's, that's what drives me is just a sense of excitement of, of, you know, building what's next, not just waiting for it to happen. Exactly.
330
00:51:11.120 --> 00:51:25.840
So as a cl- as a closer and, and a fun note, data connects us all, but also music. So here we are. Yeah. [laughs] What is your favorite band or music genre, and does it influence how you think or work in any way?
331
00:51:27.160 --> 00:51:39.520
I don't know if I have a favorite band. Um, I listen to a lot of music. But I, um, I have a lot of, I have a lot of favorite genres too. I think the, the stuff I, you know, obviously I make music. I've, um, you know,
332
00:51:40.520 --> 00:51:50.960
uh, produced, you know, house techno for, for decades. It's what I also DJ. A lot of rare disco too, um, that kind of stuff. But I, I listen to all kinds of stuff. I mean, I was listening to, uh,
333
00:51:51.860 --> 00:52:01.660
um, you know, '80s punk the other day and, uh, you know, so I, you know, and hip hop, so I listen to all kinds of music really. Um, so yeah. How about yourself?
334
00:52:02.920 --> 00:52:14.160
Well, about myself, uh, like I said, I'm a bit more the, the house fan, the deep house. Ev- everything that- Yeah... melody, that drives energy, that's, that's, that's my passion in, in the music.
335
00:52:14.360 --> 00:52:27.020
And sometimes it can be very hard techno and sometimes more melodic. Mm-hmm. Depends a bit on the mood. Uh, it's very- Exactly... it's very strange when I say this is my, my top genre, and sometimes I hate it.
336
00:52:27.280 --> 00:52:32.080
It doesn't feel like it's resonating with me. Uh, and that's the fun part of, of music.
337
00:52:32.120 --> 00:52:46.440
And if I'm going to concerts or to festivals, I'm always analyzing the people and see how they respond to the DJ and, and how that works, and that's, that's what, what I find intriguing as well on, on DJing, bringing that energy.
338
00:52:46.940 --> 00:52:49.180
And I feel- Yeah... the same in, in, in how data works.
339
00:52:49.260 --> 00:52:59.500
If you can put that [laughs] da- data into sync and, and, and bring it in, into flow, that's, that's where you bring the value, uh, to businesses or whatever you're doing. Oh, absolutely.
340
00:52:59.540 --> 00:53:10.660
And you're probably the, one of the only people who would understand this being a, a DJ as well, but there's a lot of parallels between DJing and, and data work actually, and it's, this isn't one of those, you know, what DJ, DJing taught me about B2B sales.
341
00:53:10.900 --> 00:53:23.440
Uh, but it's, it's like, um, [laughs] the, it, there's... When you're DJing, there's, you have to have crowd awareness, right? And, and you have to understand, like, what is the vibe like right now?
342
00:53:23.440 --> 00:53:30.600
'Cause you're helping create it, but you're also following it, right? It, you know, the stuff you play early on in a set isn't gonna be the stuff you play later on in a set.
343
00:53:30.620 --> 00:53:40.980
It depends on what time of the, of the, you know, usually at night you're playing. If you're doing the opening set, it's gonna be more chill. You're not gonna come out with like 170 BPM bangers, you know? [laughs] Mm.
344
00:53:41.060 --> 00:53:49.340
I hope not. [laughs] Um, you know, gabber or whatever. But then, you know, as you get later in the night, right, it's a different scene, but you have to know. But the same with data work, right?
345
00:53:49.400 --> 00:53:59.160
Like, if you know the audience you, you know, and the stakeholders you're working with, read the room, right? I think being a DJ has this invaluable thing where you, you, there's a lot of pressure to perform.
346
00:53:59.240 --> 00:54:09.700
Like, if you, if you mess the party up, that's on you actually. [laughs] Yeah. So, like, but there's a lot of, there's a lot of parallels to, to working with people in this regard, and so. Yeah. Exactly.
347
00:54:10.540 --> 00:54:23.460
Joe, it was so great having you. Yeah. Thank you. Where can people find you online? Um, website, email addresses, whatever you wanna share. Yeah. Yeah. I'm on, uh, LinkedIn, I think for better or for worse.
348
00:54:23.680 --> 00:54:34.500
Um, you can find me there. Uh, usually I have very, uh, fun profile pics, um, uh, very unserious. Uh, there's also, uh, joerice.substack.com.
349
00:54:34.560 --> 00:54:41.820
That's where I do a lot of my personal stuff, and, uh, I, I'm gonna be focusing a lot more on Substack. And, um, there's also, I have, I'm also on YouTube. You can find me there.
350
00:54:42.180 --> 00:54:56.780
Um, but I'm gonna be focusing a lot more on, um, YouTube and Substack going forward. And so LinkedIn's there, but whatever. So [laughs] thanks. Okay. Joe, thanks for your time. Great having you. Yeah. Of course. Thanks.
351
00:54:56.780 --> 00:55:10.720
[outro music] Thank you for joining us on this awesome podcast. As senior executives, data and analytics architects, and AI professionals, your time is valuable and we appreciate you choosing to hang out with us.
352
00:55:10.980 --> 00:55:17.280
If you like what you heard, please give us a thumbs up, hit the subscribe button, and leave a comment. We love hearing from our audience.
353
00:55:17.500 --> 00:55:25.480
Don't forget to spread the word on social media and let's continue to drive innovation in the industry together. Thanks for listening, and we'll catch you on the next episode.
354
00:55:26.480 --> 00:55:40.260
[outro music]