How To Bring Agile Practices To Your Data Projects

WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 7/6/2024 1:27:06 PM
Duration: 4337.480
Channels: 1
1
00:00:14.195 --> 00:00:18.350
Hello, and welcome to the Data Engineering Podcast, the show about modern data management.
2
00:00:18.890 --> 00:00:21.310
Atlin is the metadata hub for your data ecosystem.
3
00:00:21.770 --> 00:00:28.045
Instead of locking your metadata into a new silo, unleash its transformative potential with Atland's active metadata capabilities.
4
00:00:28.665 --> 00:00:32.125
Push information about data freshness and quality to your business intelligence,
5
00:00:32.620 --> 00:00:40.560
automatically scale up and down your warehouse based on usage patterns, and let the bots answer those questions in Slack so that the humans can focus on delivering real value.
6
00:00:40.985 --> 00:00:42.605
Go to data engineering podcast.com/atlan
7
00:00:43.945 --> 00:00:54.830
today, that's a t l a n, to learn more about how Atlan's active metadata platform is helping pioneering data teams like Postman, Plaid, WeWork, and Unilever achieve extraordinary things with metadata.
8
00:00:55.530 --> 00:01:02.355
When you're ready to build your next pipeline or want to test out the projects you hear about on the show, you'll need some to deploy it. So check out our friends at Linode.
9
00:01:02.815 --> 00:01:14.060
With their new managed database service, you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes with automated backups, 40 gigabit connections from your application hosts, and high throughput SSDs.
10
00:01:14.920 --> 00:01:15.740
Go to dataengineeringpodcast.com/linode
11
00:01:17.935 --> 00:01:27.395
today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services. And don't forget to thank them for their continued support of this show.
12
00:01:28.090 --> 00:01:36.670
Your host is Tobias Macy. And today, I'm interviewing Shane Gibson about how to bring agile practices to your data management workflows. So, Shane, can you start by introducing yourself?
13
00:01:37.165 --> 00:01:42.785
Hi. Yeah. I'm Shane Gibson, currently the cofounder and chief product officer of a startup called Agilentat.
14
00:01:43.565 --> 00:02:01.005
Io. And my background has been, for a while, coaching data and analytics teams, how they bring agile patterns and make them useful. Just love like to say I'm, you know, longtime listener, first time caller. So love the podcast and great to be on it. Yeah. Happy to have you here. And do you remember how you first got started working in data?
15
00:02:01.385 --> 00:02:12.310
Yeah. So it's kind of embarrassing. I've been in the data world for almost 32 years, I think, as I thought about it. And I started my career out working in a finance department doing accounts payable, so doing some of the invoice processing.
16
00:02:12.610 --> 00:02:15.685
And I didn't enjoy it. I found the work quite mundane,
17
00:02:15.985 --> 00:02:20.084
and I was lucky enough to fall into a row around the thing we call systems accounting.
18
00:02:20.385 --> 00:02:30.410
So at the time, we had a time time sheet system, and so I picked up the technology side of that. And just to date myself, I think the first server we bought was a compact 386
19
00:02:31.325 --> 00:02:38.385
SX 25. So those cost us about 50, 000 New Zealand back then and had, I think, you know, 512 k of RAM.
20
00:02:38.810 --> 00:02:45.310
So really started enjoying that working in the technology space and was lucky enough to move to another organization
21
00:02:46.175 --> 00:02:52.755
where I was put in charge of replacing the financial system, moving it off the old mainframe, the m data source across to client server.
22
00:02:53.270 --> 00:02:58.890
And as part of that, we were experimenting with what was called executive information systems back then, EIS,
23
00:02:59.190 --> 00:03:04.795
and playing around with talking forests and trees to take their financial information and make it available to our CFO.
24
00:03:05.495 --> 00:03:31.209
So after we implemented that platform, I moved jumped across to the BI vendor world. So for some of the large US vendors but based out of New Zealand. My first gig was kind of doing pace enterprise resource planning in that financial software market. And again, it wasn't my passion, but I was lucky enough at the time that the vendor had a really BI and data stack. So I jumped swiftly across into that team and spent, you know, 10 to 15 years working for those large vendors,
25
00:03:31.510 --> 00:03:34.489
but based out of New Zealand and that data and analytics space.
26
00:03:34.985 --> 00:03:57.504
After that, I founded a consulting company, Optimal BI. And so grew that to your usual type of, you know, data and analytics consulting company. I think when I was leading it, got to run about 20 consultants. So we'd go into a customer and and do that strategy work and figure out where they wanted to go and then bring the rest of the team in to help deliver it. And as part of that, I got frustrated with the standard consulting methodologies.
27
00:03:58.125 --> 00:04:18.245
I found that they didn't work. And so stumbled across this thing called Agile, which at the time for me was I thought a bit of a hairy, fairy, weird religion, you know, the whole kumbaya thing. So we experimented with with it internally with the team. And then was lucky enough to experiment with customers with their teams to see how it work. And spent the last 8 years effectively
28
00:04:18.590 --> 00:04:24.610
being a agile data coach. So working with data and analytics teams exclusively about how they can adopt agile practices.
29
00:04:25.230 --> 00:04:38.000
And then to finish it off about 3 and a half years ago, cofounded agile data dot io with a good colleague of mine, Nigel. And so been kinda leading that that product side of that company for the last 3 and a half years. And so in terms of
30
00:04:38.340 --> 00:04:38.840
the
31
00:04:39.180 --> 00:04:39.680
overall
32
00:04:39.980 --> 00:04:46.615
practice of bringing some of these agile principles into the data domain, I'm wondering if we can start by talking about some of the,
33
00:04:46.995 --> 00:04:48.455
I guess, industry verticals
34
00:04:48.755 --> 00:04:49.255
and
35
00:04:49.715 --> 00:04:56.090
existing patterns that you're coming into as you start to work with some of these businesses and say, Okay, well,
36
00:04:56.630 --> 00:05:11.945
this is what's challenging for you right now. These are some of the ways that Agile can solve that problem for you. If I look at it, everybody has a data problem. Whether you're a small organization, a big organization, whether you're in a financial services vertical, whether you're in an ensure you know, a government vertical,
37
00:05:12.630 --> 00:05:17.690
health. Right? Everybody has a bunch of data, and they want to leverage that data to make decisions.
38
00:05:18.150 --> 00:05:21.450
So from my point of view, I don't tend to work in specific verticals
39
00:05:21.975 --> 00:05:25.514
when I'm coaching teams. But what I do see is I see these
40
00:05:25.975 --> 00:05:29.435
basically 2 times that a organization will engage me as a coach.
41
00:05:29.770 --> 00:05:30.270
1,
42
00:05:30.650 --> 00:05:51.260
which is ideal was when they're starting their journey. Right? They got a way of working at the moment that's not working for them, and that's the key. Right? There you go. What we're doing right now is not working for us. We wanna make a change. And they've seen, you know, this agile way of working or talk to people who've done it before or read about it, and they wanna try to see if it it makes the world a better place in terms of their organization and the teams.
43
00:05:51.640 --> 00:05:55.980
So what we'll do is we'll start from scratch. Right? The team will start their journey
44
00:05:56.555 --> 00:06:06.095
from that point, and I will coach and help them move forward. And the other way that tends to happen is the team have been experimenting with agile for a while, normally 6 to 12 months.
45
00:06:06.500 --> 00:06:10.440
They've either had no success from the goals that they want to achieve
46
00:06:11.060 --> 00:06:16.120
or they've started off and it's been working for them and then they kinda hit a wall. Right? They get stuck
47
00:06:16.485 --> 00:06:22.265
around iterating their process. And that's the other time that I tend to get bored in to help teams change the way they work.
48
00:06:22.645 --> 00:06:23.945
As far as the
49
00:06:24.540 --> 00:06:28.800
kind of paradigms that have been dominant in the data ecosystem,
50
00:06:29.980 --> 00:06:35.845
there have particularly been in recent years a number of attempts to bring some kind of software methodologies
51
00:06:36.225 --> 00:06:41.650
into the data domain with varying levels of success, with some of the notable standouts being things like
52
00:06:42.050 --> 00:06:46.550
data ops and version control, both for the code and data versioning,
53
00:06:47.250 --> 00:06:48.150
you know, auditability,
54
00:06:48.610 --> 00:06:50.310
testing, kind of observability.
55
00:06:51.415 --> 00:07:03.240
And I'm wondering what you see as some of the aspects of agile that do map well into some of the existing data paradigms and some of the ways that it either falls short or introduces
56
00:07:03.940 --> 00:07:09.400
maybe brittleness or kind of extra work that is just not really providing value?
57
00:07:09.955 --> 00:07:35.729
Yeah. So the way I think about it now is when we talk about agile, it's a way of it's a form of mindset. And 1 of the downsides is a lot of people think agile equals scrum, and that's not true. There's a bunch of patterns out of lean, out of flow, out of XP, out of some of the other agile ways of working that have value to a data team. I think the second thing is people often focus on 1 perspective of agile. So they'll focus on something like techno call patterns that you've talked about, you know, CICD,
58
00:07:36.030 --> 00:07:38.050
version control, those kind of things.
59
00:07:38.430 --> 00:07:41.970
And the way I think about it now is I think these 4 lenses we can use.
60
00:07:42.345 --> 00:08:04.375
And I talk about patents. Right? And a patent is something that has value in a certain situation. Right? It's something that's been used before that if I look at the context of the way it was used, it fits our context. And and if we apply it, we potentially will get some value out of it. And I break those patterns down into 4 groups. So I talk about team topology organization patterns. So this is the way the teams are structured.
61
00:08:04.754 --> 00:08:12.790
You know, do we have 1 single team? Do we have multiple teams? How do they interact? How do they fit into the organization? So that whole pattern around team topologies is important.
62
00:08:13.410 --> 00:08:16.875
The second thing I talk about is process or practice patterns.
63
00:08:17.175 --> 00:08:21.995
That's, you know, what are the things the team are gonna do to get the data from the beginning to the end?
64
00:08:22.349 --> 00:08:31.715
I'm a great fan of the concept called how to make toast. So if you Google that and look at YouTube video, it's a great process to work with a team to say, actually, what is the work they do every day and how does it flow?
65
00:08:32.115 --> 00:08:34.615
The third 1 I think about is technical patterns.
66
00:08:35.154 --> 00:08:46.750
You know, those idea of version control, those idea of checking of managing tests before you deploy that idea of data modeling. Right? And in the data world, we have some unique things about that, which we can talk about in a second.
67
00:08:47.130 --> 00:08:53.585
And then the last 1 I think about is way of working patterns. How do we take all those other things and put it together and create our own way of working?
68
00:08:54.045 --> 00:08:57.505
And with teams, I really encourage them not to adopt a methodology.
69
00:08:58.120 --> 00:09:01.180
So there's a big push in the world to adopt scout agile safe,
70
00:09:01.639 --> 00:09:03.180
and I am very
71
00:09:03.880 --> 00:09:23.080
negative in my view of safe. Agile is not a methodology. Right? It's a way of working, which says we're gonna iterate. We're gonna get value to the customer early. We're gonna get feedback. And then when something's not working, we're gonna change it. And so we don't wanna pick up a methodology. We wanna craft our own way of working. But by leveraging patterns that have value, so we don't have to do it from scratch.
72
00:09:23.464 --> 00:09:24.365
1 of the interesting
73
00:09:25.065 --> 00:09:27.324
aspects of kind of the
74
00:09:27.704 --> 00:09:28.524
agile principles
75
00:09:28.985 --> 00:09:30.845
in the data world is that
76
00:09:31.500 --> 00:09:36.560
1 of the kind of predominant aspects of agile is that you want to focus on a
77
00:09:36.940 --> 00:10:01.894
fully connected end to end flow with a very narrow scope where you say, I want to, for instance, you know, add a new input form for somebody to be able to, you know, give me their email. So that means I have to have the UI. I have to implement that. I have to create the database model so that it can store the field. I have to make sure that, you know, the controller, middleware, and the web application is able to receive that input and write it to the database. I need to have tests around all of that.
78
00:10:02.435 --> 00:10:04.774
And in, you know, a web application,
79
00:10:05.315 --> 00:10:14.589
you know, workflow that's fairly straightforward. It's well understood how to actually do that end to end flow with that narrow scope. Whereas in the data domain,
80
00:10:15.134 --> 00:10:24.355
it's not always clear how to manage the appropriate chunking because before you deliver all the information to the end user, you have to think about things like governance, data modeling,
81
00:10:24.830 --> 00:10:28.290
you know, the kind of life cycle of the data, data cleanliness.
82
00:10:28.990 --> 00:10:30.770
And so I'm curious how you
83
00:10:31.390 --> 00:10:39.365
think about how to approach that question of what does that quote unquote narrow slice look like, and how do we reduce the scope
84
00:10:39.740 --> 00:10:44.960
from end to end without, you know, starting at the beginning and saying, oh, well, now I actually have to do the entire
85
00:10:45.340 --> 00:10:55.275
horizontal layer of staging the raw data across the board before I can even go to whatever the next stages are. And so you're able to do that kind of, like, deep integration instead of wide integration.
86
00:10:55.879 --> 00:10:59.420
I think there's 2 questions on there if I unpack it. There's the question of
87
00:10:59.800 --> 00:11:06.274
why do these software engineering practices that are well established and really successful seem to struggle when we apply them to the data domain?
88
00:11:06.575 --> 00:11:31.980
And then the second question is how do we thin slice? Right? How do we take this big behemoth that, you know, we typically used to spend 3 years building out and how do we bring it down into Wix? Right? And do that in a, in a successful way. So if I go back to that first 1, I've really struggled to understand why software engineering practices and agile from a software engineering point of view is difficult in the data domain. Because in theory, it's very similar.
89
00:11:32.360 --> 00:11:42.375
The best I've got at the moment, as you said, when you're building a web application, you're in control of the data. You control how that data is created, how it's entered, how it's landed, how it's stored.
90
00:11:42.755 --> 00:11:46.135
And the data domain, we effectively get given that data as exhaust.
91
00:11:46.550 --> 00:11:57.825
So we have absolutely no control and we get a massive amount of uncertainty. And that brings a lot of the problems to the data world that we don't have in the software engineering world. I think the second part is
92
00:11:58.365 --> 00:12:00.065
the tools that we have
93
00:12:00.685 --> 00:12:07.510
in the data world for adopting natural working. We're in the stone age. Right? The tools we have are not fit for purpose.
94
00:12:08.210 --> 00:12:11.110
They're based around big chunks of work happening.
95
00:12:11.490 --> 00:12:39.770
And so we've got to find ways of fixing those 2 problems. And we started to see that. We started to see our tooling get more agile in terms of the way we work. And we've found techniques to break and solve that problem around that uncertainty. We'll never solve the uncertainty unless, you know, we do go full data mesh where the software engineers are actually producing fit for purpose data. And that's a dream we've had for 30 years. I don't think we're gonna achieve it, but we might. We are let's look at it. And why don't I think we're gonna achieve it? Because
96
00:12:40.150 --> 00:12:42.330
if I'm the product owner for an organization
97
00:12:42.895 --> 00:12:45.155
and I need a new field on my form
98
00:12:45.615 --> 00:12:52.590
to go out to engage with my customer, and I have to make a trade off decision before that field turning up to this week.
99
00:12:53.070 --> 00:13:03.915
Or you doing the data work for that field to turn up next week. Unless I really am a data driven organization, I'm gonna make a trade decision, which is push that field out and give me that customer value.
100
00:13:04.375 --> 00:13:07.675
And then come and do the data. Right? And then I'll potentially
101
00:13:08.190 --> 00:13:13.170
reprioritize some other work. So I think it's an organization priority problem more than anything else.
102
00:13:13.790 --> 00:13:24.454
So if we go back to okay. So we get given this data we can't control. So has a massive amount of uncertainty. Our tools aren't fit for purpose. What techniques do we have to fix that? And so what we wanna look at is thin slicing.
103
00:13:25.130 --> 00:13:33.310
So teams will either take 1 or 2 approach. They'll thin slice, which says they'll try and break the work down into a small enough chunk that they can do end to end in 3 weeks,
104
00:13:34.565 --> 00:14:00.475
or they'll pipeline it. Right? They will break up their work to match the technology stack, and they'll pipeline the work. So you'll see them grab the data, collect it, land it. Yeah. And staging, like, whatever you wanna call it. Then they'll see them pick it up and move it into some form of other data repository and ideally model it. You know, then they'll actually go and create some metrics, and then they'll create some visualization or last mile delivery. And each 1 of those will be a set of work with handoffs and milestones in between.
105
00:14:01.399 --> 00:14:07.259
I encourage teams to thin slice. Our goal is to get a group of people to be able to go end to end with that data,
106
00:14:07.560 --> 00:14:11.295
add the value to that customer, the consumer at the end of what they do. And,
107
00:14:11.595 --> 00:14:16.015
ideally, I start where we try and get the teams to do that within a 3 week iteration,
108
00:14:16.475 --> 00:14:20.850
which is hard. Right? It's hard to go end to end in 3 weeks as a small team.
109
00:14:21.230 --> 00:14:25.009
Another aspect of that kind of end to end is
110
00:14:25.310 --> 00:14:25.810
understanding
111
00:14:26.190 --> 00:14:36.365
what is the other end, where in the web application, it's very clear that the other end is, you know, some UI functionality or some way that the user is able to interact with the product, whereas with data,
112
00:14:37.010 --> 00:14:37.910
there isn't necessarily
113
00:14:38.930 --> 00:14:47.215
a kind of cohesive end step for any given piece of effort, where in some cases it might be, I need to be able to add a new filter to this
114
00:14:47.755 --> 00:14:48.255
visualization
115
00:14:48.555 --> 00:14:49.775
in the BI dashboard,
116
00:14:50.235 --> 00:15:02.320
or it might be I need to be able to populate this, you know, data source for an API so that I can then consume that data product in some other web application or data pipeline.
117
00:15:02.860 --> 00:15:03.680
And so
118
00:15:04.475 --> 00:15:06.334
what do you see as some of the,
119
00:15:06.875 --> 00:15:08.574
I guess, most common or
120
00:15:08.875 --> 00:15:09.695
most achievable
121
00:15:10.154 --> 00:15:13.570
kind of terminal nodes in that end to end graph
122
00:15:13.970 --> 00:15:21.750
for particularly for teams that are first starting on this effort of being able to kind of transform into that agile workflow?
123
00:15:22.454 --> 00:15:28.795
Yeah. So this comes back to 1 of those process patterns, and it's around understanding requirements, understanding the value we need to deliver.
124
00:15:29.175 --> 00:15:50.600
So I've been working on a thing called the information product and the information product canvas with customers for the last 8 to 10 years and, you know, open source the canvas so we can have a link in the show notes if anybody wants to download it. And what it is, it's a way of defining a boundary of the work to be done. Right? So we go away and we talk to the stakeholders, and we wanna understand the actions and the outcomes that'll be taken with this piece of effort.
125
00:15:51.160 --> 00:16:17.995
So, you know, if you get this information at the end, what are you gonna do with it? What action are you gonna take and what's the outcome of that? So, you know, are we gonna go and look for get a flag for customers that are about to churn? What are you gonna do with it? Right? Okay. Well, we're gonna have an outbound call center to go and talk to them and give them an offer. Cool. And if that's successful, what happens? Okay. We reduce our churn, which has some financial benefit. So what we're doing in there is we're getting both the action and an understanding of the value of that piece of work.
126
00:16:18.295 --> 00:16:23.195
And the reason we want the action is often we get told the solution. Right? Not the problem we wanna solve.
127
00:16:23.570 --> 00:16:26.230
And as data specialists, we should be trying to understand
128
00:16:27.010 --> 00:16:31.910
the problem and ways we can solve it. So by asking for the action, we can sometimes come up with a different solution.
129
00:16:32.245 --> 00:16:41.020
So, well, actually, we don't need the churn flag. You know, we've already got this thing over here. If we just leverage that, we can get something to you much quicker that will solve your problem. We'll give you that that
130
00:16:41.480 --> 00:16:44.220
action. The other thing we do is we often ask for the business questions.
131
00:16:44.760 --> 00:17:00.970
And the reason we do that is we find when we ask for the action, the outcome, some people struggle. They haven't thought about it that way. But if we ask for the business questions they wanna answer, the how many, how much, how long, yeah, it comes straight off. They always had 3 to 5 straight out of the head. Right? And from there, we can help the and further and talk to them about
132
00:17:01.590 --> 00:17:17.110
it. As part of the canvas, we all wanna understand the data that we need. Right? So we use a methodology from Lawrence Call called Beam. Who does what? And we say, okay. So what core business processes are involved? So, yeah, customer orders product, customer returns product to store. And that enables us as a team to go,
133
00:17:17.590 --> 00:17:36.090
damn. Actually, you know, customer orders product, we've already moved that data into the platform. We can get something out pretty quick. Customer returns product store. Actually, that's a whole new system. Right? We haven't collected that data. And that allows us to size the data. We go, okay. Now we've got a data collection task, and we know those are hard. So the canvas allows us to have a bunch of boxes,
134
00:17:36.630 --> 00:17:37.610
gather that information,
135
00:17:38.230 --> 00:17:42.250
understand how big it is, and then go back to the stakeholders and have a trade off decision
136
00:17:42.554 --> 00:18:01.255
conversation. Say to them, well, look. We could do everything that you want, and we're estimating 3 months. Or we could do this, but first, and that'll be 2 weeks. And then we'll do the next 2 weeks. Right? And we'll just incrementally build it up. And that's what we want. We wanna be able to break those requirements down into smaller and smaller chunks to get it out, show value, and get feedback.
137
00:18:01.635 --> 00:18:09.200
Particularly on that requirements gathering piece and helping to educate the stakeholders on, you know, what is easy, what is hard, what is impossible.
138
00:18:10.140 --> 00:18:18.925
With a web application, that can be challenging enough of somebody saying, oh, 0, well, I want it to automatically know whether this is somebody that we've interacted with before. And it's like, well,
139
00:18:19.225 --> 00:18:21.885
we can do that, but that's gonna take about 2 years.
140
00:18:22.600 --> 00:18:43.009
And then in the data domain, it's, oh, well, I wanna be able to report on x, y, and z. And you say, well, actually, we're gonna have to incorporate information from 5 different sources and do entity resolution. And, you know, now we're talking another 2 years for this kind of thing versus, oh, I just wanna be able to know, you know, did they purchase or did they have something in their cart? And you say, okay. Well, we can do that pretty quickly. And just understanding
141
00:18:43.470 --> 00:18:47.809
both at the stakeholder level, but also sometimes, you know, within the data team,
142
00:18:48.190 --> 00:18:53.495
what are those kind of relative levels of effort, and what are the things that are easy versus hard versus impossible?
143
00:18:54.275 --> 00:19:01.340
Yeah. And so in the past, I've made many mistakes. You know, we fail often in agile. And 1 of the ones was trying to explain the complexity to our stakeholders.
144
00:19:01.640 --> 00:19:02.540
They don't care.
145
00:19:02.920 --> 00:19:21.170
Yeah. They just want the job done. They they're not a specialist in the data world. They have something they want. They don't understand why it takes so long. And that's reasonable. Right? It's not their domain. And often we have complexity that doesn't make sense to us. So how the hell do we justify why it takes so long to them? So this is where the role of product manager or product owner comes in, in my view.
146
00:19:21.550 --> 00:19:24.610
That is a role that sits between the team and those stakeholders.
147
00:19:24.910 --> 00:19:25.410
And
148
00:19:26.025 --> 00:19:27.965
that role is around
149
00:19:28.345 --> 00:19:32.525
facilitation and communication with the stakeholders around trade offs that need to be made.
150
00:19:33.040 --> 00:19:36.180
And then it's also about making the trade off decisions.
151
00:19:37.040 --> 00:19:43.415
So, you know, when there is something that is complex and it is gonna take some while and there is no choice,
152
00:19:43.715 --> 00:19:54.270
then the conversation then from a product owner, the stakeholders is we can do it. And this is how it's gonna take. Do you wanna wait? Or here's some some alternatives that will get us there in incremental ways.
153
00:19:54.730 --> 00:19:56.350
But you won't get everything you want.
154
00:19:56.810 --> 00:20:07.535
And what we see is when we have a good data team and a good product owner, there's a massive trust there. Right? When the team are estimating or guesstimating as I call it. Right? Because as humans, we are crap at estimating.
155
00:20:08.019 --> 00:20:20.195
When they're guesstimating the number, you know, there's a trust thing where's where the product owner's been working them for a while and goes, yep. That's that's in the right for the little I know. And then they articulate to the stakeholders that there really is no alternative.
156
00:20:20.895 --> 00:20:23.715
That's just as what it's gonna take to get their job done.
157
00:20:24.059 --> 00:20:29.360
The other interesting element of kind of doing that end to end flow is the question of
158
00:20:29.660 --> 00:20:33.664
data modeling, where how much of the modeling and kind of
159
00:20:33.965 --> 00:20:39.585
entity design and schema design do you do upfront versus how much of it is emergent? And
160
00:20:40.150 --> 00:20:53.014
when you do the path of saying, you know, we'll just do it, and then we'll let the patterns become emergent similar to how you might do in, you know, a regular software project of, you know, rate it a couple of times, figure out what are the right abstractions, refactor.
161
00:20:53.715 --> 00:21:00.520
Doing that in the data world isn't always easy or possible or it just becomes expensive because you're duplicating data
162
00:21:01.060 --> 00:21:07.480
or, you know, the operations to refactor the data and kind of rebuild it from scratch can take quite a bit of time.
163
00:21:07.865 --> 00:21:14.605
I'm wondering what you see as some of the signals for when you need to bias towards 1 direction or the other of,
164
00:21:15.070 --> 00:21:24.049
no, if we don't do the data modeling right now, we're going to regret it because it's going to take, you know, weeks or months of effort to do it right afterwards, or,
165
00:21:24.654 --> 00:21:26.755
you know, this is a small enough change set
166
00:21:27.215 --> 00:21:32.674
or a small enough problem space that we can just do whatever makes sense right now and then refactor it later?
167
00:21:33.080 --> 00:21:44.125
So I'm highly opinionated on this 1, and we're highly opinionated in our agile data product as well. I don't believe there is ever a 1 off question ad hoc piece of work. If you look at what happens,
168
00:21:44.665 --> 00:21:50.925
you get asked by a stakeholder for a piece of data or a piece of information to answer a business question. That's their first business question.
169
00:21:51.340 --> 00:22:00.555
How many customers we got? Soon as you answer that, you're gonna get the next business question. Yeah. Where are they based? What are they buying? How much are they worth? How many we're
170
00:22:00.935 --> 00:22:04.475
losing? Soon as you answer those, you're gonna get the next level of complex questions.
171
00:22:05.015 --> 00:22:06.075
Why are they leaving?
172
00:22:06.455 --> 00:22:09.195
What can we sell them that's gonna give us more revenue?
173
00:22:09.539 --> 00:22:14.919
I wanted more customers that look like them. What do they look like? Then we start getting to the really complex optimization
174
00:22:15.220 --> 00:22:27.395
and flow ones, you know, over time, how many have left from this region versus that region? Why did they leave and what could we have done to change it? Yeah. So the first question we get asked is is just the first question, and
175
00:22:27.880 --> 00:22:29.659
we know that the next ones are coming.
176
00:22:29.960 --> 00:22:36.940
The other thing we know is once we give that piece of information to a stakeholder, they're gonna ask for it again on a regular basis. It has value for them.
177
00:22:37.605 --> 00:22:53.380
So this idea that we ad hocly just go create a piece of code as a 1 off, give them the answer, and we'll never have to use that again. I think it's been proven to be wrong time and time again. You know, we've seen that in the market. We saw it with the self-service BI wave, you know, the tableaus, the clicks.
178
00:22:53.924 --> 00:22:54.825
We gave self-service
179
00:22:55.365 --> 00:23:05.470
out. Lots of people built really cool stuff, but we lost the practice of data. And then we kinda go through a way when we move back, and we're in that wave again with DBT. Right?
180
00:23:05.930 --> 00:23:07.790
And, you know, again, I'm opinionated.
181
00:23:08.250 --> 00:23:11.710
I don't like it when words get used out of context.
182
00:23:12.395 --> 00:23:16.335
So I really don't like the fact that DBT calls a chunk of code a model
183
00:23:16.795 --> 00:23:25.669
because in my past, after the last 30 years, there's not a model. It's a chunk of code. So how do we solve that problem? Well, what we do is we focus on how we model likely.
184
00:23:26.130 --> 00:23:29.830
Right? And how we have modeling that enables change as much as it can.
185
00:23:30.255 --> 00:23:40.630
You know? And if you look at people like Scott Ambler, he's been talking about agile data modeling for a long, long time. And so with the teams I've worked with, there are techniques you can use to model early, model lightly,
186
00:23:41.170 --> 00:23:43.670
and enable change as much as possible.
187
00:23:44.050 --> 00:23:47.350
But the thing that's true with every agile way of working,
188
00:23:47.895 --> 00:23:49.035
change has a consequence.
189
00:23:49.815 --> 00:24:09.285
Yeah. So we just try and reduce the consequence of that change as much as we can, but change is always gonna need effort. So there's a bunch of patents and techniques that I've seen teams use to model early, model lightly, and enable the change to the model, but they always model. And so the other lens that I use is what we call definition of done.
190
00:24:09.665 --> 00:24:11.605
So definition done is,
191
00:24:11.985 --> 00:24:14.085
a set of statements from the team
192
00:24:14.400 --> 00:24:18.580
about what the professional practice is. How do they know they've done a job
193
00:24:19.200 --> 00:24:24.645
for themselves? Right? How do they know if somebody else in their team is doing a piece of work that is done to the level they've agreed?
194
00:24:25.345 --> 00:24:26.965
And, you know, as an example,
195
00:24:27.345 --> 00:24:33.365
I would expect the definition of done for a data team to have the code is being tested and the data has been validated.
196
00:24:34.100 --> 00:24:36.680
Why would I expect that? Well, you go talk to a stakeholder,
197
00:24:37.059 --> 00:24:39.559
and you say to them, I'm gonna give you a kind of customer,
198
00:24:40.260 --> 00:24:42.680
but, actually, I'm never gonna cheat that it's right.
199
00:24:43.345 --> 00:24:47.765
Right? It's just a number. I can't prove that's how many customers we've got. I'm gonna leave that to you.
200
00:24:48.465 --> 00:24:51.125
Would that stakeholder believe that we've done our job?
201
00:24:51.460 --> 00:25:07.804
No. But we do it all the time, and it's just wrong. So there is a level of professional practice as data people that we need to do, And data modeling is 1 of those, and testing is 1 of those, and validating our data is 1 of those, and we should just do it right. It shouldn't be optional.
202
00:25:08.130 --> 00:25:15.110
The trick is how do we make it iterative? How do we make it small? How do we not go in, as they say, catalog and cocktails
203
00:25:15.804 --> 00:25:32.404
or the ocean? You know? How do we not go into that scenario where we have a single data modeler who sits in a small office scratching their chin for 9 months to come out with this beautiful enterprise data model we'll never implement. We've got to balance out, light modeling, but we still have to model.
204
00:25:35.424 --> 00:25:44.430
Prefect is the data flow automation platform for the modern data stack, empowering data practitioners to build, run, and monitor robust pipelines at scale.
205
00:25:44.890 --> 00:25:52.785
Guided by the principle that the orchestrator shouldn't get in your way, Prefect is the only tool of its kind to offer the flexibility to write workflows as code.
206
00:25:53.165 --> 00:25:53.665
Prefect
207
00:25:54.100 --> 00:26:10.705
specializes in gluing together the disparate pieces of a pipeline and integrating with modern distributed compute libraries to bring power where you need it, when you need it. Trusted by thousands of organizations and supported by over 20, 000 community members, Prefect powers over 100, 000, 000 business critical tasks a month.
208
00:26:11.390 --> 00:26:14.610
For more information on Prefect, go to data engineering podcast.com/prefect
209
00:26:16.110 --> 00:26:16.610
today.
210
00:26:16.910 --> 00:26:17.810
That's prefect.
211
00:26:21.764 --> 00:26:23.544
So in terms of
212
00:26:23.845 --> 00:26:24.345
the
213
00:26:25.524 --> 00:26:26.264
kind of
214
00:26:26.725 --> 00:26:35.039
definition of done and some of the challenges or kind of anti patterns that are maybe becoming emergent with this newer set of tooling.
215
00:26:35.980 --> 00:26:41.125
What are some of the ways that you see teams as they go through this exercise of, we're going
216
00:26:41.505 --> 00:26:44.565
to embrace some of these agile practices, we're going to do this iteratively,
217
00:26:45.345 --> 00:26:46.645
you know, how do they
218
00:26:47.520 --> 00:26:53.380
start to think about the tools that they use and the ways that they're using those tools to be able to
219
00:26:53.680 --> 00:26:57.645
kind of incorporate all of these concerns into a holistic
220
00:26:57.945 --> 00:26:59.245
kind of development flow
221
00:26:59.705 --> 00:27:01.405
without losing their minds?
222
00:27:02.105 --> 00:27:29.759
Yeah. So again, you know, opinionated. All they should do is buy an out of off the shelf platform like agile data dot I o. Right? And, not try and cobble their own. However, that's not what the market's Right? We're at the stage. We have what we call the modern data stack or or I call it the jinga stack. You know, these 25 different tools, 1 for each category, and you're coupling it together. I kinda liking it back to the old ERP days where we used to have to cobble together the payables module and the receivables module and the geo module from 3 different vendors. And we learned from that.
223
00:27:30.304 --> 00:27:38.404
But given this where we're at, right, with the majority of the teams I see building out their own technology platform using a combination of open source and closed source.
224
00:27:38.809 --> 00:27:42.350
We start focusing on only building what's valuable for now,
225
00:27:42.809 --> 00:27:47.710
but making sure that we understand the technical debt and what the cost of change will be in the future.
226
00:27:48.615 --> 00:27:56.394
So, you know, we look at creating a blueprint. And what I mean by blueprint is we draw on a piece of paper, on a whiteboard, in a Miro board,
227
00:27:56.840 --> 00:27:58.539
a bunch of very large boxes
228
00:27:58.919 --> 00:28:01.419
that have the chunks of stuff that we need.
229
00:28:01.720 --> 00:28:08.545
Yeah. And then we figure out what we're gonna build first. So we'll typically look at it and we'll go, okay. We need some way of collecting data.
230
00:28:08.925 --> 00:28:19.299
What are we gonna use to collect data? Are we gonna build something? We're gonna use the software as a service product? We're gonna use an open source thing. Right? And that's driven ideally by what's our first data source.
231
00:28:19.840 --> 00:28:28.665
And so we do that. I'll give you an example. Yeah. 1 team I worked with had a theory that the first piece of value we had to deliver was grabbing data out of SAP.
232
00:28:29.190 --> 00:28:32.330
And we know that grabbing data out of SAP is incredibly hard.
233
00:28:32.950 --> 00:28:44.645
There's a whole lot of reasons that collecting data from SAP is just a nightmare. So we looked at that, and the team estimated it's probably 3 to 6 months to get, you know, a fully automated collection process out of the SAP platform.
234
00:28:45.590 --> 00:28:55.370
But when we went and asked the stakeholders what the first information product we're gonna deliver for was, it was actually based around their call center software because that's where they had an emerging problem.
235
00:28:56.105 --> 00:29:01.085
So if the team had have gone and build out this whole collective for 6 months out of SAP first,
236
00:29:01.545 --> 00:29:08.080
and then the first information product when they were ready was actually the contact center software. They'd invested in the wrong place.
237
00:29:08.380 --> 00:29:23.690
So that's what we understand is what we're gonna build first. So they collect it normally, and then the second thing is where we're gonna store it. You know? And there's a bunch of patterns out there now. You know? Are we data lake centric? Are we, you know, data warehouse centric? You know, Snowflake, are we Databricks? Are we BigQuery? Are we Firestore?
238
00:29:24.550 --> 00:29:27.770
Firebulk, are we single store? Right. There's a bunch of patents that are reusable.
239
00:29:28.205 --> 00:29:41.190
And so we should just pick 1 and implement it as quickly as we can and then test it. Right? And then we think about what's next. And so from an agile data point of view, when teams are working, they should be constantly reviewing how they make toast their processes.
240
00:29:41.809 --> 00:29:54.775
They should be figuring out where the next major problem is or the next piece of value is, and they should iterate the way they work. So we're constantly building out our technology platform, and we're constantly changing the way we work to solve problems as they arrive.
241
00:29:55.270 --> 00:30:05.275
But we still need that blueprint. Right? We still need that big picture of where we think we might go, and that's important. But that blueprint's not 6 to 12 months of a data architecture
242
00:30:05.575 --> 00:30:07.915
or, you know, a big
243
00:30:08.215 --> 00:30:13.435
500 word document that pretends to know what we're gonna build over the next 12 months because
244
00:30:13.950 --> 00:30:15.490
everything changes. We know that.
245
00:30:15.870 --> 00:30:19.570
The other aspect of kind of allowing for
246
00:30:19.950 --> 00:30:25.395
refactoring of the data model as you explore more of the problem space is how do
247
00:30:25.855 --> 00:30:29.235
you bring in the right abstraction layers so that you can
248
00:30:29.649 --> 00:30:37.350
kind of restructure some of the foundational layers so that you can bring in, you know, data model reuse, code reuse, concept reuse
249
00:30:37.995 --> 00:30:40.415
without breaking some of those existing
250
00:30:41.035 --> 00:30:42.895
kind of end user assets
251
00:30:43.675 --> 00:31:26.520
and doing it in a way that doesn't, you know, add a whole bunch of extra work on the behalf of the developer so that it becomes a maintenance nightmare where you're just, you know, sort of like the situation where if you want to replace the foundation of a house, you first have to lift the entire house and move it, then dig out the whole foundation again, and then hope that it doesn't fall over in the process, and then relay the foundation and put the house back down. Yeah. And that's a really important concept. Right? It's this idea of an architecture of a house or I often use food analogies for some reason I kind of blame data kitchen for that. Right? The idea of ingredients, recipes, and store rooms, and and frontline service. So we've got to understand the blast radius. Right? We have to understand what our architecture looks like, what our layers are, the ones we think we're going to add later, the ones we think become
252
00:31:26.820 --> 00:31:29.925
semi immutable. Right? The technical debt for changing them is massive
253
00:31:30.405 --> 00:31:33.545
so that we understand the consequence of that change. And
254
00:31:35.205 --> 00:31:44.000
we understand we were making bets that are dangerous. Right? Where those bets are embedded as foundational pieces and to change them later is is a high cost and a high consequence.
255
00:31:44.460 --> 00:31:46.559
This is ones that are relatively disposable.
256
00:31:47.245 --> 00:31:55.025
So what's an example? You know, if I'm gonna bring in a testing framework, I could probably throw a great expectations on the side of what I'm doing, and it's relatively
257
00:31:55.860 --> 00:31:59.640
replaceable. Right? I could bring in SOTA or 1 of those other tools. Right?
258
00:32:00.020 --> 00:32:04.525
And effectively change that testing paradigm because it's not embedded. It's kind of sitting to the side.
259
00:32:05.005 --> 00:32:11.665
Whereas if I was gonna replace my cloud data analytics database, yeah, that may or may not be completely replaceable.
260
00:32:12.070 --> 00:32:23.544
I'm gonna change my modeling technique. Yeah. If I'm gonna go from Data Vault to dimensional or Data Vault to activity schema, that's probably a breaking change. Right? That's a massive change. So we need to understand the bits we're making,
261
00:32:24.005 --> 00:32:30.790
the ones that we can change easily, the ones we can't, and how we know when we need to change. And I'll give you an example for our product.
262
00:32:31.330 --> 00:32:46.375
Our entire product is based on Google Cloud. Right? It's 1 of the bets we made really early. We're not multi cloud. We decided to go with a single cloud provider. We picked Google for a whole lot of reasons. It's actually 1 of the best bets we've made. You know, it's 1 of the things I'd probably go back in hindsight and saying, damn, that was a good guess.
263
00:32:46.780 --> 00:32:52.480
And our product's based around what we call configuration. So we're metadata driven. So when you create,
264
00:32:53.100 --> 00:32:59.424
a transformation, you actually create it via rules for natural language, and that's stored in a database which tells that configuration.
265
00:33:00.365 --> 00:33:11.620
And when the transformation runs effectively, we have code that calls it config, says, what does this look like, compiles the transformation code, runs it, and then disposes of it. So we call that a manifest as a pattern.
266
00:33:12.000 --> 00:33:21.610
Now when we started out, we knew that this config is core to us, and my cofounder, Nigel, and I come from more of a relational background rather than a NoSQL background because we're that old.
267
00:33:22.650 --> 00:33:27.549
So we had a relational model for that conflict. We'd had stuff we've done before for customers when we're consulting.
268
00:33:28.170 --> 00:33:31.945
So we looked at it and we went, okay. We need a relational storage mechanism.
269
00:33:32.404 --> 00:33:37.980
We wanna keep it as low cost, as low complexity as possible. We're using BigQuery as the way of storing our customer data.
270
00:33:38.380 --> 00:33:40.960
There's no reason why we can't store that config in BigQuery.
271
00:33:41.260 --> 00:33:47.280
But we knew that that was not immutable. We knew that at some stage when we got big enough and we had enough customers,
272
00:33:47.745 --> 00:33:50.885
that actually BigQuery wouldn't be able to handle the concurrency
273
00:33:51.745 --> 00:33:56.405
of hitting that config database for us because it's not designed as a transactional system.
274
00:33:56.850 --> 00:34:12.214
So we knew we're going to have to change it. So every time we did the design and config, we did it with the idea that we were going to move it. Now, interesting enough, we decided to become 1 of the cool kids and we did a change and we moved to Google Data Store. We moved it to a NoSQL database because when we looked at the market,
275
00:34:12.595 --> 00:34:13.974
everybody was doing NoSQL.
276
00:34:14.859 --> 00:34:21.519
That was an epic failure. Now, yeah, we learned lots, but we lost, you know, a lot of time doing that.
277
00:34:21.845 --> 00:34:24.185
Why? Well, because it wasn't our natural pattern.
278
00:34:24.725 --> 00:34:27.785
Some of the features we got from a relational type database
279
00:34:28.085 --> 00:34:36.490
gave us engineering for free, Whereas in NoSQL, we had to go build it. So what we ended up doing was we ended up jumping from Google Data Store to Google Spanner.
280
00:34:36.875 --> 00:34:40.415
Right? Which is the massively scalable, relational style database.
281
00:34:41.115 --> 00:34:41.615
And
282
00:34:41.915 --> 00:34:49.089
that change, you know, it was effectively we lost all the work we did on those SQL database. We ended up going from the BigQuery pattern straight to Spanner.
283
00:34:49.390 --> 00:34:58.444
But now that's for us as immutable. Right? The cost of change of not using Spanner is massive for us, but that's okay. We get so much benefit out of that piece of technology, that pattern
284
00:34:58.744 --> 00:35:03.164
that actually we don't wanna change it. Right? And so for us, we we had a plan.
285
00:35:03.480 --> 00:35:15.545
You know? We knew we had to make a change. We had a guess at what we thought it was gonna be, but we always enabled ourselves to paint ourselves out of the corner. And so that's what teams should be doing. Right? They should be thinking about that, about what happens if
286
00:35:16.005 --> 00:35:17.704
we need to change this piece of technology.
287
00:35:18.320 --> 00:35:25.060
You know, what happens if our, you know, massive cloud analytics database vendor who has a massive loss every year
288
00:35:25.494 --> 00:35:32.714
doesn't survive or gets bought out by, I don't know, a big CRM company. What's it gonna do to us? Right? If we had to change that database,
289
00:35:33.310 --> 00:35:37.650
what would we move to and how much work would it be? So we have to keep those things in mind, I think.
290
00:35:38.110 --> 00:35:40.210
Another problem that manifests
291
00:35:40.935 --> 00:35:47.595
particularly later in the life cycle of a data product, but is something that should be considered early on, is the question of
292
00:35:48.220 --> 00:36:03.305
access control and governance and who should be able to view what data and do what with it and export it where. And it's another 1 of those things that's easiest to do early on or at least to start incorporating early on in the process rather than trying to retrofit it in afterwards.
293
00:36:04.250 --> 00:36:09.950
And in this agile way of working of saying, I want to deliver a narrow slice, you know, this can
294
00:36:10.410 --> 00:36:21.805
very easily be something that gets left on the cutting room floor of, we don't need that right now, so we'll do it later. And I'm wondering how you either encourage teams to kind of push back against that
295
00:36:22.500 --> 00:36:24.440
habit or, you know, ways
296
00:36:25.140 --> 00:36:29.480
to simplify the process of incorporating some of those governance concerns or
297
00:36:29.975 --> 00:36:32.795
understanding what are the appropriate role boundaries
298
00:36:33.175 --> 00:36:35.915
for data as it traverses its different stages?
299
00:36:36.535 --> 00:36:41.670
Yeah. So there's 2 really good examples in there, which is, yeah, our security policies for data
300
00:36:42.130 --> 00:36:43.910
and our governance way of working.
301
00:36:44.370 --> 00:36:54.065
I think I've got successful patents for the security. I've got great ideas for governance, but never been able to experiment with a team and, like, organization to see if they're right. Although we're experimenting
302
00:36:54.445 --> 00:37:00.750
a little bit within the product we're building. So if I take that security 1, what do we see? We see a natural pattern to complexity.
303
00:37:01.450 --> 00:37:06.795
We see a natural pattern of, on day 1, we want to be able to secure at a cell level.
304
00:37:07.175 --> 00:37:17.190
We wanna be able to mass in bits of data for certain users. Right? So it's a high level of complexity. And if we look at the effort, it could take 6 6 to 12 months for us to build that out in a way that works.
305
00:37:17.490 --> 00:37:21.990
So we have to question about, you know, do we really have to build that right now?
306
00:37:22.290 --> 00:37:25.465
Or can we chunk it down? Can we start off with
307
00:37:25.845 --> 00:37:32.425
an environment where only a certain small number of trusted users can come in and use the data? And there's a whole lot of built some braces
308
00:37:32.740 --> 00:37:41.480
around policies and procedures and and accountability for the people that we don't need to build a complex security model yet. You know, then can we chunk it down by
309
00:37:41.780 --> 00:37:56.300
role or group? And so we're always gonna segment our finance data versus HR data to different groups and make a really simple security model. We might get something like, no. No. We've got PII data in there. Right? And we really have to secure, you know, social security number or driver's license.
310
00:37:56.680 --> 00:38:18.079
But we're quite lucky in that there's some technology patterns out there now. Most of the cloud vendors and most of the good products have, you know, data loss protection stuff where it will go through and use machine learning to identify the columns that have that data and mask it for us. So we could probably put that in as a low cost, low well, a low effort, maybe not low cost, but a low effort, you know, component that we can replace later if we needed to.
311
00:38:18.385 --> 00:38:21.525
And we just start building up that security capability
312
00:38:21.825 --> 00:38:23.285
over time as the need happens.
313
00:38:23.905 --> 00:38:36.655
Now the downside of that is we have to refactor. Right? We have to change. So, you know, if we're using Power BI and Power BI is coming into the data and it's using a service account and it's not passing through the credentials of the user that's running the visualization,
314
00:38:37.515 --> 00:38:39.535
and we want to bring in some fine grain security,
315
00:38:39.915 --> 00:38:46.740
we've got a problem now. Right? We have to actually do the work that that identity gets passed to the database if the database is applying the security.
316
00:38:47.599 --> 00:38:50.875
But we can solve those problems, and we only solve them at the right time.
317
00:38:51.435 --> 00:38:57.295
If the organization is saying, no. We need built some braces for whatever reason. You know, maybe we're in a highly regulated industry.
318
00:38:58.075 --> 00:39:02.540
Then, you know, we estimate it and say it is 6 to 12 months to build out that capability.
319
00:39:03.320 --> 00:39:11.065
Right? That's the cost. Right? So it's a trade off decision. Yeah. Would you like us to incrementally build it out over time? Would you like us to do a big investment?
320
00:39:11.765 --> 00:39:26.244
If we're doing a big investment, I still think the team should break it down. Right? They should figure out ways of decomposing that security work into smaller moving parts where they can test and validate at each step and just, you know, build it up like a house, you know, at a room, at a layer.
321
00:39:26.945 --> 00:39:41.215
So that's the security 1, and I think there's lots of ways I've seen teams do that well. The governance 1 is 1 that I still struggle with. The best I've got at the moment is we have to break the anti pattern of governance as a bunch of committees
322
00:39:42.315 --> 00:39:47.855
that take something that's been done and has an opinion on it, which requires more work
323
00:39:48.250 --> 00:39:51.150
to be done. So they're at the end of the process.
324
00:39:51.930 --> 00:40:02.805
What we need is we need governance to be at the front of the process. And on the podcast I do, we had a great guest that gave me this kind of terminology that I love. They talked about exoskeletons and internal skeletons.
325
00:40:03.460 --> 00:40:05.400
And the way I think about it now is
326
00:40:05.859 --> 00:40:14.745
our governance groups. And and by governance groups, it might be data governance group. It might be an architecture group. Right? It's a bunch of people outside the team that have the right to set rules
327
00:40:15.125 --> 00:40:16.985
or change the work that's being done.
328
00:40:17.525 --> 00:40:24.769
So, you know, let's look at our architecture group or our data governance group. They should be setting principles that are immutable and those are exoskeletons.
329
00:40:25.230 --> 00:40:27.250
So those are the things you have to do.
330
00:40:27.869 --> 00:40:31.895
If you are gonna break those, if you're gonna go outside the boundary of the exoskeleton,
331
00:40:32.355 --> 00:40:37.095
then you need to have a conversation with those groups before you start any work. Right? You need to
332
00:40:37.450 --> 00:40:41.950
trade off with them the work you're gonna do and get agreement that you can either bypass those rules,
333
00:40:42.650 --> 00:40:43.950
do them in partial,
334
00:40:44.570 --> 00:40:46.750
or deliver them as a partial capability
335
00:40:47.515 --> 00:40:49.215
or partial compliance, I think,
336
00:40:49.595 --> 00:40:55.055
or could be completely out for whatever reason. There'll be other bills and prices put on. So what's an example of an exoskeleton?
337
00:40:55.470 --> 00:41:00.450
How about whenever we store data, that data will be encrypted on disk and in transit?
338
00:41:00.910 --> 00:41:01.410
Yeah.
339
00:41:01.710 --> 00:41:13.990
That's probably an exoskeleton from your architectural security team. And if we're gonna store data and we're gonna transmit it and it's not encrypted, we need to go have a really big conversation up front. Right? And we'll probably get told no.
340
00:41:14.290 --> 00:41:19.109
Then we talk about internal skeletons, which are patterns that we can use if they have value.
341
00:41:20.105 --> 00:41:28.205
So, you know, 1 might be we have a preference to model data for analytical reasons using Data Vault. Right? It's an internal pattern.
342
00:41:28.540 --> 00:41:36.080
So if you can do that, do it. Right? Because it makes sense. Right? There's a bunch of expertise in the organization around it. That's a well described patent.
343
00:41:36.485 --> 00:41:38.985
We know that if multiple teams create,
344
00:41:39.445 --> 00:41:45.545
hubs, we could probably conform them together if the keys match. So there's some value in your reasoning, those patents.
345
00:41:46.359 --> 00:41:49.420
But if for whatever reason you needed to go use activity schema,
346
00:41:49.960 --> 00:41:57.595
right, because you're using event streaming and it just fits for the use case that you want to do, then that's okay. Right? It's just becomes a new internal pattern.
347
00:41:58.055 --> 00:42:06.140
So that's my view around governance now is we need to be able to understand rules that are immutable, which are our principles, patents which have value, which we should reuse.
348
00:42:06.680 --> 00:42:07.420
And then,
349
00:42:07.960 --> 00:42:14.685
ideally, we should be moving to governance as code. How do we create code that kind of defines that policy
350
00:42:15.625 --> 00:42:22.569
and means we can apply it against what we're doing and it can tell us where we pass or fail. Right? That would make our life so much easier.
351
00:42:22.869 --> 00:42:26.329
Be able to do that without actually having humans having to check it.
352
00:42:26.789 --> 00:42:29.609
So that's where I'm at with complex things like security.
353
00:42:30.405 --> 00:42:31.785
Build it up step by step
354
00:42:32.325 --> 00:42:37.625
if you can. And governance, you know, turn it to rules we can't break, things we should use,
355
00:42:37.960 --> 00:42:40.619
and code that will actually test it for us.
356
00:42:41.080 --> 00:42:43.740
Another aspect of kind of the
357
00:42:44.200 --> 00:42:46.540
agile data way of working is
358
00:42:47.605 --> 00:42:50.424
the kind of question of how do you incorporate
359
00:42:51.444 --> 00:42:51.944
kind
360
00:42:52.325 --> 00:42:53.065
of collaboration
361
00:42:53.684 --> 00:42:58.080
across the different kind of capabilities in the team where
362
00:42:58.540 --> 00:43:03.040
some aspects of data engineering and data management can require
363
00:43:03.765 --> 00:43:05.625
substantial knowledge and understanding
364
00:43:06.005 --> 00:43:09.305
of the systems that you're working with, the data that you're working with,
365
00:43:09.685 --> 00:43:11.385
you know, what are the acceptable
366
00:43:12.390 --> 00:43:12.890
mutations.
367
00:43:13.430 --> 00:43:17.770
And if you're trying to sort of cross train across members of the team,
368
00:43:18.070 --> 00:43:29.480
either who don't have expertise in the domain of the data source that you're working with or don't have expertise in kind of data engineering writ large, but are quite adept at software.
369
00:43:29.940 --> 00:43:33.319
Like, what are some of the ways that you think about being able to
370
00:43:33.700 --> 00:43:36.680
kind of bring everybody up to a
371
00:43:37.045 --> 00:43:39.145
kind of shared level of understanding
372
00:43:39.445 --> 00:43:41.385
and capability and capacity?
373
00:43:42.005 --> 00:43:47.240
Yeah. So we use a patent called t shaped skills, which has been around for a long, long time. So
374
00:43:47.700 --> 00:44:04.020
when we start off with a new team and they're they're experimenting with this way of working, there's a bunch of foundational patterns we wanna put in place. So things like teaming agreements, definition of done, definition of ready, definition of done, done. Some things that we just know we need to be in place first for the team to have a chance of success.
375
00:44:04.480 --> 00:44:16.235
And 1 of those is mapping out the t skills. So t skills is based on the concept of you have the letter t. So across the top of the t is effectively breadth and vertical bar there is is depth.
376
00:44:16.695 --> 00:44:23.230
So for breadth, what we wanna do is map out the core skills a team needs to be able to deliver
377
00:44:23.610 --> 00:44:24.670
data or information
378
00:44:25.210 --> 00:44:32.905
from a end to end process. So it'll be things like facilitation skills, requirement, gathering, development of code, testing, documentation,
379
00:44:33.525 --> 00:44:34.505
release management,
380
00:44:35.070 --> 00:44:39.730
those kind of things, data bombing. Right? So we want to understand the core skills that the team should have.
381
00:44:40.190 --> 00:44:47.645
And then we want to understand the depth of capability within the team. So I have a pattern where I talk about novice practitioner,
382
00:44:47.945 --> 00:44:48.845
expert, and coach.
383
00:44:49.145 --> 00:45:11.270
Yeah. So novice is somebody who's, you know, done a little bit of it, but doesn't do it day to day. Practitioner is a day job. I'm good at it. Expert is, hey. I could actually teach you how to do it or show you how to do it. And a coach is, actually, I could teach and coach and mentor other people to do it the way I did. And there's quite a big jump in my head between an expert and a coach because sometimes experts don't want a coach. They're just bloody good at what they do, and that's okay.
384
00:45:12.050 --> 00:45:17.415
So we put that down the page and then we get everybody to map out their skills. Right? So from a documentation
385
00:45:17.715 --> 00:45:19.095
point of view, am I a practitioner?
386
00:45:19.635 --> 00:45:21.415
Am I an expert? Am I a coach?
387
00:45:21.715 --> 00:45:36.365
And then once everybody's done their t skills, we overlay it as a team. Right? As a group. Right? And when we talk about a team, you know, I subscribe to the pizza philosophy. Right? So between 4 and 9 people is an optimum size for a t team for a whole bunch of reasons.
388
00:45:36.665 --> 00:45:43.165
So we get those people. We put their skills. We overlay them. We see we've got gaps. Oh, okay. Look. We don't have any testers.
389
00:45:43.840 --> 00:45:50.900
We don't outsource testing to another team. Right? We want to have testing skills in our team. So what are we gonna do? Some people are gonna upskill
390
00:45:51.204 --> 00:46:06.035
who are interested in that. We're gonna bring another team member in. Right? We wanna fill that gap. We also wanna look where we have duplication. Right? Which is good. Now where we got 2 people that are strong at data modeling or 2 people are strong at testing or 2 people are strong at development because now we've got redundancy in the team.
391
00:46:06.755 --> 00:46:16.829
So that's what we wanna do is we wanna understand that and make sure that the team becomes self sufficient over time. And the good thing about that actually is it helps the team have a conversation about where they wanna go in their careers
392
00:46:17.290 --> 00:46:19.950
as well. 1 of the things that Agile
393
00:46:20.410 --> 00:46:32.335
does to us, which is not good, is it gets us into this factory. You know? It's funny when we talk about scrum, we talk about sprints. It's not a sprint. It's a marathon. Right? Once a team's rocking it, they're just in there day in, day out. It's monotonous.
394
00:46:32.660 --> 00:46:38.920
Right? And so we want some way for people to grow, you know, be able to get more skills. And the skills matrix or that t skills
395
00:46:39.220 --> 00:46:45.265
thing says, hey. Look. I'm actually really interested in jumping over into that facilitation space. Hey. I'm a novice, but, you know, how do I get to practitioner?
396
00:46:46.205 --> 00:46:49.825
So that's what we do when we have a team of 4 to 9,
397
00:46:50.190 --> 00:46:51.730
and then we start to scale.
398
00:46:52.110 --> 00:46:59.605
Right? And that's where we had another problem because now we go and, you know, we have 2 pod squads teams or 4. The most successful
399
00:46:59.905 --> 00:47:19.155
practice I've seen is we split our our squad into half, create 2 squads, bring in new people, and cross skill them. Now what happens is you lose what we call velocity. Right? The team the 2 squads no longer deliver as much as the 1 squad used to, but that's okay. Right? Over time, they'll build up and we'll get the the 1 +1 equals 3 behavior.
400
00:47:19.695 --> 00:47:28.180
As we build that up, we're now gonna see specialized skills that actually we don't have enough of them in the organization to have 1 person person per squad.
401
00:47:28.720 --> 00:47:35.380
And that's where we start bringing in specialized squads and seeing how they work, where they help those other squads by either upskilling them
402
00:47:35.705 --> 00:47:37.645
or by doing the work for them.
403
00:47:38.185 --> 00:47:44.605
Now there's a bunch of patents out there that help us do that. Spotify published 1, got called the Spotify model.
404
00:47:45.030 --> 00:47:47.210
We did bad things to it as a community.
405
00:47:47.510 --> 00:47:54.605
They won't share what they're doing with us anymore, which is a real loss. My favorite 1 at the moment is Jorgen's 1 called unfixed dot work.
406
00:47:55.085 --> 00:48:00.705
And the reason I like his 1 is I like the way it's described. So if you go to unfixed dot work, he has
407
00:48:01.165 --> 00:48:11.320
nice pretty pictures with boxes that are colored and a really good description of what that is. And he talks about, you know, teams that actually do the work and teams that coach other people to do the work.
408
00:48:11.715 --> 00:48:16.535
We'll see when we start scaling in the data world, we would typically see a platforms team come out.
409
00:48:16.835 --> 00:48:19.815
Definitely in the data mesh world. Right? That's the new hot thing.
410
00:48:20.190 --> 00:48:24.050
1 of the things I talk about with teams when we start to get to that level is
411
00:48:24.350 --> 00:48:29.570
the platform team is now building a platform or a product for somebody else, not for themselves.
412
00:48:30.225 --> 00:48:34.325
So they've got to bring in a whole lot of product thinking, a whole lot of new skills
413
00:48:34.945 --> 00:48:39.045
that are different to the skills when they were embedded in the squad building out their technology.
414
00:48:39.580 --> 00:48:52.805
But that's okay. Right? We just map out the t skills. What do they need as a platform team to build a product? Which is effectively what we're doing with agile data. Right? As we're building a product that other data teams can use to do the work. And I've seen lots of teams internally
415
00:48:53.185 --> 00:48:54.165
do that capability.
416
00:48:54.944 --> 00:49:02.840
As you do bring in that split between the data platform and the kind of data engineers or data products,
417
00:49:03.380 --> 00:49:04.600
what are some of the useful
418
00:49:04.995 --> 00:49:05.895
kind of interfaces
419
00:49:06.435 --> 00:49:27.865
for defining what are the boundaries where if you're somebody who's working on the platform layer and you start to try and, you know, get involved in discussions about how should you model your data or, you know, what is the right level of granularity for a data product, can say, actually, no. That's not your concern. You know, you don't need to worry about that. You know, this is the thing that we actually need from you.
420
00:49:28.244 --> 00:49:33.430
Yeah. So that's the key, right, is that the platform team are now building a capability for another customer,
421
00:49:33.970 --> 00:49:39.145
and they've got to decide where they are in their government cycle. Are they setting the, you know, the exoskeletons,
422
00:49:39.685 --> 00:49:50.500
the rules that are immutable? So our platform will only accept modeling using Data Vault, or are they building something that's internal patterns which see is our platform enables dimensional modeling, star schema,
423
00:49:50.800 --> 00:49:52.980
you know, Data Vault, Activity Schema
424
00:49:53.600 --> 00:49:58.865
to 3rd normal form. Right? So they have to be really clear about how opinionated the platform they're building is.
425
00:49:59.245 --> 00:50:00.305
The second thing
426
00:50:01.165 --> 00:50:07.290
is they need to understand how they're going to innovate that platform. What's their way of working? What's how do they make toast?
427
00:50:07.990 --> 00:50:10.330
So an example I've seen is
428
00:50:10.630 --> 00:50:11.610
with 1 customer,
429
00:50:12.150 --> 00:50:16.714
each of the squads actually did the first cut of of platform capability.
430
00:50:17.095 --> 00:50:19.194
So what happened was the teams were out building
431
00:50:19.494 --> 00:50:20.954
data or information products,
432
00:50:21.494 --> 00:50:23.674
and they'd need a new capability on the platform.
433
00:50:24.500 --> 00:50:28.520
And the problem they had was they had a bottleneck. So whenever they needed a new capability,
434
00:50:29.060 --> 00:50:36.015
they'd have to telegraph it to the platform team way early. The The platform team would have to build it in time for just when the squad needed
435
00:50:36.315 --> 00:50:39.855
it for the squad then to build out what they needed for their stakeholder.
436
00:50:40.280 --> 00:50:55.015
And that timing became a nightmare because often the information product the squad was gonna work on got changed. And therefore, the platform team were building out features that weren't needed just yet. So 1 of the techniques they used was they had good technical people in the squads.
437
00:50:55.474 --> 00:50:58.535
So they would do the first cut of the technology patterns.
438
00:50:59.020 --> 00:51:02.160
They would use them and then they would basically
439
00:51:02.460 --> 00:51:06.480
be picked up by the platform squad who would then harden them
440
00:51:07.085 --> 00:51:20.750
and then make them available to the rest of the squads. Because it's not just about building the technology, it's about all the supporting things around it, documentation, training, how do you use it, testing, All those kind of things. So that was, you know, kind of a prototype iteration and the squads.
441
00:51:21.530 --> 00:51:24.095
Harden and production wise and and the platform team.
442
00:51:24.575 --> 00:51:42.685
I still have a preference personally for the platform team to have a road map and be able to move fast enough that they can move features in and out of their road map in time for those squads. But their orchestration is often hard. So the key thing is that's your way of working. Right? You have a theory about how you're gonna make toast, how you think it's gonna work.
443
00:51:43.065 --> 00:51:44.045
Give it a go.
444
00:51:44.585 --> 00:51:54.350
When something fails, you know, your retro is let's look at that process and say, why did it fail? What are we gonna experiment with to change the way we work? And how do we know whether it made it better or worse?
445
00:51:54.650 --> 00:52:08.099
If it made it better, lock it in. Right? If it made it worse, stop doing it and experiment with something else. So there is no methodology. Right? But there are a bunch of patterns that our, teams have done that may be useful to them.
446
00:52:10.799 --> 00:52:21.335
Data engineers don't enjoy writing, maintaining, and modifying ETL pipelines all day every day, especially once they realize that 90% of all major data sources like Google Analytics,
447
00:52:21.635 --> 00:52:22.135
Salesforce,
448
00:52:22.435 --> 00:52:24.299
AdWords, Facebook, and Spreadsheets
449
00:52:24.599 --> 00:52:26.059
are already available as plug and play
450
00:52:26.599 --> 00:52:27.260
connectors with reliable intuitive SaaS solutions. Hivo Data is a highly reliable and
451
00:52:31.575 --> 00:52:38.795
intuitive data pipeline platform used by data engineers from over 40 countries to set up and run low latency ELT pipelines with 0 maintenance.
452
00:52:39.380 --> 00:52:45.799
Posting more than a 150 out of the box connectors that can be set up in minutes, HEBO also allows you to monitor and control your pipelines.
453
00:52:46.295 --> 00:52:56.050
You get real time data flow visibility with fail safe mechanisms and alerts if anything breaks. Preload transformations and auto schema mapping precisely control how data lands in your destination,
454
00:52:56.590 --> 00:53:04.974
models into workflows to transform data for analytics, and reverse ETL capability to move the transformed data transformed data back to your business software to inspire timely action.
455
00:53:05.435 --> 00:53:16.080
All of this, plus its transparent pricing into 247 live support makes it consistently voted by users as the leader in the data pipeline category on review platforms like g 2. Go to data engineering podcast.com/hebo
456
00:53:17.340 --> 00:53:22.365
data today and sign up for a free 14 day trial that also comes with 20 fourseven support.
457
00:53:24.025 --> 00:53:25.165
That question of
458
00:53:25.545 --> 00:53:31.450
accepting failure is an interesting 1 because sometimes it's not always clear whether you have failed
459
00:53:32.150 --> 00:53:43.245
or how to decide sort of when to give up, and particularly when you start getting into the sunk cost fallacy of, well, we've already put so much work into it. If we just push it a little bit further, then it will work. And, you know,
460
00:53:43.700 --> 00:53:51.560
because of the inherent complexity in the data space, what are some of the useful heuristics that you found for being able to help somebody understand
461
00:53:52.155 --> 00:53:55.615
it's not going to work, no matter how much more work you put into it versus
462
00:53:56.235 --> 00:53:59.455
you actually just haven't put enough work into it yet, and that's why it's not working?
463
00:53:59.799 --> 00:54:02.140
So the 2 know. So we'll see
464
00:54:02.519 --> 00:54:05.260
that sunk cost. That is a massive problem.
465
00:54:05.720 --> 00:54:12.724
And seems to be worse in the data space than the application space, and I don't know why for sure. But I think part of it is
466
00:54:13.025 --> 00:54:15.444
once those numbers go out, people rely on them.
467
00:54:15.900 --> 00:54:23.760
And if we do anything and those numbers change, now we've got a real problem. Right? So, you know, we had a 1000000 customers, and now we've only got 500, 000.
468
00:54:24.665 --> 00:54:37.980
Why was that? Well, was that because we changed the rule of what the definition of a customer was? Was that because our code was wrong? Was that because the source system did something and it affected the number, but the number is now actually correct, how we've been reporting the incorrect number for a while?
469
00:54:38.599 --> 00:54:42.565
So there seems to be a bigger blast impact when we file with data
470
00:54:42.945 --> 00:54:43.445
because
471
00:54:43.984 --> 00:54:44.484
effectively
472
00:54:45.105 --> 00:54:50.244
the decisions or the the consequences of the information we've used seems to be bigger.
473
00:54:50.750 --> 00:54:53.329
But if I go back to kind of the core of the question,
474
00:54:54.030 --> 00:54:57.410
the team know when they need to change, when they need to iterate.
475
00:54:57.765 --> 00:55:00.025
They just need to be given permission to do
476
00:55:00.405 --> 00:55:20.224
it. Sometimes in the team, you'll see 1 person keep because it's their baby. Keep wanting to sunk cost it. Right? Keep going and investing because, you know, you're only 10% away from being ready. You know, they're 90% done. But sometimes that last 10 percent is the hard part. Right? That's where all the effort is or you're just not gonna get there. But team behavior is really interesting. As a team,
477
00:55:20.790 --> 00:55:39.240
they will have a culture where they know, and then the team will find ways of stopping that behavior happening over time either in a polite way or in a not so polite way. But that's 1 of the benefit of teams working on a problem, not individuals. And again, what I think in the market at the moment is we've gone to hyper specialization.
478
00:55:40.260 --> 00:55:41.800
Right? We've gone to people
479
00:55:42.260 --> 00:55:45.400
really, really specializing on 1 small moving part
480
00:55:46.065 --> 00:55:52.805
or we've gone to a single person end to end, right, where 1 person picks it up and gets all the work done without any colleagues.
481
00:55:53.400 --> 00:55:55.740
And both of those patents for me are extremes,
482
00:55:56.360 --> 00:56:01.740
and they can work, but you have more chance of success if you have a small team of people working together.
483
00:56:02.415 --> 00:56:13.910
And let's be honest, it's more fun for me anyway. Yeah. A group of people working and solving problems and leveraging each other's skills and being there for the journey. That's more fun than working on your own, but maybe that's just me.
484
00:56:14.290 --> 00:56:20.150
As a kind of higher level question from your perspective of somebody who has been
485
00:56:20.675 --> 00:56:24.935
embracing these agile practices and working with teams and working in the data space,
486
00:56:25.635 --> 00:56:33.769
do you see the overall trend of the available tooling and systems and kind of infrastructure capabilities
487
00:56:34.150 --> 00:56:34.650
as
488
00:56:35.029 --> 00:56:36.010
bending towards
489
00:56:36.710 --> 00:56:44.055
kind of being conducive to agile practices, or do you see that they are, in some cases, actively harmful towards those approaches?
490
00:56:44.595 --> 00:56:47.569
I wouldn't say they're harmful because I think we can make any technology
491
00:56:47.869 --> 00:56:49.569
work. I think they've got better.
492
00:56:50.030 --> 00:57:00.685
You know? Yeah. If we look at the data space for transforming data via code, we've adopted good practices out of the software engineering space with things like version control and CICD.
493
00:57:01.224 --> 00:57:06.190
If we look at some of the front end last model tools, if we look at our visualization tools, visualization tools, very few of them
494
00:57:06.570 --> 00:57:11.390
allow us to check-in our code and vision it. Right? We're still back in the dark ages of going to that dashboard and, you know, change it and hope like, oh, you didn't break it.
495
00:57:14.494 --> 00:57:19.795
Or, you know, copy of copy of copy Tuesday just to make sure I've got 1 I can regress to.
496
00:57:20.174 --> 00:57:22.755
I think then yeah. So we're still at table stakes.
497
00:57:23.410 --> 00:57:25.990
What I don't see is I don't see our tooling
498
00:57:26.610 --> 00:57:29.590
enabling us to use the machine to do the work for us.
499
00:57:30.305 --> 00:57:36.405
Now we're still human centric. We're still bashing the problem as individuals, and we're doing it relatively manually.
500
00:57:36.945 --> 00:57:42.560
So I don't think we're well served by technology in the data space to adopt agile techniques.
501
00:57:43.260 --> 00:57:47.680
I think we in the data domain, for some reason, we love complexity at the moment.
502
00:57:48.145 --> 00:57:51.445
We love solving that complexity problem, which of the technology
503
00:57:52.065 --> 00:57:58.060
stack, not solving the complexity problem of getting information to our customers as early as possible
504
00:57:58.360 --> 00:58:05.145
to add that value and get their feedback. So I think we're focused on the wrong problem. As technologists, we love to solve those technology things.
505
00:58:05.685 --> 00:58:11.900
So I think it will change. I think we see waves. I think we'll see a move back to less complexity,
506
00:58:12.680 --> 00:58:18.380
less involvement in engineering out the technology, and more involvement around engineering out the data problems,
507
00:58:18.724 --> 00:58:20.025
but time will tell.
508
00:58:20.405 --> 00:58:21.704
And in your experience
509
00:58:22.724 --> 00:58:24.505
of working with clients,
510
00:58:24.964 --> 00:58:31.730
what are some of the, I guess, perennial questions or points of confusion that you've had to work through to help people understand
511
00:58:32.510 --> 00:58:38.475
how to think about agile approaches, how to think about proper scoping, how to think about the
512
00:58:38.935 --> 00:58:40.555
kind of useful integration
513
00:58:40.935 --> 00:58:47.890
and flow, and how to structure the work so that you can do these kind of, you know, fully vertical end to end implementations?
514
00:58:48.910 --> 00:58:49.410
So
515
00:58:49.869 --> 00:59:04.550
what we see is confusion of terminology. I mean, what we're doing is we're taking a whole lot of agile patterns, terms, and practices, and a whole lot of data terms and practices and patterns. And now we're bringing in a whole lot of product thinking and a whole lot of product terms and practices
516
00:59:04.850 --> 00:59:10.035
and patterns. And that causes confusion. You know, the difference between a product owner and a product manager.
517
00:59:10.734 --> 00:59:16.675
So what I see is I see some repeatable things that people struggle with. I see the data modeling problem.
518
00:59:17.069 --> 00:59:22.690
We can just go and create this information and whack it out to a consumer or a stakeholder without modeling the data.
519
00:59:23.069 --> 00:59:30.575
Because we think that being ad hoc is the same as being agile, and it's not. We don't wanna do ad hoc. We just wanna chunk the work we're down into smaller
520
00:59:30.954 --> 00:59:34.414
iterations that we can do faster, but still repeatably, still safely.
521
00:59:34.830 --> 00:59:37.330
I see a whole problem around build versus buy.
522
00:59:37.710 --> 00:59:52.935
I see a problem around organizations that are focused around projects and programs, and that's how they fund work to be done. And that's how they structure everything, yet they want their teams to be agile. And so they're putting these brakes on these constraints that aren't making them safer. It's just making them slower.
523
00:59:53.490 --> 01:00:22.905
1 of the core ones I see is where we have this concept of I have a concept of a heat shield, right, which is a person in the organization that's sponsoring the team to be able to adopt that agile way of working and fail as they do it. And so that heat shield fitly sits above them. Something goes wrong happens. They take most of the heat for the team and and keep the team safe. So if you don't have that role, if you don't have that person that's sponsoring it in your organization, then the team are exposed. Right? That bad behavior gets put upon them when things don't go well.
524
01:00:23.445 --> 01:00:46.160
Another 1 is I see the teams try to scale too fast, you know, rather than start off with 1 small pod squad team and prove your way of working and then try and scale. They try and do it with 4 or 10 or whatever squads at once. Right? And that's that's hard. So start off with something that, you know, adopting an agile way of working with data is hard anyway. Give yourself some chance of success by removing that complexity and uncertainty.
525
01:00:46.700 --> 01:00:53.385
Same thing boiling the ocean. Right? The idea that we're gonna go build out a platform for 12 months before we add any value to our customers is just madness.
526
01:00:53.925 --> 01:00:58.665
Another pattern that that's an anti pattern is somebody other than the team makes the promises.
527
01:00:59.060 --> 01:01:03.000
So we have a project manager who goes and tells the stakeholders how long it's gonna take.
528
01:01:03.380 --> 01:01:08.675
They're not the team. They've never done it before. The team are the only ones that can estimate or guesstimate
529
01:01:08.975 --> 01:01:17.480
with any sense of accuracy. And even then, they'll be wrong. But they should make the promises of what they're gonna deliver and win. And my last 1, which is my personal 1,
530
01:01:17.859 --> 01:01:20.040
project managers who become scrum coaches.
531
01:01:20.340 --> 01:01:26.755
Right? As a project manager, your skills and the things you're taught to do are very different to being a Servient leader as a scrum coach.
532
01:01:27.214 --> 01:01:37.280
So I find that things like business analysts make the best scrum coach. So those are some of the the areas of confusion or risk that I see a lot of teams adopt as they start their journey.
533
01:01:37.820 --> 01:01:44.085
In your own experience of working in this space and working with teams and building a product to help people
534
01:01:44.625 --> 01:01:47.285
adopt and adapt to these agile principles,
535
01:01:47.664 --> 01:01:57.030
what are some of the most interesting or innovative or unexpected ways that you've seen people try to incorporate these ideas into the way that they work with data?
536
01:01:57.485 --> 01:01:59.505
So every team that I work with
537
01:01:59.965 --> 01:02:00.785
is innovative.
538
01:02:01.245 --> 01:02:15.839
I learned early in my journey of coaching that I made a mistake. And the mistake I made was I worked with a team. They got to a level of maturity. And when I went in to help the next team, I just, for some reason, started off at that level of maturity and the team went there. They needed to go back to the beginning
539
01:02:16.195 --> 01:02:18.055
and kind of grow to that level of maturity.
540
01:02:18.675 --> 01:02:22.775
After that, I made another mistake, which was the patterns of the first team applied
541
01:02:23.250 --> 01:02:29.670
were applicable to the 2nd team. We're going to be successful. And that wasn't true. It was different context, different organization, different data, different platform.
542
01:02:30.050 --> 01:02:30.550
So,
543
01:02:31.295 --> 01:02:35.395
you know, the idea that each team can iterate and build their own way of working
544
01:02:35.695 --> 01:02:39.395
is amazing to watch, but was unexpected when I first started.
545
01:02:40.090 --> 01:02:42.670
And the other thing is the team culture is really important,
546
01:02:42.970 --> 01:02:48.030
and we have to be really cognizant of that. An example is, you know, if we talk about scrum and we do retros,
547
01:02:48.654 --> 01:02:50.674
the standard pattern is, you know,
548
01:02:50.974 --> 01:03:04.809
ever virtually or physically, we're putting stickies up on the board about what went well, what didn't go well, what we should improve. And it's a conversation. It's very verbal and it's very visual, and there's this buzz in the room, and that's great. 1 of the teams I worked with were really introverted.
549
01:03:05.615 --> 01:03:14.755
And so they didn't enjoy that type of process. They saw the value of reviewing the work they were doing and the way they were doing and iterating it, but they just didn't enjoy that standard
550
01:03:15.060 --> 01:03:17.240
pattern that we would have for doing that work.
551
01:03:17.700 --> 01:03:19.080
So what they did was
552
01:03:19.460 --> 01:03:25.434
they basically used Azure DevOps and they would have quiet time. They would sit in the room together because this is is pre COVID
553
01:03:25.815 --> 01:03:33.595
and they would bring up their laptops and they bring up the board and they would type their notes, stickies into the board and they would have absolutely no conversation.
554
01:03:34.500 --> 01:03:45.735
And then, you know, we time box them and then it was like, cool, now review it. And they would go in and drag the notes around and make comments on the notes and dot score them again with absolutely no verbal conversation.
555
01:03:46.435 --> 01:04:10.640
And then we go, right, you know, we've we've scored the things we need to focus on next. What are we gonna do about it? And they'd go and create, you know, the next set of notes about the work to be done, and they'd type over the top of each other. And it was like being in a library and it did my head in. Right. Because I was like, this is not what I'm used to. Like, where's the buzz? But for that team, it was the right fit. Achieve the goal that we wanted them to achieve, which is look at the work you're doing, figure out what's not going right.
556
01:04:11.020 --> 01:04:19.155
You know, where you're gonna iterate and iterate it. Right? That's the core principle. That's the core pattern. So for me, every time I work with a team, I see something that's that's amazing.
557
01:04:19.535 --> 01:04:23.475
And that's cool. Right? But we have to empower the teams to build their own way of working
558
01:04:23.855 --> 01:04:26.675
and encourage them to leverage patterns that have success,
559
01:04:27.040 --> 01:04:30.180
right, or may have success and experiment with them. Absolutely.
560
01:04:31.839 --> 01:04:35.859
And in your work of working with teams and
561
01:04:36.295 --> 01:04:41.755
living in this space, what are some of the most interesting or unexpected or challenging lessons that you've learned in the process?
562
01:04:42.295 --> 01:04:43.675
So there is no methodology.
563
01:04:44.160 --> 01:04:46.660
There is no out of the box way of working.
564
01:04:47.200 --> 01:04:49.700
Every team is different. Teams are amazing.
565
01:04:50.240 --> 01:04:54.365
Just empower them and enable them to get the work done. And
566
01:04:55.065 --> 01:04:56.525
typically, they always will.
567
01:04:56.985 --> 01:04:59.805
So that's the learning I constantly get.
568
01:05:00.185 --> 01:05:00.685
For
569
01:05:01.299 --> 01:05:02.279
people who are
570
01:05:02.980 --> 01:05:07.880
kind of revisiting the ways that they work or looking at how they can incorporate
571
01:05:08.180 --> 01:05:11.415
new capabilities or new practices into their
572
01:05:11.875 --> 01:05:13.655
overall workflow for delivering
573
01:05:14.275 --> 01:05:19.495
analytics and data assets? What are the cases where an agile approach is the wrong choice for a data project?
574
01:05:20.039 --> 01:05:24.460
So I don't think there ever is. I'm a bit biased on that these days. I think all the alternatives,
575
01:05:25.000 --> 01:05:31.395
not as good as an agile way of working. However, there are some things that are big warning signs that you're gonna struggle.
576
01:05:32.015 --> 01:05:34.595
So 1 of the ones is if there's no uncertainty,
577
01:05:35.220 --> 01:05:44.920
if what you're doing is repeatable, then an agile way of working may not be the best fit for you. But I've never seen it right. I've never seen teams working with data where there's not a high level of uncertainty.
578
01:05:45.605 --> 01:05:53.465
As I said, where there's no heat shield, if you're starting this journey and you don't have a senior person that can hold that heat when it hits from the rest of the organization,
579
01:05:54.450 --> 01:05:57.990
then you're going to struggle, right? There's going to be some real problems coming.
580
01:05:58.369 --> 01:06:02.710
And the last 1 is when the organization is command and control, where it's hierarchical,
581
01:06:03.385 --> 01:06:04.685
right? Where there is
582
01:06:05.225 --> 01:06:06.605
a culture of blame
583
01:06:07.145 --> 01:06:12.125
and people need to hide from that blame or serious consequences happen to them. Like they get fired,
584
01:06:12.480 --> 01:06:13.460
those kind of things,
585
01:06:13.920 --> 01:06:30.234
then your organization is not gonna support an agile way of working. It's not the culture of the organization, and your team are gonna be exposed. So have a really good think about whether whether you wanna start down that journey and whether you wanna expose the team to that. Often, I work with teams that
586
01:06:30.730 --> 01:06:44.815
are in a hierarchical organization, but we have that heat shield. Right? So the team are empowered for this new way of working. And what do we see? We we see them be successful, and we see the rest of the organization go, oh, 0, how are you doing that? Right? And they start to watch and learn and and ideally adopt it themselves.
587
01:06:45.355 --> 01:06:52.130
That's what success looks like. But, yeah, those are the probably the 3 warning signs for me that you're going into a high level of risk
588
01:06:52.510 --> 01:06:54.290
by adopting an agile way of working.
589
01:06:54.750 --> 01:06:56.130
Are there any other
590
01:06:56.695 --> 01:06:58.635
useful references or resources
591
01:06:59.095 --> 01:07:03.435
or practice exercises that you recommend for teams who want to
592
01:07:03.839 --> 01:07:09.619
dig into and understand more about how to apply agile practices to their data work that you recommend?
593
01:07:10.000 --> 01:07:16.845
So I'm trying to spend as much time as I can publishing the ones that I've used with Teams in a way that anybody else can pick it up for free.
594
01:07:17.145 --> 01:07:18.365
So if you go
595
01:07:18.665 --> 01:07:19.325
to wow.agiledata.io,
596
01:07:20.665 --> 01:07:29.660
that's a site we've created where I kind of brain dump those patents and templates as much as I can. There's a bunch of books out there that I've read over the years and approaches that
597
01:07:30.040 --> 01:07:36.765
have value. So I'm a great fan of Lawrence Core and his main theme methodology. I think that's a great way of gathering data requirements.
598
01:07:37.385 --> 01:07:54.494
I'm a great fan of of Scrum and Kanban and Lean and some of those practices, and there's lots of good books here about how you can pick those patterns up. Personally, I'm a great fan of data vault as a modeling technique. I find it's the most flexible at the moment. It's got some problems, but it's certainly the 1 that can get to change the most in my experience.
599
01:07:55.220 --> 01:08:01.240
Those are probably the core ones. I and unfixed dot work is probably the best taking topology
600
01:08:01.619 --> 01:08:08.605
explanation that I can find out there at the moment. So those are probably my go to when I point to patterns. And then the information product canvas,
601
01:08:09.305 --> 01:08:20.420
which I've published for me. Yeah. It's my go to whenever I work with a customer and try and understand what information they want delivered. So that's probably my my short list of go to stuff at the moment.
602
01:08:20.875 --> 01:08:22.335
Are there any other aspects
603
01:08:22.635 --> 01:08:28.335
of your experience of working with data teams or how to apply agile methodologies
604
01:08:28.795 --> 01:08:44.185
to data work or how to think about the kind of technical and team structures to support that that we didn't discuss yet that you'd like to cover before we close out the show? I think the main thing for me is, you know, I call out to everybody that when you find something that's worked, a pattern that's worked for your team, just sharing is caring.
605
01:08:44.620 --> 01:08:58.875
You know, take the time to write it up in a simple way and publish it out so the rest of the world can ideally see it and experiment with that patent as well. Sometimes we hold those things internally. And, yeah, if we look at it, John, everything we do is iterating on other people's work.
606
01:08:59.255 --> 01:09:05.030
That's what it's all about. We should try and pay it back and push some patterns that we've had success with
607
01:09:05.330 --> 01:09:09.910
back into the community so we can help our fellow data practitioners where possible.
608
01:09:10.265 --> 01:09:23.900
For anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest gap in the tooling or technology that's available for data management today.
609
01:09:24.360 --> 01:09:31.425
The obvious answer for me is reducing the complexity, and that's what we're focused on with agile data dot io. How do we remove a lot of the engineering
610
01:09:32.365 --> 01:09:37.810
process and practices and effort that's required and automate them? So I think that's the problem today.
611
01:09:38.270 --> 01:09:47.835
What I think the opportunity in the market is over the next couple of years is right now, everything we do is based on human effort. We don't use the machines to do the effort for us.
612
01:09:48.135 --> 01:09:50.795
And I think, you know, we're gonna see over the next
613
01:09:51.469 --> 01:09:54.130
2, 5, maybe 10 years where we start
614
01:09:54.829 --> 01:09:59.889
using algorithms and machines to recommend and do the work for us. And I'll give you an example.
615
01:10:00.190 --> 01:10:11.750
You know, when we collect a piece of data and we look at it, you know, sometimes there's a hint. Right? If it's coming from a relational database, there's a foreign or primary key. It's being flagged, and we know that's a unique key for customer.
616
01:10:12.369 --> 01:10:14.150
But in this whole event streaming,
617
01:10:14.610 --> 01:10:23.495
80% of the poorly designed systems for capturing data that don't have keys on it for us. We have to go and look at that data and figure out what the key is.
618
01:10:24.035 --> 01:10:26.935
And so, you know, using machine learning
619
01:10:27.280 --> 01:10:29.300
to actually identify a candidate
620
01:10:29.680 --> 01:10:30.820
looking at the data
621
01:10:31.200 --> 01:10:42.580
and giving us a hint of what that key is would save us so much time. And, you know, we're talking about not a simple process because it might be a concatenated key. Right? It might be 3 or 4 columns that we need to say this is actually a unique record.
622
01:10:43.140 --> 01:10:46.440
But that's what I think areas where we can actually use the machine
623
01:10:46.740 --> 01:10:55.125
to reduce the work we do. And at the moment, we're not focused on that. At the moment, we're focused on grabbing all the parts to actually build a machine that can run.
624
01:10:55.425 --> 01:11:01.470
So we gotta get over that and move on to the next phase. So that's where I think the major opportunity is is using the machines
625
01:11:01.850 --> 01:11:05.950
to automate as much of our work as we can. That's not easy.
626
01:11:06.364 --> 01:11:30.435
Alright. Well, thank you very much for taking the time today to join me and share your experiences and thoughts on how we, as data practitioners, can adopt and embrace some of these agile ways of working. So I definitely appreciate all of the time and energy that you've put into the work you've been doing with data teams that you work with and encapsulating that into your product. So thank you again for all of that, and I hope you enjoy the rest of your day. Yeah. Thank you for having me on the show. It's been great.
627
01:11:36.280 --> 01:11:46.595
Thank you for listening. Don't forget to check out our other shows, podcast dot in it, which covers the Python language, its community, and the innovative ways it is being used, and the Machine Learning podcast,
628
01:11:47.055 --> 01:11:51.555
which helps you go from idea to production with machine learning. Visit the site at dataengineeringpodcast
629
01:11:52.380 --> 01:12:02.015
dotcom. Subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a product from the show, then tell us about it. Email hosts atdataengineeringpodcast.com
630
01:12:03.595 --> 01:12:09.855
with your story. And to help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers.