WEBVTT
00:00:00.390 --> 00:00:02.910
Welcome to the DataCafe.
00:00:00.390 --> 00:00:02.910
I'm Jason.
00:00:03.060 --> 00:00:05.790
And I'm Jeremy. And today we're talking about failure.
00:00:18.000 --> 00:00:21.329
Actually, have we realised we have a birthday to celebrate?
00:00:21.359 --> 00:00:23.579
We have a birthday? Yeah,
00:00:23.640 --> 00:00:25.679
We are one. How about that?
00:00:25.739 --> 00:00:29.399
One year old on the air?
00:00:25.739 --> 00:00:32.429
It's so good. I didn't realise that a year has like crept up on us. And what a year it's been.
00:00:32.700 --> 00:00:33.570
It's been such a pleasure
00:00:33.570 --> 00:00:44.640
It has it really has, and I didn't, I didn't, I didn't know what to expect from this but I'm blown away by the fact that it's gone a year, the fact we've done what this is the 17th episode, is it and...
00:00:44.670 --> 00:00:46.350
I don't even know actually.
00:00:47.100 --> 00:00:53.759
17 episodes not bad going. And I certainly didn't think we do that many. And it's been such fun. Loved it.
00:00:54.390 --> 00:01:04.260
Yeah. And major. Thank you. And shout out to all of our brilliant guests for that month in fantastic. And I know that you've engaged with them a lot.
00:01:01.229 --> 00:01:04.560
So thanks to them for their time.
00:01:04.590 --> 00:01:11.280
Yeah, they make it what it is. And it's been it's been fantastic talking to all these really interesting people, certainly.
00:01:11.700 --> 00:01:15.450
And of course, our listeners. Thank you to all of our listeners tuning in.
00:01:15.480 --> 00:01:16.530
Absolutely.
00:01:16.980 --> 00:01:19.349
And now, what we're going to talk about today is failure.
00:01:22.620 --> 00:01:25.200
I think this is good failure sort of anyway.
00:01:25.349 --> 00:01:46.590
Yeah, I think so insofar as I've been reading these reports about why data science projects can fail, at least motivating it from a data point of view. So the data says that data projects can fail and up to 80. What is it 85% of big data projects fail according to a Gartner report in 2017.
00:01:46.620 --> 00:01:47.189
Wow.
00:01:47.000 --> 00:01:50.870
Which is massive. Yeah.
00:01:47.000 --> 00:02:12.889
And I think they had a 2020 report that said, CEOs say that the ones that land only about eight percent of them generate value for the actual success rate is really low. Gosh, and the reason why question is from the point of view of Yeah, just because something has failed, doesn't mean you haven't got a lot of learnings that then mean, you're more likely to succeed next time round. And so that's obviously not captured in these reports.
00:02:13.860 --> 00:02:22.770
That's one way. But I mean, obviously, it can go the other way, which is you've had your fingers burned. And so you never go that down that route again. Right. So true. Yeah, true.
00:02:22.800 --> 00:02:33.360
Yeah. And so I think what we can chat about today is some of the points that we could maybe avoid, or help people avoid for your own data science projects failing, potentially.
00:02:33.389 --> 00:02:43.620
Exactly. So although this is about failure, really, it's about let's see how we can learn so that we can succeed. I think that's, that's the upper.
00:02:40.020 --> 00:02:48.030
That's the real upper. Yes. So I want to play a game with you today, Jason then so if you're, if you'll indulge me,
00:02:48.960 --> 00:02:49.919
I love games.
00:02:51.060 --> 00:03:25.469
I have, I have, I reckon, six pretty broad reasons why data science projects are prone to prone to failure. And it can be combinations of those reasons. Of course, it can be more than one. But I have six core sort of reasons why. And I call those the micro reasons, but they are they're really important. But then there's a big Whopper. There's the big reason why I think a project projects fail. So I want you to maybe from your own experience, or from experience you've read about I want you to have a crack at seeing whether you can you can get some of these, and we'll have a chat about that.
00:03:25.530 --> 00:03:42.330
Okay, so this is like, data project. Bingo. I hit them all. Okay. Oh, yeah, we need all those same words, that should have been a one year gift to us.
00:03:38.340 --> 00:04:12.930
Yes. Okay, so I have to just come up with something that immediately to me is something I would like seen as a possible failure from the exposure that I've had to projects. And number one on my list is probably engagement, right? and stakeholders need to be engaged, engaged with you as a solution provider engaged with the problem, and the company engaged with the strategy of the company, because you want it to align with their overarching KPIs. And once they're on board, great, that's like a major obstacle number one.
00:04:12.990 --> 00:05:02.610
Yeah, that is definitely that is definitely one of the key reasons why projects fail. I think if you don't have engaged stakeholders, I've worked in several projects where stakeholders have fallen out of love with the idea of doing the project or never really bought into the idea of doing the project in the first place. Or don't realise sometimes sometimes some of the person said, Is this the person setting the project up having this great, oh, we should talk to the dead science team. They're really great. They're, they're not they're actually nothing to do with the pathway to, to delivering that project. And so the people who actually have to work with the tools that you develop, are going, sorry what's this, where did this come from, we didn't ask for this. And so that's a real issue. And of course, they're just as much stakeholders, in fact, really, they are the stakeholders in this case. So yeah, stakeholder engagement number one, that's a really good really good get.
00:05:00.870 --> 00:05:02.610
Well done.
00:05:03.180 --> 00:05:16.949
Oh, I should get a coffee. Oh, yeah. Just sitting on that victory, like, great.
00:05:07.560 --> 00:05:27.269
And get another coffee here. And number two, the data itself has to be clean, or at least in some form that's usable. Absolutely.
00:05:21.870 --> 00:05:30.389
I was at a talk this week and somebody referred to their data.
00:05:27.269 --> 00:05:49.110
I don't know what to call this, but they called it a swamp. It might have been a cloud or might have been a lake somewhere, but they refer to it as a swamp. And I was like, wow, yeah, there could be a swamp of data that people are saying, Let's get some scientists in to look at this swamp. And it's like all we need to prep the data better than this. Yeah.
00:05:49.139 --> 00:06:04.319
Yeah. How it's being collected, when it was collected, and tell if that collection mechanism has changed, as it always, always, almost always does feel good time. Who knows about the data?
00:05:58.980 --> 00:06:53.490
You know, I think organisations that do this, well typically have curators, you know, who are really experienced, really savvy people who look after, in sometimes individual tables, and they they know, they know everything about the columns, the metadata, the provenance of that data. Yeah. And and they can they they are the go to people for that, that data set and allow you to then say, oh, what can you tell me all about that? That how we're collecting that that longitude and latitude? And what level of accuracy we have? And when we changed equipment? How did that affect the the output? All of this stuff is just so so important, but that so often, you only discover when you're well buried within within the project, quite how parlous a state it can be in.
00:06:54.240 --> 00:06:59.220
I've engaged with some data analysts who blow me away their knowledge of the depth of the data that they have.
00:06:59.310 --> 00:07:02.819
Right? Right, exactly.
00:06:59.310 --> 00:07:08.730
Sometimes it's community knowledge, you know that a team has that data, that knowledge about the data, or the or some stakeholders have the knowledge.
00:07:08.939 --> 00:07:20.069
But it's rarely in one place, I find, you know, you often have to search and hunt high and low to find the people who don't really know. So here we go, stakeholders and data we have two good hits straight off the bat. Yeah,
00:07:20.279 --> 00:07:26.850
This is this isn't a bite episode anymore, though, we're going to be here for a while.
00:07:22.529 --> 00:07:26.850
I've a list to go through them.
00:07:27.360 --> 00:07:52.290
And number three, one of the things that I think is the actual team, the teamwork, the skills that are brought together, we talked about the need for diversity of skills and team for em, you can bring in any amount of bias or a lack of understanding, if you just have a skills gap, maybe in your team or just one person working in isolation.
00:07:52.500 --> 00:09:06.149
Yeah, yeah, you've hit hit another one here. So yeah, the science, the team who unders have that scientific knowledge, or don't maybe, you know, it's such a big subject, you can have a team of 15 people, and maybe you don't have coverage of, you know, Transformers in NLP, you know, deep learning neural networks, it requires a huge degree of expertise in a really broad area. So, I'm gonna roll into this as well, not just the not just the coverage of the scientific method, but also the ways of working with that scientific method. You know, are you? Are you a team that goes, I want to go down this rabbit hole, I'm not going to come out, until I've extracted every last percentage of goodness, or are you a team that is is is prepared to do maybe quite shallow investigation initially, and then and then deepen it as necessary? Because I think a lot of the reason some projects go go south is is when individuals maybe decide they're going to just go go after one particular problem that's fascinating them and not actually focused on on the big picture. And the science becomes sort of almost more interesting than the business problem. And that can be a bit of a bit of an issue I've seen, certainly,
00:09:06.149 --> 00:09:07.950
Yeah, it can suck people in.
00:09:07.950 --> 00:09:19.200
Yeah, it can. Good stuff. This is going really well. This is this is this isn't gonna be too long. So we got lost, we got science, we've got data, we got stakeholders what do we think?
00:09:19.559 --> 00:09:41.669
Well, one bit we were talking about there made me think of deployment, like getting what you've built, deployed in the business and productisation. And one of those skills gaps could be the ability to actually get a model, automated, whether whether it's versioned and tested and actually implemented then, for your user, whoever that might be, stakeholder, customer.
00:09:41.759 --> 00:11:10.470
Yep. Yeah. Yeah, it's absolutely the there are many issues around this. And this is probably not one this is probably several reasons but if you don't have good engineering capability, within within or adjacent to your team or within your project if you're working in a cross functional team Then then it's going to be a real struggle. And, you know, data science teams that have good engineers within them are always more capable, I think and are in a much stronger position to just to demo what they've done to stakeholders quicker. And then if you get demos, then you get feedback, you get feedback, you get improved product, you get all that virtuous circle stuff going on, don't you? Suddenly, I call out particularly in this, which is a bit of a curse of the data scientists sometimes, which is that that you're working on a problem, you're developing a model. It's a really nice model and it but it's quite computationally intensive. And, and I've seen this several times, whereby the data scientist pulls out the solution, after maybe half an hour, or an hour's worth of processing from their, from their model. And well, you know, guess what happens that the stakeholder comes to them and says, what an hour? I was hoping that was going to take two minutes. Yeah. Well, I've only got 10 seconds for that results come out to the for it to be useful. You know, the engineering doesn't just cover, being able to deploy it, it's actually got to be done in a timely way as well, which is really hard.
00:11:11.129 --> 00:11:17.129
I'm just going to run this notebook in my laptop in my locker overnight. And then and then the results, come out.
00:11:17.700 --> 00:11:30.509
Exactly. And of course, you know, this is data science, right, it tends to tends to be models, which are consuming inordinate amounts of data. So I mean, we're not, you know, there's a big scalability issue to address. It's not really one issue. It's probably two. Yeah.
00:11:30.570 --> 00:11:35.820
And which can be addressed. But yeah, it's not always immediately addressed.
00:11:32.610 --> 00:11:36.269
Yeah. So any other guests, many, how many of
00:11:36.539 --> 00:11:43.230
You've got four. You're doing really well. So it's to to go and you know, false guests, but I haven't say no surprises here yet.
00:11:44.549 --> 00:12:12.059
No surprises. Yes. Maybe over promising. And what I mean by that is sometimes you say, yeah, we'll build you. This could be echoing the rabbit hole will build you this amazing machine learning model. Turns out, they just needed something quicker, like demo that you just mentioned. He did a demo, he did a quick cycle. This is a deliverable. And something I've heard about, or people have said, is, if you aren't embarrassed by your first demo, then you've demoed too late.
00:12:13.440 --> 00:12:16.440
That's good. You over promised something and you end up spending too long.
00:12:16.470 --> 00:12:24.149
We're just, we just at that stage of the project at the moment. Yeah, we want to go back to the customer really quickly.
00:12:24.210 --> 00:12:27.629
And I'm thinking this is so simple. He's got this gets embarrassing.
00:12:28.889 --> 00:12:31.559
Sounds like you're right at that point. Yeah.
00:12:31.799 --> 00:12:44.039
That's a really good point. I like that one. It's not it's not I haven't got over promising on my list. I guess I probably would have rolled that into stakeholder. But it's it easily be ways of working but.
00:12:41.190 --> 00:12:46.889
Yeah. Okay. That's good. I like it. So we alluded to it a little bit, I get to give you a hint.
00:12:46.919 --> 00:12:55.559
So we alluded to a little bit with the skate the scaling issue, because if a computation is needed in a particular time, it's usually needed for a reason, right?
00:12:55.649 --> 00:12:58.529
Oh, is this outlining the problem from the outset?
00:12:58.830 --> 00:13:01.440
Right, right. Right.
00:12:58.830 --> 00:13:05.460
Right. So this is this is what, what is it feeding into? Right?
00:13:01.740 --> 00:14:02.129
What is what does the downstream look like? Right? What is the decision that we're actually influencing? With our Yeah, I saw, I saw a nice thread on LinkedIn a couple of days ago, where a guy was going, Oh, you know, so, so often. So often people are using their data science teams to generate insight. And you know, what, what really does that achieve, you know, if you if you're putting your insights into a dashboard, or, and he said, Well, why don't I tell you exactly what happens, you produce some nice graphs, and they get put into a PowerPoint deck. And then a month later, it gets presented as a meeting, it's like, well, you're already you're already a month behind the curve, when you do that, but probably two or three, given how long it took to run the tool, and then get it get it across to somebody. So you know, if you want impact from a data science team, you've got to design the forward process that takes that science and show which decision is going to be influenced by it.
00:14:02.129 --> 00:14:05.129
And therefore, of course, show the impact that then comes from that.
00:14:05.250 --> 00:14:06.059
I like that.
00:14:06.779 --> 00:14:09.389
That's a big one for me, I definitely preached that but a lot.
00:14:09.899 --> 00:14:31.799
I was thinking of something similar when we were talking about stakeholder engagement, about the roadmap that along that roadmap that you were talking about to get impact. There'll also be a bunch of like, gate points or decision points that may have a stakeholder who needs to be brought in to get their buy in for the value to be driven. Yep, that Yeah,
00:14:32.100 --> 00:14:58.440
Absolutely. So I mean, there's there's there is overlap in the so the stakeholder engagement Yeah, the with the decision with the impact these these of course, are all critical things that that play together and that none of these tend to sit in isolation. So the last one we have we have five now we got a we got one more to get this it's a slightly fiddly one. And it depends on the industry that you come from as to whether you care about this or that you everyone cares about it, but whether it's topper most in your mind, I think
00:14:58.860 --> 00:15:00.120
Interpretability?
00:15:00.870 --> 00:15:09.840
Ah, no pos Yeah, actually, I might have to just give that to you. Yeah. Okay.
00:15:05.759 --> 00:16:10.470
interpretability usually means that you want to know why. And if you want to know why the tool is producing the decision, typically there's a regulator sitting not a million miles away from you saying, you have to show me why you've made that decision, maybe in a medical healthcare setting, or in a, in an ethical governance setting or in a statutory setting. You know, there are lots of industries finance, for instance, where there is a huge regulation framework that has to be not just not just carefully curated and adhered to, but it has to be shown to be adhered to, you really got to make sure that you're compliant with your regulator, with your industry, with your even with your company policies, of course. So all of that has to be part of part of writing the tool. And, you know, any data scientists will know if they've done this for any length of time that you fast become very, very aware of and then expert in the regulatory frameworks that you're operating under.
00:16:10.740 --> 00:16:12.269
Yeah, that's really interesting.
00:16:12.299 --> 00:16:16.889
So there's one last one last challenge, which I think for right now, I just want my prize.
00:16:17.250 --> 00:16:20.549
You get your prize, prize. Okay, so it's not a challenge anymore.
00:16:20.549 --> 00:16:44.789
It's really just to get a last question. So those those I reckon, are the not so micro reasons for why data science projects fail. But there's, there's something which I think ties all of these together, and the companies that succeed have it, and the companies that that struggle anyway, shall we say, are ones which maybe don't have this for this particular property, you want to hazard a guess as to what that might be?
00:16:45.000 --> 00:16:48.509
Oh, like, I wanna say, investment from the top.
00:16:48.539 --> 00:17:11.880
Yeah, pretty much. I mean, it's, it's basically culture, I think, because you've got to, you've got, you've got to have that cultural desire to embrace what can be, I think, a really radical departure from your usual ways of working and, and, and also, you feel reasonably comfortable with quite quite quite abrupt change in the way that you do things.
00:17:12.210 --> 00:17:35.250
And that can be a big struggle for some companies, I think, certainly, I both worked for and and and seen this distance companies that have really struggled with that cultural standpoint, where getting getting changed into into the company and adopting this new technology, disruptive technology, and it is highly disruptive sometimes can be a real struggle for them. I don't know if you've had any experiences like that.
00:17:35.400 --> 00:18:02.730
Yeah, that point you just said about disruption's pretty key. And I've heard people talk about needing to embed an allowance for curiosity, and the psychological safety that comes with being okay with not so much failing, but experimenting, and your experiments may not deliver. But that's why you ran the experiment to see what needs to change. And any aspect of all the points we've hit on could be what needs to change.
00:18:03.210 --> 00:18:36.029
I love that and allowance for curiosity. I think that's a really nice phrase. It ties up very nicely with how a lot of good teams execute their projects, which is to say, they don't say, we will produce this model, and it will do this, they say, Can we produce a model, which actually impacts the company with 5% improvement in whatever metric and and you know, it's just as okay to come out with a no to that question is is with a with a yes to that question. Because you know, it both outcomes are likely Right.
00:18:33.990 --> 00:18:36.029
Yeah.
00:18:36.240 --> 00:18:38.700
Very good. That's been really enjoyable.
00:18:38.849 --> 00:18:49.440
Well, I guess I think it's safe to say that, you know, culturally, DataCafe has ticked all the right boxes. We're still going off for a year we have we have a high allowance for curiosity on this show. So
00:18:49.440 --> 00:19:11.430
Yes, very much so. Ah looking forward to what the future holds. Thanks for joining us today at the DataCafe. You can like and review this on iTunes or your preferred podcast provider. Or if you'd like to get in touch. You can email us Jason at datacafe.uk or Jeremy at datacafe.uk, or on Twitter at datacafepodcast. We'd love to hear your suggestions for future episodes.
00:19:16.289 --> 00:19:18.210
Can I? I can shout Bingo.
00:19:18.599 --> 00:19:22.920
You can shout Bingo.
00:19:18.599 --> 00:19:22.920
Excellent
00:00:00.390 --> 00:00:02.910
Welcome to the DataCafe.
00:00:00.390 --> 00:00:02.910
I'm Jason.
00:00:03.060 --> 00:00:05.790
And I'm Jeremy. And today we're talking about failure.
00:00:18.000 --> 00:00:21.329
Actually, have we realised we have a birthday to celebrate?
00:00:21.359 --> 00:00:23.579
We have a birthday? Yeah,
00:00:23.640 --> 00:00:25.679
We are one. How about that?
00:00:25.739 --> 00:00:29.399
One year old on the air?
00:00:25.739 --> 00:00:32.429
It's so good. I didn't realise that a year has like crept up on us. And what a year it's been.
00:00:32.700 --> 00:00:33.570
It's been such a pleasure
00:00:33.570 --> 00:00:44.640
It has it really has, and I didn't, I didn't, I didn't know what to expect from this but I'm blown away by the fact that it's gone a year, the fact we've done what this is the 17th episode, is it and...
00:00:44.670 --> 00:00:46.350
I don't even know actually.
00:00:47.100 --> 00:00:53.759
17 episodes not bad going. And I certainly didn't think we do that many. And it's been such fun. Loved it.
00:00:54.390 --> 00:01:04.260
Yeah. And major. Thank you. And shout out to all of our brilliant guests for that month in fantastic. And I know that you've engaged with them a lot.
00:01:01.229 --> 00:01:04.560
So thanks to them for their time.
00:01:04.590 --> 00:01:11.280
Yeah, they make it what it is. And it's been it's been fantastic talking to all these really interesting people, certainly.
00:01:11.700 --> 00:01:15.450
And of course, our listeners. Thank you to all of our listeners tuning in.
00:01:15.480 --> 00:01:16.530
Absolutely.
00:01:16.980 --> 00:01:19.349
And now, what we're going to talk about today is failure.
00:01:22.620 --> 00:01:25.200
I think this is good failure sort of anyway.
00:01:25.349 --> 00:01:46.590
Yeah, I think so insofar as I've been reading these reports about why data science projects can fail, at least motivating it from a data point of view. So the data says that data projects can fail and up to 80. What is it 85% of big data projects fail according to a Gartner report in 2017.
00:01:46.620 --> 00:01:47.189
Wow.
00:01:47.000 --> 00:01:50.870
Which is massive. Yeah.
00:01:47.000 --> 00:02:12.889
And I think they had a 2020 report that said, CEOs say that the ones that land only about eight percent of them generate value for the actual success rate is really low. Gosh, and the reason why question is from the point of view of Yeah, just because something has failed, doesn't mean you haven't got a lot of learnings that then mean, you're more likely to succeed next time round. And so that's obviously not captured in these reports.
00:02:13.860 --> 00:02:22.770
That's one way. But I mean, obviously, it can go the other way, which is you've had your fingers burned. And so you never go that down that route again. Right. So true. Yeah, true.
00:02:22.800 --> 00:02:33.360
Yeah. And so I think what we can chat about today is some of the points that we could maybe avoid, or help people avoid for your own data science projects failing, potentially.
00:02:33.389 --> 00:02:43.620
Exactly. So although this is about failure, really, it's about let's see how we can learn so that we can succeed. I think that's, that's the upper.
00:02:40.020 --> 00:02:48.030
That's the real upper. Yes. So I want to play a game with you today, Jason then so if you're, if you'll indulge me,
00:02:48.960 --> 00:02:49.919
I love games.
00:02:51.060 --> 00:03:25.469
I have, I have, I reckon, six pretty broad reasons why data science projects are prone to prone to failure. And it can be combinations of those reasons. Of course, it can be more than one. But I have six core sort of reasons why. And I call those the micro reasons, but they are they're really important. But then there's a big Whopper. There's the big reason why I think a project projects fail. So I want you to maybe from your own experience, or from experience you've read about I want you to have a crack at seeing whether you can you can get some of these, and we'll have a chat about that.
00:03:25.530 --> 00:03:42.330
Okay, so this is like, data project. Bingo. I hit them all. Okay. Oh, yeah, we need all those same words, that should have been a one year gift to us.
00:03:38.340 --> 00:04:12.930
Yes. Okay, so I have to just come up with something that immediately to me is something I would like seen as a possible failure from the exposure that I've had to projects. And number one on my list is probably engagement, right? and stakeholders need to be engaged, engaged with you as a solution provider engaged with the problem, and the company engaged with the strategy of the company, because you want it to align with their overarching KPIs. And once they're on board, great, that's like a major obstacle number one.
00:04:12.990 --> 00:05:02.610
Yeah, that is definitely that is definitely one of the key reasons why projects fail. I think if you don't have engaged stakeholders, I've worked in several projects where stakeholders have fallen out of love with the idea of doing the project or never really bought into the idea of doing the project in the first place. Or don't realise sometimes sometimes some of the person said, Is this the person setting the project up having this great, oh, we should talk to the dead science team. They're really great. They're, they're not they're actually nothing to do with the pathway to, to delivering that project. And so the people who actually have to work with the tools that you develop, are going, sorry what's this, where did this come from, we didn't ask for this. And so that's a real issue. And of course, they're just as much stakeholders, in fact, really, they are the stakeholders in this case. So yeah, stakeholder engagement number one, that's a really good really good get.
00:05:00.870 --> 00:05:02.610
Well done.
00:05:03.180 --> 00:05:16.949
Oh, I should get a coffee. Oh, yeah. Just sitting on that victory, like, great.
00:05:07.560 --> 00:05:27.269
And get another coffee here. And number two, the data itself has to be clean, or at least in some form that's usable. Absolutely.
00:05:21.870 --> 00:05:30.389
I was at a talk this week and somebody referred to their data.
00:05:27.269 --> 00:05:49.110
I don't know what to call this, but they called it a swamp. It might have been a cloud or might have been a lake somewhere, but they refer to it as a swamp. And I was like, wow, yeah, there could be a swamp of data that people are saying, Let's get some scientists in to look at this swamp. And it's like all we need to prep the data better than this. Yeah.
00:05:49.139 --> 00:06:04.319
Yeah. How it's being collected, when it was collected, and tell if that collection mechanism has changed, as it always, always, almost always does feel good time. Who knows about the data?
00:05:58.980 --> 00:06:53.490
You know, I think organisations that do this, well typically have curators, you know, who are really experienced, really savvy people who look after, in sometimes individual tables, and they they know, they know everything about the columns, the metadata, the provenance of that data. Yeah. And and they can they they are the go to people for that, that data set and allow you to then say, oh, what can you tell me all about that? That how we're collecting that that longitude and latitude? And what level of accuracy we have? And when we changed equipment? How did that affect the the output? All of this stuff is just so so important, but that so often, you only discover when you're well buried within within the project, quite how parlous a state it can be in.
00:06:54.240 --> 00:06:59.220
I've engaged with some data analysts who blow me away their knowledge of the depth of the data that they have.
00:06:59.310 --> 00:07:02.819
Right? Right, exactly.
00:06:59.310 --> 00:07:08.730
Sometimes it's community knowledge, you know that a team has that data, that knowledge about the data, or the or some stakeholders have the knowledge.
00:07:08.939 --> 00:07:20.069
But it's rarely in one place, I find, you know, you often have to search and hunt high and low to find the people who don't really know. So here we go, stakeholders and data we have two good hits straight off the bat. Yeah,
00:07:20.279 --> 00:07:26.850
This is this isn't a bite episode anymore, though, we're going to be here for a while.
00:07:22.529 --> 00:07:26.850
I've a list to go through them.
00:07:27.360 --> 00:07:52.290
And number three, one of the things that I think is the actual team, the teamwork, the skills that are brought together, we talked about the need for diversity of skills and team for em, you can bring in any amount of bias or a lack of understanding, if you just have a skills gap, maybe in your team or just one person working in isolation.
00:07:52.500 --> 00:09:06.149
Yeah, yeah, you've hit hit another one here. So yeah, the science, the team who unders have that scientific knowledge, or don't maybe, you know, it's such a big subject, you can have a team of 15 people, and maybe you don't have coverage of, you know, Transformers in NLP, you know, deep learning neural networks, it requires a huge degree of expertise in a really broad area. So, I'm gonna roll into this as well, not just the not just the coverage of the scientific method, but also the ways of working with that scientific method. You know, are you? Are you a team that goes, I want to go down this rabbit hole, I'm not going to come out, until I've extracted every last percentage of goodness, or are you a team that is is is prepared to do maybe quite shallow investigation initially, and then and then deepen it as necessary? Because I think a lot of the reason some projects go go south is is when individuals maybe decide they're going to just go go after one particular problem that's fascinating them and not actually focused on on the big picture. And the science becomes sort of almost more interesting than the business problem. And that can be a bit of a bit of an issue I've seen, certainly,
00:09:06.149 --> 00:09:07.950
Yeah, it can suck people in.
00:09:07.950 --> 00:09:19.200
Yeah, it can. Good stuff. This is going really well. This is this is this isn't gonna be too long. So we got lost, we got science, we've got data, we got stakeholders what do we think?
00:09:19.559 --> 00:09:41.669
Well, one bit we were talking about there made me think of deployment, like getting what you've built, deployed in the business and productisation. And one of those skills gaps could be the ability to actually get a model, automated, whether whether it's versioned and tested and actually implemented then, for your user, whoever that might be, stakeholder, customer.
00:09:41.759 --> 00:11:10.470
Yep. Yeah. Yeah, it's absolutely the there are many issues around this. And this is probably not one this is probably several reasons but if you don't have good engineering capability, within within or adjacent to your team or within your project if you're working in a cross functional team Then then it's going to be a real struggle. And, you know, data science teams that have good engineers within them are always more capable, I think and are in a much stronger position to just to demo what they've done to stakeholders quicker. And then if you get demos, then you get feedback, you get feedback, you get improved product, you get all that virtuous circle stuff going on, don't you? Suddenly, I call out particularly in this, which is a bit of a curse of the data scientists sometimes, which is that that you're working on a problem, you're developing a model. It's a really nice model and it but it's quite computationally intensive. And, and I've seen this several times, whereby the data scientist pulls out the solution, after maybe half an hour, or an hour's worth of processing from their, from their model. And well, you know, guess what happens that the stakeholder comes to them and says, what an hour? I was hoping that was going to take two minutes. Yeah. Well, I've only got 10 seconds for that results come out to the for it to be useful. You know, the engineering doesn't just cover, being able to deploy it, it's actually got to be done in a timely way as well, which is really hard.
00:11:11.129 --> 00:11:17.129
I'm just going to run this notebook in my laptop in my locker overnight. And then and then the results, come out.
00:11:17.700 --> 00:11:30.509
Exactly. And of course, you know, this is data science, right, it tends to tends to be models, which are consuming inordinate amounts of data. So I mean, we're not, you know, there's a big scalability issue to address. It's not really one issue. It's probably two. Yeah.
00:11:30.570 --> 00:11:35.820
And which can be addressed. But yeah, it's not always immediately addressed.
00:11:32.610 --> 00:11:36.269
Yeah. So any other guests, many, how many of
00:11:36.539 --> 00:11:43.230
You've got four. You're doing really well. So it's to to go and you know, false guests, but I haven't say no surprises here yet.
00:11:44.549 --> 00:12:12.059
No surprises. Yes. Maybe over promising. And what I mean by that is sometimes you say, yeah, we'll build you. This could be echoing the rabbit hole will build you this amazing machine learning model. Turns out, they just needed something quicker, like demo that you just mentioned. He did a demo, he did a quick cycle. This is a deliverable. And something I've heard about, or people have said, is, if you aren't embarrassed by your first demo, then you've demoed too late.
00:12:13.440 --> 00:12:16.440
That's good. You over promised something and you end up spending too long.
00:12:16.470 --> 00:12:24.149
We're just, we just at that stage of the project at the moment. Yeah, we want to go back to the customer really quickly.
00:12:24.210 --> 00:12:27.629
And I'm thinking this is so simple. He's got this gets embarrassing.
00:12:28.889 --> 00:12:31.559
Sounds like you're right at that point. Yeah.
00:12:31.799 --> 00:12:44.039
That's a really good point. I like that one. It's not it's not I haven't got over promising on my list. I guess I probably would have rolled that into stakeholder. But it's it easily be ways of working but.
00:12:41.190 --> 00:12:46.889
Yeah. Okay. That's good. I like it. So we alluded to it a little bit, I get to give you a hint.
00:12:46.919 --> 00:12:55.559
So we alluded to a little bit with the skate the scaling issue, because if a computation is needed in a particular time, it's usually needed for a reason, right?
00:12:55.649 --> 00:12:58.529
Oh, is this outlining the problem from the outset?
00:12:58.830 --> 00:13:01.440
Right, right. Right.
00:12:58.830 --> 00:13:05.460
Right. So this is this is what, what is it feeding into? Right?
00:13:01.740 --> 00:14:02.129
What is what does the downstream look like? Right? What is the decision that we're actually influencing? With our Yeah, I saw, I saw a nice thread on LinkedIn a couple of days ago, where a guy was going, Oh, you know, so, so often. So often people are using their data science teams to generate insight. And you know, what, what really does that achieve, you know, if you if you're putting your insights into a dashboard, or, and he said, Well, why don't I tell you exactly what happens, you produce some nice graphs, and they get put into a PowerPoint deck. And then a month later, it gets presented as a meeting, it's like, well, you're already you're already a month behind the curve, when you do that, but probably two or three, given how long it took to run the tool, and then get it get it across to somebody. So you know, if you want impact from a data science team, you've got to design the forward process that takes that science and show which decision is going to be influenced by it.
00:14:02.129 --> 00:14:05.129
And therefore, of course, show the impact that then comes from that.
00:14:05.250 --> 00:14:06.059
I like that.
00:14:06.779 --> 00:14:09.389
That's a big one for me, I definitely preached that but a lot.
00:14:09.899 --> 00:14:31.799
I was thinking of something similar when we were talking about stakeholder engagement, about the roadmap that along that roadmap that you were talking about to get impact. There'll also be a bunch of like, gate points or decision points that may have a stakeholder who needs to be brought in to get their buy in for the value to be driven. Yep, that Yeah,
00:14:32.100 --> 00:14:58.440
Absolutely. So I mean, there's there's there is overlap in the so the stakeholder engagement Yeah, the with the decision with the impact these these of course, are all critical things that that play together and that none of these tend to sit in isolation. So the last one we have we have five now we got a we got one more to get this it's a slightly fiddly one. And it depends on the industry that you come from as to whether you care about this or that you everyone cares about it, but whether it's topper most in your mind, I think
00:14:58.860 --> 00:15:00.120
Interpretability?
00:15:00.870 --> 00:15:09.840
Ah, no pos Yeah, actually, I might have to just give that to you. Yeah. Okay.
00:15:05.759 --> 00:16:10.470
interpretability usually means that you want to know why. And if you want to know why the tool is producing the decision, typically there's a regulator sitting not a million miles away from you saying, you have to show me why you've made that decision, maybe in a medical healthcare setting, or in a, in an ethical governance setting or in a statutory setting. You know, there are lots of industries finance, for instance, where there is a huge regulation framework that has to be not just not just carefully curated and adhered to, but it has to be shown to be adhered to, you really got to make sure that you're compliant with your regulator, with your industry, with your even with your company policies, of course. So all of that has to be part of part of writing the tool. And, you know, any data scientists will know if they've done this for any length of time that you fast become very, very aware of and then expert in the regulatory frameworks that you're operating under.
00:16:10.740 --> 00:16:12.269
Yeah, that's really interesting.
00:16:12.299 --> 00:16:16.889
So there's one last one last challenge, which I think for right now, I just want my prize.
00:16:17.250 --> 00:16:20.549
You get your prize, prize. Okay, so it's not a challenge anymore.
00:16:20.549 --> 00:16:44.789
It's really just to get a last question. So those those I reckon, are the not so micro reasons for why data science projects fail. But there's, there's something which I think ties all of these together, and the companies that succeed have it, and the companies that that struggle anyway, shall we say, are ones which maybe don't have this for this particular property, you want to hazard a guess as to what that might be?
00:16:45.000 --> 00:16:48.509
Oh, like, I wanna say, investment from the top.
00:16:48.539 --> 00:17:11.880
Yeah, pretty much. I mean, it's, it's basically culture, I think, because you've got to, you've got, you've got to have that cultural desire to embrace what can be, I think, a really radical departure from your usual ways of working and, and, and also, you feel reasonably comfortable with quite quite quite abrupt change in the way that you do things.
00:17:12.210 --> 00:17:35.250
And that can be a big struggle for some companies, I think, certainly, I both worked for and and and seen this distance companies that have really struggled with that cultural standpoint, where getting getting changed into into the company and adopting this new technology, disruptive technology, and it is highly disruptive sometimes can be a real struggle for them. I don't know if you've had any experiences like that.
00:17:35.400 --> 00:18:02.730
Yeah, that point you just said about disruption's pretty key. And I've heard people talk about needing to embed an allowance for curiosity, and the psychological safety that comes with being okay with not so much failing, but experimenting, and your experiments may not deliver. But that's why you ran the experiment to see what needs to change. And any aspect of all the points we've hit on could be what needs to change.
00:18:03.210 --> 00:18:36.029
I love that and allowance for curiosity. I think that's a really nice phrase. It ties up very nicely with how a lot of good teams execute their projects, which is to say, they don't say, we will produce this model, and it will do this, they say, Can we produce a model, which actually impacts the company with 5% improvement in whatever metric and and you know, it's just as okay to come out with a no to that question is is with a with a yes to that question. Because you know, it both outcomes are likely Right.
00:18:33.990 --> 00:18:36.029
Yeah.
00:18:36.240 --> 00:18:38.700
Very good. That's been really enjoyable.
00:18:38.849 --> 00:18:49.440
Well, I guess I think it's safe to say that, you know, culturally, DataCafe has ticked all the right boxes. We're still going off for a year we have we have a high allowance for curiosity on this show. So
00:18:49.440 --> 00:19:11.430
Yes, very much so. Ah looking forward to what the future holds. Thanks for joining us today at the DataCafe. You can like and review this on iTunes or your preferred podcast provider. Or if you'd like to get in touch. You can email us Jason at datacafe.uk or Jeremy at datacafe.uk, or on Twitter at datacafepodcast. We'd love to hear your suggestions for future episodes.
00:19:16.289 --> 00:19:18.210
Can I? I can shout Bingo.
00:19:18.599 --> 00:19:22.920
You can shout Bingo.
00:19:18.599 --> 00:19:22.920
Excellent
