บทถอดความฉบับเต็มของ [Bite] Why Data Science projects fail

another coffee here. And number two, the data itself has to be clean, or at least in some form that's usable. Absolutely. 00:05:21.870 --> 00:05:30.389 I was at a talk this week and somebody referred to their data. 00:05:27.269 --> 00:05:49.110 I don't know what to call this, but they called it a swamp. It might have been a cloud or might have been a lake somewhere, but they refer to it as a swamp. And I was like, wow, yeah, there could be a swamp of data that people are saying, Let's get some scientists in to look at this swamp. And it's like all we need to prep the data better than this. Yeah. 00:05:49.139 --> 00:06:04.319 Yeah. How it's being collected, when it was collected, and tell if that collection mechanism has changed, as it always, always, almost always does feel good time. Who knows about the data? 00:05:58.980 --> 00:06:53.490 You know, I think organisations that do this, well typically have curators, you know, who are really experienced, really savvy people who look after, in sometimes individual tables, and they they know, they know everything about the columns, the metadata, the provenance of that data. Yeah. And and they can they they are the go to people for that, that data set and allow you to then say, oh, what can you tell me all about that? That how we're collecting that that longitude and latitude? And what level of accuracy we have? And when we changed equipment? How did that affect the the output? All of this stuff is just so so important, but that so often, you only discover when you're well buried within within the project, quite how parlous a state it can be in. 00:06:54.240 --> 00:06:59.220 I've engaged with some data analysts who blow me away their knowledge of the depth of the data that they have. 00:06:59.310 --> 00:07:02.819 Right? Right, exactly. 00:06:59.310 --> 00:07:08.730 Sometimes it's community knowledge, you know that a team has that data, that knowledge about the data, or the or some stakeholders have the knowledge. 00:07:08.939 --> 00:07:20.069 But it's rarely in one place, I find, you know, you often have to search and hunt high and low to find the people who don't really know. So here we go, stakeholders and data we have two good hits straight off the bat. Yeah, 00:07:20.279 --> 00:07:26.850 This is this isn't a bite episode anymore, though, we're going to be here for a while. 00:07:22.529 --> 00:07:26.850 I've a list to go through them. 00:07:27.360 --> 00:07:52.290 And number three, one of the things that I think is the actual team, the teamwork, the skills that are brought together, we talked about the need for diversity of skills and team for em, you can bring in any amount of bias or a lack of understanding, if you just have a skills gap, maybe in your team or just one person working in isolation. 00:07:52.500 --> 00:09:06.149 Yeah, yeah, you've hit hit another one here. So yeah, the science, the team who unders have that scientific knowledge, or don't maybe, you know, it's such a big subject, you can have a team of 15 people, and maybe you don't have coverage of, you know, Transformers in NLP, you know, deep learning neural networks, it requires a huge degree of expertise in a really broad area. So, I'm gonna roll into this as well, not just the not just the coverage of the scientific method, but also the ways of working with that scientific method. You know, are you? Are you a team that goes, I want to go down this rabbit hole, I'm not going to come out, until I've extracted every last percentage of goodness, or are you a team that is is is prepared to do maybe quite shallow investigation initially, and then and then deepen it as necessary? Because I think a lot of the reason some projects go go south is is when individuals maybe decide they're going to just go go after one particular problem that's fascinating them and not actually focused on on the big picture. And the science becomes sort of almost more interesting than the business problem. And that can be a bit of a bit of an issue I've seen, certainly, 00:09:06.149 --> 00:09:07.950 Yeah, it can suck people in. 00:09:07.950 --> 00:09:19.200 Yeah, it can. Good stuff. This is going really well. This is this is this isn't gonna be too long. So we got lost, we got science, we've got data, we got stakeholders what do we think? 00:09:19.559 --> 00:09:41.669 Well, one bit we were talking about there made me think of deployment, like getting what you've built, deployed in the business and productisation. And one of those skills gaps could be the ability to actually get a model, automated, whether whether it's versioned and tested and actually implemented then, for your user, whoever that might be, stakeholder, customer. 00:09:41.759 --> 00:11:10.470 Yep. Yeah. Yeah, it's absolutely the there are many issues around this. And this is probably not one this is probably several reasons but if you don't have good engineering capability, within within or adjacent to your team or within your project if you're working in a cross functional team Then then it's going to be a real struggle. And, you know, data science teams that have good engineers within them are always more capable, I think and are in a much stronger position to just to demo what they've done to stakeholders quicker. And then if you get demos, then you get feedback, you get feedback, you get improved product, you get all that virtuous circle stuff going on, don't you? Suddenly, I call out particularly in this, which is a bit of a curse of the data scientists sometimes, which is that that you're working on a problem, you're developing a model. It's a really nice model and it but it's quite computationally intensive. And, and I've seen this several times, whereby the data scientist pulls out the solution, after maybe half an hour, or an hour's worth of processing from their, from their model. And well, you know, guess what happens that the stakeholder comes to them and says, what an hour? I was hoping that was going to take two

minutes. Yeah. Well, I've only got 10 seconds for that results come out to the for it to be useful. You know, the engineering doesn't just cover, being able to deploy it, it's actually got to be done in a timely way as well, which is really hard. 00:11:11.129 --> 00:11:17.129 I'm just going to run this notebook in my laptop in my locker overnight. And then and then the results, come out. 00:11:17.700 --> 00:11:30.509 Exactly. And of course, you know, this is data science, right, it tends to tends to be models, which are consuming inordinate amounts of data. So I mean, we're not, you know, there's a big scalability issue to address. It's not really one issue. It's probably two. Yeah. 00:11:30.570 --> 00:11:35.820 And which can be addressed. But yeah, it's not always immediately addressed. 00:11:32.610 --> 00:11:36.269 Yeah. So any other guests, many, how many of 00:11:36.539 --> 00:11:43.230 You've got four. You're doing really well. So it's to to go and you know, false guests, but I haven't say no surprises here yet. 00:11:44.549 --> 00:12:12.059 No surprises. Yes. Maybe over promising. And what I mean by that is sometimes you say, yeah, we'll build you. This could be echoing the rabbit hole will build you this amazing machine learning model. Turns out, they just needed something quicker, like demo that you just mentioned. He did a demo, he did a quick cycle. This is a deliverable. And something I've heard about, or people have said, is, if you aren't embarrassed by your first demo, then you've demoed too late. 00:12:13.440 --> 00:12:16.440 That's good. You over promised something and you end up spending too long. 00:12:16.470 --> 00:12:24.149 We're just, we just at that stage of the project at the moment. Yeah, we want to go back to the customer really quickly. 00:12:24.210 --> 00:12:27.629 And I'm thinking this is so simple. He's got this gets embarrassing. 00:12:28.889 --> 00:12:31.559 Sounds like you're right at that point. Yeah. 00:12:31.799 --> 00:12:44.039 That's a really good point. I like that one. It's not it's not I haven't got over promising on my list. I guess I probably would have rolled that into stakeholder. But it's it easily be ways of working but. 00:12:41.190 --> 00:12:46.889 Yeah. Okay. That's good. I like it. So we alluded to it a little bit, I get to give you a hint. 00:12:46.919 --> 00:12:55.559 So we alluded to a little bit with the skate the scaling issue, because if a computation is needed in a particular time, it's usually needed for a reason, right? 00:12:55.649 --> 00:12:58.529 Oh, is this outlining the problem from the outset? 00:12:58.830 --> 00:13:01.440 Right, right. Right. 00:12:58.830 --> 00:13:05.460 Right. So this is this is what, what is it feeding into? Right? 00:13:01.740 --> 00:14:02.129 What is what does the downstream look like? Right? What is the decision that we're actually influencing? With our Yeah, I saw, I saw a nice thread on LinkedIn a couple of days ago, where a guy was going, Oh, you know, so,

so often. So often people are using their data science teams to generate insight. And you know, what, what really does that achieve, you know, if you if you're putting your insights into a dashboard, or, and he said, Well, why don't I tell you exactly what happens, you produce some nice graphs, and they get put into a PowerPoint deck. And then a month later, it gets presented as a meeting, it's like, well, you're already you're already a month behind the curve, when you do that, but probably two or three, given how long it took to run the tool, and then get it get it across to somebody. So you know, if you want impact from a data science team, you've got to design the forward process that takes that science and show which decision is going to be influenced by it. 00:14:02.129 --> 00:14:05.129 And therefore, of course, show the impact that then comes from that. 00:14:05.250 --> 00:14:06.059 I like that. 00:14:06.779 --> 00:14:09.389 That's a big one for me, I definitely preached that but a lot. 00:14:09.899 --> 00:14:31.799 I was thinking of something similar when we were talking about stakeholder engagement, about the roadmap that along that roadmap that you were talking about to get impact. There'll also be a bunch of like, gate points or decision points that may have a stakeholder who needs to be brought in to get their buy in for the value to be driven. Yep, that Yeah, 00:14:32.100 --> 00:14:58.440 Absolutely. So I mean, there's there's there is overlap in the so the stakeholder engagement Yeah, the with the decision with the impact these these of course, are all critical things that that play together and that none of these tend to sit in isolation. So the last one we have we have five now we got a we got one more to get this it's a slightly fiddly one. And it depends on the industry that you come from as to whether you care about this or that you everyone cares about it, but whether it's topper most in your mind, I think 00:14:58.860 --> 00:15:00.120 Interpretability? 00:15:00.870 --> 00:15:09.840 Ah, no pos Yeah, actually, I might have to just give that to you. Yeah. Okay. 00:15:05.759 --> 00:16:10.470 interpretability usually means that you want to know why. And if you want to know why the tool is producing the decision, typically there's a regulator sitting not a million miles away from you saying, you have to show me why you've made that decision, maybe in a medical healthcare setting, or in a, in an ethical governance setting or in a statutory setting. You know, there are lots of industries finance, for instance, where there is a huge regulation framework that has to be not just not just carefully curated and adhered to, but it has to be shown to be adhered to, you really got to make sure that you're compliant with your regulator, with your industry, with your even with your company policies, of course. So all of that has to be part of part of writing the tool. And, you know, any data scientists will know if they've done this for any length of time that you fast become very, very aware of and then expert in the regulatory frameworks that you're operating under. 00:16:10.740 --> 00:16:12.269 Yeah, that's really interesting. 00:16:12.299 --> 00:16:16.889 So there's one last one last challenge, which I think for right now, I just want my prize. 00:16:17.250 --> 00:16:20.549 You get your prize, prize. Okay, so it's not a challenge anymore. 00:16:20.549 --> 00:16:44.789 It's really just to get a last question. So those those I reckon, are the not so micro reasons for why data science projects fail. But there's, there's something which I think ties all of these together, and the companies that succeed have it, and the companies that that struggle anyway, shall we say, are ones which maybe don't have this for this particular property, you want to hazard a guess as to what that might be? 00:16:45.000 --> 00:16:48.509 Oh, like, I wanna say, investment from the top. 00:16:48.539 --> 00:17:11.880 Yeah, pretty much. I mean, it's, it's basically culture, I think, because you've got to, you've got, you've got to have that cultural desire to embrace what can be, I think, a really radical departure from your usual ways of working and, and, and also, you feel reasonably comfortable with quite quite quite abrupt change in the way that you do things. 00:17:12.210 --> 00:17:35.250 And that can be a big struggle for some companies, I think, certainly, I both worked for and and and seen this distance companies that have really struggled with that cultural standpoint, where getting getting changed into into the company and adopting this new technology, disruptive technology, and it is highly disruptive sometimes can be a real struggle for them. I don't know if you've had any experiences like that. 00:17:35.400 --> 00:18:02.730 Yeah, that point you just said about disruption's pretty key. And I've heard people talk about needing to embed an allowance for curiosity, and the psychological safety that comes with being okay with not so much failing, but experimenting, and your experiments may not deliver. But that's why you ran the experiment to see what needs to change. And any aspect of all the points we've hit on could be what needs to change. 00:18:03.210 --> 00:18:36.029 I love that and allowance for curiosity. I think that's a really nice phrase. It ties up very nicely with how a lot of good teams execute their projects, which is to say, they don't say, we will produce this model, and it will do this, they say, Can we produce a model, which actually impacts the company with 5% improvement in whatever metric and and you know, it's just as okay to come out with a no to that question is is with a with a yes to that question. Because you know, it both outcomes are likely Right. 00:18:33.990 --> 00:18:36.029 Yeah. 00:18:36.240 --> 00:18:38.700 Very good. That's been really enjoyable. 00:18:38.849 --> 00:18:49.440 Well, I guess I think it's safe to say that, you know, culturally, DataCafe has ticked all the right boxes. We're still going off for a year we have we have a high allowance for curiosity on this show. So 00:18:49.440 --> 00:19:11.430 Yes, very much so. Ah looking forward to what the future holds. Thanks for joining us today at the DataCafe. You can like and review this on iTunes or your preferred podcast provider. Or if you'd like to get in touch. You can email us Jason at datacafe.uk or Jeremy at datacafe.uk, or on Twitter at datacafepodcast. We'd love to hear your suggestions for future episodes. 00:19:16.289 --> 00:19:18.210 Can I? I can shout Bingo. 00:19:18.599 --> 00:19:22.920 You can shout Bingo. 00:19:18.599 --> 00:19:22.920 Excellent