[Bite] Data Science and the Scientific Method の全文書き起こし

needs to be a scientific rigour that comes with forming the hypothesis, and that's what the scientific method can allow? 00:06:13.529 --> 00:07:44.730 Yeah, I, for me, it was always, it was always about the sort of the test and learn, right, I've got a phenomenon, it was rarely a phenomenon, I had to say, in computer science, it tended to be I've got a white box, not for my particular brand of computer science of black box. And, and I'm applying it to a particular data set. And I wanted to understand or a particular environment, I wanted to understand, how am I algorithm impacted and reacted to that? That data that environment? Yeah, so I might be trying to construct a ranked measure of which web page is really important or something like that. And there's loads of parameters involved in this. And some are, some are significant, and some are not. And so what I would tend to do is say, right, well, let's, let's tweak a parameter, and see what what impact it has. And then there'll be a measure at the other end of the process, which looks at the output and goes, this has had a dramatic impact, you know, I've seen a significant improvement because of that tweak in the parameter. That's the assumption from from a data science perspective, this, this transfers really well, because you've got this idea of keeping the vast body of your setup the same, changing one thing as much as possible, leaving everything else the same, and seeing, seeing what the impact is of that one, one change. And ideally, being able to infer This is a bit of an assumption that the change you then saw was as a direct result, of course, from the change that you made. 00:07:44.730 --> 00:07:48.449 Yeah, what do you get from that? 00:07:44.730 --> 00:08:15.360 Well, you get the learning, you get, you get to create a hypothesis about your environment, or about your algorithm or about your, about about your problem domain and your decision that's being impacted. And which, which then can lead to more experiments, more tweaking, and testing and learning and all of that sort of thing. So I think what we're fast getting towards is it, you start to get a nice, looping, iterative process. 00:08:15.449 --> 00:08:37.590 Yeah. Which is really important. Because even when you draw the scientific methods, you can draw it as a loop and say, you come out of one experiment with some conclusion, and that's fine. We can replicate it. And that's proven. But we also have 10 new questions. And we want to run 10 new experiments with 10 new versions of data. You know, go through that test and learn that you're talking about. 00:08:37.769 --> 00:08:56.700 And there's the problem or me maybe a problem, if for many years, the excitement, right, which is, oh, wow, I started with something I thought was really straightforward. And now I've got something which is I've got, you know, I've got 10 possible data sets and 10 possible questions on each of those data sets. So and I've got 100 investigations to carry

out. 00:08:57.120 --> 00:09:03.509 And that I think, in, in a scientific environment in especially in academic environment, yes, that's great. 00:09:03.509 --> 00:09:06.210 That's just grist to the mill. 00:09:03.509 --> 00:09:19.679 That's exactly what you're expecting you can you can divvy that up amongst your PhD students or graduate students, you could, you can have a plan for how you're going to tackle investigating and prioritising those over the next three years. 00:09:16.409 --> 00:09:22.409 Right. Yeah. But that doesn't quite work in a commercial setting and in Data Science. 00:09:23.129 --> 00:10:18.840 Yeah. And this is where I think we need at the core of our efforts, the scientific method to be understood so that we follow the correct process of forming our hypothesis and knowing that our experiment, experiment is valid, and the data is valid for the setup that we have for us. I think going into it. We also have at the outset, to know the decision to know the impact to know the cost where the line gets drawn. And how much do we need to verify our assumption Since How much does our hypothesis need to be proven? Before the impact is realised before that decisions taken? Because your business model, you know, depending on the context might hinge upon it might be something subtle, or it might be a massive change in your business operations. 00:10:20.190 --> 00:11:04.049 I think that the the change, I noticed, when I started working in industry a few years ago, now, the chain, the change, I noticed was how important it was to get a really salient and to the point hypothesis, out of the out of the box, right. Yeah, to get the exact concept that you were trying to test to trying to prove that if you did show should be true, or not true, would would enable you to access the decision access the the impacts that your your data science algorithms Exactly. But you know, see, what you didn't want was a sequence of interesting investigations, which in the end, it gave you answers to some questions. But those questions didn't really help you with 00:11:04.049 --> 00:11:48.720 Right. I think that's where our stakeholder management is so important, because when we set out to run one of these scientific experiments on the data set that we have, and to answer a question, it's exactly what you just said, we need to bring that back to the stakeholders version of the question, what is it that they're going to actually say is the hypothesis that they had that was interpreted then in the experiment set up that we adored the model that we may have built, and I think that that's where it's important for us as scientists to bring them into that way of thinking. And that's possibly where I see opportunity for a disconnect that needs to be seized upon, you know. 00:11:48.750 --> 00:13:10.320 I see that in in many projects where I have, you know, well, meaning stakeholders, approaching the team, some, you know, at a critical point in the project, and they will, they will show this disconnect very straightforwardly by just saying, how long is that going to take? Of course, yeah. And yeah, as a data scientist, and as a scientist, that that instantly causes sort of flags to go up and sort of alarms to ring because I'm going, Yeah, it's an investigation, I can't really, I can't really give you a how long, which, of course, is, you know, if they're, if they're used to doing some kind of Gantt chart based project plan, then then that causes instantly causes a bit of a problem. So I think there's some really interesting and nice modifications, then that come from that uncertainty at the heart of this data science and scientific process, which is to say, I don't know, if I knew the answer, I wouldn't have to do all this fancy stuff in, you wouldn't be paying a bunch of scientists to do this investigation. If I knew the answer. The whole point is, we don't know the answer. And we're gonna have to work out what that answer is, before we can really progress this to the next next stage of the project. 00:13:10.440 --> 00:13:40.049 Yeah. And it raises the questions when they're asking, how long will it take? Were we asking the right question in the first place? Because what you should be asking is, do you have an answer? Yes? Or what is it that we need to do to get us get confidence about it? Is it that we need more data that we need more resources that we need a new hypothesis to be visited with? Maybe a new set of stakeholders? 00:13:40.000 --> 00:13:55.659 Yeah, and sometimes you get lucky. Sometimes you can say, look, we can get you a certain quality of answer. And we can, you know, you tell us that you you tell us how long we got almost Yeah. So if you if you've got a couple of days or a week, I can get you something. 00:13:51.970 --> 00:14:09.309 So depending on the problem, if it's like an investigation, and you're sort of slowly refining your answer getting better and better. And you say, Well, I can get it to this good. And I can probably even give you an approximation of how how close this good is to what you want for your, you know, statistical matching algorithm or something. 00:14:09.940 --> 00:14:25.659 But on the other hand, if it's, if it's an investigation where you don't even know whether there is an answer, is there a signal which means that I can identify a cancer in a in a medical scan or a medical image? 00:14:22.269 --> 00:14:30.070 And I just don't know whether whether I can see. So answering the question, how long will it take is much harder then. 00:14:30.100 --> 00:14:37.480 Yeah. So I think at the core of this, we need to make sure that scientists can adhere to the scientific method boss. 00:14:38.169 --> 00:14:59.590 There is fundamental outline of what the question is upfront that needs to be agreed and need to be, compartmentalise, how much time is allowed for pure research or formation of hypothesis or gathering the data? Yeah, there's frameworks to do that. You know, we can have a whole discussion around that another time. But that iterative process. 00:14:59.620 --> 00:16:12.820 I know there are companies who do just investigative epics at the beginning of their projects where they say this epic? Yeah, it's two weeks sprint, this four weeks of sprint, whatever, we're just going to be doing investigates that's definitely one way of doing it. But I think for me, the other thing, which I didn't mention I talked to Dan was that when you're as a team, when you're setting these hypotheses, you should absolutely carry those hypotheses through to your, the way that you set yourselves tasks. And so your hypotheses should really I like, I like my heart hypotheses to be questions. I because I like them to have a yes or no answer. And for one Have either of those answers to be a valid possibility? Whereas in traditional project management, typically, you'll say, well, you'll have a block of time that's associated to put the roof on the house. The answer, if the answer to is well, we couldn't, that's not really acceptable. But in some, but in data science, it's absolutely fine to go, can we find a nice, you know, customer preference metric, which tells me what film that an individual is going to enjoy watching? Yeah. And the answer may be no, not in the way that's useful to you right now. 00:16:13.659 --> 00:16:27.460 In which case, you need to be formed the question you need to think about another way of presenting the problem, you know, there are lots of ways of them rebounding that and going okay, well, we still need to make progress, but maybe we're making sure to make progress down another another arm. 00:16:27.539 --> 00:16:42.750 I think, to summarise just off the back of your analogy there it's a case of we were putting a roof on a house but this might be a house we've never built before, and a roof and a format we've never used before. So our old timelines don't hold. 00:16:42.779 --> 00:16:45.809 Using materials that no one's ever used before. Exactly. 00:16:48.690 --> 00:16:49.200 Exactly. 00:16:49.230 --> 00:16:51.330 Thanks for chatting this out with me, Jeremy. 00:16:51.419 --> 00:16:52.860 No worries, Jason, that was fun! 00:16:54.570 --> 00:17:11.759 Thanks for joining us today at the DataCafe. You can like and review this on iTunes or your preferred podcast provider. Or if you'd like to get in touch you can email us Jason that datacafe.uk or Jeremy at datacafe.uk on twitter at datacafepodcast. We'd love to hear your suggestions for future episodes.