What does it really take to turn evidence into action?

UK Evaluation Society: You're listening to Evaluate This, a limited series podcast from the UK Evaluation Society and sponsored by Digital Science. Hello and welcome to the latest episode of Evaluate This from the UK Evaluation Society. I am Sarah Parks. I am a trustee of the UK Evaluation Society and in this episode I talk to Professor Catherine Oliver from the London School of Hygiene and Tropical Medicine about producing evidence that can be acted on. We talk about Professor Oliver's route into her work in this area, how the area is evolving and gaps in the current landscape. as well as areas of research interest, which are the UK government department's ways of telling researchers about what kind of research they are interested in. The episode also ends with a discussion around an evaluator's role in bringing evidence into action. We're kind of doing this podcast off the back of the UK Evaluation Society conference that was themed around kind of bridging the gap. evidence for action and you've done like a lot of work around evidence production and use. And I wondered if you could start off with telling us a little bit about how you got into that and how you think that area is kind of evolving over time. Well, I'm basically an amateur social scientist, so I didn't really get into this area on purpose. I sort of fell into it by mistake. So I was originally a molecular biologist, but my cell technique was really bad. I just killed everything. So I ended up getting a job doing systematic reviews and then It's actually an excellent training in social science research methods, because you have to understand every kind of research design and be able to say whether it's been done properly or not. So in that sense, I suppose I did get some training. But the Systematic Review facility that I was attached to was commissioned by various different government departments to provide Systematic Review evidence in response to needs that they articulated to us. So after a few years of doing these Systematic Reviews and sending them off, I sort of became curious about what happened when they landed on the other side of the desk. And you start to get more involved in the conversations with the policy teams as you get a bit more senior as well. And I suppose I just realized that sending over 250 page reports saying, well, the quality of the evidence wasn't very good, perhaps not that helpful. And I started to wonder if there was a different way we can approach this problem, the gap, as you say, between action and the evidence. So I ended up doing a PhD, which was about evidence use in local government, again, partly because that's what was available to apply for. But I, so I did that in Greater Manchester and I was genuinely really curious about what happened in decision making, you how did it work in local government? I didn't know anything about local government, of course, I didn't, done a very relevant degree to have any background. So I was coming at it from a very ignorant perspective. So I just went and hung around with these guys for as much as I could. I went to as many meetings as I could, public ones. I made friends with the secretariat so I could go and sort of hang out and, you know, watch the various public health meetings. I learned a lot from that. And one was it's basically all about the people. know, structures are all very well, but structures are inhabited by people who behave exactly as you would expect people to behave. And so you've got to take that human element really seriously. And the other thing that was very interesting was I was always, without exception, the only academic at those meetings, which I think is kind of extraordinary in a city of the size of Greater Manchester, which had six universities. How little outreach, I guess, universities were doing. And that was kind of fascinating, given that the conversation within universities, certainly at the time, was very much, well, we want to be helpful, but they just won't listen. Okay, so there's bit of cognitive dissonance there. And so that was back in 2012. And since then, I've just had various different research jobs, but never really let go of this problem of what the hell's going on here? You know, we've got loads of evidence. Research is being done all the time. Everyone says how great the UK research system is. mean, things are financially a little bit tight now, but frankly, we've got pots of money compared to other countries to do research. People in local government, people in central government. really want help, they're getting better and better at saying what the problems are and understanding the role that research could and should play. Why is this just not working? Like what's that about? In the last few years, people have started taking it sort of officially more seriously. So research funders, for example, have started making funds specifically available for research to policy or research to action type initiatives. So they're funding activity in this space. and I have done a few evaluations of different knowledge exchange initiatives of that kind. They said there's been this huge growth in the amount of activity in this area, but it's not terribly coordinated. I suppose if you had a cynical mindset, it's not very surprising really that every single university and practically every project is trying to do things off its own bat and competition with each other rather than. collaboratively having a needs led approach to evidence exchange. So it's still very much on the researchers producing loads of good stuff and then just chucking it out there and hoping for the best. And I feel like all of that activity is entirely divorced from the social science of evidence use, which is also really fascinating. So, you know, in the U S they've been taking this seriously as a, as an intellectual problem. ⁓ way back in the 60s and 70s really, people like Carol Weiss who is a public administration academic studying the use of evaluation specifically. around that, know, as a sort of seed area, there's been a big growth in theory and practice and we know a lot about how to do it. But again, there's this gap between, we know about how to do it and how people are actually doing it. So I think we all need to make friends a bit more. What do you think that kind of gap has? Is that gap? getting smaller or is it the same size as it was and what would we need to do to get it to be smaller? I'd like to say it's getting smaller. It's a bit hard to say because nobody really knows what good looks like. I mean, not in terms of what good practice looks like. I think we do actually know a lot about that. And some of it is basic customer service relations, you know, talk to people about what they need, set up a functional courteous relationship, you know, that is mutually beneficial and so on. And we know a lot about the practice. We don't necessarily know the good knowledge exchange when we see it from the outside. So what are we trying to achieve? I think is not always super clear. We have these very high sounding ideals like, we'll have an impact on policy and practice, or we're to make people's lives better. That's quite hard to measure. think some more practical outcomes are only really currently of interest to science policy nerds like myself. So things like, are we... funding less irrelevant research or a research activities more closely aligned with problems of importance to members of society rather than just the stuff that we care about. But you know, you go down that route, you really annoy a lot of people. Maybe we can talk about that in a second. I think it's really interesting point, thing about like, what exactly do we mean when we say take evidence to action. How do we best measure that? Because actually, as one of the things was discussed a bit at the conference was this sort of space of, know, you have to take into account all the what evidence people want to use in those situations or don't want to use or, you know, that people have certain directions they're trying to go down. so if you have evidence that's not in that direction, it's not necessarily going to be using that. I don't know. I don't know if you have any. reflections on the kind of what's a realistic bar for moving evidence into action? What should we feel good about rather than just, we want to have achieved impact. I think often we feel that we haven't done that. What you're really asking is how does, what does use look like? So, you know, I think we often start impact a very unhelpful term, I think, because it really gives the impression that something's changing. You know, there's something there's going to be observable effects. you know, a policy decision is going to be reversed or whatever it might be. But I don't think that's what use looks like. I think use can look like, you know, the changes are happening inside people's heads. That's very difficult to measure. I think it can look like a thoughtful pause in a conversation when something has landed with someone in a way that they weren't expecting. Whether that will translate into a different conversation at the meeting. that's a possible outcome of evidence use in my book, is dependent on lots of different things. But I don't think that necessarily means there's been a failure or a failure of the knowledge exchange. I think it's just, you've got to have reasonable expectations here about how decisions really work in the real world. I mean, even if in the most transactional way, I think you could say, we presented to the minister, the minister heard what we said and nothing changed. I think that is a perfectly successful knowledge exchange initiative. It's not necessarily the outcome that anyone wants, but that would be an unreasonable expectation. To change your policy direction, I think, is not up to us. This is about what the democratic mandate is about. Well, in this case, we're talking about changing to national policy, for example. And that's not on us. think Patrick Vallance, who's government chief scientist for long time, The way he always explained the role of the government chief scientists was that you have explained the evidence and you have ensured that it has been understood. Though he said that was the role, that you can answer any questions about it, but you don't tell people what to do. That is not your role as an advisor and it's not your role as a scientist. Your role is to make sure that the evidence has been heard and comprehended. And I think actually that's a very clear line. It's not a line that many people think about that much. And it's a line that a lot of academics cross all the time. And actually it's not very helpful for us, I think. I work in a non-departmental public body in evaluation and one of the things we get asked a lot is to evidence use of evaluation. So, and what the kinds of things that people want us to evidence the evaluation showed that X doesn't work. So we didn't do it the next time. And then we checked the next time that that was correct. And Hey, ho, this thing is now better. And I think, you know, there are occasionally examples of that. But for me, a lot of the things are I've had this person who's going to be involved in developing this in future in the room to discuss all of these things. And that's changed how they've thought about it. And therefore, they have made some changes going forward rather than there is this kind of evidenceable, is that a word? Chain where you can really clearly say this led to this led to this. A lot of it is what's happening in people's heads and therefore how that kind of has moved the needle on the conversation they have. I think a lot of this misunderstanding about ⁓ how the process works boils down to the fact that universities are very large organisations that employ a lot of people but unlike every other large organisation they have no sense of corporate identity or strategy or direction internally. You're just there. It's more like being a team member on a sports team that you could get sold at any moment and you you're in your, you're out, but you're doing your own thing. I think what that means is that you can be really quite senior and just have no idea how to get things through the machinery of a large organization. So I now have a managerial position or a leadership position, I should say in the NIHR, which is a reasonably large organization, but my bit of it is to connect with department of health and social care. which is a very large organization. And people now bring things to me and say, I think we should change this thing about the program. We've done an evaluation of it. Next time we do this commissioning round, let's do it this way. And I say, okay, good idea. And it's just incredibly difficult to get the organization to behave in a different way, to get meetings set up in a different way, to scrap meetings where they don't work, to tell people that the job that they've been working on for 15 years. is no longer needed in the same way. It's difficult to change the direction of an organization and I think sometimes because academics just aren't exposed to that ⁓ machinery they don't always appreciate that it really is a lot harder than it sounds. You mentioned earlier that you have done some evaluations of these kind of research to policy fellowships or research to action fellowships. Could you tell us a little bit about those, what you found, how they, what they tell us about this kind of space? I've done quite a bit of work in this space in various ways, partly because, and very admirably, research funders are really wanting to make sure that they are investing in the right area and getting back with their buck. I think sort of there's, so there's been a huge uptick in the number of initiatives from the late nineties onwards really before that there were still, you know, lot of very long lived research to policy or research practice initiatives, but they sort of bumbled along under the radar really, and were very solutions oriented. there, you know, there's quite a few in the defense space, for example, because the customer need is quite clear and so on. in the, in the, from the mid nineties onwards, there's been this big uptick and obviously very closely aligned with the impact agenda in the UK associated with the REF and so on. That's given everything a big push and really made individual academics, but primarily universities appreciate that we're going to have to tell better stories about why we're worth public investment, essentially. And for some academics, that's been a blessed relief because it meant that finally the fact that they were actually trying to do all this collaborative work, incredibly difficult, unrewarded. suddenly there were funds available to do it and they've leapt on it with a great enthusiasm. For others, I think it still comes as a bit of an unpleasant surprise that other people might be able to ask questions about the accountability of what are our activities. That's a minority these days, I would say. But this number of schemes that have come out has been quite surprising, I think. So I've evaluated a couple of fellowship schemes. There's also... Quite a lot of sort of funding instruments outside of fellowship schemes and things like seed funding that are designed to get people around a table and talk about a problem and come up with a pitch, you know, like a sort of sandpit type model. there's been longer term partnership collaborations that where, for example, the money, the research money has gone to healthcare practice organizations and from there out to universities. So that's put the problem owners in the driving seat. So there's been a various sort of iterations and they can be a bit faddy. So at the moment we're all about fellowships. I think there's something like 400 fellows and in death for it alone. I suppose what I've learned from it is because we have this very unclear sense of what we're trying to achieve, that we don't have a good theory of change for these. They're very difficult to compare. You know, we're really talking about apples and oranges. Sometimes we were talking about a £10,000 grant at the end of a project to do a bit of nice dissemination with your potential end users. Sometimes we're talking about essentially CPD experience for an academic. But we talk about them as though the goal is the same. The goal is to improve evidence use. The goal is to influence policy and practice. So I think we should be better really at trying to drill down on what our critical mode of action is and what the key ingredients are for different outcomes. We're starting to get to the point where we've got empirical evidence where we can test some assumptions. To be honest, people are very wedded to the idea that a particular intervention is good, which is why we've ended up with hundreds of fellows. The fellowship one in particular is really interesting because there's actually quite a lot of good empirical evidence about fellowships and circumvents from practice. So in the healthcare world, it's very usual for, you know, doctors to have a year out and they go and do, you know, some strategy role somewhere, or they get seconded out as an academic fellow or something like that. And there's quite a lot of evaluations of these and the evidence is pretty unambiguous, which is that the fellows really like it. They get loads out of it. They develop all sorts of new skills. It appeals to parts of their professional identity that are not fulfilled in their normal work. They develop new networks. They have a great time. And then you come back to their home organization and the home organization says, can you, well, you've had your fun. Can you get on with your day job now? basically, you know, it's difficult to release these people and the job needs doing. And so very understandably, the secundi gets a bit fed up and they apply to another one and they become serial secundis because that's the only way to continue to use those parts themselves that have been developed. So it's quite interesting thinking about that from a policy perspective, maybe that is something that we want to have in a policy world. And, you know, when I talk to people who are involved with the UKRI policy fellowship scheme, I think that is an ambition is to have a cohort of researchers across the country who are able to talk policy. It's quite an expensive way of getting there. We'll say that. So there's a lot of hidden costs for fellowships that people don't really recognize. So the Parliamentary Office for Science and Technology, they run a fellowship scheme. And I remember having a chat with them. a few years ago about the pros and cons, very sort of, I wouldn't call it an evaluation, I would definitely call it an evaluation conversation about what I was worth it about their fellowship scheme. And they said, they reckon it takes 80 man hours just to get a fellow through the door. I think that's quite a lot. So that's recruitment, it's interviews, it's everything. When you've already got a full-time day job, ⁓ I think that's a lot. then when you become a fellow, you still need management and induction and training. you know, it's not a cost free activity for a host organization. And I think often that the work associated with that gets pushed onto people who are sort of in civil service terms, G6, G7, probably more likely G7 level. So these are people who just have to suck it up basically. And so the cost continues to be unrecognized by the host organization. But when you think about the number of fellows that we're now scattering across government, I think it's quite a big ask. If the outcome is a cohort of researchers who have got sufficiently good people skills that they can talk to people from other organizations, it seems a bit hard on the civil service that they're sort of bringing us up. Is there any evidence in those examples that you gave on like impact on the host for because you mentioned clinicians right when when you're talking about the evidence but is there evidence that there's benefit for the places they go or not really? I wouldn't want to put a figure on it but I've been spoken to a lot the secondies having been a secondie ⁓ having looked at the papers that there are my best guess would be that about a third of the time a host organization gets something useful. that's very much that. So, I mean, I was a second into government office of science. I like to think that I was helpful, useful rather. And I think in part that's because I had an outstanding line manager, an incredible induction process, a very clear plan of work that had been thought about for months, blah, blah, blah. But I think also just to not to toot my own trumpet, I'm the sort of person that will just say yes to. to request for help. So you end up being, I was like the tame social scientist in the office. And they'd come over and say things like, I'm sorry to bother you, but we're supposed to be doing this piece of work on reducing research bureaucracy. Do you know anything about universities? We're like, yeah, I anything about research bureaucracy, sure. Or like I've written a report about research integrity. Can you just have a look at it? Again, nothing about it. But if you're the sort of person who's just willing to get stuck in and think, well I've got specific set of skills, I'm not actually that fast about what I deploy them on, sure. Not all academics are like that, some are much more focused on this has got to deliver for me, I've got to get a paper out of it and there's very good reasons why people are like that but you know I'm never going to get a chair at Oxford, so I've let that one go. There is, think there's an interesting factor in that it's definitely relevant to academics and I can't in my head quite think about how the extent to which it's relevant to evaluators in different parts of the evaluation community, but that kind of in the academic community becoming very involved in trying to get your evidence used can affect how well you get on in other spaces and what that kind of trajectory looks like for you. And I'm not sure I was thinking, you know, there's evaluators in lots of different spaces, but often. I think evaluates spend a lot of time doing and delivering and then giving the reports and you know maybe making nice dissemination about the reports but we're not necessarily also encouraged to spend lots of time on the and now. work out how to get this stuff that you've learned into practice that often ends up on some other people who then maybe, you know, read it, file it, forget about it. So it's really interesting. I wondered if you want to talk a little bit about your fellowship and what you did in your fellowship, because it's quite related to this kind of topic of evidence use and how we get kind of in the UK researchers to understand what government needs. Yeah. Can we talk about this issue of evaluators first? I think there's a specific tension for evaluators in this space. So ⁓ I've experienced this as an evaluator of knowledge exchange interventions, but I'm sure it's common to many evaluation activities. And that is you're very hamstrung sometimes by the relationship you have with both the intervention deliverers and your funder that can really... affect your ability to get the evidence used. I mean, that's the great benefit of being in a university and being able to say that you're an independent researcher is, you know, you got nothing to lose really. Whereas as an evaluator, you well, I mean, you've got a job to deliver, and you and you've got to do that. But you're sometimes going to say things that are not particularly popular with, you know, anyone. And that can be tricky. And it's particularly tricky in this knowledge exchange space because people feel very passionately about it. They really believe in their project. They believe in the value of knowledge exchange in general. And in some cases, they've staked their career on it and on the career of many other people. mean, I can remember Probably about 10 years ago now, I had a conversation with Maureen Dobbins. Maureen Dobbins is a Canadian health researcher and she set up the first ever randomized control trial of knowledge brokers in the Canadian healthcare system. And she, by the time she had set up this trial, recruited, got it running, and they found a null effect of the knowledge brokers. So they had all sorts of outcomes which was to do with, I don't think it was quite citation of academic research in... policy documents, but it's things like awareness of research, ability, know, people's civil servants ability to use evidence and that kind of thing. So I think it was more capabilities, but they found this null effect. And by that point she had 40 knowledge breakers working for her. It's very difficult under those circumstances to know what the morally correct thing is to do. know, what, what, what do you do? You know, It's certainly possible that the evaluation design was not the correct one and hasn't found the appropriate outcomes. know, it's possible that things will get worse if you take these people out for sure. ⁓ but you know, there's also a human element to it. You know, these are your, these are your colleagues now and you're, are you really going to publish something saying they're pointless? That's hard. Yeah. No, you're absolutely right. And it's interesting. So I've been both a consultant evaluator and an in-house kind of commissioner of evaluations. And I think that this balance sits differently for these different people, because obviously, if you're kind of the independent consultant, you have slightly more scope to worry a bit less about some of these things, I guess, but you often don't have the knowledge that allows you to actually write things that will really be actioned. Whereas if you're the internal person, however you're doing it, may have, often have the kind of like, well, I understand this thing definitely won't happen, but if I write it in this way, we might be able to get this bit to happen. And you have that, but then you are, as you say, often much more. Yeah, there's a tension on what you're comfortable saying and how that looks when you're kind of internal and how you manage that. Yeah. And whether you need to maintain those relationships afterwards as well. I mean, and you're very dependent on people behaving well. Let's put it that way. So I evaluated a knowledge exchange initiative a few years ago. Won't say which one. But we were, as the evaluation team, we were cut out from certain, we were not allowed to observe certain meetings and money that was for the evaluation was sequestered and they commissioned a separate evaluation that was going to be more favorable than ours. know, that's not good practice. You know, I didn't, I mean, I probably didn't handle that one very well, but I do know it was a very uncomfortable experience and it really. It was challenging to know what the right thing to do was. It really made me think that evaluators need to have more of a protection in some way. know, at the very least like a code of practice that we could point to to say, you know, this is just how we've got to do it. Sorry. Not sorry, actually. You know, this is what we've agreed to. This is what we've agreed to. Otherwise you're sort of just battling it mano in mano, aren't you? Yeah, I mean, it comes back to your point at the beginning that all of this is kind of predicated on humans in all of these spaces and that there will always be good behaviour and bad behaviour in these spaces and probably all of us have had some very good experiences where we've really helped people get the evidence they need and develop and learn from their interventions and ones where the findings have not. It's a sort of practice research tension, I think, because from a practice perspective, what they want to hear is you've delivered it exactly as you said you would and it's been great. From a research perspective, to have a preconceived idea of what the outcome is, nuts, isn't it? I mean, we go into it as evaluators thinking, well, maybe we'll find out, you know, and that's very uncomfortable for people who are set on success, you know. I was wondering just because... You you start off talking about local government and then obviously you've worked in the kind of the national government space in the kind of health area and then. Obviously in health, there's also kind of a lot of practice. And do you think there are differences in those spaces as to how evidence is taken on and used? Or do you think it's quite similar? course, obviously we're mainly talking about the UK context. I suppose it's different in some professions because they have a much more tightly regulated education, sign off, CPD process. you know, medicine as a profession. it's not really about application of knowledge once you get to a certain point, it's really about pattern recognition. I can say that because my husband's a clinician and he agrees, so obviously I'm correct. It's about following the guidelines and the protocols and absolutely being able to explain why you haven't, when you haven't, but it's about making decisions within a relatively contained set of circumstances actually. And that's wildly different from policy. And the fact that we try and compare them is a bit bonkers really. But, you know, when people are trying to talk about maybe we need something like a nice, fair policy. that's how we ended up with the WorkCentres because we thought the model was completely transferable. I think that, I think we all agree now that it is obviously completely different. But I think what people don't recognise outside of government is that... Government is also pretty regulated in how it seeks evidence and how it applies it. I mean, even things like the Magenta book and the Green book that dictate impact assessments and use and evaluations and how they should be done and why and what should be done with them. People have put a lot of thought into these things. I'm sure that as a professional evaluators will have views about these things. But there are structures and systems there to support the use of evidence. It's just not. ever going to be as clear-cut as it is with why did you not use this drug to treat this specific condition where we know there's an active ingredient that treats it and this is the outcome you'd expect to see it's never going to be like that. suppose a better comparison would be teaching where there still is education and training and CBD. I feel like it's more there's a recognition that there is more about you know we have the syllabus which has to be delivered but how that's done I think is less codified. So the conversation about evidence is quite different and it's actually more about the people. So the Education Endowment Fund at the moment, running a funding scheme about evidence use, looking specifically at leadership. So what's the role of leadership in promoting evidence use in schools? And I think that's a really interesting approach. So I mean, I guess the equivalent would be, can we get research use as a KPI into hospital trusts? and not an approach anyone's taken. But thinking about those sort of system level drivers, probably actually more effective than, you know, training individuals. Shall we talk a little bit about the areas of research interests? given that that's the area that you did your fellowship in. us a bit about what they are. The ARIs, Areas of Research Interest, were actually suggested quite a long time ago, I think 2014, I can't remember if it 2014, 2015, when Sir Paul Nurse did a review of the UK research councils. In it, he said, well, I'm paraphrasing, UK researchers do lots of good stuff, but... it's not really used in government. I don't know how he came to that conclusion, but this is what he said. And it would be much more likely to be used if government said what they needed. I think it's actually quite hard to argue with. If government can articulate its problems, researchers have got more of a chance of hitting the target. So I think that's fair enough. And he called them statements of knowledge, statements of need or statements of knowledge need. And they became compulsory. So I don't know what the difference is legally between compulsory and mandatory. All government departments are required to do them. And by 2017, I think about half of them had done one version. But they're really worth going and looking at those early versions. They're probably on some gov.uk archive. My favorite ones are the Foreign Office ones. And there's just a list of bullet points on a HTML site. And one of them is the future of the relationship between the UK and the European Union. That is an interesting question that we, interesting topic that we would all like to know more about. Very prescient. So about half of them had done them. They had in general been done by relatively junior people in the department. mostly hadn't been terribly comprehensive, exhaustive exercise. They'd been done quite quickly, but nevertheless they were there. And somebody, I think it was probably Jennifer Rubin, commissioned an analysis of the existing areas of research interests. So at the time she was the executive chair of the ESRC. And she observed that 80 % of them looked really terribly social sciency. And this was of course, a great interest to ESRC. who do not receive 80 % of the UKRI budget. And so she went to Patrick Valance, who was GCSA from 2018 on medicine, said, look, I think we can do more with these. Is it worth trying to get a couple of people in to see if we can improve how they're produced and improve their use? And he agreed and said these joint fellowships were set up and started at the end of 2019. So from the beginning, the fellowships had... a few purposes behind them. One of them was to try and understand a bit more about how social science could be more useful across government, more useful and more used. Clearly there was a bit of unmet potential there. The government Office of Science, which at the time, let's not forget this was before COVID, was quite small and had a, you know, It wasn't exactly running on a shoestring, it doesn't have any, well, still doesn't have any ⁓ hard power over other departments. It relies on soft influence and networking. So that's how it operates. And it holds a policy for the areas of research interest, as in it is responsible for trying to make sure others do them. as part of the science capability work that it does. So Patrick really cared about getting more scientists into government, making sure that there was science thinking across governments and not just economists. He didn't actually say that. But trying to get people who understood critical thinking, data analysis, how to interpret different data sets, trying to improve that pull through. So it was part of that for him. And then Annette and I, Annette Baraz and I, I've been working in this area for 20 years now. Annette, a little bit longer. She's been a star in the field of knowledge mobilization and evidence use research ever since her own PhD, which was also in government, looking at evidence use and criminal justice, actually. So she and I applied together because we both felt for different, I think slightly different reasons. We'd spent a long time writing papers about how we can just get along. We know all this stuff already, Ebene was listening and complaining about things. We thought, okay, like we do actually know a bit about this. Let's just go and see if we can make it work, put our money where our mouths are, et cetera. So we started this dual role, three days a week for a year initially. And the goal was, I would think about how we can get government departments to... improve evidence, improve the quality of their ARIs and Annette would work on how we can get different stakeholders to use them and to respond to them. And we've only been there three weeks when this interesting new virus was identified. So it was quite an exciting time to be hanging around the government Office of Science and in the end we were there for four years. So it was an exciting experience and very formative in many ways. But the ARIs, I think what I came to realise was the ARIs are absolutely systems intervention. You look at them and on paper, they're just a list of topics, right? So this is sort of, and Paul Ness, I think, had this very instrumental idea. They'll say the questions, researchers will do the research. Ta-da, no more complaining. That's not really what they were at all. The reasons that some government departments had done them, you know, in that sort of first tranche, were quite varied. So some of them had done because they were very sciencey departments and they just needed to get this done. They already employ 80,000 scientists and this is just a description of what they already do. You know, people are welcome to... It was sort of a transparency exercise really. Sort of, here's what we work on, here's the main themes, here's the main questions, get in touch if you think you can be helpful. And I say DEFRA was very much like that. So there's look very much like a research, a description of a research program. And then there are others that, you know, knew frankly that they needed help and they didn't have any analytical capability in house or very limited amount. Traditionally, they hadn't had much input from scientists and engineers. They weren't the sorts of areas that they would think of as being worth engaging with. So, you know, think about like DWP or MHCLG, so housing's local government communities, got that in the wrong order. Education as well. Ministry of Justice, so they're you know, less sciencey departments, but really keen for some help. And so their ARIs don't really look like research questions. They look more like, here's some problems that we know we have had for many years, and if anybody can help us. Please, please, please tell us what. you know, what we can do. And actually, interestingly, those departments have, they've really, really like got on board with the ARI process. They've, they've worked really hard on them. Some, like the DWP ones, I think they did a effectively a huge Delphi exercise where they asked every team in the department, what their questions were, what their topics were. And they hadn't like over 900. And then they were whittled down by their chief scientist team. And they were, you know, sent up to the minister for sign off and all that kind of stuff. But even just the process of doing that means that someone from the CSA team has gone to talk to every policy team and they've explained what the CSA is for, they've explained what analysts do, they've explained why it's worth engaging with research. That does a lot to change the internal culture in a department. So that was very interesting to observe that. And then there were some departments who just didn't do it. They were like, no. Like, they don't need to do that. They've got their own... DSTL or whatever it is, you know, they've got their own laboratory. They, they don't really want to say publicly. ⁓ thank you very much. What their questions are that they're tackling. Home office ditto. So one of my favorite ARIs of all time is one of the home office ones from 2018. And it is bullet point drugs. That's the whole thing. But you know, it makes sense. They, they, they do have a drugs policy team, several, ⁓ and they definitely do need some help in figuring out. what to do, but there's absolutely no way that they're going to say publicly, what would happen if we legalize? They're not going to, they're not going to say that. So what I came to realize about the ARIs is they're a bit like a Swiss army knife, you know, for, so for some departments, they're a sort of invitation to a conversation, you know, they're sort of very coquettishly drugs, you know, could you, if you're the sort of person who's happy to talk to the home office about drugs, could you come and get in touch? And they did. And, but it's under, you know, it's not in public. It's not happening in public that conversation. It's happening inside the department. For others, it was more of a management technique. So, ⁓ one department, I won't say which one. They articulated to me that, ⁓ they had a lot of trouble with the research tenders that they were managing. They kept putting research tenders out. Three years later, some research would come in and nobody had any idea what it was for, why they'd commission this research. No idea what was going on. So they said, well, we need these ARIs so that there is some kind of like management operation system so we can keep track of who's doing what and why, because by the time the research is coming, everyone's left. And then another, you know, arms length body, they did some to force their department to do them. Because, you know, there's an interesting relationship between a department and it's the arms length bodies it sponsors. And actually it's very helpful to have some. mechanism to have a conversation between the two of them about what research they're to engage in. I think what I hadn't, so I mean you probably could have worked that out in advance that different departments are going to be using them for different purposes. Actually another example that I think is quite illustrative, this department had never traditionally had a government chief scientist. This was MHCLG, think, so housing. And of course, in the wake of Grenfell, decided they really did need some scientific input. And so they decided to get a CSA. And this was therefore a department that had almost no networks into academic spaces at all. When suddenly people are asking questions like, well, what about the flammability of this kind of building material? You need to, you need to know, and you also need to talk to social scientists and people who know about, you know, social divisions and health inequalities and all that sort of stuff. So they use their ARIs, A, to build the function of the CSA within the department and make it clear to the department this was here and this was why, but also then to provide a mechanism to go out and build those networks. So lots of different purposes for lots of different reasons. But they also, and this depressed me slightly more than it should have, they also really provided a mechanism for departments to talk to each other because there just aren't channels in government for everyone to say, Okay, let's what we're to do about net zero. You know, if you're lucky, a minister will provide the spare for that. But ministers come and go. And actually, there's that's quite a high level. It's not really an operational level. So suddenly, if you've got the ARIs all together, they can see for the first time for each other, what they're working on and where they might have stuff in common. And that happens at a G7, G6 level rather than it being a perm sec level. So actually the quality of the conversation can become operationally useful. And a really good example of that was Department for Education had a question in their ARIs a few years ago about... technology enabled learning and whether you can embed health and wellbeing in it. Now I come from a sort of applied social science base, so that language is very familiar to me. MOD were listening to that conversation as they were presenting their ARIs and they managed to translate that in their heads to what MOD cares about, which is making sure that soldiers are physically and mentally healthy and all of their learning is technology enabled because they've got a great kit. they just went away and had a chat about... what they know about it. I mean, isn't that great? It's really, I think listening to that kind of variety is interesting. It shows that variety of what it looks like for things to lead to change or not, or like coming back to that kind of the action to the evidence to action piece that. that there are actually these conversations may look totally different in different spaces and what people what people are willing to share where they're willing to go to or not go to so even amongst government enormously varied and that there's some yeah that kind of what you were talking earlier on about how you know, what does use mean and how do you measure it and stuff. And actually there's a piece of that, which is like, it probably also looks different in some of these different spaces. But I hadn't, I don't think I quite realized that, you know, that idea that people were really, they, the breadth of ways of using them that weren't just like, we want you to do research on this, but actually like we need to build a network in this space. And actually all of those uses that I've described, those are all still internal. Really, they're all about making the case internally or making better use of R &D budget or making better use of their engagement resources. When you think about the external uses, it becomes even clearer why these systems interventions are not just, you know, lists of topics. So, I mean, one of the things that I'm most proud of, that I'm sure I will remain most proud of in my career, is commissioning the database, the ARI database, because up until we had that, although the ARIs were from time to time being refreshed, they were still mostly in PDFs or on HTMLs or on gov.uk. Quite literally, Annette and I were printing them out, cutting them up and saying, ⁓ look, these look similar. So we commissioned this database, which Overton designed for us, well, with us. It's a very collaborative exercise. suddenly we could have all of the live ARIs in one place, searchable by topic, searchable by method, searchable by population group, and it was really much easier to do that analysis. But I think what we realized as we started to talk to funders and researchers and intermediaries, like the National Academies about the ARIs, was how much analytical work there is to do to get them into a Well, not to change the airways, to elicit useful responses from the wider research community. And it's definitely not as simple as saying go away and do research on this. Which is why I think the invitation to a conversation description is quite a helpful one. One of the most common responses in the early days, say back in 2019, 2020, that we would hear from researchers was, well, it's not a very well-written research question. I wouldn't put it like that. Or, yeah, but they haven't asked about X and I think they should, I think government should be focusing on X. Okay, but this is what they're actually asking about. And so a lot of the process work that I did around the ARIs was sort of explaining to people, you know, entry level knowledge mobilization stuff, which is answer the question that's there when you've got your foot in the door, then tell them that they're wrong, but do it nicely. Or, you know. say yes and not yes but ⁓ and try and come up with a 500 word summary of what we know and say and if there's any further conversation I'm happy to help rather than sending the three papers that you've written on it you know stuff like that and I think some people have have started to do that I guess perhaps we didn't appreciate we don't really know what was helpful for ⁓ know, different government departments are in a really different space. So some of them want to have lots of, you know, 150 people email going, I'd like to help. And, and here's my paper. And others really don't because they've got this very formed science system that, they've got specific points of entry and specific gaps. You know, they're actually pretty well served most of the time and they want, they've got specific things they need doing. And they want an academic who knows specifically about this or has got this skill. The foreign office ones, I like to laugh about those, but the truth is that the foreign office, the FCA, or no, it's FCDO, no, but when it was FCO, had very different evidence needs from other departments. So they weren't commissioning multimillion pound programs and evaluations like DFID was. They tended to need people who knew a lot about a small topic, but were able to reach them very quickly. So actually their evidence use... mechanism was the AHRC fellowship scheme. So they had a sort of network of academics who went in, were seconded, and then of course they had PhD students and there was multiple cohorts. And it looked a bit sort of flimsy from the outside, but if you're the FCO and what you need is to get hold of someone who knows about piracy off the coast of Mauritius, that's perfect. And that is typically the kind of thing that would come up in my experience. One of the things that's making me think a little bit is that it's also worth saying like these are things we are interested in and may be willing to do something about. Which I think is a, possibly, yeah. But I think that's an interesting thing in the evaluation space because often when you evaluate things, I think there are things that people might be willing to change and things that people are very much not willing to change or not able, yeah. And, and ARIES in a way is a way of saying here are some of our slightly open windows. are an interesting beast and I think they do take a lot of work, typically a year's work for someone in the department here. run the process of consultation and shortlisting and all the rest of it because they do have to get ministerial sign-off by and large which is quite a heavy lift and it is really important therefore for departments that they see the value of it. One of the other things, one of the functions that the ARI database enabled was we could use a large language model to match ARIs semantically with grants so basically you could see are they grants that look similar in topic and question and so on to the questions that government's asking. That's obviously not saying is there relevant evidence out there, but it's sort of, I thought it was quite a neat way of saying potentially, potentially is there, is there relevant evidence out there that we could connect? And I can't remember the exact thing. think it was 87 % of ARIs could be linked to grants in the UCLRI database. So for me, that was... quite exciting to see the untapped potential there, that even if the research isn't exactly matching the questions that being asked by government, there are certainly people who know a lot about very closely related problems. But, you know, surely there's more that we could be doing to make that happen. The other thing that we can see now, which is really exciting from an evaluation perspective, is you can see causality. So you can see areas of research interest, because they've been around for a decade now. You can see AI documents that are being cited in grant applications that have led to publications that have been cited in policy documents from the same department. So you can see it's actually generated. sort of relevant research activity, relevant on purpose, not just by coincidence, which is what we had before. So we've just come up with this data in the last week, actually. So I'm really looking forward to sharing that back to the departments and understanding the degree to which that helps them continue to make the case for investing in this systems intervention. On that topic, are you able to see that some of the, you know, how much are ARIs changing over time? I guess is one of my questions. Obviously there's political motivation to that as well. So it's not, and are you able to see that some are like, I'm going to say done. I know that's not how research works, but you know what I mean? That they, we've sort of, some areas have become more covered and therefore are no longer ARIs. Do you see what I mean? Well, we have an archive, so you can, you know, empirically investigate that. I probably don't have the quantitative skills to do that. to a degree that I would like, having spoken to the team who do the analysis with us, don't change that. mean, departmental portfolios don't change that much. You do get influx of buzzwords and key interests. obviously AI is very heavily represented in the current crop of ARIs, which I don't think will surprise anybody. But know, things like workforce retention, that's not going away. That's been an issue since the start of the welfare state, you know, and, ditto things like employment issues, know, keeping unemployment high, productivity, economic growth and resilience, particularly in the regions, you know, this more climate resilience and adaptation. These things are not going anywhere. Is it, guess in my head, there's an interesting thing about being able to, agree that not going anywhere, but being able to show that that evidence base has built over time, know, like link kind of linked to your causality thing. We are starting to cover off some of those. Yeah. So I think this comes back to my, the, what I was saying about the degree of analysis this tech, takes. So yeah, we have the ARIs now we have the database, we can do these like fancy statsy things that some of these trends and patterns, but to really show where there's a body of evidence to really synthesize it and do the work that is going to make it policy accessible, to put it in front of the right policymakers and have those long-term conversations, that is, know, no one's doing that job at the moment. And I think various small groups are doing small bits of it, but it's certainly not coordinated. And the knowledge exchange that we have in the UK ⁓ is still very research driven rather than being demand or need led. We talk about being need led, but really it's still very researchy. And certainly we don't have the synthesis capability that is oriented towards addressing bodies of policy knowledge. Again, we have a couple of small groups that do it. But I think we need a big ⁓ piece of infrastructure that all this evidence together, which feels like such a huge win. No hanging fruit, frankly. This has been a super interesting, quite wide ranging conversation, I guess. And we've talked quite a lot around both kind of evaluators getting evidence into action, but also quite a lot in that kind of the academic-y space. Obviously there are lots of evaluators who also sit in the academic-y space. I just wondered if we kind of finish off by thinking about kind of reflecting on all of this. Are there key things evaluators might want to think about in getting their evidence into? whatever we've now decided use means and what their role is kind of in achieving that. I mean we've covered some of these things by just thinking specifically in that kind of evaluation space if you have any thoughts. I think there's huge potential for evaluators as a profession to have more traction in government so back in probably 2021 I can't remember. we did a, and I ran with Go Science an exercise called Rebuilding Resilient Britain, which I now regret that name. But this is what with the name we gave out our exercise and we basically pulled out themes from the ARIs and we established task and finish groups to do some of this work that I was just describing. So to synthesize evidence around some key messages, identify gaps that funders can pick up and then deliver that within a sort of six week period. was really good fun. One of the things that allowed us to do was look across the ARIs. A significantly large minority of questions were about evaluation, not just, you know, can we evaluate this or what's the impact of that? It was more, how do we embed evaluation capability in government? How do we ensure that the next range of applicants to the Civil Service have evaluation expertise? And I've always thought about the ARIs. They're a really untapped resource for university education planners. You can look at them and you can see very clearly what government wants is for the research community. to have a certain set of skills and they want those skills to be more readily available to government and basically at the moment their skills in data science and evaluation and I think there's such a big offer to be made around are there university courses that you can take to understand what evaluation is, to know how to apply it, to help you navigate some of these tensions that we've been talking about, I doubt it but so for me that would be an incredibly helpful thing for the evaluation community to think about. I think there's a nice link to the UK Evaluation Society in some ways, which obviously does provide. offers in terms of how to develop and I think not just kind of technical skills but that also like what does evaluative thinking look like which isn't just a skill that's evaluated I think that links to your skills point right it's not that you need I mean you do need expert evaluators but you also need a lot of people in general to understand what evaluation can and can't do what you can and can't expect from evaluators and those those kinds of things. And also what's proportionate in terms of evaluation. So conversation I have a lot with policy colleagues is, you know, what's good enough for evaluation. And you might think it would be the other way around, but often they want bells and whistles evaluation. You're like, well, maybe, but this is like a syllabus change for one, you know, profession, very small profession. Do we really need to spend a million and a half pounds on an evaluation for that? What's going to be enough? What's the critical piece of knowledge that you need that's going to allow you to continue to invest in it or to help you move to the next stage? And that's a really difficult question for people and helping people develop those skills, I think would be great. If you have questions or feedback following this episode, we'd love to hear from you. Contact details for the UK Evaluation Society can be found in the show notes for each episode. Our thanks to Professor Oliver for taking the time to create this for us and we hope you'll join us again for the next episode of Evaluate This. Evaluate This is brought to you by the UK Evaluation Society, sponsored by Digital Science and co-produced with Swanky Creative. Please visit evaluation.org.uk for more information.