When does “inclusive data” start to do harm? Exploring the intersection of data and identity.

UK Evaluation Society: You're listening to Evaluate This, a limited series podcast from the UK Evaluation Society and sponsored by Digital Science. Hello and welcome everyone to Evaluate This, the UK Evaluation Society podcast. I am Dr Fiona Lana, Vice Chair of the UK Evaluation Society and your host today. I am joined by Dr Kevin Guyon, writer and researcher who explores the intersection of data and identity. Kevin is a Chancellor's Fellow at the University of Edinburgh, Director of the Gender and Sexuality Data Lab and author of Queer Data Using Gender, Sex and Sexuality Data for Action. and rainbow trap, queer lives, classifications and the dangers of inclusion. The UK Evaluation Society 2025 conference in Glasgow focused on the importance of data for vector for evaluation. And Sheffield 2026 is looking at the next steps in the process, the importance of getting evaluation findings into decision-making. Kevin, so your work spans all three parts of the process. How does it do this? Yeah, I think for me data, a key role across many parts of the evaluation process and I hope that my work and my research is able to speak to how data can be used and sometimes misused in a variety of ways. So across a range of different evaluation approaches, ⁓ capturing data that kind of collects a range of views, experiences from those who may be the loudest in the room but also those who aren't is going to be very key. But what I'm particularly interested in also in my work is how after we have that data set. after we've gone out and collected information about a group or a community, how do we then use that data for action? So my research particularly looks at the lives and experiences of lesbian, gay, bisexual, trans and queer communities in the UK. And over the past few years, I've kind of been collecting information and data around lives and experiences, particularly in areas such as creative industries and in data collection exercises like the census. And I think for me, it's been interesting to see how the collection of data alone is not enough. So actually going out and collecting information about people's opinions, views, feedback, can be a stepping stone to taking action, can be a stepping stone to actually bringing about change in policy, in legislation, in decision making, but on its own, just collecting the data is only one part of the process. So right now I'm really interested in that kind of longer kind of afterlives of data, where it goes, how it's used, how it's misused, that initial kind of data gathering exercise. So the data is collected and then it doesn't just sit there, it is used or it's misused. How you collect that data, does that impact how the next steps in the process work? Yeah, certainly. So I think for me, we don't spend enough time thinking about the spaces and places where we capture information about people's lives and experiences. I'm really interested in the kind of environmental effects of different data cultures, different organizational cultures, and how they might excite, enthuse people to engage in the process, or quite the opposite, turn people off from engaging in sharing data about themselves. So I think we should be spending more kind of time thinking about the cultures in which we ask people to share information about who they are and what they think, particularly on topics like my research, so on and equality, diversity, inclusion topics, we can't expect a neutral kind of playing field where across all spaces and places, all communities will feel equally willing and able to share data for an evaluation process. So I think it is important for us to reflect on the communities, the cultures we build within our organizations and whether or not they facilitate or create barriers to effective evaluation. But more broadly, also, I'm interested in the kind of back and forth between data people share about themselves and the impact that has on their environments. And what I mean by that is the sense of, I'm trying to give one clear example, a clear example is around ⁓ communities' relationships to the police. So my work looks at LGBTQ communities and historically there are kind of contested relationships between this community and police forces for a variety of reasons. So in those instances, we need to think actually these communities might not be willing to share information about their life, their experience with an organization like the police. And rather than just trying to devise better methods or more effective ways to kind of reach these hard to reach communities, I think we instead have to take a few steps back and think how actually the historical legacies, the kind of power differentials, the questions of historical harms all play into our contemporary. of present-day approaches to capturing information. So in a previous life, ⁓ my kind of background and restructures history and I think there's a lesson here in thinking about the history, the historical relationships between people collecting data and the communities are trying to collect data about and not treating this as a neutral apolitical space. Actually it's all packed with politics, power and history which I think is worth us considering in more depth when engaging any type of evaluation exercise. So leaving from that, one thing which we have in the evaluation process is the way that we collect that data. So we predetermine what we're looking for. So we have categorizations, have tick boxes, we love a good quantitative ⁓ way of, you know, categorizing our data, classifying it. How do you approach that when that kind of categorization isn't always compatible with personal identity? you know, asking somebody to be in a box as such. Yeah, I think there's probably two strands to your question there. I think it's a fantastic question in this sense of, as evaluators, are we always going out looking for problems that we think exist already? Are we always going out with preconceived notions about what's a problem, what needs to be counted, what needs to be categorized, what needs to be evaluated? Or is it possible to do this work in a genuinely open-ended way, where we're genuinely open to what we might discover in this process? I think that's probably quite unlikely. I think in most cases, people conduct a type of evaluation, a type of research more broadly to go out and look for something that based on their own biases and assumptions, they think exists. So both in terms of kind of problems, like I'm interested in my work, how we capture evidence or information about things understood as problems or problematic and how problems or things that are problematic are often only problems for some people. actually what counts as a problem, what doesn't count as a problem is quite subjective. But in the second strand your question was also interesting in terms of the different demographics of communities we try to engage in our work. So my work particularly looks at gender, sex and sexuality, particularly the lives and experiences of LGBTQ communities. And again, as evaluators, as researchers, if we go out thinking we have a preconceived idea, about who is an LGBTQ person, that's going to inform and shape the type of research we do. It's probably going to limit the potential findings, the potential discoveries we might do with our research, because it's always going to incorporate our feedback into our existing biases and assumptions about what sort of life and experience this person might have. So think there is a need for us to be reflexive in the work we do. actually not just thinking about how do we design more accurate or more detailed evaluation tools, but spending as much time thinking about ourselves. Actually, what do we bring to process? What sort of biases and baggage do we bring to the research we do? And how ultimately will that have a shaping effect on the type of findings, outcomes we lead to in our work? When it comes to the reporting mechanisms of evaluation data, we've set it up correctly. We've got an unbiased way of approaching it. Are there any limitations or implications we have for evaluation? So we have, know, particular way of collecting data. Are there any implications that we have ⁓ in different sectors, for example? Yes, a great question. And I, my work looks at kind of queer approaches to data. how might we take ideas and concepts from queer studies and queer theory and apply to data, particularly quantitative data, so kind of numerical data and It's often ⁓ people feel excited by this idea of actually kind of rethinking how we count, how we measure, how we calculate things, how can we kind of try and break or subvert some of those rules. However, people then also hit a kind of brick wall where at some point they're expected to then report or share information with funders, with regulators, with kind of accreditation bodies and present their data in particular ways. And often these ways are less ⁓ bold or radical. than I might call for in my research and work. So I think there's a gap between what people would like to try and do, what they're excited about, what they think actually might better reflect and represent the views and experiences of the communities they're evaluating, while at the same time having to kind of change that data in ways that fits the boxes of the people that they have to report to. And there's no easy answer to this. I would love. us all to be kind of going out there and bending the rules, subverting the rules as much as possible. But when it comes to applying for funding and competitive processes, when it comes to reporting to regulators, often there's limits in how much people can bend or subvert the rules when it comes to reporting data. So I always try and think of a kind of middle way, a kind of middle path forward. And I often like to use analogy of kind of floors and ceilings. And what I mean by that are If you have to report data on identity characteristics, particularly gender, sex, sexuality in a particular way for a funder, a regulator, that's kind of your floor. That's a bare minimum. The basics you have to do, your male, female, sexual orientation, often things aligning with legislation, those types of things. But that doesn't need to be an endpoint. So actually I encourage evaluators and researchers to kind of go beyond that bare minimum and think of that kind of ceiling, kind of aim for more. So on top of your bare minimum, actually have some questions that maybe ask a more interesting, a more innovative question on identity demographics that actually maybe has a write-in box or a free text box and allows people to share their experiences in a kind of qualitative sense. So never just stopping for what you have to do, but trying to kind of transcend that bare minimum in ways that maybe actually offers better ways of representing and reflecting people's experiences. Yeah, that's really interesting because within evaluation, we're looking at how we can potentially use AI. in that approach in terms of, know, quantitative is really easy to report because you can, there are numbers and it's easy to fill out a spreadsheet, but qualitative is like, you say, it's where the richness is and the nuance. Um, but we're finding that, um, by having AI and large language models that can go through large text bodies, we could potentially open up in all different areas of, you know, getting more narrative work and then using the large about language models to help us filter through that. Um, Can you think of, know, how does that sound for your sector where the AI has some barriers to, know, what kind of data is in there that informs it is going to be what it's viewed back out at you. there's already a missing data kind of situation. Yeah. So the topic of AI and, ⁓ kind of identity data think is so, ⁓ so topical right now. It's a really hot topic. ⁓ I think there's a few elements to this. I'm not somebody who's all doom and gloom. I don't think it's kind of all negative. While at the same time engaging in any kind of AI technology, particularly generative AI is going to come with some risks and some dangers that I think we need to all be aware of. I'm first going go back to your kind of earlier point about numbers and how the kind of the combination of qualitative and quantitative data is key to really changing people's views and opinions on problems, on things understood as problematic. In my own work, I'm skeptical about the impact of just presenting numerical data. decision makers actually presenting them a statistic percentage a data point on its own is not necessarily going to change people's views on a matter. think where it's going to be most impactful is packaging that number with real world testimonies, those lived experiences, those stories, those kind of first hand accounts of where initiatives and activities have impacted people in a variety of ways. Now there is potential here to also use AI with the analysis stage. of qualitative data. However, I am concerned about how some of the kind of histories that are encoded in these large language models might create testimonies of kind of adopt approaches to analysis that present a very monolithic account of people's lives and experiences, particularly those in marginalized and minoritized communities. So with large language models, I like to think that what they bring into being, what they generate or create is not out of thin air. It's based on historical data about people's lives and experiences. It's based on data that exists about people from existing data sets. And often those data sets are very one-sided. So my work particularly looks at LGBTQ communities in the UK. And as you can imagine, a lot of the data out there ⁓ is not painting the full picture of these lives and experiences. There's not that there's not as much data about queer joy, happiness, excitement, love, where there's a lot of data around harassment, hate speech, ⁓ kind of violence and those types of topics. So I'd be concerned that using something like a large language model or generative AI in the qualitative analysis stage might bring some of those stereotypes and tropes into full view. And if not kind of painting a necessarily one-sided approach of these communities. I'd also worry that it might just air into stereotypes. So if we think about kind of testimonies of LGBTQ communities, it might all be pride parades and rainbow flags. And while that paints a picture of some parts of the community, it's only one part of the community. And I think that richness, that depth, that kind of nuance and insights is not there yet with these technologies. So think there does have to be a human in the loop to kind of bring out the kind of the full color. of the richness of qualitative data and not just rely on generative AI. Thank you. really, really insightful. I think you're mentioning how, you know, there's an absence of data. So not just when it comes to AI, but is missing data for queer queer groups. So, you know, we really want to evaluation to be informing decision-making and our conference is coming up as kind of looking at that. But If we're missing data, are we going to be informing decision-making in the right way? If the is not there, we can't evaluate it. What we're evaluating and what we're bringing out there is not going to be reflected. Missing data as a concept, know, what does that really look like for your area of practice and what should we do if the data that we need isn't there? Yeah. I love topic of missing data. I could speak for the next three hours, all about missing data and how I think there are some... kind of knee jerk reactions people have when they hear the term missing data. And the knee jerk reaction is we have to go out and plug those data gaps. The kind of the common belief is if there is missing data, then the right thing to do is to kind of find that data and pop it into this spreadsheet or the data set. And I think in some spaces and places there are benefits to that. So in some parts of the world and some sectors and some geographies, there is no data collected about queer lives, queer experiences. And I think in those spaces and places, there's a really clear justification for plugging that missing data to show that we exist, to demonstrate proof of our existence and to kind of fill in kind of data gaps that have probably explicitly and intentionally not been filled. However, the mission of kind of plugging data gaps or finding missing data, that mission can become all consuming and it can become all consuming in a way. where we actually lose sight of what data is intended to do. So collecting data about a problem on its own is not going to solve that problem. So simply having a fully populated data sets about LGBTQ experiences in the UK on its own is not going to do anything to necessarily improve people's experiences of being LGBTQ in the UK. It's a stepping stone for action. It's a means to kind of justify an evidence base to go on and do something. So I have seen in my work, ⁓ in the UK, particularly sectors like higher education, where we've done quite a lot of research in recent years. Actually, there's quite a lot of data on most marginalized communities. There's quite a lot of detailed disaggregated data. There's a lot of data that intersects a variety of characteristics. And in many situations where we see maybe a small data gap or missing data, actually, we maybe don't need to be spending all of our time, money and energy trying to plug that gap. actually our time and energy might be better spent using our existing data, using what we know about the problem to actually go out there and do something about it. So sometimes in meetings, equality and diversity spaces, I sometimes ⁓ squirm when somebody kind of becomes obsessed with plugging data gaps or going out and finding missing data. And while that can have benefits in some spaces and places, I think we also have to think critically about what is the purpose of data. And the purpose of data is not just to have neat and tidy, comprehensive, complete data sets. It's to use that data to do something. And sometimes we don't always need to have fully complete data sets in order to go out and do something with them. when it comes to, you know, why we might not be counting that data, you said that there might be really good legitimate reasons as to why not. I can think of an example that in some countries it's illegal to be gay. So therefore by going to collect that data, we could be putting people at risk. know, thinking in maybe a UK centric situation, you know, what are your experiences of why people don't want to share or, or what's going on? It means that they're not comfortable to feel that they can share that even where it is legal to be gay, queer in this country. Yeah. It's a really interesting topic and something which my kind of current research and writing is exploring this idea of disclosing or sharing information about who you are and that might relate to things like gender, sexuality, it can as much relate to disability, religion, belief, race, ethnicity, the kind of spaces and places that make it possible or impossible to share information about who you are. And I think in the UK, there has been a bit of a mood in the past 10, 20 years where we've maybe lost sight of the risks and dangers of a company disclosing or sharing information about who you are. This might be in workplaces, it might be in kind of organizations with the government or the state. There are risks and dangers that always accompany sharing information about your identity, your life, your experience. And I think we need to kind of pay maybe more attention to the reasons why people might not wish to share information about who they are. As you highlighted with your question, there might be legal reasons in some contexts. There may be risks around kind of family, culture, society, the people live in or community. But also, I'm also keen to give space to those who don't want to share information about their gender, sexuality, race or disability with an evaluator or the state or the government or a census or an employer. I think people should have agency to choose to refuse to share data if they wish to do so. It's not a moral imperative. It's not a sign that you're being secretive or elusive. Actually, the decision to share or not to share information about your identity characteristics I think we should give agency to that position. I think there has been a tendency in some spaces and places to see those people who kind of choose to refuse as being difficult or awkward or kind of people who are just a stick in the mud. And actually that framing of these communities is quite harmful and quite problematic. Cause I think we lose sight of some of the kind of ethical dilemmas around disclosing or sharing information. And also the sense of where that data might go. So you might think actually, I trust the evaluation team or the research organization that's currently collecting this information about who I am. But I have no idea where that data set, where that row in that Excel file will move to, will transport to in the future. And the risks that might accompany that, the political winds are changing, might change in an even more kind of regressive direction. And we can see how quickly a data set that you shared in a space and place that felt safe. can easily become weaponized or a kind of data set that's dangerous when the political moods change. So yeah, one of my kind of cold arms in my books and my writing, my research, is to kind of acknowledge the existence of those who choose to refuse and not see them as a hard to reach group, but as people who are acting out form of agency. They're choosing to say thanks, but no thanks. I don't want to disclose information about who I am and nor should they have to. You mentioned politics there. You've also mentioned how if data is missing, you know, people can be obsessed about plugging these gaps and they're saying, well, maybe you don't need to plug those gaps, deal with what you've got. know, politically, how do you, you see anything playing out? We have the data, we have evidence, we have stuff, we have enough. You know, do we see action being made or do we really see this as a stumbling block for action still currently? Yeah, I think this kind of dimension of our work definitely sits in kind of question of politics. It's around not just about how we design more effective methods, how we design more savvy kind of analytical tools, or how we kind of make amazing visuals or graphs with our data. Actually, what I'm interested is how do we take that data and use it for action? And that takes us into the world of policy, of politics, of activism. ⁓ of communications, I think there's a lot more space for thinking about how do we communicate data with different audiences. And I think that political dimension to work is also really key, particularly, I mean, my research is on gender, sex and sexuality, which is a very kind of political topic, often for the wrong reasons at this moment in time. So again, thinking through how do we pose these more kind of contrarian positions, like we don't need more data, we have enough data in a climate that is very much around data maximalization, very extractivist approach to data, a belief that data will somehow resolve this myriad of social problems that we're encountering. And I think it's kind of, we need to re-examine and reimagine what data is, particularly quantum data. Actually, what can data achieve? And actually, when do we need to see the limits of data and actually place more emphasis maybe on kind of human connection, community, solidarity with other kind of communities who are maybe different from us? and not just seeing this kind of counting categorization metrics approach to the world as being a solution. I think it can help to a degree, but it's not going to be our savior. And I think that maybe goes against some of the kind of political messaging right now, particularly in this kind of moment of hype around AI, around large language models. All of this is premised on more and more and more data. And actually, I don't think that is going to solve these kind of deeply entrenched systemic issues. In some situations, it might make it worse. So I think to kind of try and steer back to your question in a way, I think maybe part of the way forward is a, an education around data, a re-education, kind of learning about what data can do, the limits of data, actually how something like numerical data might seem to us as being apolitical, historical objective. But actually it's nothing but actually numbers, how we count, how we categorize, how we measure. are equally constructed. They're also representing one view of the world at the detriment of others. And think having that kind of critical eye, that critical lens, when we think about numerical data, when we think about quantitative data, and not just seeing this as kind of an apolitical science, I think that will hopefully create new ways of thinking and using data that embed the politics that I think should be there throughout. I'm going to play devil's advocate a bit, certainly from an evaluator point of view. You mentioned ethics and ethics of whether you collect the data. But we have data limitations and we shouldn't use those as an excuse not to act. We should use what we've got, understand it, understand its context and people have reasons to not share data. We want to give people agency. When evaluators don't have data and even if they take those things into consideration, which we want to inform decisions. But these people are not going to be represented in their data set potentially, or if we don't have their voice or don't know what they think, we're going to fail to some point of representing them, which means decisions are going to be made without them ⁓ included. And, you know, is that, is that ethical? Yeah, that's a fantastic question. And something people actually ask a lot, and I kind of get on my soapbox about needing more data. There's always somebody in the audience who asks, okay, so are you kind of arguing that we need no data? Can we devise? structures and systems that don't have an evidence base that aren't evidence led or data driven. I often like to kind of imagine this as a kind of pendulum and this kind of swing of the pendulum and it can swing one way and that's we have more than than enough data. have enough data to get on with the problem, but the data can the kind of pendulum can also cause harm when it swings too far in the opposite direction where we have no data. And why it can cause harm is we don't have data about the difference in society about different communities, about different lives and different experiences. Then unfortunately, the society that we live in often tends default to a white, male, cisgender, heterosexual, non-disabled norm. The kind of normative human is somebody who represents this kind of masculine kind of version of life and experience. And we don't want that kind of default either. And I think there is a risk or a concern that if we didn't collect any data, if we didn't undertake any kind of demographic breakdown when doing evaluation work, if we didn't try and reach a diversity of communities, then you're right, we would be in embedding and enshrining this kind of default human, which only represents a small minority of the kind of richness of people's lives and experiences. And that's how it was in the past. So that's how before maybe kind of 40, 50 years ago, when people actually made kind of moves around embedding kind feminism, anti-racist work, kind of engaging a variety of different communities in research, in evaluation. We didn't acknowledge difference, we just saw this kind of default human. So again, I just like to kind of people to imagine this pendulum and just think we don't want to swing too far where we're being obsessed with trying to plug data gaps and spending all of our time trying to kind of find communities who maybe don't want to engage in the work that we do, while at the same time collecting no data or actually not kind of acknowledging difference or collecting data about difference can also cause a variety of harms. So it's that sweet spot we need to aim for between those two extremes. So we find the sweet spot, you know, there is still some data that's not there potentially. How that data is used is important as well. So there's a concept that we have in evaluation about misuse of data, you know, it'll be misusing data to be represented in evaluation. Are we not at risk at that as well? Isn't that an ethical issue? might, if we don't have the whole data set, might we be leveraging this data? What can we do about that? Yes. I think it's an interesting thing around, particularly in that space, we get into challenges around small numbers as well. So particularly when people are maybe trying to do the right thing, they're trying to adopt maybe an intersectional approach to their, the research that trying to engage a variety of communities and not just taking one. a gay man as representative of a broad sweep of LGBTQ experiences, for example, it's people who are trying to do the right thing. And I think you then to kind of encounter some challenges in the analysis and youth phase, where maybe your numbers are getting quite small, you may be at risk of revealing information by particular individuals, when you maybe aggregate, or you're trying to kind of present data on race and gender or race and sexuality, you're kind of cutting up your data set in ways that can maybe identify or risk identifying individuals. So there are ethical issues in that space. And then in terms of kind of misrepresenting, so this misuse or miscommunication of data, again, presenting a data set as something that is actually not really telling us. So you might have gone out and spoken to four people who identify as disabled, ⁓ presenting that then as representative of a kind of a very significant large community, again, it's kind of misrepresentation of what you have. And I think what's interesting here is lots of these decisions are not made by the communities themselves. So actually, once it reached the kind of the data analysis and data use stage of the pipeline, these are decisions that are being made by the researcher, the evaluator, the research team, they're deciding what to aggregate, what to segregate, what buckets to put people into. ⁓ If you have too small, too few people, ⁓ you might aggregate a few groups together. You might remove a few bars, charts from the graph. All of these decisions are made after the fact. So something I'm also interested in my work is how can we give more voice to kind of the communities who are being researched? Actually, how do they want their data to be analyzed and used? Are they open to being aggregated or disaggregated in certain ways? If you've written a write-in option, in the kind of other please specify box, how do you want that data to be counted and used throughout the process? And that's something we don't do very well at this moment in time. Lots of these decisions are made kind of behind the scenes. They're made with very little transparency. So something I would call for in terms of people engaged in research and evaluation is just try and think about all the power that you have at those stages of the process and just try and be again, reflexive, try and engage communities, voices who may be no more about the topic than you do. And not to see these as objective decisions, but actually it's all meshed and kind of power about who counts and who doesn't. You've kind of alluded to this a bit, but how can organizations balance inclusion and consent and safety and utility in how they are collecting this data, evaluating it and informing decisions. Yeah. So there are a few kind of spinning plates for people to think about when engaging in any data around identity. I like you mentioned the question there about safety, about utility, about the kind of the use and misuse of data. And also something which is interesting is are people confident and brave enough to say, we don't need more data. Actually, are you brave enough to justify your organization's approach by saying, we're not going to collect data about a particularly maybe marginalized or minoritized group. for X, Y, Z reasons. It might be around safety. It might be around bringing visibility to maybe a numerically small community. It might be around awareness that actually, even if you engage in a fantastic kind of research exercise and there's no way to kind of analyze or present this data without revealing information about those individuals, it might just be too numerically small to do anything meaningful with it. So I'd love to see more organizations kind of making decisions, maybe not to collect data. and justifying it for ethical reasons. In my work on LGBTQ communities, one of the reasons often is around kind of re-traumatizing or collecting data about topics that are quite traumatic. ⁓ So for example, on conversion practices, so kind of efforts to kind of change people who are LGBTQ, ⁓ it's a very traumatic topic to discuss, to share information about. So when you're asked by a researcher or evaluator to speak about your experience of this topic, ⁓ can be really emotionally kind of draining. So it makes sense not to then go and ask that person to tell that story two, three, four, five times actually make sense to use that existing data set, that existing research and repurpose it actually say we've got those testimonies. have that data. We don't need to go out and collect it again. So a big champion of making use of kind of secondary data sets, a big champion of not just feeling that you personally. have to be the primary researcher on every topic, actually leaning into the work that other fantastic researchers have done in their data collection. So I think these are all decisions that organizations should be thinking about before going out and collecting data. And I think there is often just this gut reaction that when we're aware of a problem or we think something is problematic, the knee-jerk reaction is we have to go out and collect data about that problem. So I always ask us just to pause. press the pause button and just think, actually, how will that data help us address the problem? Has anybody else already collected data about that problem? We can maybe repurpose or reuse that data to help illuminate or kind of explore the challenges we're navigating. And also ensuring that by collecting data about that problem, we're not inviting more dangers for the community under investigation. So in my work, particularly on kind of trans communities in the UK, we're not shining a spotlight. on a group that's kind of at risk of being kind of attacked in the media, attacked by politicians, actually bringing people into research, kind of collecting data about people is also an act of visibility in some regard. And visibility can be great, but it can also be a danger and a risk. So again, just thinking how all of this has a double edge to it. It's not all good, it's not all bad. So I think we need to be thinking critically about the work we do in this space. So if we... You know, we notice as missing data and making those kind of decisions and we're not going to collect them. Evaluators and taking a next step back. Well, we've got the data we've got, we've collected what we have. Do evaluators have a responsibility to question the origins of missing data and address that in an evaluation as well? Or is it, mentioned transparency before, you know, how, how should evaluators go about recognising that there is that an internal. decision or is that something that needs to be really flushed out within an evaluation should have more prominence ⁓ in write-ups and kind of methodologies? I would love that. I think the sense of having kind of evaluation forms which allow evaluators to provide narratives on the reasons they collected or didn't collect data would be really positive development. And I can imagine how that could look with the kind of the box inputs, the kind of the numerical data, then the writing box below seeing the context and narrative, the history to it. And I think it may be also just engage requires people to think outside of their kind of disciplinary area. So maybe an evaluator is very knowledgeable about designing research methods, or maybe engaging different communities, but maybe you're not an expert on the history of LGBTQ equalities in the UK, for example. So again, it might require you to kind of engage or do a bit more background reading and research to kind of explain why certain communities, groups, individuals, maybe chose or didn't choose to participate in your research exercise. That might be around thinking about the kind of political mood, the organisational culture that you're engaging, the time and place where you're asking people to share information about who they are. And I think we saw some of this starting to come into discussion, particularly around during the COVID-19 pandemic. So again, people engaging this type of evaluation work were very quickly asked to kind of use maybe new methods. So moving from in-person to online, asking people to share information about themselves during moments, which were extremely traumatic, extremely ⁓ unsure of what kind of was coming down the pipeline, job insecurity, family insecurity, health problems. I love this idea of having a writing box, just caveat and explain the context within which data was collected and why the results you're presenting tell a particular story. And I think why I like that idea is because it kind of adds a caveat to the numbers. So as I mentioned with an earlier answer, when we see numerical data, when we see a headline percentage, we often treat that as this gold standard objective representation of reality. But it's often not that it's often presenting one version of reality. It's one account of the social world that's brought into view through the tools, methods, context within which we're operating. having that kind of descriptive narrative box on the valuation form. just remind the reader that the numbers we're seeing are not presenting the full picture. They're presenting a picture, maybe a helpful picture, but there's always going to be people missing, stories missing, and nuance missing that I think that box might help explain. So it's interesting because we, when we're looking at evaluation, we're often looking to reduce burden. The data that you collect, what you ask for needs to be proportionate to use and utilization. It's also making sure I think a really important message to say, well, actually there are examples such as this, which is really important to include it, that the burden is proportionate to the value that it can bring. And yet to kind of challenge the evaluation community to kind of, you know, what can you do to really address this missing data issue in a different way other than trying to gather more data. Okay. Do you have any examples of, you know, decisions that could be misleading if we don't assess missing data critically. know it's there. We just give it, okay, we don't have it. Okay, fine. Are there any examples you have of where this could be a problem? Yeah, so I'm going to go for a big example. And the big example is the national census. So in the UK, in 2021, 2022, there were three censuses took place. One in Scotland, one in England, Wales, one in Northern Ireland. And they really exciting because for the first time there were new questions. on sexual orientation and in England, Wales and Scotland on transgender identity. And this was the first time the question had asked the census to ask these questions. It was one of first census in the world to have these questions included. And while this was a great achievement, a real milestone in a variety of ways, there are also some dangers and risks in terms of how we report and speak about the numbers that were published from the census. So following the count, we then kind of had this headline figure and I'm talking in the context of Scotland that I believe four percents of the population age 16 and plus identify as lesbian, gay, bisexual or sexual orientation other than straights. That's a real kind of headline statistic and you can see how that headline stat will trickle down in a variety of ways. So benchmarking exercises, people trying to figure out what's a representative kind of sample of representative community for their work. There's a lot of kind of tentacles that kind of percentage will have. shaping work in a variety of ways. However, we also have to take a few steps back and remember this was a voluntary question. This is a question that was not compulsory, so it was people can choose to engage with it or not engage with it. ⁓ It was a question asked in a moment in time in UK society where there was an increasing ⁓ proliferation of anti-LGBTQ voices across media, across politics, where understandably many people would not feel safe telling the states how they identify or whether they identify as LGBTQ+. So I always like just to kind of zoom out and take that percentage with a grain of salt or to think about who is missing from that percentage, who for a variety of reasons didn't want to share information about their sexual orientation or gender identity. And again, kind of trying to stop that percentage from taking on a life of its own. And unfortunately, due to the nature of a census, that type of percentage will take on a life of its own. It will inform benchmarking exercises, targets, quotas, key performance indicators. But going back to the source, need to remember this is a number that has a background, a history, and a politics to it. So I think that's a really clear example how something like census data, I've spoken particularly about sexual orientation and gender identity. But all types of census data can be kind of embedded and enshrined as this kind of natural fact, where actually the numbers are kind of constructed in particular ways that present one account of society rather than the picture. Yeah, it's fascinating because it's wonderful that we have that question in the data, but the fact that it's optional is also, I imagine, really important because if you could imagine the other direction where it's compulsory and people have to fill it out. Have you seen any examples where that has impacted data quality or had effect on the community and then not wanting to engage in things going forward? Yeah, definitely. mean, for all types of data exercises, I'm a big champion of prefer not to say another box and a writing box. think all questions should allow people ⁓ to actively voice their disengagement. So I love a prefer not to say a box. While at the same time providing opportunities for many people will just take the form and throw it in the bin, they won't even engage to that extent. And in something like the census, although those questions are voluntary, there were other questions which are compulsory. So the question on sex in the census is a compulsory question. And if you don't answer that question, you're at risk of being fined. And there were some problems around that in the UK, particularly for individuals who identify as non-binary, so they don't identify as being male or female. And they were forced to pick a box, were forced to identify as male or female in the census. So again, we can see how on one hand, this kind of inclusive census had new questions, kind of engaging a broader mix of kind of a broader diversity of society, while at the same time, the same document had compulsory questions that had a very kind of a binary view of the world. So this kind of, again, double edge to things, and it's kind of, it's not all good, it's not all bad. It just is kind of critical thinking about what types of lives are brought into the spotlight and what types of live racks may be pushed further into shadows through things which are presented as being inclusive. So if we imagine how the next census could be, so the future that we have, what does that look like? How can we address this tension of we need to have equitable data practices, we need to collect right amount, at least the right amount of data. What does the future of data look like for you and therefore how it can help evaluators make informed better decisions? Yeah, so the future of data. I'm trying to think of something exciting to kind of answer this. So we've spoken a lot about my critique of more data, so just collecting more and more data for the sake of it. I also in my research and writing, I'm quite critical of adding more and more categories. as well. And what I mean by that are particularly my work on kind of data around sexuality, sexual orientation. In the past 10 years, we've seen a kind of proliferation of different identity categories for sexuality, for gender, for relationships. And again, it kind of makes sense why that's happened, because it's a response to what happened in the past. In the past, there were too few categories. So what we've done is we've added more and more response options. However, thinking again, a kind of pendulum metaphor, there's a risk it can swing too far in the opposite direction, where we're presented by more and more kind of granular boxes to put yourself into. And there's a risk that we reach a situation where everyone has their own individual unique box, but there's nobody else in that box with us. And there's, I think there are some risks and dangers of that, because we start to lose sight of what connects us. We lose sight of the kind of solidarity across difference. Actually, you and I, are not identical, we have differences, but we also have a lot in common as well. So part of my work and kind of what I hope the future direction of data will not just kind of embrace this more and more kind of category approach where we have 50 options for sexuality, 50 different genders. Actually, I think we need to be thinking about what are the kind of connections across these points of difference. And I think particularly when it comes to evaluation exercises, again, if you have 50 categories and you have very small counts for each of those categories, it's actually difficult to use that data in ways that's meaningful as evidence-based or to bring about change. You might have a far more impactful data set with fewer categories and far kind of higher counts within each category. So yeah, this kind of turn to more and more categories, I think, is going be a risk. And I hope that the future of data will kind of acknowledge and embrace difference, but not see difference as be all in end all. Is it interesting kind of this is related to categorization, you know, essentially, isn't it's like, okay, lots of different categories, we would then probably have to aggregate later to make use of it. But then, which people who are filling in, know, they've had their say in the survey, they've represented themselves, but then they don't have any control later. So the raw data has got the absolute, their absolute truth in it, but then the researchers could aggregate it later into groups that maybe they don't agree with. Whereas alternatively, when we give fewer categories, you could just say that we are imposing our own construct for people to choose from. And yet I guess why is the second the better option? Yeah, I think I like the idea of people having agencies decide what box or bucket they want to put themselves into. I think it's more transparent if that's something they do rather than something that happens behind the scenes. during the data analysis stage. And particularly if people are kind of conducting the research evaluation, explain their logic, explain why they're only offering five options rather than 15, because they know that having five options will create larger counts and will create a more usable data set. And I found in my work, actually, when people know how data is going to be used, they're actually generally quite happy to essentialize themselves or to kind of narrow the kind of put themselves in a particular bucket or particular box, if they think that will bring about benefits for their community. So actually if they know that by identifying as a gay man, for example, and being in a larger category will create data that goes and helps those communities, actually that kind of strategic approach to the use of data, I think is a real strength and a real bonus. And it all goes back to showing, explaining your workings. So rather than doing something secretive or kind of behind the scenes, actually be upfront. in your kind of introduction to your question, explain why you're presenting these boxes, explain how you've arrived at those response options and how that data will be used and how this decision will hopefully make a data set that's kind of going to bring about more action. So how can evaluators advocate for this better data collection? You what else can they do aside from, you know, explaining themselves a bit more about their thought processes? Yes, I think a big challenge of this work is lots of stuff I've been calling for kind of almost goes against the job description of what these people are being asked to do. So I'm somebody who researches and writes about data and quite a lot of what I call for is like we need maybe less data, we don't need as much data. So actually I'm doing myself out of a job sometimes. But I think that's a really helpful thing to be thinking, to be kind of critical, to be thinking about your own positionality in this work, the own politics that you bring to it. Again, kind of I'm interested also in being able to recuse yourself from discussions that are beyond your competency. So I do a lot of work on gender and sex data and I'm always alarmed how little some people know about this topic who are in positions of power to make really kind of important decisions about how gender and sex data is counted and categorized across public institutions, governments, large organizations. I'd love to imagine a future where people are aware of how little they know about a topic and feel able to step back and kind of step out of a decision-making space and give that seat to the table to somebody who's maybe better informed either through experience or expertise about the topic. Like I always think in meetings, we are able to recuse yourself if we have a conflict of interest, we're able to say, can't be involved in discussion because there's a conflict of interest. I'd love to have a feature where we can do that with data as well. we can go, actually, I don't know enough about this topic to make a decision on it. I want to give my seat at the table to somebody else who's better placed and better informed to make those calls. So I think, yeah, it goes back to the idea of being kind of self-reflexive, knowing the limits of your own knowledge and your own kind of expertise in topic and not just treating these kind of decisions as technical or scientific when actually they're always political. I absolutely agree, you having the informed people in the room is unbelievably important. But there will be a lot of evaluators out there who also want to be able to contribute as much as they can in the role they have. So are there any skills or mindsets that evaluators should be developing in this area around data gaps or potentially in the queer data space? Yeah, so I think my big call would be engaged with there's a growing field of critical data studies, particularly emerging out of North America, it's a growing movement also in the UK, ⁓ kind of an amazing collective of scholars, activists, practitioners, artists engaged in critical thinking. about data, what data does, what data can achieve, and how it intersects a variety of characteristics, whether it's my work on queer data, work on data feminism, work on black data. There's a lot of critical discussions about data, particularly quantitative data. So I'd encourage evaluators to tap into those conversations, those discussions, read outputs, blog posts, podcasts with folk writing, researching, and working in that space. And hopefully, it'll just kind of create those light bulb moments where you start to think differently about numerical data. You start to see politics everywhere. And I think it's a really healthy kind of approach to any type of data work. And I think we'll just make better researchers and evaluators. Fantastic. Thank you. Perhaps we can have some links to some of those areas to put place in the show notes for people. A huge thank you to Dr. Kevin Guyane for that absolutely fascinating conversation and introduction to your work. I think there's a lot for our listeners to take forward from our conversation. The key messages I'm taking away are that missing data is meaningful. It isn't just absent. It has a story behind it. All data has a backstory and we really need to be thinking about that when we decide what to collect, what our research questions are and what data we're going to be using. Data practices really shape an evaluation and there's a lot more that evaluators can know and can use to inform decision making going forward. I think we really need to look at what's not in the spreadsheet and why it's not there. And also how can we complement that with qualitative data to bring much more nuance? Thank you again to Dr. Guyon. Information on Kevin's books and the relevant links will be in this episode's show notes. For more exploration of equity, diversity and inclusion in evaluation, please do join us at this year's conference panel session, Those Three Words. On day one of the conference, I will be talking to Reema Vadolia, Sophie Wall and Celia Shatra-Arris about the impact on decision-making when evaluations are not equitable. Evaluate This is brought to you by the UK Evaluation Society, sponsored by Digital Science and co-produced with Swanky Creative. Please visit evaluation.org.uk for more information.