התמלול המלא של Ethics & Innovation: Serving, not Harming and Connection, not Convenience

of rabbit holes, but I find care as a concept something really fascinating because it just is something that means so much in English. But if you go into German, for example, there are lots of different words for the various concepts that get packed into care as the English word. Susannah de Jager: Thank you. We will go into lots of those rabbit holes, but I want to start with the broader lens initially. Why do you think that care in that broadest way that you just articulated, zooming out has become such an important way to consider some of the impacts on our society of AI? Caroline Green: Yes. So interesting that you're saying that because we've seen an adoption of the language of care by some of the big tech companies. So what I mean by that is, OpenAI also Anthropic or so, they're actually saying, "We want our AI systems to be caring. We want them to ensure that the users have agency and are autonomous and we want to support them with their well-being." Those are words of caregiving, and that caregiving communities use a lot. And OpenAI very recently also published a policy paper in which they say that people who've been displaced, workers who've been displaced by AI should be going into care. So that adoption of language of care is there. And why is that? Care is a concept, what it invokes in us, is a feeling of something positive. That there's somebody there who cares for us somebody who we can trust in a way. So it's very value loading, right? And what's fascinating about that is that caregiving which is often seen as something that specifically women do in their private homes. So they care for their children, they care for older people, they may care for their partners, that has been really politicised. And those are areas of life that haven't actually been given a lot of attention by policymakers and by politicians, by tech companies even. So it's really fascinating that this value language of care is now being adopted by big tech companies, and something that I find quite concerning. On the other hand, I think there's an opportunity now to bring out the value of care for flourishing societies and that we really should be leaning into what care means and how we can build caring societies and communities and how AI can help us do that. Susannah de Jager: Fascinating. And I suppose therefore, if we start from one place, then we perhaps dig into some of the realities. So obviously at the moment, there's a lot from the large tech companies and in the public discourse that is quite negative and is talking about the loss of jobs that you just touched upon and potential displacement. How do you see these things and how do you see the impact to date and potentially evolving from now? Caroline Green: I have different viewpoints on this. So obviously, we need to have these really important conversations, because we can already see how AI is impacting workplaces, how it's already impacting young people, how they learn what it means for them to then enter the job market. But I think that often these big narratives, all the way to existential risk from AI, are not enough counterbalanced by some of the realities that we also see in our lives that can be really positive. So if we go back to care. A lot of us, I would say, we do experience care in our families, in relationships. Obviously not all of us, but there are pockets of care that we all experience or can experience or in our communities. The amount of times I come across projects where people are really interested to build something that will help others. So I think it's important for us to remember that this is also a reality, and that we need to counterbalance those big doom and gloom narratives, which are important to have. I'm not denying that we shouldn't have them because it will help us to plan to now put in place some responses to what we're seeing. But let's go also to what we've got and that we value and lift that and learn from it and then create spaces and responses based on what we know works well already in our communities, in our families, and our societies. Susannah de Jager: Can you give some examples? Because I know when looking through some of your work, it's such an interesting space because you've got this hugely human touch element that one could assume we shouldn't displace. But actually there were some really encouraging examples where that can be supplemented in some places, not entirely replaced, but to quite a high degree, that the frequency of touch in some environments could be massively enhanced by AI and I, like you, found that quite hopeful. Caroline Green: Yeah, I can tell you a lot of different examples from my work. So I think one of the most influential examples for myself or experiences this year has been a journey to Dharamsala in India. And spent some time there a week or so with some of my collaborators. We wanted to learn about the local community. So the particularly the displaced community of Tibet who lives there together with His Holiness, the Dalai Lama, and we wanted to understand how they use AI, if at all, how that's working in their lives, how they're also building tools to preserve their language and so on. But what happened during that week is that actually mostly we spend time in spaces that have got nothing to do with AI, but that are very much about the human. So monasteries, and nunneries, for example, care homes for older Tibetans, and a school that had been built by a Buddhist monk who saw the plight and the suffering of the local children from the really underserved and very poor communities and who then built the school for those children very much based on secular religion and ethics. What that has taught me is really very much that, first of all, what it means to be human. Namely, that we are incredibly creative people and beings that we can change from one day to another if we choose to do so. That we can be compassionate and I think there's so much hope and so

much power in that and AI can serve us to do that. When we then spoke to, for example, the nunnery, there and how they use AI. So they've got their classes where they teach also very young nuns already and all sorts of Buddhist ethics and English and so on. And they say, well, we want them to not use AI until we feel that they really know what the core is for whatever way they're teaching. But we are now using it for ourselves to create some of the curriculums to help us translate things and what that has done is that it has actually taken some of these nuns out of their nunnery, and it helps them to go into the world and teach people about their ethics. I think that's something incredible for all of us and for them as well. And then there's one community called Monlam IT who have built these incredible AI systems to preserve the Tibetan language. There is a possibility that the Tibetan language will be lost and so they created a system where they digitised all these beautiful old scriptures where people can now access these scriptures, can have them translated. There's like a translation tool from Tibetan into English. This is worth so much and not just in terms of translating the preservation of language. It's about culture, it's about identity, and here, AI can help people to preserve what they value the most and what we as humans value the most. So that gave me a lot of hope. Susannah de Jager: It really does and actually I looked into this work because I thought that it was such a fascinating trip that you had been on and the Dalai Lama's kind of first elicited response on AI focused on exactly as you just said, the fact that we can change our minds being so much of what is human. And it's interesting that has already come up in this mini-series on ethics and AI because Lionel Tarassenko, who I interviewed, said that he had changed his voting pattern on the social media ban in the House of Lords. And actually he was saying some people see that as weakness, but that when you're presented with something and it changes your mind, it's one of the strongest things one can do. But I love this idea that's coming through from multiple themes that our ability to re-evaluate, giving changing inputs is a huge strength of the human condition not, as it's often cast, a weakness. Why does it matter in your work, the ways in which we are distinct and human? Why is it so important to be able to define this? Caroline Green: I ask myself that question all the time, and sometimes people say to me, "Oh, well why do we need to be special?" And I think, we need to define and understand ourselves and realise that we are special because that also means we have responsibility. You know, we are special. We have an intelligence that no other animal or species has and with that comes responsibility. That responsibility also lies in understanding what makes us distinct. What it is that we value about being human, right? That's a very helpful way for me to think about the value of this whole question of what does it mean to be human in the age of AI. But it also means it's okay, so if we understand that we are special, what is it that we want to protect? And what values are important to us when it comes to caring for each other, having relationships, to education, to work? And when we've then got that foundation, that can actually help us to understand the ethics of AI and to formulate ways to protect what it is that we value. For example when it comes to caregiving in this country, we have a very good understanding what good quality care, for example, in a care home or in somebody's own home looks like, right? It's about person-centeredness. It is about privacy. It's about supporting people to be who they want to be. That helps us to frame what type of AI tools might be good here to support that. It also helps us understand the risks and what the harms could look like and then to create frameworks to protect people from that. Susannah de Jager: And you have created a framework. So I'd love to dive into that now because you've led us into that beautifully and there's this real kind of conceptual split that you've highlighted there between things that are just tasks, perhaps, and can be really expedited and made more efficient and then things that are relational and that we must really protect and be very mindful about that. So I'd love you to dive into the framework that you've created around that. Caroline Green: Yeah. So the framework you're referring to came out of a project called the Oxford Project on the Responsible Use of Generative AI in Adult Social Care and that started a couple of years ago where we convened at Oxford a group of leaders in the care community. So that's formal caregiving now. People who run care homes, but also policymakers who create policies around adult social care. And we invited them to understand better how people in care services are already using AI tools and generative AI had just dropped, and care workers and others in the care community started using these. But not knowing how to do that safely, what some of the risks are, there were no statements out there that could guide people, for example, from the Care Quality Commission, which is the regulator of health and social care services. So as a community, we said, "Okay, we need to find frameworks, and we need to help people to understand how to use this safely and responsibly." So we then created a big project with over 100 people working on this. We were very collaborative working with people who are actually drawing on care and support, and that's really important for this work. Care workers, tech providers, everyone coming together to make sense of what AI means to them and then where are the different pockets where people actually need guidance and what could these frameworks look like? The framework is based on a definition on what responsible AI in social care actually means and that definition is really

important because it gives the foundation for everything we do. Whenever we think about creating a new AI tool for this space, or deploying it, or then looking out for benefits and risks, we go back to that definition. And the definition is all about that AI systems should be serving and not harming people who are drawing on care and support and to ensure that their human rights and their well-being and safety is protected and that they can flourish, right? So it's all about these values, and about the people. And then we also created something very practical which will actually help care providers to ask some of the difficult questions before they and during their deploy AI systems in their caregiving. And this work has now led to a new alliance called the AI in Care Alliance, which is a community of good practise of people in the care community, again, where we come together and now develop this and other frameworks. We develop our understanding and we have some tricky questions as well. Susannah de Jager: Ooh, what are some of the tricky questions? Caroline Green: Well, I think some of the tricky questions are what you really asked just now. Where are the administratives tasks or where are those tasks where we say AI can just take over and where do we stop? So those are not clear cut. If we, for example, take a carebot. We get quite a few companies that are developing bots which often look very humanoid robot-like. So they might look like a little child, very sweet with big eyes that's supposed to be deployed in care homes and it goes round on its little wheels and then chats with the older residents. At the same time, while chatting, it will look at them and it will take lots of data. So we'll see if somebody's demeanour has changed, what are they saying? Might that be an indication that they have got some cognitive decline? But at the same time, it's supposed to be entertaining and help people to feel less isolated. So that's a tricky question here, right? So is there's we're mixing up some administrative tasks together with some very relational tasks and some people would say, "Well, I wouldn't want my grandmother to talk to a robot all day. I want them to have a human caregiver that will give them that touch." And others will say, "Well, actually, they will otherwise just be sitting in front of the TV. So what's the difference having a little box they can talk to or sit in front of the TV?" So here, those are tricky and difficult questions. Where do we start? Where do we stop with AI and then robots and technology more broadly when it comes to caregiving? Susannah de Jager: And I think this defines the space more broadly and these discussions that I've been having is that we're running at pace with things developing around us and so it's imperfect. We are not going to set up a regulatory framework that's perfect day one or even day 100 because it's such a fast pace. So in that framework, how are you helping people measure that? It's

evident to me that the example you just gave, both answers could be true. What are you developing to help us have a more objective stance on that? Caroline Green: Yeah. So first of all, when, again, let's go back to caregiving. In this country, formal care is actually a highly regulated sector. So we've got a lot of laws and regulations that formal care services are subject to and that they are also being measured, inspected for through the regulator. So the regulator, the Care Quality Commission, has only recently published a statement in which they say that they still inspect services for the quality of care. They don't inspect the AI systems as such. So really what that means is that care providers who are deploying these systems, they are responsible to keep the good quality care. But it's their responsibility and they have to understand what some of the implications are. As you just said, that is sometimes really tricky when it comes to the right to privacy, for example, person-centered care and so on. So how does our framework help people? There are some questions you can ask that will help with particular, for example, safety questions. So everybody in a care service needs to keep the people who live there safe. Now, an AI system that has distinct safety kind of risks, right? And so some of the questions you would need to ask is actually to the tech companies that are selling you this product. How have you built it? What are some of the privacy considerations and so on? So that's something that our framework has started to do, but that now as a group in the Care Alliance, we're trying to unwrap much more. So that's where everything, starts. The care providers are able to ask these questions to the tech companies who then owe them a transparent answer to that, right? But then when it comes to some of these relational issues. So some of the care homes I've worked with who are now deploying AI systems, they are saying, we're working with people. So we're doing pilots here where we've got care residents and where we've got care workers, and they know that we're piloting this, and they've got very much a voice in this. So they'll tell us how they feel about having this product here, how it's working out for them, what are some of the kind of fears they have around it, and how is it benefiting them? And I think that's really important this co-production approach where we understand the lived experiences of those who are subject to AI systems and then that people also act on that. To give you another example, if a care home puts an AI surveillance camera in and they pilot it, and then they see that the care workers in the residence, and they tell them that they're really uncomfortable with this. I would then want to see that they take it down again, or that they find a solution so that people feel safer. Susannah de Jager: This Oxford Plus miniseries is brought to you by Equinox, Equitable Innovation Oxford. Equinox is a major regional partnership established by the University of Oxford to drive inclusive innovation and sustainable economic growth. Bringing together universities, local authorities, government, industry, investors, and community organisations, Equinox provides a unified voice for Oxfordshire, championing innovation that delivers economic impact alongside social value. More than 80 organisations have signed the Equinox charter, committing to atract investment, remove barriers to growth, and ensure the benefits of innovation are shared more widely. By aligning Oxfordshire's world-leading research, enterprise, and talent, Equinox aims to unlock nationally significant growth, reinforce the county's position as a global innovation hub, and ensure that success creates skilled jobs, attracts investment, and delivers lasting prosperity across the region. We've spoken a bit, Caroline, about care homes and clearly that's a particular demographic. But I know that you're also studying quite a lot about the implications of an ageing society more broadly. I'd love to hear a little bit about how you see that space in relation to these discussions. Caroline Green: Yeah. So we are living in a world of demographic ageing. Generally, the world is becoming older and even though we've known that for a really long time, is starting to be one of those big doom and gloom narratives "Oh my God, we actually have to do something about this now" because there are lots of problems related to ageing populations. That's the narrative. I think of it as it's not the ageing population or older people that's the problem here. It's that we don't have the systems or even in the sort of the society in order to ensure that everybody, no matter what age they are, they get the right services that they need and deserve. So here, caregiving, again, is one of those areas I'm particularly interested in, because even though people are living for longer it also means that we have more people living with care and support needs and at the moment, the care system is extremely under pressure, right? So that's seen as a big issue and a big problem and it is, obviously, because people are not getting the care and support they need and AI and tech more broadly is, I think, quite often considered by policy and also by care providers as the panacea to that issue. Now, I would argue that we have to be very careful about this, because AI can be part of the solution to build better systems to support people in their own homes, for example, but it cannot be the whole solution. Ageing, first of all, is something that we all do. We are relational beings. We need each other. We need community. We actually know through so many studies from around the world of what it is to help us age in a way that we are happy and healthy. Technology and AI can always only be part of that but we need whole systems and societies that are less ageist that look at intergenerational fairness and some of these issues of justice and so on.

So I think there's a lot of work to be done here, and it's really exciting, because AI, again, can also help us in all of this. So we are about to start a project with the civil society working within the United Nations Human Rights Council's process towards a new convention on the rights of older persons. So this is really exciting, because it's the first convention that's being drafted in the new age of new, powerful generative AI. So we are going to be using some of these systems to try and lift the voices of older people around the world who usually don't get heard in a place like the United Nations or in processes like the convention drafting. That's a project that excites me on so many levels, because AI can be used for good here. But at the same time, we can also understand some of the ways that older people engage with AI, what might be some of their concerns, how can we support them, and then also create projects and systems around them in order to make sure that there's digital accessibility and so on. Susannah de Jager: And this is obviously a very interesting stage to be looking at and you've got the feedback loops, but we are talking about people who are developed and they are at the end of their life. Now, if we flip it and you take it to people at the beginning of their lives, children in development, there's a whole different set of considerations that are less proximate and that are future dated and we don't yet know the impacts of potentially involving AI in youth care or any educational youth environment. I know that you're doing some work on this as well at the moment. I'd love to hear a bit about that and then perhaps the ways in which it differs. Caroline Green: Yeah, so we've got a collaboration with the BBC ongoing at the moment. Where we are specifically trying to find out how young people, so people aged 16 to 25, are viewing or considering AI in their everyday lives. How they serve, so AI systems serve, as a source of information and if they feel that this type of information is trustworthy. The kind of data that we capture is we've got interviews with young people. But we've also got screenshots. So they have their phones and they will, during the day, they will choose to take screenshots of if they have, for example, got a question, how they then navigate finding a response to that question using AI, but also other avenues as well. So that's really fascinating. So we're at an early stage of getting the results in. But very broadly, I think we're starting to see a picture where some people really feel like the AI is all knowing and they can trust AI because it will go through all of the different sources and then find the information that you would normally just find through other sources anyway. We are also seeing that some younger people, they will have AI as part of their journey to find a response to something, but they'll also check in with people around them. So trusted people be it their parents, be it their peers, and then at some point they decide that they've got a response that they're happy with. So there's a bit of a mixed picture there, but what I'm concerned about is if young people actually feel that AI is the God of all knowledge because we need to understand here, obviously, like where does the information come from that they've been given? There's so many issues around that, right? So an over-reliance on AI, I think, is a big concern and I think generally that's also a sticky point where people are concerned about the professional lives in the future. Is an over-reliance of, for example, healthcare providers on AI systems losing their own expertise to be able to tackle questions? Susannah de Jager: Yeah. It's a very interesting point because I spend quite a lot of time. I've got a 12-year-old and a nine-year-old explaining to them that I think it's 25% of kind of resources that are being drawn upon for most chat bots are Reddit and Wikipedia and if you and I think back to when we might look at Reddit for a review or Wikipedia for something, you'd view it with quite a high degree of caution and as much as you can have a slight wisdom of crowds things going on, actually, it's just well repackaged to make it sound more believable than perhaps we should really take. It's like very educated people. I sometimes get accused by my many sisters of being very good at dressing up bullshit as a fact and I feel like chat bots are great at that. They package it brilliantly and it is a concern. Caroline Green: It is a concern and that also takes me back to the point around care language and the adoption of care language by large language models because if something is caring or sounds caring, we tend to trust it more too. Susannah de Jager: And do you think that we could disintermediate that? So when I had the conversation with Audrey, what I thought was very interesting was that she was talking about a social media platform that rewards agreement rather than outrage, which is such a simple switch. Should we be trying to leverage the big tech companies to remove the language of sycophancy of care to make it more evident that this is just data being sent to us rather than a person? Caroline Green: Wow, so that's an interesting question because, as you said, people can also choose what they want the language to be and shouldn't there be more of an educational kind of aspect to all of this public education where people also know and then make a choice? So give them the agency to also choose on how these systems, the kind of language that they adopt, rather than putting it into the hands of big tech companies to take the sycophancy away. So I think that there's a big public education piece here as well that's really important also for educational settings like schools and I am also really concerned about vulnerable groups. So people living with mental health conditions or people who are in a vulnerable spot and who, or, maybe at the first

stages of cognitive decline and are then using these systems and they get taken down this rabbit hole and they feel that it's a human they talk to. So those, I think, are specific cases. But I think the public education piece is really important and to make sure that people know that they can also choose the kind of language that they are using, or that their large language models will be giving them or whatever systems they're using. Susannah de Jager: Reminds me of historians being taught to look at the source material and always think about history's always written by the victors and I think we all need to now take a more careful lens to, quite frankly, everything we see. So we've obviously spoken about some of your work looking at really quite specific areas and now I'd love to hear a little bit about your work with Audrey Tang, the Ambassador for digital in Taiwan, because it takes a lot of what you've learned from these specific environments, but creates a wider lens. So please take us into that. Caroline Green: Yes. So Audrey Tang and I, so Audrey is an Inaugural Fellow at the Institute for Ethics and AI's Accelerated Fellowship Programme, which I lead and last year in September we also recorded a podcast. During that podcast about about her work we started coming together with my work on care and then her work in AI and we felt that there's a real need and space to engage with some of the philosophy around care, so the ethics of care and how that could inform a new approach to AI and how AI sits in the world and how it can actually serve us and communities and humanity, right? So from that podcast onwards, we had a lot of conversations and, we started engaging in that more. We started reading more and more of the work in ethics of care, and specifically that of Joan Tronto, who came up with a specific ethics of care approach, which is all about taking the ethics of care out from a more domestic sphere. That's where ethics of care originated as a philosophical way of thinking and took it into more of a political and much broader sphere. So we thought that would be a good start to thinking about this and we felt that so much of a narrative of this singularity that's about to come and form the all powerful AI system that is going to control us. That's one of the big narratives that we are seeing at the moment. And that sort of seems to be the trajectory whether we like it or not. Because of Audrey's work, we know that it doesn't have to be the default trajectory, that there's actually very much, a way where we can have AI systems that serve a particular purpose for a particular community, even a particular family, and that those AI systems should be created in order to serve the relational health of a community. And what we mean by that is really, like, caring relationships. It's people being able to work with each other, live with each other, also with their differences of perspectives, of opinions, and so on. So that's really the starting point of civic AI is that we say, what do kind of humanity do we want? What kind of society do we want? And that is one of communities coming together, people coming together, deliberating with each other of what it is that they need, and what the future should look like. So civic AI is all about proposing a sociotechnical trajectory that moves us away from the singularity and that serves that whole idea of caregiving, relationally healthy communities. Susannah de Jager: And I think that's something that's come through this conversation and threads through your work, but really is so evident to be able to go from the more specific and the smaller to this wider lens that you're describing are these two expressions you use a lot, which is _"Serving, not Harming"_ and _"Connection, not Convenience" _and I think that whether you choose to go really magnified in on a specific or you zoom right out, I think that those two pairs of words for me really elucidates so much on what it is we should be thinking about when we look at any system that we bring into any environment, quite frankly. And I love that Audrey's work also, and experience, talks about how we can do that in practise and both the ability of these systems and the way that we can make sure that these inputs are democratic. So whether it's through your research on the ground or a broader lens, your point is so well made and rather hopeful, which is we can do this. The tools do exist. Casting it back at you, how do you feel when you look forward? Are you hopeful? Are you an optimist? Are you worried about whether we'll succeed in doing this? Caroline Green: So for me, it's mixed. I am hopeful because I have the pleasure of working in the ethics of AI. So I am around people every day who think through these very difficult questions and who really want to also find solutions and those are people from very different backgrounds. They're not just academics, you know who are philosophers or sociologists or so. They are people in the private sector. They are people in civil society. There are policymakers, politicians who are thinking about this and then that also internationally. So that gives me hope because I know what as a humanity, and we are terrifying in one way, but we can also create the most amazing things and we can respond to things. So that gives me hope. What also gives me hope is, as you just said, Audrey's work has shown that it's possible to have a future with AI that really serves us as humanity with the values like human rights and well-being and safety and so on. So that also really gives me hope. What does concern me though is I do think we will see some real damage done. Some harms that right now we might not even know the risks exist because AI is very new in some of those spaces. So we'll need to understand these risks and I think even though we wouldn't really want to see these harms I think we will see some harms that we just weren't even able to imagine. So that concerns me. Then also in the middle, I am also excited. I am excited about AI and what it can do, right? I think it's an incredible technology. So it's about finding those ways to really ensure that we've got control and the grip over the potentially negative effects of it and really lean into the benefits however we define those. Susannah de Jager: If people are listening who, like myself are lay people, are there going, how do we differentiate between the slight scare mongering that we're seeing sometimes in the public discourse, more cynical perhaps, but commercially loaded kind of agendas? How should we be thinking about our place in this discourse and where we can add value as individuals? Caroline Green: So I think as individuals we can all educate ourselves as much as possible on what these technologies actually are. What the real risks are, what also the benefits to them are, so that the control of those narratives is taken back to the people. I think there is time for deliberation for people to discuss and debate more. I also think that it's time for all of us to really start thinking about what it is that we actually value, about our lives, about the relationships we have, about the education of our children. What is it the future that we want to see? And then see how we can hold people accountable. Whether that's our politicians and what they do, our civil society leaders, but also the tech companies. So I think it's really a time of democracy. Often people feel like, "Oh, this tech world. I don't understand AI much. I don't understand. I'm not very techy." I hear that so often or even people stepping away saying, "Oh, I don't wanna hear anything about this." But it's not true. We don't need to become experts in computer science to be able to make our voices heard here and to know what it is that's important to us. So I think that's really important for everybody is to just really lean into that kind of civic space. Susannah de Jager: Thank you. I've really enjoyed this. Caroline Green: Me too. Thanks for having me. Susannah de Jager: Thank you for listening to this episode of Oxford Plus, hosted by me, Susannah de Jager If you wanna keep up with all things Oxford Plus, visit our website, oxfordplus.co.uk or sign up for our newsletter on Substack. Oxford Plus is a podcast produced by Story Ninety-Four.