TRANSCRIÇÃO 🔗
WEBVTT
00:00:00.080 --> 00:00:01.600
Is data science really neutral?
00:00:01.760 --> 00:00:05.040
Why can algorithms cause harm even when they're designed to help?
00:00:05.200 --> 00:00:08.880
And how can we make sure that data science is a force for social justice?
00:00:09.039 --> 00:00:11.039
Well, that's exactly what we will be discussing today.
00:00:11.199 --> 00:00:12.560
Hi, my name is Katarina Doria.
00:00:12.800 --> 00:00:19.679
I'm an AI ethicist, and this is Bots Don't Cry, the podcast where every Tuesday I'll teach you critical AI literacy with other critical AI experts.
00:00:19.839 --> 00:00:25.120
And if you love the show and want to listen to this ad-free and help us continue this work, consider supporting us in Patreon.
00:00:25.280 --> 00:00:26.239
Link in the description.
00:00:26.399 --> 00:00:29.440
And today we are super duper lucky to learn from Dr.
00:00:29.519 --> 00:00:30.079
Ben Green.
00:00:30.239 --> 00:00:30.399
Dr.
00:00:30.559 --> 00:00:39.520
Ben Green is an assistant professor of information and public policy at the University of Michigan and a faculty associate at Harvard Berkman Klein Center for Internet and Society.
00:00:39.679 --> 00:00:44.399
He has published opads and outlets including The New York Times, the Wall Street Journal, and The Hill.
00:00:44.560 --> 00:00:53.759
And he's also the author of The Smart Enough City and his newest book, Algorithmic Realism, is a bold new guide to building technologies that serves people and promotes social justice.
00:00:53.920 --> 00:00:56.159
So thank you so much for joining us today.
00:00:56.399 --> 00:00:57.520
Thank you so much for having me.
00:00:57.679 --> 00:00:58.960
Really excited to be here.
00:00:59.280 --> 00:00:59.759
Me too.
00:00:59.840 --> 00:01:00.719
I am super excited.
00:01:00.880 --> 00:01:08.560
I think that a lot of people will understand more why they should care about data science and the impact in our world and that change is possible.
00:01:08.719 --> 00:01:13.200
So to start our conversation, I would love for us to start with the basics.
00:01:13.359 --> 00:01:16.239
That is pretty much why should people care about data science?
00:01:16.560 --> 00:01:16.799
Yeah.
00:01:16.959 --> 00:01:25.760
Well, people should care about data science because data science is the field of research and practice that creates algorithms.
00:01:26.000 --> 00:01:29.840
And algorithms are making all sorts of decisions around us.
00:01:29.920 --> 00:01:33.120
And I think most people have experience with seeing some of this, right?
00:01:33.280 --> 00:01:54.079
Whether it's the Facebook recommender algorithm that decides what shows up on your newsfeed, an evaluation algorithm from an employer that reads your cover letter and resume and decides whether you move on in the process for applying for a job, or an algorithm used by police to identify someone who was involved in a crime.
00:01:54.239 --> 00:02:00.319
Algorithms are involved in making all sorts of decisions that affect our lives in small and large ways.
00:02:00.400 --> 00:02:05.200
And so it's important to understand who are the people actually involved in building these systems and what do they do.
00:02:05.519 --> 00:02:10.479
I think even before they start asking those questions, I think the first one that they have is but what is an algorithm?
00:02:10.639 --> 00:02:13.520
What is this word that I keep hearing all the time and everywhere?
00:02:13.840 --> 00:02:18.879
And algorithm is sort of a funny term because it's used in multiple different ways.
00:02:19.039 --> 00:02:23.919
So, sort of broadly, an algorithm is just a set of instructions.
00:02:24.080 --> 00:02:28.479
So even like a cookbook recipe could be described as an algorithm.
00:02:28.719 --> 00:02:36.639
And in data science, in the more sort of technical side of the field, an algorithm usually refers to a process for learning from data.
00:02:36.800 --> 00:02:40.159
But more colloquially, that's actually not what we mean by algorithm.
00:02:40.319 --> 00:02:44.240
Kind of what we talk about as an algorithm are the outputs of those processes.
00:02:44.400 --> 00:02:55.439
So the Facebook recommender algorithm or a predictive policing algorithm or a medical diagnostics algorithm is actually the outputs of what the computer has done.
00:02:55.599 --> 00:03:00.879
It's really the process that the computer uses to then make decisions about new data.
00:03:01.039 --> 00:03:04.240
So that's sort of more broadly how we talk about algorithms today.
00:03:04.400 --> 00:03:13.439
And sometimes these are called like automated decision systems because often these are tools that are used to help people make decisions.
00:03:13.599 --> 00:03:22.400
It either provides information to a decision maker, like a doctor or a judge, or they in some cases will just make a choice or take an action all on their own.
00:03:22.639 --> 00:03:30.479
I think here you're already touching in so many different super interesting parts, which is algorithms are being used today to make decisions on people's lives.
00:03:30.639 --> 00:03:33.840
And this is all part of the field of what we call data science.
00:03:34.080 --> 00:03:49.120
So here, what I see around me is that many people assume, because this is data science and we're dealing with data, that this somehow would be neutral, that this would somehow be a really hard science, and that if an algorithm made a decision, that would be the truth.
00:03:49.280 --> 00:03:50.400
Is that really the case?
00:03:50.560 --> 00:03:52.159
Are algorithms really neutral?
00:03:52.240 --> 00:03:54.639
Is it something that we should be trusting by face value?
00:03:54.800 --> 00:03:56.000
What can you tell us about that?
00:03:56.319 --> 00:03:57.360
Definitely not.
00:03:57.680 --> 00:04:01.520
And and really, like the notion of science is I would say a misnomer.
00:04:01.680 --> 00:04:03.919
Like data science isn't really a science.
00:04:04.159 --> 00:04:19.279
It
Load full transcript
is more of an engineering field than something that I would describe as a science because it's not really trying to, I mean, really what the field of data science is doing is trying to understand how algorithms function on a mathematical, technical level.
00:04:19.360 --> 00:04:20.639
That's a lot of what the field does.
00:04:20.800 --> 00:04:30.639
And then there's a more sort of engineering side of it, which is saying, oh, we just want to build tools to provide utility for some business context or some decision-making process.
00:04:30.720 --> 00:04:44.000
But it's really not a science in the sense that it's actually often quite divorced as a field from what are the impacts of these tools and what are sort of like the broad generalized knowledge, how data is constructed and how data is functioning.
00:04:44.240 --> 00:05:11.759
And I would say, like, one way to think about my work and my book is actually trying to make it more scientific as a field by saying if you're gonna be building algorithms that are making interventions in the world, you need to have a better understanding about what impacts that system is actually going to have, rather than making claims about the social benefits of an algorithm based only on mathematical details that you've tested and that often don't provide a very good picture of the system.
00:05:11.920 --> 00:05:15.040
So that kind of answers your question about science.
00:05:15.120 --> 00:05:21.360
And so I think that helps to contextualize why we shouldn't trust algorithms as these neutral artifacts.
00:05:21.600 --> 00:05:23.360
And there's sort of different ways to think about this.
00:05:23.519 --> 00:05:25.759
I mean, data is is often incorrect, right?
00:05:25.920 --> 00:05:31.040
Data is messy and may not accurately represent the world.
00:05:31.279 --> 00:05:35.680
But also data is a reflection of problems that are already present in society.
00:05:35.839 --> 00:05:43.519
So often, even if data is accurate, that accuracy is reflecting the presence of inequality and injustice.
00:05:43.680 --> 00:06:00.000
And so if we just make decisions based on that, we might be accurate in terms of predicting someone's likely social outcome, but we shouldn't just want to recreate that because that would be just projecting past inequalities into the future and then justifying them as inevitable.
00:06:00.240 --> 00:06:10.959
And here you're already hinting the idea that what you see in the field of data science is that there is a disconnect between the things that are being studied and the context in which the things that are being studied exist.
00:06:11.279 --> 00:06:17.199
What have you been seeing in terms of that disconnection between decision and output and the world that exists?
00:06:17.360 --> 00:06:22.240
And how are you thinking about the changes that need to be made for that connection to exist?
00:06:22.480 --> 00:06:29.920
So it really comes down to how people in data science think about what counts as rigorous, what makes for a good algorithm?
00:06:30.240 --> 00:06:41.439
And right now the field is really focused on the technical details of systems, by mean which I mean sort of like the technical notion of algorithm that I was describing earlier, where they're saying, okay, what's the inputs to the system?
00:06:41.600 --> 00:06:42.560
What are the outputs?
00:06:42.720 --> 00:06:48.240
Do those outputs appear to be making accurate predictions or judgments about the world?
00:06:48.399 --> 00:06:50.720
Are they fair in some technical sense?
00:06:50.959 --> 00:06:55.759
And they don't really think about, okay, but what actually happens when you put that system into practice?
00:06:55.920 --> 00:07:02.639
When you give that algorithm to a doctor or HR manager, what do they do with it?
00:07:02.720 --> 00:07:04.480
Does it actually help them do their work?
00:07:04.639 --> 00:07:06.639
How does it change their decisions and so on?
00:07:06.720 --> 00:07:10.800
Those types of questions just don't really get that paid that much attention to.
00:07:10.959 --> 00:07:20.800
If you go through your data science undergraduate program, you're really not taught that much about how to look at human users or understand how a system gets used in practice.
00:07:20.879 --> 00:07:22.800
There might be some high-level discussion of it.
00:07:22.879 --> 00:07:30.639
People might say, yeah, you have to think about the users of a system and make sure that you're building ethical tools, but there's very little training in terms of how that actually works.
00:07:30.720 --> 00:07:33.839
And the same is true in academia on the research side.
00:07:33.920 --> 00:07:43.519
Like writing a paper that says, I've built a new system and it increases the performance of this model by, you know, a few percentage points, that's going to be cool.
00:07:43.600 --> 00:07:44.800
You're going to get a nice paper.
00:07:44.959 --> 00:08:01.920
But if you say, I built a system for, you know, a specific public health agency and here's how I worked with them, you probably could publish that somewhere, but that is just not looked on remotely as favorably as a paper that's doing the sort of quote unquote like pure technical modeling.
00:08:02.319 --> 00:08:17.759
What I started seeing here is that you have started seeing data scientists as political actors and that they should acknowledge that this is what they are and the impact that they have in the world that is far beyond just numbers and data and the systems that they're creating.
00:08:18.000 --> 00:08:23.680
Why do you think it is so important for data scientists to understand the responsibility that they carry in their hands?
00:08:24.000 --> 00:08:26.800
I think there's really two different sides to this.
00:08:27.040 --> 00:08:37.039
The first is to say that whether or not you want to be, you are acting politically, like you have agency as a data scientist, whether or not you're seeking it out.
00:08:37.200 --> 00:08:44.960
Now,
because data scientists do often have this idea that, oh, I'm neutral, I'm doing science, I'm just working with data.
00:08:45.120 --> 00:08:53.279
And if they think of what they're doing as working with these mathematical formulas as algorithms, then it can be hard to see the politics in that.
00:08:53.440 --> 00:09:01.840
But often data scientists are building systems that are affecting someone's rights, someone's liberty, whether or not someone gets a job.
00:09:01.919 --> 00:09:11.279
And even what you see on your newsfeed feels small compared to those things, but it actually has a really huge impact on society, how these those algorithms are constructed as well.
00:09:11.440 --> 00:09:15.360
And data scientists don't always have complete control over those systems.
00:09:15.519 --> 00:09:38.000
There are managers and business leaders and other people who are giving them directives, but they they do have quite a bit of agency in terms of deciding how do you evaluate this system, how do you measure engagement or accuracy or the types of things you might be trying to evaluate your system on or incorporate into the model as considerations like socioeconomic status.
00:09:38.080 --> 00:09:39.440
And so you have to make those judgments.
00:09:39.519 --> 00:09:41.120
So that's kind of like the first reason.
00:09:41.279 --> 00:09:45.120
But but the second reason is more of a call to action beyond that.
00:09:45.200 --> 00:09:58.399
It's like we as a field should be taking this power much more seriously and really be creating a culture where the public interest and social justice in particular is really foundational to the ethos of the field.
00:09:58.639 --> 00:10:05.840
That what we think of as good data science is the same as data science that is good for the world.
00:10:06.000 --> 00:10:12.159
That we can't just say, oh, well, some people like to do good things, but other people will just be kind of morally neutral.
00:10:12.320 --> 00:10:26.720
Uh, we should really be pushing people in this field to take this responsibility seriously and think that part of your kind of like ethical obligation as a practitioner or researcher in data science is to use your skills for good.
00:10:26.879 --> 00:10:35.120
And I think that's really part of the shift that my book is trying to create in data science is like we need to teach this field differently.
00:10:35.279 --> 00:10:37.519
We need to create different structures within research.
00:10:37.679 --> 00:10:51.279
We need regulations as well that kind of push this more to the forefront so that people who graduate from college or a master's program and become data scientists have a real sense that part of my mission in this profession is to do good for the world.
00:10:52.159 --> 00:11:01.279
So, how do you think that we can bring data scientists closer to the knowledge of, hey, let's focus on the world, let's see beyond the numbers.
00:11:01.440 --> 00:11:03.200
How do you think we can bridge this gap?
00:11:03.519 --> 00:11:18.639
I appreciate that question because I do think for me, because I have that foundational approach to my work and training was in that field, I think it allows me to try to communicate some of these more critical and social topics to technologists well.
00:11:18.799 --> 00:11:34.559
That's like what I try to do, to be sort of that bridge where it's like I can read the critical literature and the news stories and try to help say, okay, here's how this would actually be used in practice because I have a little more of that experience than some of the like social science scholars do.
00:11:34.720 --> 00:11:39.519
So that's very much the direction that my book takes is to say, okay, let me be that bridge.
00:11:39.679 --> 00:11:46.639
Let me try to take these ideas and make them feel digestible and actionable to technologists.
00:11:46.799 --> 00:11:55.279
Because part of the problem in terms of changing the field is that, you know, these critiques get thrown at technologists and they don't entirely reject them.
00:11:55.440 --> 00:12:04.000
Like a lot of technologists are like, yeah, I get that there are harmful impacts and that algorithms can be biased and there are ethical concerns, but I don't really know what you want me to do with that.
00:12:04.159 --> 00:12:06.639
And that's that's a lot of the conversations that I have.
00:12:06.879 --> 00:12:11.519
It's not so much data scientists telling me like, I'm neutral and I don't care about these things.
00:12:11.679 --> 00:12:15.759
It's more them telling me, this is hard and I don't really know what you want me to do.
00:12:16.000 --> 00:12:19.919
This doesn't really seem that practical, and that's not my job to make these decisions.
00:12:20.080 --> 00:12:39.600
And so my work is really trying to say, okay, let me take that seriously and give you something that is more tractable and actionable, but at the same time, doesn't dumb down the idea because that's the tricky thing is to a lot of the work in sort of ethical AI, quote unquote, is so actionable that it actually kind of misses the point.
00:12:39.919 --> 00:12:52.480
And so you have to find that middle ground between taking the idea seriously without saying, and now there is a single metric that you can evaluate your algorithm based on, and you've taken care of all of the social effects of your system.
00:12:52.639 --> 00:13:00.879
So finding the way to bring that together is very important and kind of a central goal for me in all of the research that I do.
00:13:01.039 --> 00:13:09.360
So here we're talking a lot about the operationalization of culture or social studies and social science within data science.
00:13:09.440 --> 00:13:19.120
It's being able to say, okay, this is not only, hey, I don't want to be neutral, but these are the things that you could be doing to make sure that your algorithms, your work is including that conversation.
00:13:19.360 --> 00:13:27.840
Could you tell us a little bit more about
what is that operationalization that you have created to support data scientists in adding social sciences to their work?
00:13:28.159 --> 00:13:35.360
So the broad guide or manual that I lay out is in terms of this concept of the data science workflow.
00:13:35.440 --> 00:13:42.480
And this is a concept that already exists within the field, but my intervention is to say the way that we think about this workflow is far too narrow.
00:13:42.559 --> 00:13:47.679
There are not actually enough steps and really important considerations are left out.
00:13:47.919 --> 00:13:54.159
And it's really hard to kind of engage with social questions within that scope.
00:13:54.240 --> 00:14:04.159
Because if we're trying to deal with like ethics or power or bias, the work ends up being centered on well, how do we incorporate that into the modeling step or the data curation step?
00:14:04.240 --> 00:14:06.000
And there's only so much you can do there.
00:14:06.240 --> 00:14:11.919
So my version of the workflow adds a lot of steps both before and after the actual development.
00:14:12.000 --> 00:14:16.960
So there's upstream work that I call scoping, which is really just about figuring out like what should we actually do?
00:14:17.120 --> 00:14:24.720
So first you have to figure out what is the problem that we're even trying to address, which is like a hugely overlooked aspect of data science work.
00:14:24.879 --> 00:14:27.600
You know, what is the real social problem we're dealing with here?
00:14:27.759 --> 00:14:30.240
And then what do we actually want to do about it?
00:14:30.399 --> 00:14:39.440
What type of system should we build, or what type of support should we provide to this group of people based on the skills and competencies that we can bring to the table?
00:14:39.600 --> 00:14:44.399
And all of that needs to happen before you actually start creating a model or algorithm.
00:14:44.559 --> 00:14:48.879
And a lot of those things are happening implicitly within data science work, right?
00:14:48.960 --> 00:14:54.799
Like there is an implicit understanding of there's a problem and we're gonna build a tool, but it doesn't really get thought about that much.
00:14:55.039 --> 00:15:05.440
Data scientists often work on pretty superficial understandings of a challenge or problem and then assume, well, the thing to do is to just build a predictive model about the specific decision here, right?
00:15:05.600 --> 00:15:14.799
And so this is to try to like destabilize that process and really situate that if you're gonna build something, it should actually be useful to a real problem and a real context.
00:15:14.960 --> 00:15:18.000
And if you don't, all the work you do is kind of useless.
00:15:18.159 --> 00:15:25.440
You might build an algorithm that's really cool technically, but if it doesn't engage with a real problem, then it's all kind of a waste of time.
00:15:25.600 --> 00:15:28.960
And then you build a system, okay, great, you're not done.
00:15:29.200 --> 00:15:38.879
You have to figure out if you want to implement that system, really think about what are the social impacts of this going to be, implement it, and then evaluate what those impacts are.
00:15:39.120 --> 00:15:47.279
If we want to judge whether our efforts as data scientists were successful, we need to look at did we actually help address some real problem?
00:15:47.600 --> 00:15:58.080
We can't just say, I built an algorithm and it's really accurate and it runs really efficiently, which is kind of the status quo right now for a lot of uh papers and research and products.
00:15:58.240 --> 00:16:02.960
And we see today, all of these AI tools are like what gets touted are the benchmarks.
00:16:03.120 --> 00:16:13.440
And it's like, but that doesn't tell you whether this actually helps people, even setting aside like whether it's good for the world, whether it helps people just like do a job that you may or may not even agree with.
00:16:13.600 --> 00:16:16.080
But is it actually functionally useful for people?
00:16:16.240 --> 00:16:27.440
Is the last step of this is to do that evaluation because that at the end of the day, I argue, is the way that we need to judge algorithms, not their technical properties, but what are the social impacts?
00:16:27.679 --> 00:16:35.840
When we think about the impacts, the social impacts, again, we go into ideology and the understanding that nothing is ever neutral and that the world is embedded into ideology.
00:16:35.919 --> 00:16:42.159
But we live in capitalistic systems and AI companies, they want to make a lot of money and they don't care about our lives whatsoever.
00:16:42.399 --> 00:16:48.399
So I'm wondering, what do you think we can say in terms of the people who say, I we want to build fast and break things?
00:16:48.639 --> 00:16:55.360
And whenever we start adding culture and ideologies and we have to make sure about the impact, it makes it slower because of course it makes it slower.
00:16:55.440 --> 00:16:56.399
It should be slower.
00:16:56.559 --> 00:17:06.079
But what can we say in terms of ideology and the importance of adding this step, regardless if this is going to take longer in the entire process of data science?
00:17:06.240 --> 00:17:06.960
Does it make sense?
00:17:07.359 --> 00:17:08.079
Yeah, totally.
00:17:08.240 --> 00:17:12.160
And I think there's two ways that I would think about this question.
00:17:12.400 --> 00:17:31.279
First, I would say the sort of like narrower answer is to say that I think in a lot of contexts, setting aside social justice or you know, the public good, I think there's a real case to be made for this process because companies and organizations writ large have wasted so much time and money building and adopting AI systems, right?
00:17:31.359 --> 00:17:36.160
Like I think about how many millions of dollars companies wasted over the last year on token maxing, right?
00:17:36.319 --> 00:17:39.599
Like I could have probably told them that this was
not gonna work so well.
00:17:39.680 --> 00:17:41.440
You know, like I was skeptical from the start, right?
00:17:41.599 --> 00:17:56.400
So it's like if you're a company and you're just trying to like get more clients or create efficiency, still having a system where you're thinking about what type of AI tool to build or adopt is gonna be valuable because you're gonna save yourself a lot of wasted time of building a model that doesn't work.
00:17:56.559 --> 00:18:01.440
So that's a big, a big piece of this that I would say to a lot of companies in terms of the work they do.
00:18:01.599 --> 00:18:10.960
But there is a sort of larger issue that a lot of companies are thinking about a certain type of impacts, which is making money, and they don't care about the larger social impacts.
00:18:11.119 --> 00:18:15.599
And that is not a problem that can be entirely solved by this framework.
00:18:15.759 --> 00:18:22.720
It's not the case that well-meaning data scientists at OpenAI can just build better systems if only they were more ethically trained.
00:18:22.880 --> 00:18:28.079
I think there's maybe a little bit they could do, but these are like larger problems of power and capitalism.
00:18:28.319 --> 00:18:40.079
The easiest implementation of my framework certainly is in research or nonprofit or government, where there's less of that profit motive that says, actually, we don't care about the social harms because it's gonna make us more money.
00:18:40.240 --> 00:18:51.680
So for that, I think we need sort of shift what the corporate labs are doing, is much more an answer for regulation and public pressure and data scientists organizing, right?
00:18:51.759 --> 00:18:58.799
Like worker organizing in tech companies is important, data scientists refusing to build certain systems, learning how they can work together.
00:18:58.960 --> 00:19:07.839
But the other piece of my point about we were talking about earlier about data science being political or data scientists being political is like the choice of where to work is political.
00:19:08.079 --> 00:19:12.000
And like, yes, there are constraints, you know, people need jobs, they need money.
00:19:12.079 --> 00:19:12.640
I get that.
00:19:12.799 --> 00:19:18.960
But a lot of data scientists go into working for these companies without much internal struggle or searching elsewhere, right?
00:19:19.119 --> 00:19:21.599
You could just not go work for these companies.
00:19:21.680 --> 00:19:24.720
And we've seen it a few companies, Palantir or Flock.
00:19:24.960 --> 00:19:28.160
There's been a little bit more people leaving or not going into them.
00:19:28.240 --> 00:19:30.000
But I, you know, that's that's too narrow.
00:19:30.160 --> 00:19:50.799
I think it's a real shame on the field that so much of our culture tells data scientists that, like, yeah, the best job to get is to go work for these massive companies that we've just seen in many cases, like years and years of evidence that they truly do not care about the social good and are willing to, as they say, risk destroying the world and ending humanity.
00:19:50.960 --> 00:19:58.400
So I think we need to create a culture where people are forced to think more deeply about where they choose to go, you know, leverage their talents.
00:19:58.640 --> 00:20:02.079
So at the same time, we have the choice to work in a place is political.
00:20:02.319 --> 00:20:11.440
We also have people who are working in companies that are able to push certain policies, they're able to raise awareness and work within these companies to make a change.
00:20:11.680 --> 00:20:13.119
There's also everyday people.
00:20:13.279 --> 00:20:23.039
What do you think is the power that everyday people have in terms of pushing for your work or, you know, data science that is closer to social studies to be pushed forward?
00:20:23.119 --> 00:20:26.960
What can we do to help you help data scientists understand that this is important?
00:20:27.119 --> 00:20:29.839
I think a lot of people are already doing this, right?
00:20:29.920 --> 00:20:39.519
We're seeing protests against data centers, protests against AI in schools, in hospitals, for public benefit systems and policing.
00:20:39.599 --> 00:20:41.440
Like people are already pushing back on that.
00:20:41.519 --> 00:20:47.279
And it's been really inspiring to me to see just how many of these efforts are out there and how successful they've been.
00:20:47.440 --> 00:21:01.599
I think a big piece of it is, you know, is organizing it and really trying to build power to help decision makers and people with authority, especially policymakers, like really see that there is pressure to change how they're managing these systems.
00:21:01.759 --> 00:21:04.400
And we've, you know, really seen that over the last year.
00:21:04.559 --> 00:21:10.079
And, you know, particularly, you know, where I am in the US, the politics around AI has really changed a lot.
00:21:10.160 --> 00:21:15.920
Um, and it's much harder for any politician of either party to sort of just be like gung-ho about AI.
00:21:16.079 --> 00:21:18.240
That's not gonna do very well for you.
00:21:18.400 --> 00:21:28.960
But I think it's really important to reject the narratives or like defeatist ideas about inevitability that often sort of fall into these conversations.
00:21:29.119 --> 00:21:36.720
Not just the inevitability about a particular project, but like the sense of resignation that, like, oh, well, AI is here to stay and AI is going to have a certain direction.
00:21:36.960 --> 00:21:39.440
We can fight a small thing, but it's not really gonna change much.
00:21:39.519 --> 00:21:41.359
And I think that's completely not true, right?
00:21:41.519 --> 00:21:55.359
This is one of the key tactics that the industry uses is to try to give this image of inevitability and urgency that tells companies and governments you have to adopt AI right now, or else you'll fall behind.
00:21:55.440 --> 00:21:58.000
And whether you do
it now or later, like it's gonna happen.
00:21:58.079 --> 00:21:59.279
So you may as well get on board.
00:21:59.440 --> 00:22:02.880
And You know, I think there's very different directions that this field could go.
00:22:03.039 --> 00:22:08.559
And it's important not to cede control over that and to really push for a different direction of AI development.
00:22:08.799 --> 00:22:14.880
One of the things that really frustrates me about the conversation about AI right now is that it's very one-dimensional.
00:22:15.039 --> 00:22:20.960
We sort of assume that AI is going on a certain path and maybe we slow it down, right?
00:22:21.119 --> 00:22:23.680
Pacing is sort of a term of the last month or two.
00:22:23.839 --> 00:22:35.359
But that still assumes we're going to the same place, which is these big labs with massive power just building tools that are at least in theory supposed to replace workers and kind of replace everyone's tasks and create AGI.
00:22:35.440 --> 00:22:40.640
And then people who are, you know, respond to skeptics saying, oh, well, if you don't want AI, you just want no innovation.
00:22:40.720 --> 00:22:41.920
You want to shut down everything.
00:22:42.079 --> 00:22:46.000
What I want to say is like there's other directions that AI development can go.
00:22:46.160 --> 00:22:51.200
The question is not AI versus no AI, it's which direction should AI go?
00:22:51.279 --> 00:22:58.640
Like we can pursue a different model for AI development that is actually going to be much more socially beneficial than what we have today.
00:22:58.799 --> 00:23:03.119
So it's not a single train track and we can pump the brakes or not.
00:23:03.359 --> 00:23:08.960
We're at a terminal where there's many directions we can go in and we have to pick which of those tracks we want to pursue.
00:23:09.039 --> 00:23:31.680
And that's, I think, just opens up the conversation beyond this very binary view of we can have AI or not, sort of plays into the industry's hands by allowing them to be the people who own innovation, as opposed to innovation being this much more widely spread thing that there's many different forms of innovation and we get to choose or we should be able to choose, like which one we want.
00:23:32.160 --> 00:23:33.759
Because we are part of the conversation, right?
00:23:34.000 --> 00:23:43.599
Silicon Valley really made this entire marketing scheme that makes it seem that they and only they are the ones who can decide where data science goes, where artificial intelligence goes.
00:23:43.680 --> 00:23:52.960
And it goes to say what you were saying at the start of this conversation that many times people who are outside data science understand the impacts of data science way more than data scientists.
00:23:53.039 --> 00:24:02.160
So we don't need a degree, you don't need a master's, you don't need a PhD in data science to understand the values that you would like to be followed in artificial intelligence or data science.
00:24:02.319 --> 00:24:04.559
So that's a huge marketing that happens today.
00:24:04.720 --> 00:24:12.400
And then there's another marketing that I wanted to ask you because today in the industry, I see that it's the classic ethics washing.
00:24:12.559 --> 00:24:23.359
We see that a lot of companies they want to say that they're doing good or they're thinking about these things, even though they really couldn't care less about us or making sure that the impacts are not destroying the world or destroying people.
00:24:23.519 --> 00:24:29.759
What can you tell everyday people to maybe watch out for so they are not ethics washed in data science?
00:24:30.720 --> 00:24:32.559
Yeah, that's such a great question.
00:24:32.720 --> 00:24:39.599
Uh, number one is just to be skeptical of any claims about benchmarks and safety and alignment.
00:24:39.680 --> 00:24:48.960
The companies really push these sort of narratives of how powerful their systems are that sort of say, like, oh, well, it passed this benchmark or this formal assessment.
00:24:49.119 --> 00:24:56.799
So that's like one thing to just be aware that, yes, they may have done that, but there is a huge gap between passing the bar exam and being a good lawyer.
00:24:56.960 --> 00:24:59.759
And the same is true across all of these other benchmarks.
00:24:59.920 --> 00:25:06.880
And so they want you to sort of see the test that they've passed and then suddenly believe, wow, A, I can do all of these things.
00:25:06.960 --> 00:25:08.720
I need to adopt it or it's going to replace me.
00:25:08.799 --> 00:25:10.240
So that's like one big piece of it.
00:25:10.400 --> 00:25:18.720
Broadly, I'd say the companies often will try to deflect criticism by suddenly saying, like, oh, but we care about safety, we care about ethics, we want regulation.
00:25:18.880 --> 00:25:25.519
And that is a very common tactic of the tech industry to try to maintain control.
00:25:25.839 --> 00:25:40.720
It goes back to, you know, various fields of science even before big tech was a thing, where science and technology fields and companies will often suddenly embrace regulation or responsibility when they realize that they have to do something and that by they can co-opt it.
00:25:40.799 --> 00:25:49.680
And so that's often what we're seeing with the safety conversations, or even when anthropic or so will say, like, we really need regulation, or when Facebook has said we need privacy laws.
00:25:49.759 --> 00:25:55.440
It's like, yeah, you want the laws, but you want it in a way that you get to write because you realize that there's enough pressure building.
00:25:55.680 --> 00:25:59.039
That's definitely another big, big piece of this.
00:25:59.200 --> 00:26:06.960
And then maybe one final element that has been core to a lot of my work is this idea of human oversight or human in the loop.
00:26:07.039 --> 00:26:11.680
And this comes up a lot of the time for organizations that are adopting AI.
00:26:11.839 --> 00:26:16.880
And then the
public will say, look, well, we don't want you making these decisions just based on AI, right?
00:26:16.960 --> 00:26:29.039
Whether it's who gets a job or who gets arrested or who gets benefits, whatever it might be, there's often this sort of distinction that gets drawn between like fully autonomous systems and having a human in the loop.
00:26:29.200 --> 00:26:33.920
But human oversight is this very thin safeguard that really just doesn't work.
00:26:34.000 --> 00:26:36.799
And like I mean doesn't work in a pretty direct empirical sense.
00:26:36.960 --> 00:26:38.559
I've done a lot of experimental research.
00:26:38.640 --> 00:26:42.400
This was core to my PhD, and a lot of other people have studied this as well.
00:26:42.559 --> 00:26:51.200
Just looking at can people make good assessments of whether an algorithm is accurate or appropriate or fair in a given case.
00:26:51.359 --> 00:26:54.079
And by and large, the answer is no, they cannot do that reliably.
00:26:54.240 --> 00:27:01.680
When a government or a company says, oh yeah, we're using algorithms, but there's a human in the loop and they want you to trust them, you shouldn't trust them.
00:27:01.839 --> 00:27:07.119
You should say, show me the evidence that that human in the loop can actually do the thing that you're telling them to do.
00:27:07.279 --> 00:27:25.839
The craziest example of this that just happened a few weeks ago was there was a story that CNN published about essentially how the US military almost started a war with China because some officer or intelligence analyst believed that there was a Chinese ship that had military parts or nuclear parts.
00:27:26.000 --> 00:27:29.759
So the military almost invaded that ship, which would have been a huge international incident.
00:27:29.920 --> 00:27:36.799
It turned out that that information was conceived by whatever AI chatbot this person was using.
00:27:36.960 --> 00:27:38.160
And then they passed it along.
00:27:38.319 --> 00:27:51.440
So this is just one example of how, like, yeah, we always say we're gonna review the case and do our diligence and all of that, but it is just such a common phenomenon that people defer to the system and don't actually pay as much attention.
00:27:51.599 --> 00:28:00.880
And that's happening even in this highest possible stakes circumstance of like, should we or should we not take a military action against an adversary country?
00:28:01.039 --> 00:28:03.920
But this happens in far more mundane ways as well.
00:28:04.160 --> 00:28:05.759
I think this will be a great call out to people.
00:28:05.920 --> 00:28:09.599
Don't trust human on the loop, human in the loop, or all the different variants.
00:28:09.839 --> 00:28:13.680
Here you are also hinting, I at least I could sense the hinting of regulations.
00:28:13.839 --> 00:28:19.920
A lot of AI regulations today, or the ones that are being currently written, they add human in the loop as like a big part.
00:28:20.000 --> 00:28:25.759
Like, no, we have to make sure that if a system is a high risk, we need to have humans in the loop, as if there was a solution.
00:28:25.920 --> 00:28:37.920
So I can only imagine that you in your book or you and your experience, you have a different take to what we need in terms of regulations to make sure that data science is good for humanity and that we're able to promote justice.
00:28:38.160 --> 00:28:45.200
How do you think that a good regulation for AI in data science could be to make sure that this is good for us and not harmful?
00:28:45.519 --> 00:28:50.079
There are a few, a few important directions that my that my research really leads me to.
00:28:50.240 --> 00:29:03.839
So certainly in the context of human in the loop, often that is sort of a band-aid for systems that often really just shouldn't be deployed in the first place and is sort of meant to defray the criticism.
00:29:04.000 --> 00:29:15.200
So one of the big pieces, especially for public sector algorithms, is that agencies should be required to provide much more upfront justification of what they actually want to do.
00:29:15.359 --> 00:29:18.960
That is subject to some form of public review and approval.
00:29:19.119 --> 00:29:27.359
And that can be like, okay, you're saying you want to develop an automated system to make judgments about benefits fraud.
00:29:27.440 --> 00:29:28.960
You don't just get to do that.
00:29:29.200 --> 00:29:50.319
You have to describe why you want to do it, why you think it will be beneficial, and how you validated the system that it's actually going to work and validated the sort of operational workflow, and then provide that to city council or whatever the appropriate board might be within a given jurisdiction and let people really say, okay, do we think this is a worthwhile thing to do?
00:29:50.480 --> 00:30:04.160
So that would really open up much more conversation, give the public a lot more insight into what's happening, a lot more authority, and also place the accountability and the pressure on organizational leaders who are making these decisions.
00:30:04.240 --> 00:30:11.039
Because the other problem with human in the loop is that when something goes wrong, it's really easy to point fingers at the low-level person, right?
00:30:11.200 --> 00:30:17.839
The intelligence analyst who didn't double check the work or the detective who didn't realize that the facial recognition match was wrong.
00:30:18.000 --> 00:30:19.359
But really, it's not on them.
00:30:19.519 --> 00:30:29.279
It's on the leader who pushed these systems into operational use without having really validated are they appropriate and will they work in practice.
00:30:29.440 --> 00:30:34.240
So that's definitely one piece is much more of this upfront analysis and review.
00:30:34.640 --> 00:30:39.920
Certainly, I think all of these AI companies need to be just held liable for what their tools are doing.
00:30:40.079 --> 00:30:45.359
It's kind of remarkable that these AI agents
are doing crimes and then it's like, they went rogue, there's nothing we can do.
00:30:45.519 --> 00:30:54.319
Like, no, you could, you could just, you could just penalize the the people who built this system, like the CEOs of these companies for building tools that are doing crimes.
00:30:54.480 --> 00:30:56.480
That's actually not not so hard.
00:30:56.640 --> 00:31:04.400
I think the other piece, the sort of broader regulation conversation, is to think about, okay, how do we also create a different type of ecosystem for AI development?
00:31:04.559 --> 00:31:10.079
So it's not just like, yeah, we need safety regulation or things like that, but that's still very focused on what the companies are doing.
00:31:10.319 --> 00:31:18.960
We need to create regulation or funding streams and other types of support to get the types of AI development that will actually be good for society.
00:31:19.119 --> 00:31:23.680
Because that's not going to come from open AI and anthropic whether or not there's safety regulation.
00:31:23.839 --> 00:31:28.480
So we need to try to help the people who actually want to build systems that are going to make your life better.
00:31:28.720 --> 00:31:31.920
Um, and of course, you know, in the US, we're doing the opposite of that.
00:31:32.079 --> 00:31:38.000
We've cut so much funding from research agencies or federal funding agencies like the National Science Foundation.
00:31:38.079 --> 00:31:39.519
So we're kind of doing the opposite.
00:31:39.680 --> 00:31:43.519
But I think that, you know, the AI that we need is gonna come from researchers.
00:31:43.599 --> 00:31:52.559
It's gonna come from often like government bodies that have more staff and resources to build technology in-house so that they can actually build things that make your lives better.
00:31:52.720 --> 00:32:18.559
I think we need more, more competition, more smaller companies that are actually building more domain-specific, like narrow AI models that are actually addressed at helping doctors do medical diagnostics or treatment or, you know, things, things like that, um, or companies that are like actually trying to build AI that improves education rather than just get as many eyeballs as possible and consumers and just get tons of money from all the school districts.
00:32:18.720 --> 00:32:29.279
So part of that is the larger sort of monopoly conversation where these companies are so big and they've pursued this model of AI development that is so resource intensive that it's very hard to compete.
00:32:29.359 --> 00:32:30.400
It's kind of crazy.
00:32:30.559 --> 00:32:33.279
Just, you know, we were talking about this how AI isn't inevitable.
00:32:33.359 --> 00:32:43.039
It's like kind of incredible when you step back and say, you know, these companies, especially the newer labs like OpenAI and Anthropic, have poured billions of dollars into data centers.
00:32:43.279 --> 00:32:51.039
They've stolen so much data to train their models, they are sucking up
so much natural resources to build these systems.
00:32:51.200 --> 00:32:55.119
And the craziest thing about all of that is that they're losing billions of dollars per year.
00:32:55.279 --> 00:33:00.000
They've done all of this and they've gotten such favorable regulatory environments and they're still not making a profit.
00:33:00.160 --> 00:33:01.839
It's it's really remarkable.
00:33:02.000 --> 00:33:07.279
And so, again, like it's not clear that this is gonna be around, at least in this manifestation in a few years.
00:33:07.440 --> 00:33:16.400
So I think more fuel to the fire of fighting back and really seeing that this is not inevitable because it's really not clear that this is even just economically viable for these organizations.
00:33:16.640 --> 00:33:23.200
We can kind of say it's clearly not socially viable and environmentally viable, but it's it's probably not even an economically viable.
00:33:23.359 --> 00:33:25.440
So that that foundation is not so strong.
00:33:25.759 --> 00:33:26.559
You're too kind.
00:33:26.640 --> 00:33:28.559
I would say we don't know if they're going to be in the future.
00:33:28.640 --> 00:33:30.000
I dare to say that they're not.
00:33:30.079 --> 00:33:35.279
And in five years or 10 years, I would love to re listen to this podcast episode to see if I was right or not.
00:33:35.519 --> 00:33:36.480
It is not viable.
00:33:36.640 --> 00:33:37.519
Like it's impossible.
00:33:37.680 --> 00:33:45.599
Like, I definitely don't think that generative AI is the future, and I do think that the future will be narrow AI with artificial intelligence models that actually help people.
00:33:45.759 --> 00:33:53.920
So I am sure that many data scientists will be listening to this podcast today and they will listen to this and say, okay, now I understand things could be different.
00:33:54.079 --> 00:33:55.680
I learned this, but where do I start?
00:33:55.839 --> 00:33:56.480
Like, what can I do?
00:33:56.640 --> 00:34:03.039
What are maybe the three things that they can do if they want to make sure that their work is not that narrow and more ambitious?
00:34:03.200 --> 00:34:03.680
What can they do?
00:34:03.920 --> 00:34:15.599
If I could pick out a few things that kind of span a couple of different categories, I think on like a sort of methodological level, one big piece is just spend time assessing the social impacts of your systems.
00:34:15.840 --> 00:34:20.400
Actually go out and really think about like what is this doing in the world?
00:34:20.559 --> 00:34:37.199
Whether you're providing a tool for a client that is a company, whether you're building a system to improve healthcare or anything else, actually run a study or do an evaluation that says, does the system actually affect the social outcome that you care about?
00:34:37.360 --> 00:34:41.440
I'd say another broad piece is just talking to people from other fields, right?
00:34:41.519 --> 00:34:54.960
Whether that's like the stakeholders who might be the users of your system, but also the people who are going to be affected by it and really trying to understand their understanding of the world and what they're actually looking for and ask a lot of questions.
00:34:55.119 --> 00:35:05.039
I think it's not just like go talk to people, but but ask questions and really recognize the sort of lack of knowledge that you often have as a data scientist about a specific domain.
00:35:05.199 --> 00:35:13.119
But the sort of like broad thing, maybe finally, I would say is don't view this as an all or nothing practice.
00:35:13.280 --> 00:35:16.639
I sort of described earlier like this whole framework and sequence of steps.
00:35:16.800 --> 00:35:21.039
Don't view it as I need to go from current practice to doing all of that at once.
00:35:21.119 --> 00:35:23.840
Like you can take this as a one step at a time sort of thing.
00:35:24.000 --> 00:35:29.599
Don't expect to be able to just think about data science in an entirely new way in a week.
00:35:29.760 --> 00:35:31.039
It's going to take time.
00:35:31.119 --> 00:35:39.679
And you may need to think about how do I do more good in my job or how do I align my career to have more of a social good focus within it.
00:35:39.840 --> 00:35:56.880
So don't feel intimidated or don't feel this is impossible, but instead be like, okay, let me take one chunk of this that feels accessible and just focus on doing that for a bit and then move on and do the next thing and kind of just view this as a gradual process.
00:35:57.199 --> 00:35:59.360
That's lovely because change has to be sustainable.
00:35:59.519 --> 00:36:10.159
And many times you want to change everything so fast that we just get overwhelmed, change a little bit of the things that you're doing, and then slowly and gradually you're going to get to a place that you're looking at the bigger picture and asking questions.
00:36:10.320 --> 00:36:11.519
So, so super important.
00:36:11.920 --> 00:36:29.599
The other thing is that for everyday people who are listening, because I do think that I have more everyday people followers than data scientists, as a data scientist yourself and someone who has studied this in so many different angles, what do you think everyday people can do in their day-to-day to protect themselves from this algorithmic harm that is happening systemically around the world?
00:36:29.840 --> 00:36:33.119
I think it's, you know, it's often difficult as an individual to do this, right?
00:36:33.280 --> 00:36:41.360
I mean, I think one thing that I would say is just to be very careful about one's own use of, you know, generative AI tools, right?
00:36:41.440 --> 00:36:44.559
And we were talking earlier about, you know, the challenges of human in the loop.
00:36:44.639 --> 00:36:45.840
Like that applies to everyone.
00:36:46.000 --> 00:36:54.800
It is very easy to overestimate one's own ability to use AI without deferring to it and offloading critical thinking.
00:36:54.960 --> 00:37:15.599
And I,
in my own work, I'm very hesitant to use AI for anything that I view as like substantively important to my research or thinking because I guess just have like a hard-won humility where I've seen so much evidence that shows even experts, even people who know that you're gonna defer to a system, still do it.
00:37:15.679 --> 00:37:27.280
And so, you know, I'm very careful about that and would say to listeners, like, be very careful about sort of doing the easy use of these AI tools and assuming that you won't defer to it or you won't offload critical thinking.
00:37:27.440 --> 00:37:28.159
So that's a big piece.
00:37:28.320 --> 00:37:32.159
Then the other, the other element is just find a way to organize around this.
00:37:32.239 --> 00:37:51.280
So find ways to get involved in ideally a local fight, whether it's a data center in your community or the public school system or, you know, in your workplace, collaborating with your with your colleagues to push back on that or create better guardrails around AI or surveillance technologies are being used.
00:37:51.440 --> 00:37:58.559
And, you know, we're seeing now that AI regulation is a major part of conversation for electoral candidates.
00:37:58.639 --> 00:38:09.280
And so even beyond like a specific AI fight, you know, supporting candidates who are critical of AI and want to push back on the industry rather than just grease the wheels for the industry is gonna be important.
00:38:09.440 --> 00:38:17.199
So there's a lot of things to do, but they're gonna be collective political responses rather than a sort of like individual checklist.
00:38:17.599 --> 00:38:18.800
We are stronger together.
00:38:18.960 --> 00:38:22.480
We love to say this in the podcast because at the end of the day, this is harming all of us.
00:38:22.639 --> 00:38:31.599
And if we want to change things and if we want to fight big tech because they're billionaires and they have so much money, we have to get together because that's the only way that we will be able to fight them.
00:38:31.840 --> 00:38:37.039
Last question is there anything else that you would like everyday people to know about that we have not discussed?
00:38:37.280 --> 00:38:41.440
I think one thing that's important to say is just like AI.
00:38:41.599 --> 00:38:50.639
We should even just be careful in how we use AI because it's such a broad term and it refers to so many different types of systems that it can often be worth unpacking.
00:38:50.719 --> 00:38:52.880
Like, well, what kind of AI are we talking about?
00:38:53.039 --> 00:39:02.639
You know, everything from a logistic regression that predicts whether it's gonna rain tomorrow to Chat GPT, to an AI, like that's all AI.
00:39:02.800 --> 00:39:10.239
And so these types of systems or these different types of methodologies and tools have different capabilities and benefits and risks.
00:39:10.400 --> 00:39:20.239
But when we talk about AI in this broad way, it also can limit our ability to have that conversation about directionality because it feels very binary if we just talk about AI.
00:39:20.400 --> 00:39:27.679
Whereas if we break that down and say, well, there's these different types of AI and different ways of building AI, and we did mention, you know, narrow versus general AI is one category.
00:39:27.840 --> 00:39:42.960
But if once you sort of break down, you can say, oh, well, here's a component of the field of AI that is more promising and we can pursue, whereas here's this other type of system that, you know, is more worth being concerned about and we should try to slow down or we should try to move away from that development.
00:39:43.119 --> 00:39:56.559
So I'd say just generally for all of us, but especially as a consumer of news or policy debates, when you hear AI being discussed in a broad umbrella way, you know, try to dig in and say, what kind of AI are they talking about?
00:39:56.639 --> 00:40:03.840
And would this conversation be stronger and more precise if we actually broke this down into those different categories?
00:40:04.159 --> 00:40:08.239
This is such good advice because one, we have to criticize the right things.
00:40:08.320 --> 00:40:16.960
And the second thing is because if we criticize the whole thing and we say we hate AI, again, we're narrowing our minds to which the things that we could be doing with artificial intelligence.
00:40:17.119 --> 00:40:19.280
You don't hate AI, you hate one part of AI.
00:40:19.440 --> 00:40:24.480
Let's think about the wonderful other ways that we could create artificial intelligence that is good for you, me, and everyone else.
00:40:24.639 --> 00:40:27.199
So that is a great way to finish the podcast.
00:40:27.280 --> 00:40:31.519
I think it's pretty clear for everyone that data science is not neutral whatsoever.
00:40:31.599 --> 00:40:43.280
And it is so important for us to think about data science in terms of the harms that it does for everyday people and what we can do in the process to make sure that we're thinking about those questions at the start and not at the end when people are harmed.
00:40:43.360 --> 00:40:45.119
So thank you so much for sharing that with us.
00:40:45.280 --> 00:40:46.239
Yeah, thank you so much.
00:40:46.320 --> 00:40:47.760
This is a really fun conversation.
00:40:47.840 --> 00:40:48.800
I enjoyed it a lot.
00:40:49.039 --> 00:40:49.519
Thank you.
00:40:49.679 --> 00:40:59.519
So if you have stayed until the end and you listened to our conversation until now, the emoji for today will be the headphones, just because Ben is using really cool headphones today.
00:40:59.679 --> 00:41:04.239
So if you listen until the end and you heard this, what I'm saying, don't forget to comment the headphone emoji.
00:41:04.320 --> 00:41:09.039
With that being said, it was a pleasure having you all today in Box Don't Cry, and I'll see you next Tuesday for another episode.
00:41:09.199 --> 00:41:10.239
Bye bye, everyone.