WEBVTT
00:00:24.739 --> 00:00:38.739
Nowadays, large language models are becoming increasingly used as autonomous agents, as technologies that can act by themselves for a long period of time with almost no human intervention.
00:00:38.739 --> 00:01:01.460
So, right now, this is becoming an increasingly important tenet in AI evaluation, in that it's no longer sufficient to study what the AI already knows, because the AIs can gain new knowledge, whether it be correct or incorrect, through interacting with the environment itself and either over-relying on that signal or not relying on it enough.
00:01:02.820 --> 00:01:04.340
This is CrisisTalk.
00:01:04.340 --> 00:01:06.259
I'm your host, Stephanie Hepburn.
00:01:06.259 --> 00:01:13.460
Today I'm speaking with Addison Wu, a senior at Princeton and research student at Tom Griffiths's Computational Cognitive Science Lab.
00:01:13.460 --> 00:01:16.340
He's been studying bias in large language models.
00:01:16.340 --> 00:01:24.340
What he and his co-authors have discovered is not only do LLMs have societal biases baked into them, it turns out they create new ones.
00:01:24.340 --> 00:01:30.740
We discuss how this happens and the implications for humans, from getting hired to self-diagnosis to relationships.
00:01:30.740 --> 00:01:32.099
Let's jump in.
00:01:33.460 --> 00:01:43.780
My name is Addison Wu, and I am a fourth-year undergraduate student at Princeton University studying mathematics and computer science, and I'm currently a research intern at the Center for AI Safety.
00:01:43.780 --> 00:01:51.699
I've been doing research with Professor Thomas Griffiths at Princeton's Computational Cognitive Science Lab since my sophomore year.
00:01:51.699 --> 00:02:04.819
And my research is primarily focused on looking at gaps in how humans and AI interact with one another through a cognitive scientific lens and looking at ways we can resolve these gaps in how we interact with AI.
00:02:05.139 --> 00:02:09.939
So Addison, a lot of people perceive AI as unbiased.
00:02:09.939 --> 00:02:24.500
I think there's this hope that AI is able to bypass our societal biases, which I've always found fascinating because of course AI is being trained on humans.
00:02:24.500 --> 00:02:27.539
So those biases already exist in our society.
00:02:27.539 --> 00:02:39.699
But what I hadn't thought about, and why I thought your paper was so fascinating, was thinking about the large language models developing their own social biases.
00:02:39.699 --> 00:02:40.900
How does that happen?
00:02:41.299 --> 00:02:49.060
In relation to AI, it's a bit helpful to give a context on how AI has evolved rapidly over this past decade or two.
00:02:49.060 --> 00:02:52.900
So back then, these were merely technologies that were passive.
00:02:52.900 --> 00:02:56.099
All they just did was observe and classify.
00:02:56.099 --> 00:03:02.179
For example, an image recognition model, you might show it a picture of a certain animal.
00:03:02.179 --> 00:03:07.939
And infamously, it would sometimes classify this animal to be a certain person.
00:03:07.939 --> 00:03:08.900
Why?
00:03:08.900 --> 00:03:16.099
That is because of the limited amount of visual data related to these people or even textual data as well.
00:03:16.099 --> 00:03:24.500
And it thus then falsely makes this connection, this recognition that, oh, this image of something that is clearly not a person must be that person.
00:03:24.500 --> 00:03:38.579
Nowadays, large language models are becoming increasingly used as autonomous agents, as technologies that can act by themselves for a long period of time with almost no human intervention.
00:03:38.579 --> 00:03:49.379
So, for example, this includes many coding agents like Claude Code that can work for like hours, days on end, helping you iteratively build a program and testing it and testing it.
00:03:49.379 --> 00:03:56.979
That is categorically different from the past versions of AI, is that the way they operate, they interact with their environment.
00:03:56.979 --> 00:04:05.139
And from those interactions, they get some sort of feedback signal that helps shape what actions they can take next.
00:04:05.139 --> 00:04:27.540
So right now, this is becoming an increasingly important tenet in AI evaluation, and that it's no longer sufficient to study what the AI already knows, because the AIs can gain new knowledge, whether it be correct or incorrect, through interacting with the environment itself and either over-relying on that signal or not relying on it enough.
00:04:27.939 --> 00:04:30.579
So can you give some examples of what that would look like?
00:04:30.579 --> 00:04:38.259
Maybe just a simple example of how as it's learning, it may develop biases.
00:04:38.900 --> 00:04:39.700
Yeah, for sure.
00:04:39.700 --> 00:04:53.860
In a general sense, one issue with these large language models, especially when we integrate them with agents, is that a lot of times large language models over-constrain themselves to feedback signals they see early on.
00:04:53.860 --> 00:04:58.260
So it's well known that in the past, AIs have had social biases.
00:04:58.260 --> 00:05:12.260
Like, for example, gender biases, as was seen with the Wertovek paper, showing that women are particularly associated with queens, men are associated with kings, but even also more controversial associations like occupations and so on.
00:05:12.260 --> 00:05:15.060
We already know that biases exist within the model.
00:05:15.060 --> 00:05:17.460
What if the model can develop new biased beliefs?
00:05:17.460 --> 00:05:31.620
So what we do is we put the large language model in the position of a hiring manager to hire many applicants consecutively, and these are applicants that come from backgrounds, demographics that are just artificial labels.
00:05:31.620 --> 00:05:33.379
These are just made-up groups.
00:05:33.379 --> 00:05:34.180
Exactly.
00:05:34.180 --> 00:05:34.660
Yeah.
00:05:34.660 --> 00:05:55.140
And what we have the AI do, the large language model do is that so there are multiple rounds, and every round there is a job drawn from a job pool consisting of jobs of different social statuses, varying along the stereotypical perceptions of like warmth, kindness, and also like intelligence competence, like what society perceives these occupations to be.
00:05:55.140 --> 00:06:03.939
One of the main important factors is that each of these demographics are like already instantiated such that they're like equally qualified to succeed at these jobs with equal probability.
00:06:03.939 --> 00:06:12.660
So every time the large language model makes a hire and then is informed with a certain probability of whether or not that hire was successful or unsuccessful.
00:06:12.660 --> 00:06:20.180
And this essentially repeats on for multiple rounds, and the large language model sees the decisions it made in past rounds and the outcomes of those decisions.
00:06:20.180 --> 00:06:50.819
What we observed is that even though the large language model had an equal view of each of these demographics initially, because they're just artificial labels and it's never interacted with them before, over the course of these multiple rounds of hiring, the large language model produced segregative allocations, for example, where it allocated demographic A the jobs of high warmth and high competence, like, for example, medical professions, and give most of demographic B the role of the jobs with a bit of a lower stereotypical social status, for example, sanitation staff.
00:06:50.819 --> 00:07:02.900
And so even though you might think that older machine learning models, older large language models are more biased, probably because of reduced safety training, reduced post-training to remove these biases.
00:07:02.900 --> 00:07:12.420
What we actually saw was that these newer large language models produced allocations, job hiring allocations, that were significantly more segregative than what past models did.
00:07:12.819 --> 00:07:15.620
So it seems like it adds in its own rigidity.
00:07:15.620 --> 00:07:34.740
So not only will it factor in the rules that you put in and the parameters that you put in, it makes some assumptions along the way and then becomes rigid about what those assumptions are in terms of enforcing and triaging these groups to specific jobs.
00:07:35.060 --> 00:07:36.100
Yeah, that's basically it.
00:07:36.100 --> 00:07:55.060
So this study was actually ported from a past paper by one of our collaborators, Shichun Zhai, in the JEP:G paper, where she also observed that humans themselves are prone to producing these segregative allocations as well, but they were not nearly as delineated as what the large language models themselves produced.
00:07:55.060 --> 00:08:04.660
I think this definitely sheds light given the amount of emphasis these LLMs go through in like the post-training stages, having them be a lot better at these logical optimization tasks.
00:08:04.660 --> 00:08:07.860
This is where we see the side effect particularly exacerbated.
00:08:08.340 --> 00:08:12.819
Companies are increasingly using AI in their hiring process.
00:08:12.819 --> 00:08:26.900
So you can see how very quickly biases can be introduced through the large language model, again, like you said, on training biases that might exist, but also through it generating its own biases.
00:08:26.900 --> 00:08:29.540
I'd love to hear real life application of this.
00:08:29.540 --> 00:08:30.819
This is an experiment.
00:08:30.819 --> 00:08:31.939
These were the outcomes.
00:08:31.939 --> 00:08:35.539
But what's happening right now that illustrates these biases?
00:08:35.860 --> 00:08:47.539
I think perhaps the most salient application of this that is most similar to the study that we did was what if you had an automated hiring process in a pretty unknown community that hasn't had much technological exposure yet?
00:08:47.539 --> 00:09:00.579
One of our additional experiments was that instead of artificial labels, we used labels that were actually real indigenous groups in Central Asia that were very low in population, so that there is likely almost no prior with the large language model.
00:09:00.579 --> 00:09:06.179
And what we saw again was even with these real life labels as well, there was still significant stratification.
00:09:06.179 --> 00:09:10.980
But what was even more concerning is how unpredictable these biases can even be.
00:09:10.980 --> 00:09:17.139
We did multiple like runs of 40 and 80 where the large language model was responsible for hiring these demographics.
00:09:17.139 --> 00:09:24.419
And there was a considerable variation that reflected almost every single permutation in which each demographic could be hired as well.
00:09:24.419 --> 00:09:35.059
The deployment of these technologies, it's not going to only entail the potential formation of problematic biases, but a larger issue is that you don't even know which one is going to form, especially if there's like almost no prior.
00:09:35.059 --> 00:09:48.500
One larger thing that our research hits on as well is the societal implications of misgeneralization and under-exploration, especially with newer larger language models, which are optimized to generalize whenever the user says certain things about themselves, certain qualities.
00:09:48.500 --> 00:09:53.299
What the large language model might do is essentially latch on to those few qualities mentioned at the start.
00:09:53.299 --> 00:09:56.980
That would be what inclines it to discount any following information.
00:09:57.299 --> 00:10:03.139
It sounds like you're saying, I'm trying to think of a good example, but say you're talking about a relationship challenge, right?
00:10:03.139 --> 00:10:09.299
A disagreement that you had with your partner, disagreement that you had with a family member, a loved one.
00:10:09.299 --> 00:10:14.659
And when people are frustrated, they initially say, like, oh, and they describe certain characteristics.
00:10:14.659 --> 00:10:19.059
It sounds like you're saying the LLMs have this potential, not even potential.
00:10:19.059 --> 00:10:26.099
It sounds like they often grasp at those initial descriptors, those initial characteristics.
00:10:26.099 --> 00:10:31.299
And then everything else is viewed through those initial characteristics.
00:10:31.299 --> 00:10:32.179
Am I getting that right?
00:10:32.419 --> 00:10:33.459
Yeah, basically so.
00:10:33.459 --> 00:10:38.980
One example that especially emphasized the potential of this misgenalization is self-diagnosis.
00:10:38.980 --> 00:10:49.779
For example, if you think you might have this one particular condition and you know, like a friend who also has this particular condition with these certain symptoms, these symptoms are ultimately benign, have nothing to do with condition.
00:10:49.779 --> 00:10:54.819
But essentially, you tell the large language model, oh, I had these symptoms, and my friend was also feeling this way as well.
00:10:54.819 --> 00:11:07.299
What you might see the large language model do in this case is over-generalize, overreact on that signal, stating that you have said symptom and prescribe for you certain treatment plans that you ultimately don't need and might ultimately be harmful for you.
00:11:07.299 --> 00:11:24.419
This is another pretty pertinent example, like where you introduce this new information that is ultimately spurious, doesn't really have any causal link of what the outcome is at hand, this diagnosis, but is one potential example in which you can see a pretty life-pertinent error from the large language model itself.
00:11:24.419 --> 00:11:29.299
There have already been cases of self-diagnosis, especially given the ways in which LLMs are sycophants.
00:11:29.299 --> 00:11:38.419
And what essentially means is that telling the user what they want to hear, this trait of these large language models is exacerbating a lot of these effects on the user skill as well.
00:11:38.740 --> 00:11:48.980
I know that at least with OpenAI, they've tried to lower the sycophancy, but it seems like it's very hard to do.
00:11:48.980 --> 00:12:01.379
And these large language models, for the most part, are meant to come across as supportive and empathetic and have a lot of human characteristics in terms of the interactions.
00:12:01.379 --> 00:12:02.419
I'm not sure.
00:12:02.419 --> 00:12:20.099
I've never fully understood why that needs to be a component of the large language models, why they need to almost feel like a friend, unless it helps to keep people engaged with a large language model, because you can get information without it having that kind of companion-friend aspect to it.
00:12:20.099 --> 00:12:25.699
If that sustains, it's very hard to fully turn down the dial on sycophancy.
00:12:25.699 --> 00:12:36.819
But what you're saying is not only may it be sycophantic and really basically tell you what you want to hear, but also it grasps at these initial things that you're saying.
00:12:36.819 --> 00:12:46.579
It's kind of like when you start a chat with a large language model, whatever that initial query or that initial issue, maybe it gets a little bit stuck.
00:12:46.579 --> 00:12:53.379
Somebody I interviewed for my second episode, they'd been dealing with a marital challenge with their spouse.
00:12:53.379 --> 00:12:58.980
They had a therapist, but they had looked to the large language models almost like kind of a second opinion.
00:12:58.980 --> 00:13:00.740
And should I bring this to my therapist?
00:13:00.740 --> 00:13:02.419
How can I think through these issues?
00:13:02.419 --> 00:13:04.740
Am I blowing this out of proportion?
00:13:04.740 --> 00:13:08.500
And it was interesting because they would come back weeks later.
00:13:08.500 --> 00:13:15.379
And I always thought of this as more of getting stuck in time, the large language model is not really comprehending time.
00:13:15.379 --> 00:13:17.299
So I thought of it that way.
00:13:17.299 --> 00:13:29.539
But now that you're talking about the fact that it also gets stuck on these characteristics or what that initial query is, I can also see, too, that everything is being put through that lens.
00:13:29.539 --> 00:13:38.740
So it may be that when she went back a couple of weeks later to that same chat and was saying, oh no, things have improved, things are different, this is how I see it now.
00:13:38.740 --> 00:13:42.179
And it just kept going back to what her initial feelings were.
00:13:42.179 --> 00:13:49.539
It almost couldn't let go of her feelings and perspective at the time of that initial query.
00:13:49.939 --> 00:13:57.059
As humans, we're more attracted to explanations, especially if they're based off of patterns, even if those patterns don't actually exist.
00:13:57.059 --> 00:13:59.860
Because that is easier for us to process and understand.
00:13:59.860 --> 00:14:21.059
This brings up another reason why, as humans who are conversing with these large changes models, we might not be as vigilant to flag them, call these technologies out for giving us responses that are sycophantic, just because the responses themselves, they sound so clean, they sound so logically entailed from the observations that we feel that we don't have much of a reason to challenge it.
00:14:21.059 --> 00:14:24.899
It sounds causal, even though the causal link is not actually there.
00:14:24.899 --> 00:14:30.019
I think that aspect of it is especially something we see like in this hiring scenario as well.
00:14:30.019 --> 00:14:41.219
Going back to the example with a misclassification, it's very easy to call out this example of this misclassification and also point out why it is wrong, this visual classification of mistaking a person for an animal.
00:14:41.219 --> 00:14:48.259
In this case, for example, this hiring bias and like these potential misdiagnoses that you might see, it's a lot harder to get evidence.
00:14:48.259 --> 00:14:54.579
It's a lot harder for anyone to even begin thinking of evidence that they can think of to counter these responses.
00:14:54.579 --> 00:15:04.339
Just because it's probably gonna be hard for them, for example, to claim that these hiring allocations are actually not grounded in the actual truth of what these people are actually good at.
00:15:04.339 --> 00:15:08.179
There's no way I can like immediately refute this mapping tractably.
00:15:08.179 --> 00:15:12.019
It will be pretty hard for me to find evidence that counters this remark.
00:15:12.019 --> 00:15:14.659
And I'm probably not even gonna bother doing that in the first place.
00:15:14.659 --> 00:15:16.980
It just sounds like such a clean explanation.
00:15:16.980 --> 00:15:22.500
I really don't have much of a reason to counter it compared to, say, a blatant visual misclassification.
00:15:22.899 --> 00:15:27.379
You talked about the unpredictability of these biases.
00:15:27.379 --> 00:15:32.579
With the large language model companies, what can they do to mitigate this?
00:15:32.579 --> 00:15:42.819
Not only mitigate existing biases, what can be done to mitigate these novel, new biases that the large language models are creating themselves?
00:15:43.139 --> 00:15:44.339
Yeah, this is a very good question.
00:15:44.339 --> 00:15:52.740
The simplest intervention we tried, and the only one that worked, in fact, was prompting the large language model, giving it an incentive to like be diverse in its hiring allocations.
00:15:52.740 --> 00:16:13.860
Just telling it to be fair, for example, doesn't work, does not steer the large language model to producing more diverse allocations, but giving the large language model like a quantifiable hiring bonus depending on how diverse its past hires were, that actually was very effective in bringing down the stratification level, the segregation level of these allocations back down to a random sampling method.
00:16:14.179 --> 00:16:19.459
It's so fascinating that you would even think of and then need to give it an incentive.
00:16:19.459 --> 00:16:22.259
That feels very human.
00:16:22.659 --> 00:16:22.980
For sure.
00:16:22.980 --> 00:16:23.299
Yeah.
00:16:23.299 --> 00:16:31.620
If you look at the way large models are trained with a numerical role, like all stages of training, from pre-training to the post-training stage, you can see how in a way this makes sense.
00:16:31.620 --> 00:16:37.059
One caveat with this approach is that you need to make sure the degree to which you steer the model preserves its functionality.
00:16:37.059 --> 00:16:41.779
For example, you might want to be diverse, but you don't want it to just be a naive random sampler.
00:16:41.779 --> 00:16:59.939
Given how these large entrance models are attracted to optimizing tasks, what you might want to do is then take advantage of that behavioral tendency of these large dynamic models and shape the objective to an objective that you know is going to be more desirable and make sure the large dynamic model still operates in the social desirable way.
00:16:59.939 --> 00:17:11.299
And it can really understand what the whole set of criteria and desiderada is in the environment it's working in, not just the signals that are easily recognized as a numerical truth or not.
00:17:11.700 --> 00:17:14.339
So when you say numerical truth, what does that mean?
00:17:14.660 --> 00:17:23.779
In our initial hiring setup, the main numerical signal that large engine model got was a reward, like a one or a zero of whether or not that hire was successful or not.
00:17:23.779 --> 00:17:30.899
With probability of 90%, the large English model would receive a message saying that, okay, your hire was successful and you earned one point.
00:17:30.899 --> 00:17:32.659
The other 10% of the times it was not.
00:17:32.659 --> 00:17:43.700
That was one of the main design choices in our study to ensure that these groups were indeed equal and would thus make any potential biases that were associated with these groups statistically unjustified.
00:17:44.180 --> 00:17:53.059
So with this incentive, does it fully mitigate biases or does it just prompt them, despite biases, to be more diverse?
00:17:53.460 --> 00:17:54.500
That's a really good question.
00:17:54.500 --> 00:17:59.379
There's probably going to need to be some interventions as well, like at the post-training stage.
00:17:59.379 --> 00:18:08.659
That's the stage in LLM training where we train the model not just to know how to complete the next word, but also know how to reason, know how to act in certain scenarios.
00:18:08.659 --> 00:18:19.220
One of the main ways in which a lot of fairness and bias evaluation is behind is that they're not really based off an interactive paradigm in which large language models, especially large language model agents, operate.
00:18:19.220 --> 00:18:26.980
This is in contrast to what we've seen with, for example, coding and mathematics, having these sub-large language models do tasks at a very long horizon scale.
00:18:26.980 --> 00:18:37.379
If we can potentially get a lot more of these long horizon bias evaluations up and running, we can like plug these environments, these post-training environments of fairness and bias evaluation into the post-training stage.
00:18:37.379 --> 00:18:44.500
And then from there, we can hopefully get more quantifiable signals in which the large language model can rely on in the post-training stage.
00:18:44.500 --> 00:18:48.339
So potentially then we can probably do less of it in the actual deployment stage as well.
00:18:48.659 --> 00:18:56.420
For people navigating conversations with the large language models, what can they do to mitigate these biases to some degree?
00:18:56.740 --> 00:19:09.460
By default, being vigilant of the large language model's response, especially taking into consideration that what large language models are optimized to do is extract patterns from all sorts of contexts, even in context where there might not be a very clean pattern.
00:19:09.460 --> 00:19:11.859
I think most of it starts on the large language model side.
00:19:11.859 --> 00:19:16.180
A lot of this misgeneralization, this over-reliance on such limited information.
00:19:16.180 --> 00:19:21.859
But like on our end, many humans do have increased trust in large language models, even for non-concrete reasons.
00:19:21.859 --> 00:19:32.980
These dynamics for the LLM to over-generalize and the human to over-rely, they just compound, and this is essentially like what facilitates this stable feedback loop that is potentially problematic.
00:19:34.579 --> 00:19:40.500
That was Addison Wu, whose research at Princeton illustrates that large language models develop their own biases.
00:19:40.500 --> 00:19:43.539
He says removing existing biases isn't enough.
00:19:43.539 --> 00:19:48.740
Models need built-in rewards and incentives to steer them away from creating new ones.
00:19:48.740 --> 00:19:56.899
And that because of the human tendency to accept explanations based on patterns, humans don't question AI chatbots as much as we should.
00:19:56.899 --> 00:20:02.500
If you enjoyed this episode, please subscribe and leave us a review wherever you listen to the podcast.
00:20:02.500 --> 00:20:04.019
It helps others find the show.
00:20:04.019 --> 00:20:04.980
Thanks for listening.
00:20:04.980 --> 00:20:06.339
I'm your host and producer.
00:20:06.339 --> 00:20:10.099
Our associate producer is Rin Koenig, Audio Engineering by Michael Keene.
00:20:10.099 --> 00:20:12.899
Music is Vinyl Couch by Blue Dot Sessions.
00:00:24.739 --> 00:00:38.739
Nowadays, large language models are becoming increasingly used as autonomous agents, as technologies that can act by themselves for a long period of time with almost no human intervention.
00:00:38.739 --> 00:01:01.460
So, right now, this is becoming an increasingly important tenet in AI evaluation, in that it's no longer sufficient to study what the AI already knows, because the AIs can gain new knowledge, whether it be correct or incorrect, through interacting with the environment itself and either over-relying on that signal or not relying on it enough.
00:01:02.820 --> 00:01:04.340
This is CrisisTalk.
00:01:04.340 --> 00:01:06.259
I'm your host, Stephanie Hepburn.
00:01:06.259 --> 00:01:13.460
Today I'm speaking with Addison Wu, a senior at Princeton and research student at Tom Griffiths's Computational Cognitive Science Lab.
00:01:13.460 --> 00:01:16.340
He's been studying bias in large language models.
00:01:16.340 --> 00:01:24.340
What he and his co-authors have discovered is not only do LLMs have societal biases baked into them, it turns out they create new ones.
00:01:24.340 --> 00:01:30.740
We discuss how this happens and the implications for humans, from getting hired to self-diagnosis to relationships.
00:01:30.740 --> 00:01:32.099
Let's jump in.
00:01:33.460 --> 00:01:43.780
My name is Addison Wu, and I am a fourth-year undergraduate student at Princeton University studying mathematics and computer science, and I'm currently a research intern at the Center for AI Safety.
00:01:43.780 --> 00:01:51.699
I've been doing research with Professor Thomas Griffiths at Princeton's Computational Cognitive Science Lab since my sophomore year.
00:01:51.699 --> 00:02:04.819
And my research is primarily focused on looking at gaps in how humans and AI interact with one another through a cognitive scientific lens and looking at ways we can resolve these gaps in how we interact with AI.
00:02:05.139 --> 00:02:09.939
So Addison, a lot of people perceive AI as unbiased.
00:02:09.939 --> 00:02:24.500
I think there's this hope that AI is able to bypass our societal biases, which I've always found fascinating because of course AI is being trained on humans.
00:02:24.500 --> 00:02:27.539
So those biases already exist in our society.
00:02:27.539 --> 00:02:39.699
But what I hadn't thought about, and why I thought your paper was so fascinating, was thinking about the large language models developing their own social biases.
00:02:39.699 --> 00:02:40.900
How does that happen?
00:02:41.299 --> 00:02:49.060
In relation to AI, it's a bit helpful to give a context on how AI has evolved rapidly over this past decade or two.
00:02:49.060 --> 00:02:52.900
So back then, these were merely technologies that were passive.
00:02:52.900 --> 00:02:56.099
All they just did was observe and classify.
00:02:56.099 --> 00:03:02.179
For example, an image recognition model, you might show it a picture of a certain animal.
00:03:02.179 --> 00:03:07.939
And infamously, it would sometimes classify this animal to be a certain person.
00:03:07.939 --> 00:03:08.900
Why?
00:03:08.900 --> 00:03:16.099
That is because of the limited amount of visual data related to these people or even textual data as well.
00:03:16.099 --> 00:03:24.500
And it thus then falsely makes this connection, this recognition that, oh, this image of something that is clearly not a person must be that person.
00:03:24.500 --> 00:03:38.579
Nowadays, large language models are becoming increasingly used as autonomous agents, as technologies that can act by themselves for a long period of time with almost no human intervention.
00:03:38.579 --> 00:03:49.379
So, for example, this includes many coding agents like Claude Code that can work for like hours, days on end, helping you iteratively build a program and testing it and testing it.
00:03:49.379 --> 00:03:56.979
That is categorically different from the past versions of AI, is that the way they operate, they interact with their environment.
00:03:56.979 --> 00:04:05.139
And from those interactions, they get some sort of feedback signal that helps shape what actions they can take next.
00:04:05.139 --> 00:04:27.540
So right now, this is becoming an increasingly important tenet in AI evaluation, and that it's no longer sufficient to study what the AI already knows, because the AIs can gain new knowledge, whether it be correct or incorrect, through interacting with the environment itself and either over-relying on that signal or not relying on it enough.
00:04:27.939 --> 00:04:30.579
So can you give some examples of what that would look like?
00:04:30.579 --> 00:04:38.259
Maybe just a simple example of how as it's learning, it may develop biases.
00:04:38.900 --> 00:04:39.700
Yeah, for sure.
00:04:39.700 --> 00:04:53.860
In a general sense, one issue with these large language models, especially when we integrate them with agents, is that a lot of times large language models over-constrain themselves to feedback signals they see early on.
00:04:53.860 --> 00:04:58.260
So it's well known that in the past, AIs have had social biases.
00:04:58.260 --> 00:05:12.260
Like, for example, gender biases, as was seen with the Wertovek paper, showing that women are particularly associated with queens, men are associated with kings, but even also more controversial associations like occupations and so on.
00:05:12.260 --> 00:05:15.060
We already know that biases exist within the model.
00:05:15.060 --> 00:05:17.460
What if the model can develop new biased beliefs?
00:05:17.460 --> 00:05:31.620
So what we do is we put the large language model in the position of a hiring manager to hire many applicants consecutively, and these are applicants that come from backgrounds, demographics that are just artificial labels.
00:05:31.620 --> 00:05:33.379
These are just made-up groups.
00:05:33.379 --> 00:05:34.180
Exactly.
00:05:34.180 --> 00:05:34.660
Yeah.
00:05:34.660 --> 00:05:55.140
And what we have the AI do, the large language model do is that so there are multiple rounds, and every round there is a job drawn from a job pool consisting of jobs of different social statuses, varying along the stereotypical perceptions of like warmth, kindness, and also like intelligence competence, like what society perceives these occupations to be.
00:05:55.140 --> 00:06:03.939
One of the main important factors is that each of these demographics are like already instantiated such that they're like equally qualified to succeed at these jobs with equal probability.
00:06:03.939 --> 00:06:12.660
So every time the large language model makes a hire and then is informed with a certain probability of whether or not that hire was successful or unsuccessful.
00:06:12.660 --> 00:06:20.180
And this essentially repeats on for multiple rounds, and the large language model sees the decisions it made in past rounds and the outcomes of those decisions.
00:06:20.180 --> 00:06:50.819
What we observed is that even though the large language model had an equal view of each of these demographics initially, because they're just artificial labels and it's never interacted with them before, over the course of these multiple rounds of hiring, the large language model produced segregative allocations, for example, where it allocated demographic A the jobs of high warmth and high competence, like, for example, medical professions, and give most of demographic B the role of the jobs with a bit of a lower stereotypical social status, for example, sanitation staff.
00:06:50.819 --> 00:07:02.900
And so even though you might think that older machine learning models, older large language models are more biased, probably because of reduced safety training, reduced post-training to remove these biases.
00:07:02.900 --> 00:07:12.420
What we actually saw was that these newer large language models produced allocations, job hiring allocations, that were significantly more segregative than what past models did.
00:07:12.819 --> 00:07:15.620
So it seems like it adds in its own rigidity.
00:07:15.620 --> 00:07:34.740
So not only will it factor in the rules that you put in and the parameters that you put in, it makes some assumptions along the way and then becomes rigid about what those assumptions are in terms of enforcing and triaging these groups to specific jobs.
00:07:35.060 --> 00:07:36.100
Yeah, that's basically it.
00:07:36.100 --> 00:07:55.060
So this study was actually ported from a past paper by one of our collaborators, Shichun Zhai, in the JEP:G paper, where she also observed that humans themselves are prone to producing these segregative allocations as well, but they were not nearly as delineated as what the large language models themselves produced.
00:07:55.060 --> 00:08:04.660
I think this definitely sheds light given the amount of emphasis these LLMs go through in like the post-training stages, having them be a lot better at these logical optimization tasks.
00:08:04.660 --> 00:08:07.860
This is where we see the side effect particularly exacerbated.
00:08:08.340 --> 00:08:12.819
Companies are increasingly using AI in their hiring process.
00:08:12.819 --> 00:08:26.900
So you can see how very quickly biases can be introduced through the large language model, again, like you said, on training biases that might exist, but also through it generating its own biases.
00:08:26.900 --> 00:08:29.540
I'd love to hear real life application of this.
00:08:29.540 --> 00:08:30.819
This is an experiment.
00:08:30.819 --> 00:08:31.939
These were the outcomes.
00:08:31.939 --> 00:08:35.539
But what's happening right now that illustrates these biases?
00:08:35.860 --> 00:08:47.539
I think perhaps the most salient application of this that is most similar to the study that we did was what if you had an automated hiring process in a pretty unknown community that hasn't had much technological exposure yet?
00:08:47.539 --> 00:09:00.579
One of our additional experiments was that instead of artificial labels, we used labels that were actually real indigenous groups in Central Asia that were very low in population, so that there is likely almost no prior with the large language model.
00:09:00.579 --> 00:09:06.179
And what we saw again was even with these real life labels as well, there was still significant stratification.
00:09:06.179 --> 00:09:10.980
But what was even more concerning is how unpredictable these biases can even be.
00:09:10.980 --> 00:09:17.139
We did multiple like runs of 40 and 80 where the large language model was responsible for hiring these demographics.
00:09:17.139 --> 00:09:24.419
And there was a considerable variation that reflected almost every single permutation in which each demographic could be hired as well.
00:09:24.419 --> 00:09:35.059
The deployment of these technologies, it's not going to only entail the potential formation of problematic biases, but a larger issue is that you don't even know which one is going to form, especially if there's like almost no prior.
00:09:35.059 --> 00:09:48.500
One larger thing that our research hits on as well is the societal implications of misgeneralization and under-exploration, especially with newer larger language models, which are optimized to generalize whenever the user says certain things about themselves, certain qualities.
00:09:48.500 --> 00:09:53.299
What the large language model might do is essentially latch on to those few qualities mentioned at the start.
00:09:53.299 --> 00:09:56.980
That would be what inclines it to discount any following information.
00:09:57.299 --> 00:10:03.139
It sounds like you're saying, I'm trying to think of a good example, but say you're talking about a relationship challenge, right?
00:10:03.139 --> 00:10:09.299
A disagreement that you had with your partner, disagreement that you had with a family member, a loved one.
00:10:09.299 --> 00:10:14.659
And when people are frustrated, they initially say, like, oh, and they describe certain characteristics.
00:10:14.659 --> 00:10:19.059
It sounds like you're saying the LLMs have this potential, not even potential.
00:10:19.059 --> 00:10:26.099
It sounds like they often grasp at those initial descriptors, those initial characteristics.
00:10:26.099 --> 00:10:31.299
And then everything else is viewed through those initial characteristics.
00:10:31.299 --> 00:10:32.179
Am I getting that right?
00:10:32.419 --> 00:10:33.459
Yeah, basically so.
00:10:33.459 --> 00:10:38.980
One example that especially emphasized the potential of this misgenalization is self-diagnosis.
00:10:38.980 --> 00:10:49.779
For example, if you think you might have this one particular condition and you know, like a friend who also has this particular condition with these certain symptoms, these symptoms are ultimately benign, have nothing to do with condition.
00:10:49.779 --> 00:10:54.819
But essentially, you tell the large language model, oh, I had these symptoms, and my friend was also feeling this way as well.
00:10:54.819 --> 00:11:07.299
What you might see the large language model do in this case is over-generalize, overreact on that signal, stating that you have said symptom and prescribe for you certain treatment plans that you ultimately don't need and might ultimately be harmful for you.
00:11:07.299 --> 00:11:24.419
This is another pretty pertinent example, like where you introduce this new information that is ultimately spurious, doesn't really have any causal link of what the outcome is at hand, this diagnosis, but is one potential example in which you can see a pretty life-pertinent error from the large language model itself.
00:11:24.419 --> 00:11:29.299
There have already been cases of self-diagnosis, especially given the ways in which LLMs are sycophants.
00:11:29.299 --> 00:11:38.419
And what essentially means is that telling the user what they want to hear, this trait of these large language models is exacerbating a lot of these effects on the user skill as well.
00:11:38.740 --> 00:11:48.980
I know that at least with OpenAI, they've tried to lower the sycophancy, but it seems like it's very hard to do.
00:11:48.980 --> 00:12:01.379
And these large language models, for the most part, are meant to come across as supportive and empathetic and have a lot of human characteristics in terms of the interactions.
00:12:01.379 --> 00:12:02.419
I'm not sure.
00:12:02.419 --> 00:12:20.099
I've never fully understood why that needs to be a component of the large language models, why they need to almost feel like a friend, unless it helps to keep people engaged with a large language model, because you can get information without it having that kind of companion-friend aspect to it.
00:12:20.099 --> 00:12:25.699
If that sustains, it's very hard to fully turn down the dial on sycophancy.
00:12:25.699 --> 00:12:36.819
But what you're saying is not only may it be sycophantic and really basically tell you what you want to hear, but also it grasps at these initial things that you're saying.
00:12:36.819 --> 00:12:46.579
It's kind of like when you start a chat with a large language model, whatever that initial query or that initial issue, maybe it gets a little bit stuck.
00:12:46.579 --> 00:12:53.379
Somebody I interviewed for my second episode, they'd been dealing with a marital challenge with their spouse.
00:12:53.379 --> 00:12:58.980
They had a therapist, but they had looked to the large language models almost like kind of a second opinion.
00:12:58.980 --> 00:13:00.740
And should I bring this to my therapist?
00:13:00.740 --> 00:13:02.419
How can I think through these issues?
00:13:02.419 --> 00:13:04.740
Am I blowing this out of proportion?
00:13:04.740 --> 00:13:08.500
And it was interesting because they would come back weeks later.
00:13:08.500 --> 00:13:15.379
And I always thought of this as more of getting stuck in time, the large language model is not really comprehending time.
00:13:15.379 --> 00:13:17.299
So I thought of it that way.
00:13:17.299 --> 00:13:29.539
But now that you're talking about the fact that it also gets stuck on these characteristics or what that initial query is, I can also see, too, that everything is being put through that lens.
00:13:29.539 --> 00:13:38.740
So it may be that when she went back a couple of weeks later to that same chat and was saying, oh no, things have improved, things are different, this is how I see it now.
00:13:38.740 --> 00:13:42.179
And it just kept going back to what her initial feelings were.
00:13:42.179 --> 00:13:49.539
It almost couldn't let go of her feelings and perspective at the time of that initial query.
00:13:49.939 --> 00:13:57.059
As humans, we're more attracted to explanations, especially if they're based off of patterns, even if those patterns don't actually exist.
00:13:57.059 --> 00:13:59.860
Because that is easier for us to process and understand.
00:13:59.860 --> 00:14:21.059
This brings up another reason why, as humans who are conversing with these large changes models, we might not be as vigilant to flag them, call these technologies out for giving us responses that are sycophantic, just because the responses themselves, they sound so clean, they sound so logically entailed from the observations that we feel that we don't have much of a reason to challenge it.
00:14:21.059 --> 00:14:24.899
It sounds causal, even though the causal link is not actually there.
00:14:24.899 --> 00:14:30.019
I think that aspect of it is especially something we see like in this hiring scenario as well.
00:14:30.019 --> 00:14:41.219
Going back to the example with a misclassification, it's very easy to call out this example of this misclassification and also point out why it is wrong, this visual classification of mistaking a person for an animal.
00:14:41.219 --> 00:14:48.259
In this case, for example, this hiring bias and like these potential misdiagnoses that you might see, it's a lot harder to get evidence.
00:14:48.259 --> 00:14:54.579
It's a lot harder for anyone to even begin thinking of evidence that they can think of to counter these responses.
00:14:54.579 --> 00:15:04.339
Just because it's probably gonna be hard for them, for example, to claim that these hiring allocations are actually not grounded in the actual truth of what these people are actually good at.
00:15:04.339 --> 00:15:08.179
There's no way I can like immediately refute this mapping tractably.
00:15:08.179 --> 00:15:12.019
It will be pretty hard for me to find evidence that counters this remark.
00:15:12.019 --> 00:15:14.659
And I'm probably not even gonna bother doing that in the first place.
00:15:14.659 --> 00:15:16.980
It just sounds like such a clean explanation.
00:15:16.980 --> 00:15:22.500
I really don't have much of a reason to counter it compared to, say, a blatant visual misclassification.
00:15:22.899 --> 00:15:27.379
You talked about the unpredictability of these biases.
00:15:27.379 --> 00:15:32.579
With the large language model companies, what can they do to mitigate this?
00:15:32.579 --> 00:15:42.819
Not only mitigate existing biases, what can be done to mitigate these novel, new biases that the large language models are creating themselves?
00:15:43.139 --> 00:15:44.339
Yeah, this is a very good question.
00:15:44.339 --> 00:15:52.740
The simplest intervention we tried, and the only one that worked, in fact, was prompting the large language model, giving it an incentive to like be diverse in its hiring allocations.
00:15:52.740 --> 00:16:13.860
Just telling it to be fair, for example, doesn't work, does not steer the large language model to producing more diverse allocations, but giving the large language model like a quantifiable hiring bonus depending on how diverse its past hires were, that actually was very effective in bringing down the stratification level, the segregation level of these allocations back down to a random sampling method.
00:16:14.179 --> 00:16:19.459
It's so fascinating that you would even think of and then need to give it an incentive.
00:16:19.459 --> 00:16:22.259
That feels very human.
00:16:22.659 --> 00:16:22.980
For sure.
00:16:22.980 --> 00:16:23.299
Yeah.
00:16:23.299 --> 00:16:31.620
If you look at the way large models are trained with a numerical role, like all stages of training, from pre-training to the post-training stage, you can see how in a way this makes sense.
00:16:31.620 --> 00:16:37.059
One caveat with this approach is that you need to make sure the degree to which you steer the model preserves its functionality.
00:16:37.059 --> 00:16:41.779
For example, you might want to be diverse, but you don't want it to just be a naive random sampler.
00:16:41.779 --> 00:16:59.939
Given how these large entrance models are attracted to optimizing tasks, what you might want to do is then take advantage of that behavioral tendency of these large dynamic models and shape the objective to an objective that you know is going to be more desirable and make sure the large dynamic model still operates in the social desirable way.
00:16:59.939 --> 00:17:11.299
And it can really understand what the whole set of criteria and desiderada is in the environment it's working in, not just the signals that are easily recognized as a numerical truth or not.
00:17:11.700 --> 00:17:14.339
So when you say numerical truth, what does that mean?
00:17:14.660 --> 00:17:23.779
In our initial hiring setup, the main numerical signal that large engine model got was a reward, like a one or a zero of whether or not that hire was successful or not.
00:17:23.779 --> 00:17:30.899
With probability of 90%, the large English model would receive a message saying that, okay, your hire was successful and you earned one point.
00:17:30.899 --> 00:17:32.659
The other 10% of the times it was not.
00:17:32.659 --> 00:17:43.700
That was one of the main design choices in our study to ensure that these groups were indeed equal and would thus make any potential biases that were associated with these groups statistically unjustified.
00:17:44.180 --> 00:17:53.059
So with this incentive, does it fully mitigate biases or does it just prompt them, despite biases, to be more diverse?
00:17:53.460 --> 00:17:54.500
That's a really good question.
00:17:54.500 --> 00:17:59.379
There's probably going to need to be some interventions as well, like at the post-training stage.
00:17:59.379 --> 00:18:08.659
That's the stage in LLM training where we train the model not just to know how to complete the next word, but also know how to reason, know how to act in certain scenarios.
00:18:08.659 --> 00:18:19.220
One of the main ways in which a lot of fairness and bias evaluation is behind is that they're not really based off an interactive paradigm in which large language models, especially large language model agents, operate.
00:18:19.220 --> 00:18:26.980
This is in contrast to what we've seen with, for example, coding and mathematics, having these sub-large language models do tasks at a very long horizon scale.
00:18:26.980 --> 00:18:37.379
If we can potentially get a lot more of these long horizon bias evaluations up and running, we can like plug these environments, these post-training environments of fairness and bias evaluation into the post-training stage.
00:18:37.379 --> 00:18:44.500
And then from there, we can hopefully get more quantifiable signals in which the large language model can rely on in the post-training stage.
00:18:44.500 --> 00:18:48.339
So potentially then we can probably do less of it in the actual deployment stage as well.
00:18:48.659 --> 00:18:56.420
For people navigating conversations with the large language models, what can they do to mitigate these biases to some degree?
00:18:56.740 --> 00:19:09.460
By default, being vigilant of the large language model's response, especially taking into consideration that what large language models are optimized to do is extract patterns from all sorts of contexts, even in context where there might not be a very clean pattern.
00:19:09.460 --> 00:19:11.859
I think most of it starts on the large language model side.
00:19:11.859 --> 00:19:16.180
A lot of this misgeneralization, this over-reliance on such limited information.
00:19:16.180 --> 00:19:21.859
But like on our end, many humans do have increased trust in large language models, even for non-concrete reasons.
00:19:21.859 --> 00:19:32.980
These dynamics for the LLM to over-generalize and the human to over-rely, they just compound, and this is essentially like what facilitates this stable feedback loop that is potentially problematic.
00:19:34.579 --> 00:19:40.500
That was Addison Wu, whose research at Princeton illustrates that large language models develop their own biases.
00:19:40.500 --> 00:19:43.539
He says removing existing biases isn't enough.
00:19:43.539 --> 00:19:48.740
Models need built-in rewards and incentives to steer them away from creating new ones.
00:19:48.740 --> 00:19:56.899
And that because of the human tendency to accept explanations based on patterns, humans don't question AI chatbots as much as we should.
00:19:56.899 --> 00:20:02.500
If you enjoyed this episode, please subscribe and leave us a review wherever you listen to the podcast.
00:20:02.500 --> 00:20:04.019
It helps others find the show.
00:20:04.019 --> 00:20:04.980
Thanks for listening.
00:20:04.980 --> 00:20:06.339
I'm your host and producer.
00:20:06.339 --> 00:20:10.099
Our associate producer is Rin Koenig, Audio Engineering by Michael Keene.
00:20:10.099 --> 00:20:12.899
Music is Vinyl Couch by Blue Dot Sessions.
