Interview with Margaret Mitchell: Stochastic Parrots and the LLM Quandary

General Purpose Systems, Large Language Models (LLMs) and Agents are now the most debated topics today. They are touted as enabling significant progress since LLMs were introduced three years ago.

The irony of progress has meant hiccups—for these systems and the capabilities they produce, these hiccups are life threatening, economically, mentally, and personally. The hype peddled by frontier companies are distractions from these very real consequences. The power that comes from a handful of Big Tech companies, backed by the U.S. administration, shades them from any accountability.

The misdeeds of these systems are well documented from teen suicide, AI psychosis, addiction, misinformation, copyright infringement, the data center energy guzzling, and more recently, serious infrastructure security risks from agent attacks. The latest OpenAI agent unauthorized attack of Hugging Face has created a media firestorm that threatens existential threat to mankind and an AI that needs to be controlled.

Before that, the markets were convinced of extraordinary gains, and have been very optimistic despite the bubble talks, and circular financing. Now investors face a dual risk:

…the prospect that AI could pose an existential threat to humanity, and a possible slowdown championed by some of the industry's biggest names.

We’ve seen this scene play out before: In 2023 Future of Life’s Open Letter to pause AI development was signed by the who’s who of Silicon Valley, who called for regulatory oversight and development of governance standards. That didn’t happen.

The warnings that Margaret Mitchell, Timnit Gebru, Emily Bender and Angelina McMillan-Major detailed in their paper, “The Dangers of Stochastic Parrots: Can Language Models be too Big?” did not amount to effective legislative and AI governance progress to mitigate the risks they outlined until AFTER these harms had already played out.

Our discussion with Margaret Mitchell, a computer scientist and one of the earliest leaders in AI ethics, demystifies LLMs and their implications.

This event was done in collaboration with The AI Fellowship Toronto.

System Malfunction is a reader-supported publication. All my posts are currently free. My intention is to share my knowledge without paywalls. To receive new posts and support my work, consider becoming a subscriber.

Margaret Mitchell is a researcher and Chief Ethics Scientist at Hugging Face. She also co-authored, with Emily Bender, Timnit Gebru and Angelina McMillan-Major, “The Dangers of Stochastic Parrots: Can Language Models be too Big?”

This paper came out in 2021, before the mainstream introduction of Open AI’s ChatGPT. Mitchell shares that large language models have been in development over the past five-ten years. In the 1990’s language models, which were smaller, were being developed across all kinds of tasks. And she was part of this language modelling work, as she explains,

I did my thesis on generating language based on visual input, and this was using language models. And so, I was in a position in 2020 where I saw that the growth of large language models introduce a ton of different issues—and potential benefits—that were being overlooked [in the tech industry, people tend to focus on the positives.]

As someone who had been developing language models for a long time, I saw that they were about to take off—and from my specific position in the industry—without any academic writing on harms and risks.

Mitchell and her colleagues decided to document these harms and risks to develop some grounding as they evolved. What they recognized was the massive potential with language models becoming fluent, as she states,

Previously, language models could be used for small chunks of text. Now language models, because they were large, could capture much larger contexts, and then that means that they could generate entire essays that sounded fluent in ways we could never do before.

Once we started seeing that fluency, it was clear to us that people were going to start imbuing it with human-like intelligence.

She added that this new paradigm of LLMs was based on the massive scraping of internet data without consent(which she explains that she contributed to) and would surface legal and societal rights issues in its wake.

At the same time, Mitchell knew that this fluency could be used to generate things that looked factual, and attribute these systems as individual entities, extensions of our human likeness. She adds,

We use that as part of our own expressions of humanness… It was clear to me that people were going to love this, and they were not going to know the full picture about what these things are.

The paper argued that LLMs do not understand the “concepts underlying what they learn." It also cited significant risks when it came to LLM’s carbon emissions and financial costs as models are fed more data. Moreover, the disproportionate data capture that favored the western, more economically rich nations meant that smaller nations with less internet access would have far less representation.

This paper was the catalyst that saw Google executives respond negatively and demand retraction, which eventually led to her team co-lead, Timnit Gebru being forced out of Google.

Scraping Internet at Scale Does Not Equate to A Representation of the World

Before LLMs, smaller language models were developed from curated text data from licensing agreements with trusted institutions like The Wall Street Journal. The LLM paradigm shifted to massive scraping of internet data without consent, as she explains,

So, it went from the paradigm where you decide on the desirable data inputs, with appropriate licensing to a new recognition that with Web 2.0 the growth of blogs and social media, you could get tons of text, without any legal protection.

Going into 2019, we saw companies stop curating altogether, with the prevailing notion to retrieve whatever the Internet provided, as opposed to focusing on high-quality resources.

And so that created a situation where we were just all getting as much data as we possibly could, regardless of any other considerations, because it meant improving downstream benchmark performance and the perceived fluency.

The data being scraped was skewed to represent the viewpoints of people who were on the internet the most. ChatGPT tended to select from the US, Western Europe and parts of East Asia. This paper which introduces “silicon gaze” to explain how LLMs reproduce and amplify inequalities argued

bias is not a correctable anomaly but an intrinsic feature of generative AI, rooted in historically uneven data ecologies and design choices.

The structural features mean that training data has already been shaped by “centuries of uneven information production” giving advantage to English-language and stronger digital presence.

Mitchell adds that data ingested from sites like Wikipedia are populated by North American men, and by some estimates, between the ages of 17 and 35. Also among the top 10 cited domains on LLMs, Reddit, Linkedin, Medium, Youtube, Google, Forbes are largely U.S. centric, with audiences predominant within the Global North.

Mitchell adds that Global North hegemony—disproportionately represented in the data including misogynistic viewpoints from predominantly male users on Reddit, a narrow view of ‘worthy’ content published on Wikipedia, and influential figures across blogs and social networks—are what these systems are learning from, as she adds

Women's viewpoints are generally not well represented and the viewpoints of people who are non-white are generally not well represented.

AI is Not Neutral

In the Stochastic Parrots paper, Ruha Benjamin, Abebe Birane and Vinay Uday Prabhu were cited for their report where they highlighted issues with LLMs and AI systems that scrape massive web datasets like Common Crawl, containing toxicity, and hate speech, and argued that filtering or keyword blacklists are ineffective at identifying harmful content without inadvertently censoring or harming groups,

“Feeding AI systems on the world’s beauty, ugliness and cruelty but expecting it to reflect only the beauty is a fantasy.”

Mitchell explains this encoded bias that has resulted from the development of LLMs,

The thing that happens with slurs, toxic or obscene language is that they are also associated with things you do want to have represented.

A classic case of this are communities where common terms are also used in other communities as epithets.

For example, ‘gay porn’ is disproportionately more commonly found on the internet than ‘straight porn.’ The word, ‘porn’ is marked as obscene language. The word, ‘gay’ may be marked as a slur.

By trying to filter out the word, ‘gay’ risks also disproportionately erasing communities that are already marginalized, such as the LGBTQ.

Mitchell adds that despite the flawed curation that can ingest misogynistic, abusive content, and child sexual abuse (CSAM) material from the web, there is this reflection of tech optimism that doesn’t fully reflect the data. The supposed neutrality of AI just doesn’t bear out. She points out that biases against women and against Black people are far too common,

When you do a search for great fiction authors, it’ll tend to skew to white men. So, it’s these kinds of subtle effects that you don’t notice, but influence your perception of the world.

She explains that at least with traditional search engines, extraction techniques ensured a solid match between the search query and the search result—content from ‘real people.’ With LLMs it’s different:

LLM searches run on probabilistic sequences, meaning that pieces of text are being stitched together that sound confident and factual because they’re trained on confident and factual data, but they’re not grounded—directly connected to a clear source that can verify it’s true.

You can click on those citations and you can tell they’re post hoc because they don’t actually support what the summary is saying.

This post hoc analysis means the citations were specified after “the data is seen,” and were not part of the original query, thus making them highly susceptible to false positive results. Mitchell concludes that the interface design that gives the appearance of factuality but,

unlike single models, these larger systems can pull in all the bells and whistles, additional rules, models and algorithms. Right now, they’re based on LLMs, which have many issues. Your foundation is one of quicksand.

So, using LLMs as neutral knowledge sources risks reinforcing the inequalities these systems will mirror.

AI Fluency and the Illusion of True Human Understanding

The pace at which users are embracing chatbots for mental health is staggering:

* A study by CognitiveX found that one in three people use AI chatbots primarily for “fear of judgement or social stigma.”

* 43.75% of people prefer AI chatbots to discuss mental health issues first rather than approaching a trusted person.

Mitchell addresses sycophancy and the Eliza Effect (1966) where individuals attribute a human-like effect from chatbots:

Eliza was a simple rule-based chatbot, and people felt they could really talk to it like a therapist. It was designed to put forward sentences that sounded like Rogerian Therapy and so people’s tendency to impute a human mind when they see human-like language means they do feel like they’re talking to someone they can trust.

These conversations are increasingly private and when people disclose things to chatbots, now their experience is like having a private conversation with something, as Mitchell says,

… that is picking up their language because it’s stitching things together probabilistically, repeating back that language so it creates entrainment [synchronizing their rhythms or flows] in psycholinguistics. And that builds relationships.

These private feelings and confidential settings are not private because they’re going through large company APIs determining how language is aligning to them. Mitchell and her co-authors addressed this in their paper because they were concerned people would fall for this illusion,

As humans have evolved to communicate with other humans via language. I think when people worry about the illusion of human likeness, they may be misunderstanding that this isn’t something you can reason yourself out of.

It’s something that our brains are designed to do. Our brains are designed to have a sense of this other person there and to impute into that intentionality, feelings, emotions and all sorts of things we experience as humans.

We wanted to flag this was going to happen and there was no way out of it unless people took this seriously in their designs and handled this very foreseeable issue.

This did not happen and has led to the current day, AI Psychosis.

Read more here about one company tackling AI Psychosis from chatbots.

Semantic Associations and Fluency Create Coherence

What does the word ‘understanding’ mean from a technical perspective? Mitchell points out we use words like reason, chain of thought—things we understand from our human experience and attribute to something that’s fundamentally not human. She illustrates,

It understands in a neural network computer way.

It does not understand in a human way.

And that’s where people are getting a little bit confused.

Does understanding mean that it can take something you say, expand it into a bunch of related concepts and give you something back that is relevant to what you said? If so, then yes, these systems can do that.

But if understanding means that it can feel what you’re saying—that it can empathize and can match your experience to its own human lived experience—it can’t do that because it’s not human.

The language we’re using is creating more confusion, in our ability to understand what these systems are because we are already attributing human-like tendencies. On top of that, the more we anthropomorphize the chatbot technical behaviors, the more we lose the ability to tell the difference.

Each word, Mitchell explains, has many other natural associations. For example, cat is associated with dog, but also associated with fur. Language models have been exposed to the word, ‘cat’, often co-occurring with the word, ‘dog,” and that’s recorded probabilistically. She continues,

So if you input the word, ‘cat’ the network associates to dog, to pet, to fur, to tiger—this whole network of semantically related concepts. Once it’s connected to this larger network, it can stitch together other phrases related to those concepts that can further extend.

So, if I say, ‘I love cats’ it can say, ‘dogs are nice.’

It has associated ‘love’ to ‘nice’ as an associated related concept and it’s been trained on syntactically well-formed sentences so that these higher level semantic concepts can now be put forward in a fluent English text string.

She concludes that current AI systems can respond with hyper relevant things to what we input because they’re trained to do so. Mitchell emphasizes that we don’t give ourselves enough credit when we get fooled by these systems or when believe we should be smarter as she points out,

This is not about being smart. It’s just how our brains have developed and then how the systems are trained. They are trained to sound coherent and to pull together related semantic concepts. Many times, they can say things that are appropriate.

The $300 Billion Data Center Dilemma: Will We Need Them?

The pace of AI progress has not scaled to the same level of expected efficiency. The transformer models combined with neural network architecture continue to emit high levels of CO2. The Stochastic Parrots paper warned that “compute to train the largest deep learning models has increased 300k times in six years, a far higher pace than Moore’s Law.”

Every action in an AI system requires some amount of energy. So many computations can really slow down the system as they get hotter. You need data centers that run cooling systems, to maintain compute without overheating.

Mitchell explains the direct correlation between these growing systems and the demand for data centers,

If I can say the word ‘cat’ and it’s a small network, then it can only output ‘dog.’ However, if it’s a larger network, then it can associate it with sentences about dogs. That requires more computations and more storage and greater access to all these different kinds of stored representations.

And then as it gets larger, now it’s associating entire documents and then doing the processing over all of the sequences relevant to those documents i.e. those strings of text and then formulating further what to return to the user.

The larger and larger it gets, at the point of inference, the more computations it’s doing, the more things it has access to, the more computations on the server side is required, which is the reason for more powerful systems, more and more servers and more data centers.

Many have warned of the parallel of these data center buildouts to the first internet bubble. The astonishing multi-hundred billion dollar data center investments (to be exact, $335 billion dollar market in 2025) that frontier models have promised will return within the coming decade are still far too expensive to work with within current infrastructures.

Mitchell, however, calls out the tremendous work in the development of smaller models that can run locally on device, which does not require data centers. If smaller models endure in the coming years, will we see this data center boom collapse?

Generated AI is Eating Itself

After a chatbot has been deployed, Mitchell confirms that to create more effective systems it is a technically strategic imperative to treat chatbot and user interactions as gold to make the system better. For free accounts, this data is used for training; and for premium accounts users can choose to disable their interactions for use in training.

However, the more that generated content proliferates on the web, the more it creates this homogenizing effect that can lead to model collapse.

She alludes to the introduction of the long tail effect, coined by Chris Anderson, founder of TEDX, who argued the value of promoting niche or less popular products can collectively build better markets compared to popular products, given a larger market distribution. This has been the massive effect of the social web as popularity did not regress to the rich and famous, but instead, to early influencers from social platforms, which created new niche and local market opportunities as these distributed networks evolved.

As Mitchell describes, the risk is the shattering of the long tail:

The various ways which things can be talked about and expressed and the various topics that people talk about get chopped off when general purpose generations occur. And so, if you keep chopping off the long tail, you are creating a situation where you learn less and have less diversity. It’s called a Ouroborus—a snake eating its own tail.

How long will it take before our systems converge to this mean? Will synthetic generations prevail? Mitchell says it is the case that tons of original data on the internet has now been mined, and the content that is being produced is increasingly generated—not novel!

Five Years Later Since the Paper: “You must change consumer demand to put safeguards in place!”

Mitchell is hopeful. Since their paper, awareness has heightened, and companies have been putting in fixes to mitigate the risks. She acknowledged more legislation has been introduced to identify at what age is appropriate to be introduced to these systems, to lessen the effects of over-dependence on companion bots.

On the technical side, post-training techniques are being used to ‘reshape’ models as she explains

LLMs out of the box have a ton of different biases and are problematic in ways of handling the world that are easily exposed. Post training techniques like Reinforcement Learning with Human Feedback (RLHF) and Constitutional AI are used to move the biases of the system to be associated with the values you want it to represent.

Models can be jail-broken, and post-hoc fixes can be applied that allow developers to move the model into post-training spaces for remediation.

She adds that input prompts and generated outputs can also be put through classifiers to output better responses.

Overall, is there enough being done? Mitchell responds,

I really want to say yes but I don’t know if I honestly can.

It’s amazing that safety is a value that people are talking about. 10 years ago, the idea of ethics in AI was laughable. Over these 10 years people have not only realized that ethics can play a role, but they’ve honed in on critical ethical values such as safety and fairness and made those priorities in the design.

But she also acknowledges the most powerful systems—the most ubiquitous—are commercial and will always have a profit motive that flies in the face of safety precautions. Companies that constrain their systems will lose customers to competitors who don’t. The ‘safety’ company then marginalizes themselves out of the market.

There’s very real pressure to survive as a company, to push the boundaries beyond what might be the most ethically-well informed solutions and prioritize profit.

I would say by and large, there isn’t enough being done. Companies that had prioritized safety are now walking that back in the face of competition. That’s a concern.

While there are powerful voices trying to change the norms, you have to essentially change the market. You must change consumer demand to put in place the kinds of safeguards that I think a lot of us would want to have.

Margaret Mitchell’s Advice:

* When you’re interacting with something that’s AI, treating it like a monolith can be detrimental to figuring out how to work with it.

* Understand it in terms of its components: the data, the various models underneath and recognize that each comes with its own risks and benefits.

* To minimize overreliance on these systems, try and formulate your own answers and thoughts before turning to the system to retrieve them.

* This will keep your skills sharp and combat cognitive degradation.

* It will reveal how systems can be wrong when it does create an answer and will shatter some of the all-knowing illusion that many attribute to them. Mitchell defines this as ‘Nuance-Smoothing.’

* For GenZ, who are more vulnerable to the job and mental health impacts of Generative AI they either hate AI or are excessively optimistic. The latter may accelerate their use to the point they don’t think about ethics or safety as they build. Mitchell says:

I’m not happy that there is a sense in the younger generations that all AI is bad. It means they don’t know that when they look up driving directions to get somewhere, and when it gives us the estimate of traffic congestions and how long it will take for the best route—that’s AI… and when you go through your photos and you can easily find your friends—that’s AI.

She argues that there are cool uses like leveraging AI to detect Tuberculosis more effectively than two people; and in Nigeria, using AI can detect vaccine spoilage;

When we say AI, we mean ChatGPT. Mitchell is adamant that ChatGPT/Generative AI is but a sliver of Artificial Intelligence:

That misses the point that there’s this other realm of predictive AI that most of us are using every day and benefitting from.

It also misses the point that AI that’s being developed NOW is not the only path.

She relates to a lot of the booing and negativity about the current trajectory of generative AI and is helping develop solutions to counter its effects. She worries, however, that people who put all AI in one bucket, without having a historical perspective means that people cannot grasp what AI could be and that may not include this current path of Generative AI. She adds it is up to the newer generations to do things differently.

Barry Hillier, an audience member and one who is a strong user of current AI systems, recognizes this moral disparity among the generations and put it succinctly,

If you’re not using and understanding and becoming engaged in the use of AI, then what’s happening is you’re relegating how it’s designed and how it’s used to a very small group. And that group is not going to have your interests.

Whereas if we collectively decide how it needs to be done, we are able, through literacy and use, to determine what we do want out of the products, and what we expect from them.

Mitchell emphasizes literacy and not letting powerful players steal the narrative of what AI is and what it can be.

The Dire Consequences of Generative AI: Who Decided We Needed Them?

Eva Navarro Lopez, an AI ethicist and policy expert was less optimistic as she questioned,

We haven’t talked about why we need to use these tools for everything that we do. Who decided this?

She spoke about the dismantling of the wider field of AI and its being reduced to these current Generative AI technologies. And contrary to Mitchell’s position that policies for fairness and ethics are increasing, Lopez felt they were instead disappearing from government: inclusion, diversity with ploys of ethics washing from companies lobbying for government support to impose technologies on its citizens.

Mitchell clarified that 10 years ago fairness in machine learning was a new idea. It emerged but that doesn’t mean things have improved,

I’ve made statements about the long-term historical trajectory of AI and then there’s also what’s happening now. And I am completely aligned with you.

The issue of ethics washing and responsibility is a really serious one.

It was amazing I was able to start an ethical AI team at Google—a breakthrough considering what the tech industry was interested in. That ended up being completely shut down and what followed was a reactive force against ethical considerations.

Mitchel argues that countries are handling this differently, prioritizing human values to different degrees however this is being quelled,

The desire to put out more powerful systems is overriding ethical considerations. What’s worse is that companies will say they have safety teams, but the those who are grappling with these issues are often disempowered. They serve as a marketing tool to push off regulators. They’re also treated poorly, struggling to say sane in a situation that can be very crushing.

Today’s frontier models need to be open to criticism, and frankly, need to get out of their own way, and there are no signs of this happening today.

Five years ago, the Stochastic paper was released. It’s been cited thousands of times in academic research. The uncanny prescience of this research has been instrumental in bringing awareness to the downstream repercussions that these systems have created.

But we’re far from where we should be in shaping systems that work for all of us.