AI Impersonation

This week we talk about robo-Biden, fake Swift images, and ElevenLabs.

We also discuss copyright, AI George Carlin, and deepfakes.

Recommended Book: Debt: The First 5,000 Years by David Graeber

Transcript

The hosts of a podcast called Dudesy are facing a lawsuit after they made a video that seems to show the late comedian George Carlin performing a new routine.

The duo claimed they created the video using AI tools, training an algorithm on five decades-worth of Carlin's material in order to generate a likeness of his face and body and voice, and his jokes; they claimed everything in this video, which they called "George Carlin: I'm Glad I'm Dead," was the product of AI tools.

The lawsuit was filed by Carlin's estate, which alleges these hosts infringed on the copyright they have on Carlin's works, and that the hosts illegally made use of and profited from his name and likeness.

They asked that the judge force the Dudesy hosts to pull and destroy the video and its associated audio, and to prevent them from using Carlin's works and likeness and name in the future.

After the lawsuit was announced, a spokesperson for Dudesy backtracked on prior claims, saying that the writing in the faux-Carlin routine wasn't written by AI, it was written by one of the human hosts, and thus the claim of copyright violation wasn't legit, because while the jokes may have been inspired by Carlin's work, they weren't generated by software that used his work as raw training materials, as they originally claimed—which arguably could have represented an act of copyright violation.

This is an interesting case in part because if the podcasters who created this fake Carlin and fake Carlin routine were to be successfully sued for the use of Carlin's likeness and name, but not for copyright issues related to his work, that would suggest that the main danger faced by AI companies that are gobbling up intellectual property left and right, scraping books and the web and all sorts of video and audio services for raw training materials, is the way in which they're acquiring and using this media, not the use of the media itself.

If they could somehow claim their models are inspired by these existing writings and recordings and such, they could then lean on the same argument that their work is basically the same as an author reading a bunch of other author's book, and then writing their own book—which is inspired by those other works, but not, typically anyway, infringing in any legal sense.

The caveat offered by the AI used to impersonate Carlin at the beginning of the show is interesting, too, as it said, outright, that it's not Carlin and that it's merely impersonating him like a human comedian doing their best impression of Carlin.

In practice, that means listening to all of Carlin's material and mimicking his voice and cadence and inflections and the way he tells stories and builds up to punchlines and everything else; if a human performer were doing an impression of Carlin, they would basically do the same thing, they just probably wouldn't do it as seamlessly as a modern AI system capable of producing jokes and generating images and videos and audio can manage.

This raises the question, then, of whether there would be an issue if this AI comedy set wasn't claiming to feature George Carlin: what if they had said it was a show featuring Porge Narlin, instead? Or Fred Robertson? Where is the line drawn, and to what degree does the legal concept of Fair Use, in the US at least, come into play here?

What I'd like to talk about today are a few other examples of AI-based imitation that have been in the news lately, and the implications they may have, legally and culturally, and in some cases psychologically, as well.

—

There's a tech startup called ElevenLabs that's generally considered to be one of the bigger players in the world of AI-based text-to-voice capabilities, including the capacity to mimic a real person's voice.

What that means in practice is that for a relatively low monthly fee you can type something into a box and then have one of the company's templated voice personas read that text for you, or you can submit your own audio, creating either a rapidly produced, decent reflection of that voice and having that read your text, or you can submit more audio and have the company take a somewhat more hands-on approach with it, creating a more convincing version of the same for you, which you can then leverage in the future, making that voice say whatever you like.

The implications of this sort of tech are broad, and they range from use-cases that are potentially quite useful for people like me—I've been experimenting with these sorts of tools for ages, and I'm looking forward to the day when I can take a week off from recording if I'm sick or just want a break, these tools allowing me to foist my podcasting responsibilities onto my robo-voice-double.

In my opinion these tools aren't there yet, not for that purpose, but they're getting better all the time, and fast, and that the consumer-grade versions of these things are as good and accessible and easy to use and cheap as they are, today, suggests to me that I'll probably have something close to my dream in the next year or two, maybe sooner.

That said, this startup has gotten some not great mainstream attention, of late, alongside the largely positive press it's received for being a popular tool for making marketing videos and generating voices for characters in video games, because it was apparently used by someone to generate an audio recording that sounds a lot like US President Joe Biden, and that recording was then used to make robo-calls to voters across New Hampshire, encouraging them not to vote in the democratic primary there, and to instead save their vote for November—which is not a thing you have to do, but this is being seen as a portentous moment in politics nonetheless, because although AI-generated images and videos and audio clips have been used in other recent elections around the world, with varying, still mostly low-key levels of impact, the upcoming presidential election in the US in November is being closely watched because of the stakes involved for the country and for the world.

The folks running ElevenLabs have said they suspended the person who created the fake Biden audio clip from their service, and though the company recently achieved a valuation of more than a billion dollars and is, again, being generally seen as one of the leaders in this burgeoning space right now, this news item points at very tangible, already here risks for this sort of company, as there's a chance, still theoretical at this point, but a chance that has now become more imaginable, that this sort of deepfake audio or video or image could cause some kind of political or international or even humanitarian catastrophe if deployed strategically and at the right moment.

This political AI story arrived shortly before another torrent of relevant news about a deluge of what we might call explicit material—I'm going to try to avoid saying pornographic so as not to trigger any distribution filters on this episode, but that's the type of material we're talking about here—featuring AI-generated versions of performer Taylor Swift.

The most recent update to this story, as of the day I'm recording this, is that the social network formerly known as Twitter, now called X, has had to completely remove users' ability to search for the words Taylor and Swift on the platform, because efforts to halt the posting of such images and videos were insufficient due to the sheer volume of media being posted.

One such image attained 45 million views, hundreds of thousands of likes and bookmarks, and about 24,000 retweets before it was taken down by X's staff, 17 hours after it was originally shared.

Reports from 404 Media suggest that these images may have originated in a Telegram group, Telegram being a pseudo-social network that operates a lot like WhatsApp, and on 4chan, which is a forum that's basically dedicated to creating and sharing horrible and offensive things.

Most of the images shared were not deepfakes, where an existing image has another person's face plastered over it, but instead original AI-generated, let's say "adult" works, based on Swift's likeness.

The Telegram group recommends folks use Microsoft's AI image-generator, which is called Designer, to make these sorts of images—and though Microsoft has put limitations in place to try to keep people from making this sort of content, prompt-hackers, folks who enthusiastically figure out ways to bypass limitations on how AI tools respond to different prompts, telling them what to make, have figured out ways around most of these blocks, including those related to Taylor Swift, apparently, and those related to nudity and the other violatory themes that were incorporated into many of these images.

Like ElevenLabs, Microsoft isn't thrilled about this and has said they're looking into it and are figuring out ways to prevent this from happening again in the future, including outright banning users who make these types of images.

It's worth mentioning, though, that Taylor Swift, as a very famous and successful woman, has long been a target for this sort of thing, even before AI was used, back when folks were just photoshopping their fantasies and sharing those comparably less-sophisticated images in similar forums and on similar platforms.

It's important to note here, too, that Swift isn't the only person dealing with this kind of violation.

All sorts of people, men and women, though mostly women are also having their likenesses turned into explicit imagery and video content, and though this is an extrapolation on the way things have always been—the creation and distribution of revenge porn has plagued, again, mostly but not exclusively women since the dawn of the internet, and people have been making sometimes satirical, sometimes just intentionally vulgar images of other human beings since the dawn of pictographic communication.

Back in November of 2023, there were reports of teenage boys using these sorts of AI tools to create fake nude photos of their female classmates without those classmates' knowledge (or, obviously, permission).

The outcry following these revelations was substantial, as these were underage girls being turned into explicit images by their peers, which is creating all sorts of legal, interpersonal, and psychological problems, including but not limited to issues related to the creation of images featuring sexualized children, and issues related to the victimization of people via what amounts to completely fabricated revenge porn.

There are really substantial and tricky layers to all of this, then, because while mimicking someone's voice for political purposes is in some ways the same as reproducing someone's facial features in order to portray them in adult situations, there are additional concerns when the content being generated makes it seem as if the portrayed people are doing or saying something that they didn't do or say, and it's even more complicated when the human beings in question are of a protected class, like children.

There's also the question of degrees:

To what degree is this better or worse, or maybe the same, as people creating these types of images with Photoshop, or drawing them in a sketchbook with a pencil, rather than using AI to create realistic images?

How similar does a character in one of these images have to look to a real person, be they Taylor Swift or a classmate, in order for it to be, in the legal sense, a violation of their rights? How about a violation of their sense of personal security?

How explicit must a generated character's youth be for that character to count as underage, in the eyes of the law?

And how much protection does a normal, non-famous person have over their image, and should the legal consequences for violating that image be greater or less than the consequences for violating the image of a public figure who makes a living off their name and look and voice and persona?

It's a big tangle of questions, all of them related to potentially quite traumatic and scarring experiences for the people being targeted and portrayed in this way.

At the moment there are no clear answers about the legalities of all this, just a lot of in-the-works court cases and legal theories, and periodic pronouncements by government officials that we need to do something—but many of those same representatives are also slow-walking actual action on the matter due to a lack of legal precedent, an inability to do much about it, in a practical sense (because of the nature of these tools), and because some of them worry about stifling the fast-growing AI industry in their jurisdictions with regulations that may not actually address these issues but which would hamper potential productive uses of the same tools; throwing out the good stuff to try to hobble the bad, but not actually managing to do anything about the bad, so it's only the good that suffers.

One potential upside of Swift being targeted like this, if there can be said to be an upside to something that, again, is often traumatic and scarring for those afflicted—is that the US government finally seems to be moving more aggressively to do something because of her status, though the nature of that something is still unclear at this point.

The White House press secretary said that the government is alarmed by these reports, though, and they believe Congress should take legislative action, as there are no federal laws on the books that can keep someone from making or sharing these things at the moment, boggling as that may seem.

Research from 2019 found that something like 96% of all deepfake videos are non-consensual, explicit videos of this kind, and they're mostly of women, and there are thousands of known sites dedicated exclusively to sharing such content and teaching people to make more of it.

We're living through a tumultuous period in this regard, then, and are awash with flashy new technologies that grant everyday people heightened powers to create both incredible and harmful things.

We will almost certainly see some of these ongoing court cases establish new policy in the coming year, though it will likely be several years before actionable legal, and concomitant practical technological solutions to these sorts of problems start to roll out—at which point the same denizens of the internet who are bypassing today's restrictions on such things will get to work finding ways around those new barriers, as well.

Show Notes

https://mashable.com/article/fake-biden-robocall-creator-suspended-from-ai-voice-startup-elevenlabs

https://www.wired.com/story/biden-robocall-deepfake-elevenlabs/

https://www.bloomberg.com/news/articles/2024-01-26/ai-startup-elevenlabs-bans-account-blamed-for-biden-audio-deepfake

https://www.404media.co/ai-generated-taylor-swift-porn-twitter/

https://www.theverge.com/2024/1/25/24050334/x-twitter-taylor-swift-ai-fake-images-trending

https://www.usatoday.com/story/news/nation/2024/01/26/was-deepfake-taylor-swift-pornography-illegal-can-she-sue/72359653007/

https://variety.com/2024/digital/news/x-twitter-blocks-searches-taylor-swift-explicit-nude-ai-fakes-1235889742/

https://www.wsj.com/tech/x-halts-taylor-swift-searches-after-explicit-ai-images-spread-06ef6c45

https://www.bbc.com/news/world-us-canada-68123671

https://lawandcrime.com/high-profile/high-school-student-allegedly-used-real-photos-to-create-pornographic-deepfakes-of-female-classmates/

https://arstechnica.com/tech-policy/2023/11/deepfake-nudes-of-high-schoolers-spark-police-probe-in-nj/

https://www.morningbrew.com/daily/stories/2023/11/04/teens-are-terrorizing-classmates-with-fake-nudes

https://www.ft.com/content/0afb2e58-c7e2-4194-a6e0-927afe0c3555

https://arstechnica.com/ai/2024/01/george-carlins-heirs-sue-comedy-podcast-over-ai-generated-impression/ 

https://www.nytimes.com/2024/01/26/arts/carlin-lawsuit-ai-podcast-copyright.html?unlocked_article_code=1.Qk0.GtfO.azJzGDa58AVv&smid=nytcore-ios-share&referringSource=articleShare

https://www.sciencedirect.com/topics/social-sciences/fair-use