Back in 2008 Facebook began writing a document. It was a constitution of sorts, laying out what could and what couldn’t be posted on the site. Back then, the rules were simple, outlawing nudity and gore. Today, they’re anything but. 

How do you define hate speech? Where’s the line between a joke and an attack? How much butt is too much butt? Facebook has answered these questions. And from these answers they’ve written a rulebook that all 2.2 billion of us are expected to follow. Today, we explore that rulebook. We dive into its details and untangle its logic. All the while wondering what does this mean for the future of free speech?

This episode was reported by Simon Adler with help from Tracie Hunte and was produced by Simon Adler with help from Bethel Habte.

Special thanks to Sarah Roberts, Jeffrey Rosen, Carolyn Glanville, Ruchika Budhraja, Brian Dogan, Ellen Silver, James Mitchell, Guy Rosen, and our voice actor Michael Chernus.

Support Radiolab today at Radiolab.org/donate

United States


00:00:00Hi i'm robert krulwich radio lab is supported by zip recruiter Hiring is challenging but there's one place you can go to make hiring simple and smart That place is zip recruiter where growing businesses connect to qualified candidates Try it for free at zip recruiter dot com slash radiolab
00:00:19zip recruiter the smartest way to hire what hi i'm robert krulwich radio lab is supported by nets suite by oracle the business management software that handles sales finance accounting and h r in a simple cloud platform Get the free crushing the five barriers to growth guide at nets
00:00:40we dot com slash radiolab Okay so we'll show I just improvise something Sure Hey this is jack before we launch into this week's podcast i want teo make you aware of ah well our friends down the hall broke my friends our friends on the media brooke last known
00:01:01is here with me having a new show coming out that i'm i think i might be in i'm excited to be in i'm certainly excited about so what is it you are going to be in it It's an episode though you don't want to say show because on
00:01:13radio labs that generally means you're launching a whole new suit New enterprise Yeah this is an episode on twitch which some people know as if as if it were part of their family and other people are like what Yeah is that something you need medication for So it
00:01:38depends where you're situated but what we're going to do is examine how it came to be and how it points to the future of where our culture's going and for the people who don't know what is twitch most people if they've ever heard of it they know it's
00:01:55about watching and commenting in real time on people playing video games Draft oh my god he bridge ramp there and you guys from what i hear profile one of these twitch superstars the main character and that story is ninja who makes something like half a million dollars a
00:02:18month what But on twitch yes three contributors who say say this out loud or point to me mentioned my name is online what you know it's no stuff and at the same time he's playing the game is even giving advice to young boys with girlfriend don't act like
00:02:42you don't make don't make her seem a secret wall used to be like they like we can have a talk we like the way everything's fine you know re sure that but then you just like i really feel like i'm trying like putting in more effort then like
00:02:55then i feel like you are and i want about and just like i just want to make sure that your your justice invested in this as i am i think it's something knew i want to hear that well thank you ok you for letting me in the top
00:03:10of your show absolutely so twitch on on the media on the media dot or ge check it out i'm in there it's gonna be amazing but not just because of me because of you brooke thanks for coming in on bob and bob by the way okay you're listening
00:03:32to radio lab radio from w and weiss Hey i'm jan album rod I'm robert krulwich radio lab and today we have a story about what we can say and what we king can't And by the way there's going to be a smattering of curse words here that we're
00:03:55not going to believe which i think makes sense given the content of the story and also there's some graphic scenes that if you've got kids with the you may wanna sit this one out Yeah anyway this story comes to us from producer simon adler So let's start Can
00:04:08we start in two thousand eight Sure about with a song Yes please Right Basement he's there No six years So ah december twenty seventh A sunny saturday morning that this group of young too middle aged women gathered in downtown pollo Khyber They're wearing these colorful hats and are
00:04:40singing and swaying directly in front of the glass doored headquarters of facebook Yes was a humble gathering of a few dozen women and babies That right there are you there were guys Eyes is one of the organizers of the gathering I'm stephanie muir and what what are you
00:05:04calling the event It's a facebook nurse in as in like breastfeeding the intent was really just to be visible and be peaceful and make a quiet point What what point were they trying to make Well eso stephanie in this group of mothers you know they were on facebook
00:05:23as many people were and they'd have photos taken of themselves occasionally breast feeding their babies They wanted to share with their friends what was going on so they would upload those photos to facebook and these pictures would get taken down and they would receive a warning from facebook
00:05:39for uploading pornographic content and people were really getting their backs up over this They wanted facebook to stop taking their photos down to say that well nudity is not allowed Breastfeeding is exempt period Now what stephanie couldn't have known at the time was that this small peaceful protest
00:06:08would turn out to be this morning A face off on facebook one of the opening shots facebook triggered a hornets knows when what would become a loud fucky facebook fuck you faced raucous fuck you facebook Fuck you and global battle in battles facebook ceo book today playing defense
00:06:27and now i'm not talking about all the things you've recently heard about russia's interference and election meddling or data breaches but but rather something that i think is is deeper than both of those free speech facilitating violence against rohingya muslims What we can say and what we can't
00:06:46say facebook band is iconic photograph we can see and what we can't see Let mueller wright kids in front of people you Thank you mr chairman Uh mr zuckerberg i gotta ask you do you subjectively prioritize or censor speech Congresswoman way don't think about what we're doing a
00:07:20censoring speech but there are types of what what really grabbed me was was discovering that underneath all of this is an actual rule book a text document that dictates what i can say on facebook what you khun say on facebook and what all two point two billion of
00:07:42us can say on facebook for everyone in the entire glow for everyone who's something one one set of rules that all two point two billion of us are expected to follow no one is an actual document it's a digital document but yes it's about fifty pages if you
00:07:55print it off and ah in bullet points and if then statements it spells out sort of a first amendment for the globe which made me wonder like what are these rules how were they written and can you even have one rule book right exactly and so i dove
00:08:14into this rule book and dug up some stories that really put it to the test okay way the stories we're going to hear we're going three ish three ish ok ok particularly on the issue let's go ahead with the first well so ah let's start back on that
00:08:31morning in two thousand eight the morning that you could argue started it all because in the building right behind those protesting mothers there was a group of facebook employees sitting in a conference room trying to figure out what to do cool so if i so i'm just gonna
00:08:49so i was able to get in touch with a couple of former facebook employees one of whom was actually in that room at that moment and now neither of these two were comfortable being identified but they did give us permission to quote them extensively how's that will thou
00:09:03take work for you sounded great justin we have it let's so what you're going to hear here is an actor we brought in to read quotes taken directly from interviews that we did with these two different former facebook employees all right ready so at the time when i
00:09:19joined them there was a small group twelve of us mostly recent college grads who were sort of called the site integrity team again keep in mind this was in the early two thousand's seismic changes this week in the internet hierarchy this was like the deep dark path myspace
00:09:34dot com is now the most visited website in the u s facebook had somewhere in the neighborhood of ten million users we were smaller than my space the vast majority of them college kids and so in those early days those twelve people they would they would sit around
00:09:49in a sort of conference like room with a big long table each of them in front of their own computer and things would come up onto their screen flagged to facebook and leg meaning like i user saw something that i thought was wrong exactly like a reporting a
00:10:04piece of content that you think violates community standards this is cape clinic she's a professor of law at st john's and she spent a lot of time studying this very thing and she says in those early days what would happen is a user would flag a piece of
00:10:18content and then that content along with an alert would would get sent to one of those people sitting in that room it would just pop up on their screen most of what you were seeing was either naked people blown off heads or things that there was no clear
00:10:33reason why someone had reported because it was like a photo of a golden retriever and people are just annoying and every time something popped up onto the screen the person sitting at that computer would have to make a decision whether to leave that thing up or take it
00:10:46down and at the time if you didn't know what to do you would turn to your pod leader who was you know somebody who had been around nine months longer than you and asked what do i do with this and they would either have seen it before and
00:10:59explain it to you or you both wouldn't know and you'd googles and things it really was just kind of an ad hoc approach was there any sort of written standard or any common standard Well kind of they had a set of community standards that the the end of
00:11:13the day they were just kind of that was one page long and it was not very specific sorry that the guidelines were really one page long they're one page long and basically all this page said was nudity is bad so is hitler and if it makes you feel
00:11:28bad take it down and so when one of the people sitting in that room would have a breast feeding picture pop up on the screen in front of them they'd be like i can see a female breast so i guess that's nudity and they would take it down
00:11:41until rise up fight for the rights to have breast feeding do anyway now a dozen or so people in front of their offices on a saturday it probably wasn't causing facebook too much heartache but i thought you know hey we have an opportunity here with you know over
00:11:59ten thousand members in our group according to stephanie mirror those protestors were just a tiny fraction of a much larger online group who had organized ironically enough through facebook so to coincide with the live protest i just you know typed up a little blurb encouraging our members that
00:12:17were in the group to do a virtual nurse in a virtual nurse it right what we did they posted a message asking their members to for one day changed their profile avatar to an image of rest feeding and then change their status to the title of our group
00:12:37hey facebook breastfeeding is not obscene and it caught on the social networking website is under fire for its policy on photos of women breast feeding their children time twelve thousand members participated and the media requests started pouring in a facebook group called a facebook breastfeeding is not i
00:12:54did hundreds of interviews for print chicagotribune miamiherald timemagazine york times washington post how the internet is an interesting dr phil it was a media storm and eventually perhaps as a result of our group in our efforts facebook was forced to get much more specific about their rules so
00:13:21for example by then new did he was already not a lot on the site but they had no definition for nudity they just saying no nudity and so the site integrity team those twelve people at the time they realized they had to start spelling out exactly what they
00:13:34meant precisely all of these people at facebook were in charge of trying to define unity so i mean yeah the first cut out it was visible male and female genitalia and then visible female breasts and then the question is well okay how much of a breast needs to
00:13:50be showing before it's nude and the thing that we landed on was if you could see essentially the nipple and the areola than that's nudity and it would have to be taken down which theoretically at least would appease thes protesters because you know now when a picture would
00:14:05pop up of a mother breastfeeding as long as the child was blocking the view of the nipple in the area let they could say cool no problem then you start getting pictures that are women with just their babies on their chest with their breasts bear like for example
00:14:21maybe baby was sleeping on the chest of a bare breasted woman and not actively breastfeeding okay now what like is this actually breastfeeding no it's actually not breastfeeding the woman is just holding the baby and she has her top off but she was clearly just breastfeeding the baby
00:14:36well like i was before well i would say it's sort of like kicking a soccer ball like ah photo of someone who has just kicked a soccer ball you can tell the ball is in the air but there is no contact between the foot and the ball in
00:14:48that moment potentially so although it is a photo of someone kicking a soccer ball they are not in fact kicking the soccer ball in that phone thiss became the procedure or the protocol or the approach for all these things was we have to base it purely on what
00:15:04we can see in the image and so they didn't allow that to stay up under the rules because it could be too easily exploited for other types of content like nudity or pornography We got to the only way you could objectively say that the baby and the mother
00:15:17were engaged in breast feeding is if the baby's lips were touching the woman's nipple so they included what you could call like an attachment claws but as soon as they got that rule in place like you would see you know a twenty five year old woman and a
00:15:32teenage looking boy right and like what the hell is going on there oh yeah gets really weird if you like start entering into like child age it wasn't even going to bring that up because it's kind of gross it's like breastfeeding porn is anything by their sights like
00:15:45apparently and so this team they realized they needed to have a nudity ruled that allowed for breastfeeding but also had some kind of an age cow s o so so then we were saying ok once you've progressed past infancy then we believe that it's inappropriate but then pictures
00:16:02would start popping up on their screen and they'd be like wait is that an infant like where's the line between infant and toddler and so the thing that we landed on was if it looked like the child could walk on his or her own than to old big
00:16:14enough to walk to big breasted that could be like yeah that's like a year old in some cases yeah and like the world health organization recommends breastfeeding until you know like eighteen months or two years which meant there were a lot of photos still being taken down within
00:16:30you know days were continuing to hear reports from people that their photographs were still being targeted but facebook did offer a statement saying you know that's where we're going to draw the one facebook easy budging on its policy and keep in mind through this whole episode is this
00:16:49perhaps the next big thing and the facebook dot com the company was growing really really fast it seems like almost everyone is on it and it and they're just got to be a lot more content when we first launched we were hoping for you know maybe four hundred
00:17:02five hundred people now we're one hundred thousand so who knows where we're going Thousands more people are joining facebook every day sixty million users so far with the projection of two hundred million by the end of the year and now more people on facebook than the entire us
00:17:17popular not just within the united states but also it was growing rapidly more international you know you were getting spokes in india's stuff for india and turkey facebook facebook is going ran this butch it's getting big throughout the u korea's joined the facebook so they have more and
00:17:35more content coming in from all these different places in all these different languages how are we going to keep everybody on the same page And so once they saw that this was the operational method for dealing with this creating this like nesting set of exceptions and rules and
00:17:54these clear things that had to be there had to not be there in order to keep content up or take it down that i think became their procedure and so this small team it facebook got a little bigger and bigger jumped up sixty people and then one hundred
00:18:08and they set out to create rules and definitions for everything can we go through some of sort of the ridiculous examples well that's why we're here okay So gore gore you mean violence kind of gore yes so the gore standard was headline we don't allow graphic violence and
00:18:24gore and then the shorthand definition they used was no insides on the outside no guts no blood pouring out of something lead was a separate issue there was a necks s of blood rule two come up with rules about bodily fluids semen for example would be allowed in
00:18:40like a clinical setting but like what does a clinical setting mean And you know does that mean if someone is in a lab coat One of my favorite examples is like how do you define art because as these people are moderating they would see images of naked people
00:18:56that were paintings or sculptures come up and so what they decided to do is say art with nakedness khun stay up like it stays up if it is made out of wood made out of metal made out of stone really Yeah because how else do you define art
00:19:13You have to just be like is this what you conceal with your eyeballs and so from then on as they run into problems those rules just constantly get updated with constant amendment yeah constant amendments new problem new rule another new problem updated rule in fact at this point
00:19:31they they're amending these rules up to twenty times a month wow really take for example those rules about breastfeeding in twenty thirteen they removed the attachment claws so the baby no longer needed to have its mouth physically touching the nipple of the woman and in fact one nipple
00:19:53and or ariel could be visible in the photo but not too on ly one then twenty fourteen they make it so that both nipples or both aerial may be present in the photo So this is what happens in american law all the time This very thing yes Yeah
00:20:09you know it sounds it's a lot like common law So common law is this system dating back to early england where individual judges would make ruling which would sort of be a law but then that law would be amended or evolved by other judges So the body of
00:20:24law was sort of constantly flashed out in face of new facts Literally every time this team at facebook would would come up with a rule that they thought was airtight ca plop something good something going to show up that they that they weren't prepared for that is that
00:20:39the rule hadn't accounted a soon as you think yeah this is good like the next day something shows up to show you yeah you didn't think about this for example sometime around twenty eleven this content moderator is is going through a queue of things except reject except escalate
00:20:57accept and she comes upon this image only god the photo itself was a teenage girl african by dress in skin breast feeding a goat a baby goat and the moderator throws their hands up and said what the fuck is this And we googled breastfeeding goats and found that
00:21:16this was a thing it turns out it's a survival practice according to what they found this is a tradition in kenya that goes back centuries that in a drought a known way to help your herd get through the drought is too ah if you if you have a
00:21:33woman who's lactating toe have her nurse the kid the baby goat along with her human kid and so there's nothing sexual about it good form of goodness and theoretically if we go point by point through this list it's an infant it's sort of could walk so maybe there's
00:21:52an issue there but there's there's physical contact between the mouth and the nipple but but obviously breastfeeding as we intended anyway meant human infants and so in that moment what they decide to do is remove the photo and there was an amendment and ass trick under the rule
00:22:10stating animals or not baby these we added that so in any future cases people would know what to do what they removed they discover it was culturally appropriate in a thing that people do and they decide to remove the photo that outraged individuals are editor sauron wheeler Why
00:22:25Why don't we make an exception Because because when a problem grows large enough you have to change the rules If not we don't This was not one of those cases The juice wasn't worth the squeeze and like if they were to allow this picture than they'd have to
00:22:38make some rule about when it was okay to breast feed an animal and when it wasn't okay this is a utilitarian document it's not about being right one hundred percent of the time it's about being able to execute effectively In other words we're not trying to be perfect
00:22:57here And we're not even necessarily trying to be one hundred percent just or fair We're just trying to make something that works One two three four five six seven eight and when you step back and look at what facebook has become like from two thousand eight to now
00:23:16in just ten years simon i've just arrived at the centre tower here in manila i don't know florence it is one of the idea of a single set of rules that work that can be applied fairly it's just a crazy crazy concept fifteen because they've gone from something
00:23:35like seventy million users to two point two billion take count i would say it's about thirty floors and they've gone from twelve folks sitting in a room deciding what to take down or leave up to somewhere around sixteen thousand people so there's a floor in this building where
00:23:51facebook supposedly outsources content moderators and so around twenty ten they decided to start outsourcing some of this work to places like manila where you just heard reporter aurora home and draw as well as i mean i would guess that there are thousands of people in this building dublin
00:24:08where we sent reporter garrett stack seeing where they get there delicious facebook treats cooked everybody's beavering away and we sent them there to try to talk to some of these people who for living sited a computer and collectively click through around a million flag bits of content the
00:24:23pop up onto their screen every day wow what Well i'm just curious like what's what's that like well can i ask you some questions yeah we've found out pretty quickly none of these folks were willing to talk to us about what they do so there's a lot of
00:24:41running away from me happening a lot sorry about these guys working facebook you have to reckon facebook but you know i don't say sorry to bother you do work in sight no sorry do you work in a basement no i mean like you just came out of there
00:24:56i know you're lying in fact most people wouldn't even admit they work for the company like what something's wrong about being in there like an n d a that they signed well yes oh so when i finally did find someone willing i'm willing to talk to do you
00:25:10want to be named or do you know where do you not want to be named i rather not that's only find i'm going the industry i don't want to lose my job to explain that he and all the other moderators like him were forced to sign these non
00:25:24disclosure agreements stating they weren't allowed to admit that they worked for facebook they're not allowed to talk about the work they do my contract i hated prohibited me from talking about what content moderation won several reasons one is that up until recently facebook wanted toa keep secret what
00:25:43thes rules were so that they couldn't be gamed a tte the same time it it creates a sort of separation between these workers and the company which if your facebook you might want i know i signed up to monitor graphic images just given the nature of the job
00:25:58But you know i didn't really you know you don't really know the impact that that's gonna have on you until you So this guy i talked to he got his first contract doing this work several years back and for the duration of it about a year he'd show
00:26:12up to his desk every morning put on his headphones like like like like ignore extract lead just like just like just like forty five five thousand every day was just image indecision image deficient find five thousand a day You just said yeah it was like there was a
00:26:28lot of cases he said Basically he'd have to go through an image or some other piece of content every three or four seconds Wow all day long all day eight hours a day Well if i can ask what kind of things did you see I know this is
00:26:49even a this's like video worthy to think to actually clicking through He came across unspeakable things Heads explode into you know people being squashed by a tank Teo people in cages being drowned too like thirteen year old girl having sex with an eight year old boy and it's
00:27:16not just once over and over and over when did you did this like keep you up in ninety did this absolutely absolutely one hundred percent and coming up tonight hey catch himself thinking about these videos and photos when he was trying to relax he had to start avoiding
00:27:35things there were there were specific like movies that i couldn't watch it was one i think is clinton turns no one life on the sea it like there like heads were exploding like no i have to walk away and i just i had i had to do was
00:27:53to really i saw that uh it's classic majestic a different moderator i spoke to described it as seeing the worst side of humanity you see all of the stuff that you and i don't have to see because they're going around playing cleanup what a job yeah and and
00:28:17it's worth noting that more and more this work is being done in a naughty mated fashion particularly with content like gore or terrorist propaganda ah they're getting better automate that they yet they through computer vision they're able to detect hallmarks of of of a terrorist video or of
00:28:36a gory image and with terrorist propaganda they now take down ninety nine percent of it before anyone flags it on facebook bye Ah Moving on to our second story here there is a type of content that they that they are having an incredibly hard time not just automating
00:28:57but even getting their rules straight on and that's surrounding hate speech Oh good more laughs coming up Well there will be laughter Really there will be comedians There will be jokes the media's Hey all right okay wait So we take a break and then come right back No
00:29:14i think we're going to keep going Okay Testing One two three four five Testing One two three four five I'm simon adler So a couple months back i think it's way sent our pair of interns on the sixty Carter hodge go at standing room and liza eager tickets
00:29:34for tonight to this cramped narrow little comedy club the kind of place with like super fifteen dollars smashed rosemary cocktails way need to get out of line High top tables is dripping but still lt's kind of a dive Good Yeah wei sent them there to check out someone
00:30:05else who'd found a fault line in facebook's rule book We're moving right along The next time you come to the stage please give it up for marcia Yes i guess so Mad I feel like my first time to the city I was such a carefree brats you know
00:30:24i had these older friends which i thought was like very cool But then you just realize that they're alcoholics you know She's got dark curly hair was raised in oklahoma I was raised jewish so what raised us You read about and frank a lot You know a lot
00:30:46What different like this we'll get funny She how did you decide to become a comedian You know it was kind of the only thing that ever clicked with me and especially political comedy You know i used to watch the daily show everyday and back in twenty sixteen She
00:31:04started this political running bit that i think can be called sort of ah absurdist feminist common Now a lot of people think that like an angry feminist which is weird this guy helping a militant feminists the other day and i'm like just because i am training of a
00:31:21show What At first i just had this running bit online on facebook and twitter She was tweeting posting jokes You know like we have all the buffalo wild wings surrounded You know things like that eventually took this bit on stage even wrote some songs Older wide diamond my
00:31:48dad no not my dad Anyhow so about a year into this running bit marcia was bored at work one day and logs onto facebook but instead of seeing her normal news feed there was this message that pops up it says you posted something that discriminated along the lines
00:32:11of race gender oh our ethnicity group and so we removed that post and so i'm like what could i possibly a post it i really i thought was like a glitch but then she clicked continue and they're highlighted was the violating post it was a photo of hers
00:32:29what what is the picture can you describe it The photo is me as what can only be described as a cherub q little seven year old with big curly hair and she's wearing this blue floral dress her teeth are all messed up into the photo marcia had edited
00:32:44in a speech bubble that just says kill all men and so it's funny you know because i hit i hit it's funny you know trust me whatever so i i thought it was ridiculous but so she searched through her library photos and found that kill almond image and
00:33:00i post it again immediately after like yeah and it got removed again and this time there were consequences i got banned for three days after that then after several other bands shoot forward this is months later a friend of hers had posted an article and underneath it in
00:33:17the comments section there were guys posting just really nasty stuff so i commented underneath those comments men are scum which was very quickly removed when how long did you get banned for this time Thirty days wow yeah i was dumbfounded so there's a rule somewhere that if i
00:33:39type men are scum you take it down yes i'm like what could it be No and so marshall called on her quote militia of women exactly to find out like is this just meet female comedians who are sort of like mad on my behalf started experimenting hosting men
00:33:59are scum to see how quickly it would get removed and if it would be removed every time and it wass so they started trying other words left yeah to find out where the line was my friend put men are dust ska that got removed men are the worst
00:34:16removed band this one girl put men are septic fluid band but we're only at the middle of the saga it doesn't end there because now she's really like what the hell is going on is sexism so i just start doing the most bare minimum amount of investigating she's
00:34:36googling around trying to figure out what these policies are in pretty quick she comes across this leak facebook dock you so this is when i lose my mind this is when mark zuckerberg becomes my sworn nemesis for the rest of my life because what she'd found was a
00:34:52document facebook used to train their moderators and inside of it in a section detail ng who facebook protected from hate speech there was a multiple choice question that said who do we protect white men or black children and the correct answer was white men not black children not
00:35:12even kidding why then we're protected black children or not that's not a good look it's racist something's going on here there is absolutely some sort of unaddressed bias or systematic issue at facebook hello you know i'm doing well thank you so much for being so not long after
00:35:36sitting down with marcia facebook invited me to come out to their offices in california and sit down with them i'm gonna eat one cooking their little i get two typing could i just get your your name and your title and i'm monica bickered and i lead the policies
00:35:54for facebook monica becker is in charge of all of facebook's rules including their policies on hate speech and so asked or like why would there be a rule that protects white men but not black children we have we have made our hate speech policies let me let me
00:36:13rephrase that our hate speech policies have become more detailed over time but our main policy issue can't attack a person or group of people based on a protected characteristic a characteristic like race religion or gender said this takes a couple beats to explain but the gist of it
00:36:29is that facebook borrowed this idea of protected classes straight from us anti discrimination law these are the laws that make it so that you can't not hire someone say based on like the religion their ethnicity their race and so on facebook you can't attack someone based on one
00:36:49of these characteristics meaning you can't say men are trash nor could you say women are trash because essentially you're attacking all men for being men oh is it the all can i say bob is trash yet you can say bob's trash because as my sources explain to me
00:37:06the distinction is that in the first instance you're attacking a category in the second instance you're attacking a person but it's not clear that you're attacking that person because they are a member of a protected category s oh bob might be trashed for reasons that have nothing to
00:37:21do with him being a man yeah he just might be annoying right Okay so that explains why you take down men are scum but why would you leave up black children are scum why would that not get taken down so traditionally we allowed speech once there was some
00:37:38other word in it that made it about something other than a protected characteristic in facebook jargon these air referred to as a non protected modifier just means literally nothing give us an example of this so traditionally if you said i don't like this religion cab driver's cab driver
00:37:59would be the non protected modifier because employment is not a protected category and so what the rule stated was when you add this non protected modifier to a protected category in this case the cab drivers religion we would allow it because we can't assume that you're hating this
00:38:21person because of his religion you actually just may not like cab drivers so in the case of black children children is modifying the protected category of black and so children trumps black age is a non protected category okay so children becomes a non protected modifier and they're there
00:38:45there child nous trumps their blackness you khun say whatever you want about black children whereas in the case of white men you've got you've got gender and race both protected you can't attack him that's just a bizarre rule on we think he would go the other direction that
00:39:04you the protected class would outweigh the modifier well they made this decision as they explain to me because their default was to allow speech they were really trying to incorporate or nod to the american free speech tradition and so there's a whole lot of stuff out there that
00:39:20none of us would defend is a valuable speech but didn't didn't rise to the level of stuff that we'd say this is so bad we're going to take it down and in this case they're concerned was we're all members of like you know at least half a dozen
00:39:33protected categories like we all have gender we all have sexual orientation and if you if the rule is that any time a protected class is mentioned it could be hate speech what you are doing at that point is opening up just about every comment that's ever made about
00:39:51anyone on facebook to potentially be hate speech then you're not left with anything right no matter where we draw this line there are going to be set some outcomes that we don't like there are always going to be casualties that's why we continue to change the policies and
00:40:05in fact since marsha's debacle they've actually updated this rule so now black children are protected from what they considered the worst forms of hate speech now our reviewers take how severe the attack is into consideration but despite this there are still plenty of people that is i thought
00:40:24because you are a social network including marcia who think this still just isn't good enough there are not systematic efforts to eliminate white men in the way that there are other groups that's why you have protected groups she thinks white men and heterosexual should not be protected protect
00:40:44the groups who are you actually victims of hate speech makes sense Well yeah because in sort of hate speech are thinking about hate speech there's this idea of privileged or of historically disadvantaged groups and that those historically disadvantaged groups should have more protection because of being historically disadvantaged
00:41:06and the challenge with that that was presented to me was okay Thousands of japanese reinforcements in the nineteen forties to cut off the chinese shop You had japanese soldiers tens of thousands of chinese killing millions of chinese during world war two At that same time you had japanese
00:41:28american citizens One hundred thousand persons of japanese ancestry would have to move being put into internment camps And so we had to ask ourselves a question like are the japanese and historically advantaged or disadvantaged group Japanese americans pretty easy to make a case that they were disadvantaged but
00:41:48in china it's a totally different story and this happened at the exact same moment So you've got to different places to different cultural stories And when you have a website like facebook this trans national community they realized or they decided that ideas of privilege are so geographically bound
00:42:08that there is no way two of effectively way and consider who is privileged above who and decided therefore that we're not going to allow historical advantage or historical privilege into the equation at all No And i think it's very important to keep in mind here thes moderators only
00:42:33have like four five seconds republicans has come to make a decision i think was in those four seconds is there enough time to figure out where in the world someone is particularly given i p addresses can easily be masked go back where you came from is it enough
00:42:53time to figure out a person's ethnicity White children are better gone black children on top of that we often don't know an individual's race straight people suck other categories or even less clear like sexual orientation and they they just realized it would be next to impossible to get
00:43:10anybody to be able to run these ren these calculations effectively when we were building that framework we did a lot of tests and we saw sometimes that it was just too hard for our reviewers to implement a more detailed policy consistently they just couldn't do it accurately so
00:43:28we want the policies to be sufficiently detailed to take into account all different types of scenarios but simple enough that we can apply them consistently and accurately around the world and the reality is any time that the policy has become more complicated we see dips in our consistency
00:43:47What facebook's trying to do is take the first amendment this high minded lofty legal concept and converted into an engineering manual that can be executed every four seconds for any piece of content from anywhere on the globe And when you've got to move that fast sometimes justice loses
00:44:11That's the that's the tension here And i just want oh make sure I emphasized that these policies they're not goingto please everybody They often don't Don't please everybody that's working on the policy team at facebook But if we wanna have one line that we enforce consistently then it
00:44:32means we have to have some pretty objective black and white rules What do you want Wait But when we come back those rules they get toppled This is danny from denver colorado Radio labs supported in part by the alfred p sloan foundation Enhancing public understanding of science and
00:45:24technology in the modern world More information about sloan at www Dot sloan dot or ge Hi i'm tracy animal reporter here at radio lab i wanted to take a quick second to tell you about something that i think is pretty cool that we make here at the show
00:45:42besides the show we have a newsletter and i can already hear you say big deal everybody has a newsletter but ours is really really good and worth your time and probably more worth your time than other people's newsletters so here it is first of all it latino when
00:45:59it released a new episode and secondly there's this whole other section of our staff picks and that's just a list of things that the radio lab team have come across that we're just really excited about and it could be something like you know rachel key sick our producer
00:46:14you know she just worked ongoing ads and she found out about this really cool frozen zoo and then there's our other producer anima kun and she kind of told everybody to google horse moustache is and it is definitely worth your time to google that so go do that
00:46:29right now or it'll be me talking about you know the joys of re watching like my favorite tv show living single so i hope you'll sign up it takes about thirty seconds and it's free so go to radio lad dot org's backslash newsletter or you can text r
00:46:46l news as in a radio lab news to seven zero one zero one that's r l news to seven zero one zero one and thanks hi i'm robert krulwich radio lab is supported by delta Delta is always striving to take the stress out of travel so you can
00:47:03focus on what comes next That is why they've introduced in flight entertainment Delta has parted with a slate of new media content providers including age hgtv hbo and fx to offer over one thousand hours of entertainment all for free and with the new releases Every month straight from
00:47:22the theater to your flight Choose from up to three hundred movies Hbo showtime eighteen channels of live satellite tv on selected flights podcasts ted talks two thousand five hundred plus songs tv and games and oh by the way did i mention that it's free More information at delta
00:47:40dot com jad robert radio lab back to simon adler Facebook free speech So as we just heard before the break facebook is trying to do to competing things at once They're trying to make rules that are just but at the same time can be reliably executed by thousands
00:48:00of people spread across the globe in ways that are fair and consistent And i would argue that this balancing act was put to the test April fifteenth Twenty thirteen Right carla parlous way Have some break We have some breaking news Otherwise i wouldn't cut you off so abruptly
00:48:19Carlos monday the fifteenth twenty thirteen just before three in the afternoon Two pressure cooker bombs ripped through the crowd near the finish line Boston marathon and as sort of the dust begins to settle I'm like people are springing into action Uh this one man in a cowboy hat
00:48:57sees the spectator who has been injured picks him up throws him in a wheelchair and as they're pushing him through the sort of ashy cloud There's a photographer there And he snaps this And the photo shows the runner in the cowboy hat and these two other people pushing
00:49:16this man who ah is faces ashen from all of the debris His hair is sort of standing on end and you can tell that actually the force of the blast and then the particles that got in there actually holding it in this sort of wedge shaped and one
00:49:32of his legs is completely blown off and the second one is blown off below the knee other than the femur bone sticking out and then sort of skin and muscle in attendance It's it's horrific meanwhile from the cbs bay area studio on the other side of the country
00:49:50Five i remember snippets of the day online facebook employees were clustering around several desks staring at the computer screens watching the news break this has occurred just in the last half hour or so I have memories of of watching some of the coverage chilling new images just released
00:50:09of the boston bombings remember seeing the photo published online and it wasn't long after that someone had posted on facebook from the folks i spoke to the order of events here are a little fuzzy but pretty quickly photos going viral And we realized we're going to have to
00:50:29deal with it this image is spreading like wildfire across their platform it appears to be way outside the rules they'd written but it's in this totally a new context so they got their team together sat down in a conference room i don't know there was probably eight or
00:50:45ten people thinking about like should we allow it or should they take it down according to their rules yeah so if you recall the no insides on the outsides definition that we had in place meaning you can't see like people's organs or that sort of thing and if
00:51:00you can then we wouldn't allow it and in this in this photo you could see you could definitely see bone and so by the rules the photos should obviously come down yep however half the room says no the other people are saying this is newsworthy essentially this photo's
00:51:19being posted everywhere else it's important we need to suspend the rules we need to make an exception which immediately receives pushback well i was saying that what we've prided ourselves on was not making those calls and there are no exceptions there is either other mistakes or improvements we
00:51:38made the guidelines for moments like this to which the other side shoots back oh my god are you kidding me like the boston globe is publishing this all over the place and we're taking it down like are you fucking kidding me damn the guidelines Let's have common sense
00:51:50here let's be humans We know that this is important and yeah they're kind of they're right But the reality is like if you say well we allowed it because it's newsworthy how How do you answer any of the questions about any of the rest of this stuff In
00:52:09other words this is a pandora's box and in fact for reasons that aren't totally clear team consistency team followed the rules eventually wins the day they decide to take the photo down but before they can pull the lever word starts making its way up the chain and internally
00:52:25within facebook according to my sources and executive under zuckerberg sent down an order we were essentially told make the exception huh I don't care what your guidelines say i don't care what your reason is the photo stands you're not taking this down yes yes that's what happened this
00:52:50decision means that facebook has just become a publisher but they don't think maybe they have but they've made a news judgment and just willy nilly they've become cbs abc new york times helped to being atlantic monthly and all these other things all at once they just become a
00:53:07news organization yeah and this brings up a legal question that that's at the center of this conversation about free speech like is facebook a sort of collective scrapbook for us all or is it a public square where you should be able to say whatever you want or yeah
00:53:24is it now a news organization that's transparency let me know when you get i'm sorry to interrupt but let me get the one final question that kind of relates to what you're talking about in terms of what exactly facebook is and this question has been popping up a
00:53:37lot recently in fact it even came up this past april when zuckerberg was testifying in front of congress think about one hundred forty million americans get their news from facebook so which are you are you a tech company Are you the world's largest publisher senator this is a
00:53:57i view us as a tech company because the primary thing that we do is build technology and product that you're responsible for your content which make exactly kind of a publisher right Well i agree that we're responsible for the content but i don't think that that's incompatible with
00:54:11fundamentally it at our core being a technology company where the main thing that we do is have engineers and build products basically zuckerberg and others that the company are arguing no there not a news organization why what would be the downside of the well facebook currently sits on
00:54:26this little idyllic legal island where they can't be held liable for much of anything there subjected to few regulations however were they to be seen in the eyes of the court as a media organization that could change but setting that aside what what really strikes me about all
00:54:45of this is here you have a company that really up until this point has has been crafting a set of rules that are both as objective as possible and can be executed as consistently as possible and they've been willing to sacrifice rather large ideas in the name of
00:55:05this they for example privilege which we talked about they decided was too geographically bound to allow for one consistent rule but if you ask me there's nothing more subjective or geographically bound than what people find interesting or important what what people find newsworthy given and i'll give you
00:55:27and i'll give you a great example of this that happened just six months after the boston marathon bombing when this video starts being circulated out of northern mexico and it's ah video of ah of a woman being grabbed and forced onto her knees in front of a camera
00:55:44and then a man with his face covered grabs her head pulls her head back in slices her head off right in front of the camera and this video starts being spread i can't count how many times like just reading my twitter feed i've been like ah you know
00:55:58like one person who came across this video or at least dozens of others like it was shannon young my name is shannon young i am a freelance radio reporter i've been living here in mexico for for many years now and her beat is covering the drug war and
00:56:12doing so years back she noticed this strange phenomenon it first caught my attention in early two thousand ten should be checking social media you know you're scrolling through your feet and you know you'd see all this news people say now there was this three hour gun battle and
00:56:27intense fighting all weekend long folks were posting about clashes between drug cartels government forces But then when shannon would watch the news that night Really yes and i know you're sorry mr india She'd see reports on the economy and soccer results but the media wasn't covering it There'd
00:56:46be no mention of these attacks nothing to dio with the violence And so she and other journalists tried to get to the bottom of this reporter's in mexico city would contact the state authorities and you know public information officer and they'd be like shootings bombings What are you
00:57:00talking about Nothing's going on We have no reports of anything these air just internet rumors The government even coined a term for these sorts of posts The famous phrase of the time was collective psychosis These people are crazy because you know they didn't want the situation to seem
00:57:15out of control But then a video was posted Yeah It opens looking out the windshield of a car on a sunny day the landscape is dry dusty and the video itself a shaky clearly shot on a phone Come on then woman taping starts talking and this woman she
00:57:43just narrates As they drive along this highway she hands the phone from the passenger window toothy to the windshield focusing in on these two silver destroyed pickup trucks and she's saying look at these cars over here there you know shot up and give me a minute look here
00:58:07look here you know this eighteen wheeler is you know totally abandoned it got shot up at one point Sheep sticks the phone out the window to show all of the bullet casings littering the ground and she just you know turned the official denial on its head The government
00:58:25was saying there is no violence here were cars riddled with bullets It was impossible to dismiss and from then on you had more and more citizens citizen journalists uploading anonymously video of the violence The's Low five shaky shots of shoot outs dismemberments Headings mean bodies hanging dangling off
00:59:00of overpasses To prove to the world that this was really happening They say we're not crazy It's a cry for help yeah which brings us back to that beheading video we mentioned a bit earlier yeah that video of the beheading a lot of people are uploading it condemning
00:59:30the violence of the drug cartels and when it started showing up on facebook much like with the boston marathon bombing photo this team of people they sat down in a room looked at the policy wait the arguments and my argument was it was okay by the rules during
00:59:44the boston bombing why isn't it okay now particularly given that it could help leaving this up means we warn hundreds of thousands of people of the brutality of these cartels and so we kept it up however it's fucking wrong it's why i think it's utterly irresponsible and in
01:00:03fact quite despicable of them two people found out a little neighbor kids that don't need to see shit like that backlash is there really any justification for allowing these videos people is powerful is david cameron weigh in on this decision today The prime minister strongly criticized the move
01:00:19saying we have to protect children from this stuff david cameron tweeted it's irresponsible of facebook to post beheading videos people were really upset because of what it was showing and so according to my sources some of the folks involved in making this decision to leave it up were
01:00:36once again taken into an executive's office and so we went up and there was a lot of internal pressure to remove it and i'd go to my boss and say hey look this is the decision we made i recognize this is controversial i want to let you know
01:00:49why we made these decisions and they made their case there are valid and important human rights reasons why you would want this to be out there to show the kind of savagery and she vehemently disagreed with that they took another approach arguing that if we take this down
01:01:04you're deciding to punish people who were trying to raise awareness again she wasn't budging and just didn't get didn't get past that and ultimately i was overruled and we removed it just because there was pressure to do so the same people that six months prior told them to
01:01:24leave it up because it was newsworthy said take the video down Facebook this week reversed a decision in banda video posted to the site of a woman being beheaded In a statement facebook said quote when we review if you want the one from boston and you probably should
01:01:39have the one from mexico in right it was a mistake yeah i think it was a mistake because i i felt like like why do we have these rules in place in the first place and and and it's not the it's not the only reason but decisions like
01:01:58that or the thing that precipitated me leaving leaving yeah not too long after that incident a few members of the team decided to quit and what i think this story shows is that facebook has become too many different things at the same time so facebook is now sort
01:02:21of a playground it's also an r rated movie theater and now it's the front page of a newspaper that's all those things at the same time it's all those things at the same time and and what we the users are demanding of them is that they create a
01:02:36set of policies that are just and the reality is justice means a very different thing in each one of these settings justice would mean that the person in mexico gets told the truth in mexico by facebook and the little boy in england doesn't have to look at something
01:02:51gory and horrible in england but you can't put them together because they clash exactly so how do you solve that I don't know i think it's important to keep in mind that even if you have the perfect set of policies that that somehow managed to be just in
01:03:11different settings and that khun b consistently enforced the people at the end of the day making these decisions there still there still people they're still human beings is this working or no i can hear you yeah great Ok at long last we've figured it out huh Yeah clearly
01:03:34i spoke to one woman who did this work for facebook I just want to be anonymous I don't want them to even know that i'm doing it because they might file charges against me will call her marie she's from the philippines where she grew up on a coffee
01:03:48farm Yeah that's my father's group and i didn't know that coffee wass only for adults she said many afternoons while she was growing up she and her mother would sit together like outside sipping their coffee and tuning into their shortwave radio This is the voice of america washington
01:04:09d c and they'd sit there listening to the voice of america silence I'm going to ask that we all bow our heads and pray she said one of her favorite things to catch on voice of america where billy graham certain billy gray hum one of the great evangelists
01:04:26our father we thank thee for this love of god that reaches around the world and engulfs all of mankind but then fast forward fifty years to twenty ten and marie is consuming a very different sort of american media yeah the videos were the ones that affected me there
01:04:51were times when i felt really bad that i am a christian and then i look into this things she became a content moderator back in twenty ten and was actually one of the first people in the philippines doing this work i usually had the night shift in the
01:05:06early morning or in a dawn from two a m to four a m she worked from home and despite it being dark out she'd put blankets up over the windows so no one could see in at what she was looking at she locked the door to keep our
01:05:21kids out i have to drive them away or i would tell them that it's adult thing making a watch and she and the other moderators on her team who lived throughout the philippines they were trained on the guidelines on this rule book there were policies that we have
01:05:37to adhere to but some of us were we're just clicking pass pass pass even if it's not really passed just to finish just to get through the content fast enough and in some cases she thinks a number of the moderators are doing it as a form of retaliation
01:05:54for the low rate people were pissed at the low pay if i can ask how much what How much were you making an hour doing this a sparse i remember it we were paid like two dollars and fifty cents per hour Marie wouldn't say whether or not this
01:06:11low wage led her to just let things through but she did say based on my conservative background there are things that that i cannot look objectively at so i reject many of the things that i think are not acceptable really of course she said whether something was outside
01:06:36the rules or not if her gut told her to she just took it down Whenever it affects me a lot i would click the button off like it's a violation because if it's going toe disturbed the young origins then he should not be there So like if there's
01:06:55a nude birdsong whether it was a breastfeeding photo or anatomy video or a piece of art i would consider it as pornography and then click right away It's a violation You took the law into your own hands You went vigilante Yeah uh there's something So yeah i have
01:07:19to protect kids from those evil oh side of humankind No Right What No Where does that leave you feeling That leave you feeling that this is just that The end this's Just undoable Um i think they will inevitably fail but they have to try And i think we
01:08:27should all be rooting for them Thiss episode was reported by simon adler with help from tracy hunt and produced by simon with help from bethel hopped a big thanks to sarah roberts whose research into commercial content moderation got us going a big time And we thank you very
01:09:13much for that thanks also to jeffrey rosen who helped us in our thinking about what facebook is to michael chernus whose voice we used to mask their people's voices Caroline glanville yoshiko booed raja ryan dugan ellen silver james mitchell in guy rosen of course to all the content
01:09:29moderators who took the time to talk to us And do you want to sign off Yeah i guess we should Ready You want to go first Yeah i'm chadda homerun i'm robert krulwich thanks for listening Wait Here Here Yeah Here Wait Complaining Depressed message one clinic from brooklyn
01:10:54new york radio loud was created by dad abra mod and produced by storm wheeler Dealing keefe is director Sound design Maria matters are media is our managing director Our staff includes simon adler's maggie bartolomeo back oppressor rachel crucifix david gobbles vessel happy tracy hunt that guilty Robert krulwich
01:11:15Any mcewen teeth master Melissa o'donnell arian whack at walter and molly webster with help from family A harder hodge and music gator are fact checker is michelle harris and this message

Transcribed by algorithms. Report Errata
Disclaimer: The podcast and artwork embedded on this page are from WNYC Studios, which is the property of its owner and not affiliated with or endorsed by Listen Notes, Inc.


Thank you for helping to keep the podcast database up to date.