We try to tell the difference between correlation and causation.
United States


00:00:00support for this podcast and the following message come from a template for understanding big debt crises a new book from ray Dalio founder of Bridgewater associates available as a free PDF on principles dot com and for purchase on Amazon Hey just had that this episode originally aired in
00:00:19April twenty thirteen ten years ago Charlie wheeling was living in Chicago and his wife faced a classic dilemma for parents in cities they're trying to figure out where to send the daughter to kindergarten and then we got wind that there was a Chicago public school in our neighborhood
00:00:36that was doing some interesting things in that seem like it would be a great place to send our oldest daughter to school seems like an easy choice but Charlie was raised out in the suburbs of Chicago and to people in the suburbs of Chicago the city's public school
00:00:51system seem like horror a terrible place to send your kids so my mother I mean I think the best metaphor I can offer is that she was stricken to the point that she kind of had to lie on the couch with a compress on her head at the
00:01:04prospect of her grandchildren her beloved grandchildren spending their formative years in a Chicago public school literally willing was teaching public policy at the time at the university of Chicago so he had something that a lot of us wish we had when we were having fights with our parents
00:01:18access to data and a grasp of statistics and Charlie started to tell his mother that she was falling prey to one of the classic fallacies of our time sure the Chicago public school system might have lower test scores higher dropout rates than private schools are even schools in
00:01:34suburbs but that doesn't mean that the Chicago public school system caused those low test scores are high dropout rates Horton thing in figure out whether a school is good not good somewhere between is how much value is added to the kids who walk in the front door and
00:01:53in suburban areas in in public schools and some charter schools the kids who walk in the front door have motivated highly educated affluent parents so those kids are gonna do very well no matter what happens and in the Chicago public schools you had a really tough demographics so
00:02:10they walked in the front door with a lot of disadvantages this argument Charlie was having with his mom is over this really basic questions that we all struggle with all the time whether we know it or not that basic question what causes what in the human brain is
00:02:25really program to answer this question constantly I mean this is how we have survived to dominate this planet right what made that noise bear made that noise what because my hand hurt fire because my hand hurt we're so eager to figure out what causes what that we often
00:02:41get it wrong %HESITATION I wore my like you have to the game and my team won so I am never taking off this happened hello and welcome to planet money I'm Jacob Goldstein and I'm Robert Smith today on the show what causes what or if you are the
00:02:59fancy about it correlation and causation support for this program comes from simply safe home security simplisafe is completely wireless protection for your home the company was founded by an electrical engineer whose friends were burglarized they thought most home security systems at the time were too complicated and too
00:03:20expensive so he built simply save the system can be self installed in under an hour and with simplisafe there are no annual contracts learn more at simply save dot com slash planet money I'm of your Eisenberg join me on NPR's ask me another as we challenged contestants and
00:03:38celebrities to nerdy word games music parodies and wonderful trivia find is every week on the NPR one apple and where ever you listen to podcasts our guest today Charlie Wieland you may know him as the author of a book called naked economics we've talked about it before on
00:03:54the show truly has a new book out it's called the get statistics at Charlie did you choose the title naked statistics so that people wouldn't really notice the statistics part of things we did in fact I did a book signing and hand over and I saw a woman
00:04:07walking out of the store with fifty shades of grey and they could statistics I thought okay she may be sorely disappointed with one or the other so the book sort of covers a lot of ground in statistics we're not gonna try and cover it all today we're really
00:04:21gonna focus on this one thing on this one question what causes what and really this question is at the core of how everybody makes all kinds of decisions I mean if you think about it every decision government makes every decision a parent makes with their kid almost any
00:04:37decision you can think of it's trying to call some outcome and statistics can be very helpful in trying to figure out what causes what but as you show in the book statistics can also totally screw you over yeah we want to start with a specific example in your
00:04:52book Charlie and also a really serious issue which is that older women especially after menopause they have an increased risk of certain diseases and of course researchers are passing by this and and they they wanted to look into it so so tell us what happened when there was
00:05:06a huge longitudinal study it was a study of nurses over many many years decades even it was found that women as they approached menopause who were given hormone replacement therapy which is estrogen had lower rates of heart disease and other diseases that are prominent killers of women and
00:05:25you know to put out this wasn't just a random connection this wasn't boy people wearing purple shoes have less click cancer there was at least some biological corroboration Lisa theory that would make sense which is look what happens as you age is a woman your estrogen levels go
00:05:40down we're seeing this association between lower estrogen and bad outcomes so then estrogen therapy if it shows that has positive effects on things like heart disease would appear to be a beneficial strategy and this was like a big serious study I mean this was done by Harvard people
00:06:00looked at this is a large sample size and doctor saw this and they said well if there is a correlation between reduced risk of disease and estrogen we're gonna start prescribing estrogen right made medical sense to got to the point where not only were doctors comfortable pursed being
00:06:16estrogen for thousands and thousands of women to be kind of became what you did when you got older but it there was even a movement to start giving estrogen to men now scientists looked at this and they said okay %HESITATION there may be a correlation between taking estrogen
00:06:33having reduced heart disease but there could be a lot of other factors here maybe the people who are more likely to go there doctor ask for estrogen are just healthier people maybe there are other reasons we don't know about and so the site is did something such as
00:06:46do all the time and it's called a randomized trial that explain to me what they did they took another large group of women and they randomly assigned them to one of two groups just like your seventh grade science project the treatment group where the control group and one
00:07:01group got the estrogen the other group did not and what they found was was really scary scary and tragic they found that the opposite appears to be true which is that the women who got estrogen therapy in this controlled experiment actually had higher rates of heart attack stroke
00:07:19breast cancer and other things that were quite damaging and then when they looked back at the number of women who had been prescribed estrogen they assumed that thousands of women most likely had these events or died prematurely because of what had been really fundamentally a correlation and not
00:07:38a causal relationship and in fact the step there is a causal relationship it went the other direction that estrogen was actually bad for the health of women and this is obviously a particularly large particularly tragic case of of seeing correlation and mistakenly thinking that there is causation but
00:07:55this kind of error it happens all the time it happens everywhere I mean when we talk about a lot on the show is the stock market or to the extent we talk about the stock market you know that the daily moves the stock market is up today or
00:08:07it's down today basically every news story about the stock market if it's up they just find some correlation find some good news and and assume that that good news is causing it to go up on the same day if the stock market we're down they would find some
00:08:21other news some bad news and assume that bad news is causing it to go down yeah on any give a thousand things are having so I read a story not long ago in the Wall Street journal the admittedly was a little tongue in cheek but in the run
00:08:33up to the masters it pointed out that when Tiger Woods was winning tournaments the Dow Jones industrial average consistently did better well that I should be tracking his golf scores and not watch the masters in other tournaments you know with your your broker on the line %HESITATION and
00:08:52of course that is clearly just correlation and I feel like when we make this jump when we wanna go from finding correlations to actually figuring out causes that's when we sort of get to the edge of statistics right it seems like statistics is amazing at figuring out correlations
00:09:09and even you know getting rid of extraneous variables and whatever but it as far as I can tell statistics on its own is not enough to get us to causation to get the causation it seems like people who are studying the stuff alternately need to come up with
00:09:23sort of clever tricks internally in your book you have a number of experiments that researchers have done to get to this question of what causes what and and whether what they loved was about rhyme and that's basically crime in in DC in the district of Columbia because this
00:09:39is huge policy question about cops and crime if you just hire more cops will crime go down and it's really hard to tease out because obviously high crime cities have an incentive to hire a lot of cops and so the the date is really messy but you describe
00:09:54a way in which they figured out this problem they figured out in the district of Columbia right so there were some very clever researchers and they said it will we need to find an example where we get a lot of police for reasons unrelated to crime that we
00:10:12would think is a relatively rare occurrence why the world would you create a lot of police who didn't have a lot of crime and they happened upon and this is what great researchers do the terrorism alert system so you know all those colors orange red whatever else they
00:10:25are by law because Washington DC is likely to be a terrorism target when the terror alert level goes to orange then extra police are put on the mall in other parts of Washington to protect against terrorists has nothing to do with street crime or things like that so
00:10:45they could then ask on orange alert days when their extra police on the streets for reasons unrelated to street crime what happens to street crime does it go down and in fact it does it turns out that when you have the extra police there for terrorism related reasons
00:11:02there on the streets they make the streets safer and things like murder robbery assault go down and that to my mind is a very clever way of establishing a causal relationship between at least in Washington DC between more police and less crime as long as we assume that
00:11:17the robbers aren't hiding in their rooms because they're afraid of the elevated terror level it is entirely possible you know as he in that vein where these they considered was is it possible that tourists were less likely to visit Washington or to go out in the ballot so
00:11:33there's less violence on High Street infants are there fewer victims and you know what they did they checked that hypothesis by looking at ridership levels on the metro system and they actually were not diminished on high torque tear day so they suggested the number of victims was largely
00:11:49unchanged even in this very clever universe of finding this very particular time there are ways you could trip up I mean that fewer tourists hypothesis seems plausible and they tested it and know they're the same number of tourists yes it's so you're you're no even then it's tough
00:12:05to establish causality but you're building a case it's it's almost like circumstantial evidence in a court room where you're getting closer to the point where you can be persuaded that yes in fact this person did that crime or a causes be or that police reduce crime after the
00:12:24break we asked Charlie we live in is it worth the money to send your kids to eat ivy league school and how do you figure that out support for NPR and the following message come from the business platinum card from American Express platinum enhances life's moments both big
00:12:42and small so that you can do business to the fullest don't do business without it I'm Linda Holmes there's more stuff to watch these days than you can ever get to that's why we make pop culture happy hour twice a week we give you the low down on
00:12:57what's worth your time and what's not find pop culture happy hour on the NPR one apple or wherever you get your podcasts so Charlie how old your kids now they're now fifteen twelve and ten so you pretty soon you're going to be facing a question that a lot
00:13:13of people face which is which is colleges and where to send your kids to college and we're told that clever ways to tease this out %HESITATION you touch a book about basically schools like Harvard and Yale these these big name schools is it worth while to send your
00:13:29kid there will it make their life will it will it cause great success if you some to those schools rather than other schools and and this is another one that's problematic issues like cops and crime right and you can imagine why this is so hard to study people
00:13:42are not randomly assigned to Harvard verses a local community because the kids who get into Harvard our highly talented that's why they got into Harvard so when you compare that the wages are life outcomes of people went to Harvard versus those who went to lesser schools it's that
00:14:00is the the classic apples to oranges comparison there's really no legitimacy to the that comparison whatsoever on the other hand it's a very difficult thing to study either Harvard nor most families would agree to participate in the experiment where you had a fifty percent chance of being sent
00:14:16to Harvard a fifty percent chance of being sent to a community college or not being able to go to college at all so it's hard to tease out the value added at these highly selective schools but you describe in your book away that some researchers manage to actually
00:14:30if not do that at least come close right with a very clever experiment in this was done by two researcher Stacy Dale and Alan Krueger they found a large group of students who had been accepted to a group of schools that they described as the highly selective schools
00:14:47so it's the Ivy League and a whole bunch others you would recognize and those same students had also been selected %HESITATION admitted to a group of schools that were less traditionally competitive less highly selective so they were good enough to get into Harvard some of them subsequently went
00:15:05to the highly selective schools and some of them didn't and what that sets up it's a nice comparison group where everybody in the sample is smart enough to get into what we describe usually is the very best schools some went to those schools some didn't so we can
00:15:21then compare their life outcomes ten or fifteen years later end it turns out that there doesn't appear to be any great advantage in fact I think no statistically significant event at all in terms of wages for most students going to the highly selective schools with one notable exception
00:15:39which is for minority students there does appear to be in a bad but for the vast majority of folks in the sample there was no real huge advantage to getting into and going to one of these highly selective schools well so you have three kids you are a
00:15:54professor of public policy you know statistics you wrote an entire book on statistics are you now the dad that saying you don't need to go to Harvard you need to go to Yale University Chicago well it given that I taught at the university of Chicago I now to
00:16:07teach at Dartmouth I went to Princeton I what do you know I'm I'm told by hardly yeah I know I know I think there's a pressure to go to school I'm not going to pressure them at all I think that's a take away is that I do think
00:16:19there are a lot of benefits in terms of that work of fact and and things like that that are a positive outcome related to going to those schools but I'm certainly not going to put pressure on them and I'm certainly not going to cling to the delusion that
00:16:35the thing that's going to shape the trajectory their lives most is whether they get in or don't get in because really in this goes all the way back to the way I felt about the Chicago public schools it's going to be the talent that they bring to the
00:16:47front door that matters most so what will end up happening with your kids go into Chicago public schools and how did your brother and a feeling about well there's a nice little interlude here in that we got an objective measure of how they were doing in the Chicago
00:17:03public schools because after they've been in school for six years so my oldest started and the Chicago but she was in kindergarten I had a sabbatical and we moved to Hanover New Hampshire Hanover New Hampshire is a privileged little oasis that spends an ungodly amount on its public
00:17:20schools they would qualify is very good schools by any superficial measure so these are the good good certainly in my mother's side this is exactly where they should have been the first place so for one year all three kids in first grade fourth grade and sixth grade transferred
00:17:38to the Hanover public school so we can now see whether they're illiterate and have all kinds of other problems and in fact they did extraordinarily well in the hand or schools would suggested that they were getting a very fine education in the Chicago public schools %HESITATION and that
00:17:53the value added there was excellent did you convince your mother about the Chicago public schools you know to her universal she was always a little wary about safety that was not really an issue either but it wasn't just about the eight hours I would say in her defense
00:18:08that she really did come around and %HESITATION and in part because of our experience and because you know was a genuine concern on her part you half the grantor that that when she realized the kids were turning out okay she has actually shifted her opinion and she is
00:18:24a little more skeptical of people who pay extraordinary amounts were devote great effort to try and get into a private school when the kids three years old into thinking that this is going to radically change that child's life so I do think you know I'm not sure that
00:18:39she be eager to have be send if you she still than what my daughter going to one of the highly selective high schools in Chicago but she did come around to some extent so I'll give her credit for that that seems like a perfect place to wrap this
00:18:51up Charlie will and thank you so much for walking us through the world of statistics that was good to be back with you thank you I should say that we have established that there is a high correlation between appearing on planet money and professional success exactly I'm glad
00:19:06to be a part of that all because there have have a question about correlation and causation or do you have an idea for an episode send us an email planet money at NPR dot org we're on Facebook Twitter and Instagram also at any money she McKean produce this
00:19:30re run our editor is brain our staff and our supervising producer Alex

Transcribed by algorithms. Report Errata
Disclaimer: The podcast and artwork embedded on this page are from NPR, which is the property of its owner and not affiliated with or endorsed by Listen Notes, Inc.


Thank you for helping to keep the podcast database up to date.