WEBVTT
1
00:00:02.080 --> 00:00:05.679
Welcome back to the Reality Gap. So today we look
2
00:00:05.759 --> 00:00:10.000
at the phrase studies show. You've probably heard it a
3
00:00:10.039 --> 00:00:16.089
thousand times, and it sounds authoritative, scientific, almost unquestionable. But
4
00:00:16.109 --> 00:00:19.149
here's a question we rarely ask. Did this study actually
5
00:00:19.250 --> 00:00:24.250
show what the headline claims? More times than not, it doesn't.
6
00:00:24.969 --> 00:00:27.910
Consider a headline announcing that drinking coffee reduces the risk
7
00:00:27.929 --> 00:00:32.170
of developing a disease by 30%. Sounds impressive, but suppose
8
00:00:32.189 --> 00:00:34.609
the underlying research found that people who drank coffee had
9
00:00:34.649 --> 00:00:36.990
a lower incidence of the disease than those who did not.
10
00:00:37.899 --> 00:00:38.920
That's an important difference.
11
00:00:39.219 --> 00:00:42.219
Researchers may have identified a correlation, not demonstrated that coffee
12
00:00:42.280 --> 00:00:45.060
caused the reduction. Most of the time, a lot of
13
00:00:45.100 --> 00:00:47.799
these studies are correlational or animal studies that we see.
14
00:00:49.100 --> 00:00:52.329
Back to the coffee drinking study, perhaps coffee drinkers exercised
15
00:00:52.369 --> 00:00:55.689
more frequently. Perhaps they had different dietary habits, incomes, or
16
00:00:55.770 --> 00:00:56.670
access to health care.
17
00:00:56.909 --> 00:00:59.350
I know what you're saying. Well, maybe they controlled for that.
18
00:00:59.590 --> 00:01:00.030
They could have.
19
00:01:01.380 --> 00:01:03.679
And that would make it stronger, but it's still correlational.
20
00:01:04.260 --> 00:01:07.400
Researchers can statistically adjust for some of these variables, but
21
00:01:07.480 --> 00:01:11.909
observational studies can never eliminate every alternative explanation. And this
22
00:01:11.969 --> 00:01:14.890
is where the first reality gap really appears, is the
23
00:01:14.930 --> 00:01:18.829
famous line, correlation is not causation. Now consider a second problem,
24
00:01:19.370 --> 00:01:22.230
a study involving 40 college students who completed a memory
25
00:01:22.269 --> 00:01:26.060
test after listening to classical music. The researchers discovered a
26
00:01:26.079 --> 00:01:31.319
statistically significant improvement. A headline announces scientists prove classical music
27
00:01:31.379 --> 00:01:32.180
makes you smarter.
28
00:01:32.859 --> 00:01:33.439
But did they?
29
00:01:34.060 --> 00:01:37.000
The study might have measured performance on one particular memory
30
00:01:37.060 --> 00:01:40.340
task and might not have measured intelligence at all. And
31
00:01:40.420 --> 00:01:43.420
even if the result was statistically significant, the improvement might
32
00:01:43.459 --> 00:01:47.060
have been too small to matter in everyday life. So
33
00:01:47.099 --> 00:01:51.359
statistical significance does not automatically establish practical importance or what
34
00:01:51.379 --> 00:01:52.640
we call the effect size.
35
00:01:54.209 --> 00:01:55.810
How about the problem of generalization?
36
00:01:55.829 --> 00:01:58.689
A study involving 40 college students does not necessarily tell
37
00:01:58.730 --> 00:02:02.510
us what will happen among children, older adults, people without
38
00:02:02.650 --> 00:02:07.799
college degrees, or neurological disorders. Yet headlines frequently transform findings
39
00:02:07.819 --> 00:02:13.699
from really narrow populations into universal claims. We are particularly
40
00:02:13.740 --> 00:02:16.180
receptive to research that confirms what.
41
00:02:15.800 --> 00:02:19.159
We already believe as well. This is called confirmation bias.
42
00:02:19.199 --> 00:02:21.639
If you believe social media is destroying society, you're more
43
00:02:21.699 --> 00:02:24.460
likely to notice research suggesting.
44
00:02:23.879 --> 00:02:26.530
Negative psychological effects without.
45
00:02:26.250 --> 00:02:29.789
Really even doing due diligence on checking out the studies
46
00:02:29.830 --> 00:02:38.419
and how good they are. Both findings might contain useful information.
47
00:02:38.479 --> 00:02:41.580
Neither necessarily captures the entire picture. The problem becomes more
48
00:02:41.639 --> 00:02:44.080
complicated when scientific findings move through the media.
49
00:02:44.840 --> 00:02:46.560
Researchers publish the study, and a lot.
50
00:02:46.460 --> 00:02:52.800
Of times they're good publishing, and they're honest about the findings. But, however,
51
00:02:52.819 --> 00:02:56.509
they have limitations and qualifications that the media doesn't represent.
52
00:02:57.810 --> 00:03:01.449
So a university press simplifies the findings. A journalist writes
53
00:03:01.530 --> 00:03:04.849
a compelling headline. An influencer turns that headline into a
54
00:03:04.870 --> 00:03:07.449
30-second video. By the time the information reaches you and
55
00:03:07.490 --> 00:03:11.289
millions of others, the original scientific uncertainty has disappeared.
56
00:03:12.430 --> 00:03:14.930
This does not mean science is unreliable. It's quite the opposite.
57
00:03:15.569 --> 00:03:19.629
Science becomes powerful because its conclusions remain open to scrutiny.
58
00:03:20.490 --> 00:03:23.310
What's actually unreliable are the sources you're getting it from
59
00:03:23.650 --> 00:03:26.819
most of the time. So the next time someone tells
60
00:03:26.840 --> 00:03:29.280
you that studies prove something, ask three questions. What did
61
00:03:29.300 --> 00:03:33.199
the researchers actually measure? Who participated in the study? And
62
00:03:33.240 --> 00:03:36.759
does the evidence establish causation or merely an association?
1
00:00:02.080 --> 00:00:05.679
Welcome back to the Reality Gap. So today we look
2
00:00:05.759 --> 00:00:10.000
at the phrase studies show. You've probably heard it a
3
00:00:10.039 --> 00:00:16.089
thousand times, and it sounds authoritative, scientific, almost unquestionable. But
4
00:00:16.109 --> 00:00:19.149
here's a question we rarely ask. Did this study actually
5
00:00:19.250 --> 00:00:24.250
show what the headline claims? More times than not, it doesn't.
6
00:00:24.969 --> 00:00:27.910
Consider a headline announcing that drinking coffee reduces the risk
7
00:00:27.929 --> 00:00:32.170
of developing a disease by 30%. Sounds impressive, but suppose
8
00:00:32.189 --> 00:00:34.609
the underlying research found that people who drank coffee had
9
00:00:34.649 --> 00:00:36.990
a lower incidence of the disease than those who did not.
10
00:00:37.899 --> 00:00:38.920
That's an important difference.
11
00:00:39.219 --> 00:00:42.219
Researchers may have identified a correlation, not demonstrated that coffee
12
00:00:42.280 --> 00:00:45.060
caused the reduction. Most of the time, a lot of
13
00:00:45.100 --> 00:00:47.799
these studies are correlational or animal studies that we see.
14
00:00:49.100 --> 00:00:52.329
Back to the coffee drinking study, perhaps coffee drinkers exercised
15
00:00:52.369 --> 00:00:55.689
more frequently. Perhaps they had different dietary habits, incomes, or
16
00:00:55.770 --> 00:00:56.670
access to health care.
17
00:00:56.909 --> 00:00:59.350
I know what you're saying. Well, maybe they controlled for that.
18
00:00:59.590 --> 00:01:00.030
They could have.
19
00:01:01.380 --> 00:01:03.679
And that would make it stronger, but it's still correlational.
20
00:01:04.260 --> 00:01:07.400
Researchers can statistically adjust for some of these variables, but
21
00:01:07.480 --> 00:01:11.909
observational studies can never eliminate every alternative explanation. And this
22
00:01:11.969 --> 00:01:14.890
is where the first reality gap really appears, is the
23
00:01:14.930 --> 00:01:18.829
famous line, correlation is not causation. Now consider a second problem,
24
00:01:19.370 --> 00:01:22.230
a study involving 40 college students who completed a memory
25
00:01:22.269 --> 00:01:26.060
test after listening to classical music. The researchers discovered a
26
00:01:26.079 --> 00:01:31.319
statistically significant improvement. A headline announces scientists prove classical music
27
00:01:31.379 --> 00:01:32.180
makes you smarter.
28
00:01:32.859 --> 00:01:33.439
But did they?
29
00:01:34.060 --> 00:01:37.000
The study might have measured performance on one particular memory
30
00:01:37.060 --> 00:01:40.340
task and might not have measured intelligence at all. And
31
00:01:40.420 --> 00:01:43.420
even if the result was statistically significant, the improvement might
32
00:01:43.459 --> 00:01:47.060
have been too small to matter in everyday life. So
33
00:01:47.099 --> 00:01:51.359
statistical significance does not automatically establish practical importance or what
34
00:01:51.379 --> 00:01:52.640
we call the effect size.
35
00:01:54.209 --> 00:01:55.810
How about the problem of generalization?
36
00:01:55.829 --> 00:01:58.689
A study involving 40 college students does not necessarily tell
37
00:01:58.730 --> 00:02:02.510
us what will happen among children, older adults, people without
38
00:02:02.650 --> 00:02:07.799
college degrees, or neurological disorders. Yet headlines frequently transform findings
39
00:02:07.819 --> 00:02:13.699
from really narrow populations into universal claims. We are particularly
40
00:02:13.740 --> 00:02:16.180
receptive to research that confirms what.
41
00:02:15.800 --> 00:02:19.159
We already believe as well. This is called confirmation bias.
42
00:02:19.199 --> 00:02:21.639
If you believe social media is destroying society, you're more
43
00:02:21.699 --> 00:02:24.460
likely to notice research suggesting.
44
00:02:23.879 --> 00:02:26.530
Negative psychological effects without.
45
00:02:26.250 --> 00:02:29.789
Really even doing due diligence on checking out the studies
46
00:02:29.830 --> 00:02:38.419
and how good they are. Both findings might contain useful information.
47
00:02:38.479 --> 00:02:41.580
Neither necessarily captures the entire picture. The problem becomes more
48
00:02:41.639 --> 00:02:44.080
complicated when scientific findings move through the media.
49
00:02:44.840 --> 00:02:46.560
Researchers publish the study, and a lot.
50
00:02:46.460 --> 00:02:52.800
Of times they're good publishing, and they're honest about the findings. But, however,
51
00:02:52.819 --> 00:02:56.509
they have limitations and qualifications that the media doesn't represent.
52
00:02:57.810 --> 00:03:01.449
So a university press simplifies the findings. A journalist writes
53
00:03:01.530 --> 00:03:04.849
a compelling headline. An influencer turns that headline into a
54
00:03:04.870 --> 00:03:07.449
30-second video. By the time the information reaches you and
55
00:03:07.490 --> 00:03:11.289
millions of others, the original scientific uncertainty has disappeared.
56
00:03:12.430 --> 00:03:14.930
This does not mean science is unreliable. It's quite the opposite.
57
00:03:15.569 --> 00:03:19.629
Science becomes powerful because its conclusions remain open to scrutiny.
58
00:03:20.490 --> 00:03:23.310
What's actually unreliable are the sources you're getting it from
59
00:03:23.650 --> 00:03:26.819
most of the time. So the next time someone tells
60
00:03:26.840 --> 00:03:29.280
you that studies prove something, ask three questions. What did
61
00:03:29.300 --> 00:03:33.199
the researchers actually measure? Who participated in the study? And
62
00:03:33.240 --> 00:03:36.759
does the evidence establish causation or merely an association?
