1
00:00:00,520 --> 00:00:03,580
JOHN: Students with strong
critical thinking skills can make
2
00:00:03,590 --> 00:00:07,870
better decisions that have real
consequences on their quality of life.
3
00:00:08,260 --> 00:00:12,769
In this episode, we explore an approach
to developing these skills that appears
4
00:00:12,780 --> 00:00:18,859
to have a persistent effect on students’
ability to avoid common cognitive biases.
5
00:00:24,139 --> 00:00:24,259
[MUSIC]
6
00:00:24,490 --> 00:00:28,810
Thanks for joining us for Tea for
Teaching, an informal discussion
7
00:00:28,810 --> 00:00:31,940
of innovative and effective
practices in teaching and learning.
8
00:00:32,529 --> 00:00:35,920
REBECCA: This podcast series is
hosted by John Kane, an economist…
9
00:00:36,389 --> 00:00:38,519
JOHN: …and Rebecca Mushtare,
a graphic designer…
10
00:00:38,869 --> 00:00:42,169
REBECCA: …and features guests doing
important research and advocacy work
11
00:00:42,309 --> 00:00:46,060
to make higher education more inclusive
and supportive of all learners.
12
00:00:49,149 --> 00:00:49,269
[MUSIC]
13
00:00:55,090 --> 00:00:56,579
Our guest today is Michael Bishop.
14
00:00:56,799 --> 00:00:59,809
He is professor of philosophy
at Florida State University.
15
00:01:00,020 --> 00:01:04,669
His research interests focus on two areas:
rationality [how should we think about
16
00:01:04,669 --> 00:01:08,779
important things] and happiness [what is
lasting happiness, and how can we get it].
17
00:01:09,010 --> 00:01:12,459
He is also a co-author of a recent
study investigating the impact
18
00:01:12,459 --> 00:01:16,029
of critical thinking classes
on common cognitive biases.
19
00:01:16,500 --> 00:01:17,120
Welcome, Michael.
20
00:01:17,470 --> 00:01:17,890
MICHAEL: Thank you.
21
00:01:17,890 --> 00:01:18,839
Glad to be here.
22
00:01:19,159 --> 00:01:20,209
JOHN: Thank you for joining us.
23
00:01:20,779 --> 00:01:23,979
Today's teas are:… Michael,
do you have tea by any chance?
24
00:01:24,389 --> 00:01:25,629
MICHAEL: No, not today.
25
00:01:25,629 --> 00:01:31,639
I usually drink tea in my office, and when
I do, it's usually a Chinese black tea.
26
00:01:31,939 --> 00:01:33,219
REBECCA: Always a nice choice.
27
00:01:33,470 --> 00:01:37,139
Today I have an English breakfast
tea, nothing that exciting,
28
00:01:37,610 --> 00:01:41,220
JOHN: And I just have nothing that
exciting either, a Lady Grey today.
29
00:01:41,599 --> 00:01:43,020
REBECCA: We got to step up our game, John.
30
00:01:43,509 --> 00:01:46,859
So, Michael, we've invited here today
to discuss the field experiment that
31
00:01:46,860 --> 00:01:51,129
is described in your co-authored paper
entitled "Critical Thinking Classes Can
32
00:01:51,129 --> 00:01:55,429
Reduce Common Biases: Results from a
Field Experiment.” But before we discuss
33
00:01:55,430 --> 00:01:59,449
the study, can we start a little bit
with the concept of critical thinking?
34
00:01:59,519 --> 00:02:00,919
You teach critical thinking.
35
00:02:00,930 --> 00:02:01,939
What are you trying to teach?
36
00:02:01,939 --> 00:02:04,029
And what makes someone
a good critical thinker?
37
00:02:04,489 --> 00:02:05,979
MICHAEL: Yeah, I think
that's a great question.
38
00:02:06,219 --> 00:02:11,450
One of my favorite sayings is the
philosopher George Santayana said that a
39
00:02:11,450 --> 00:02:15,349
fanatic is someone who, having lost sight
of their goals, redoubles their efforts.
40
00:02:15,649 --> 00:02:20,759
And so, in teaching critical thinking,
I think it's important to start with
41
00:02:20,759 --> 00:02:23,069
an idea of what you want to teach.
42
00:02:23,069 --> 00:02:25,239
And so there's a lot of possible answers.
43
00:02:25,569 --> 00:02:29,669
Some people think that a good critical
thinker is someone who uses certain
44
00:02:29,750 --> 00:02:35,199
rational methods or rules like logic or
probability for the scientific method.
45
00:02:35,239 --> 00:02:38,759
Other people think that a rational
thinker would be an expert debater,
46
00:02:38,949 --> 00:02:44,369
someone who can detect fallacies and
build a coherent case for their views.
47
00:02:44,940 --> 00:02:48,819
Other people view good critical
thinkers as virtuous thinkers;
48
00:02:48,929 --> 00:02:53,769
someone who is careful,
open-minded, curious, intellectually
49
00:02:53,789 --> 00:02:55,479
honest, intellectually humble.
50
00:02:55,899 --> 00:03:00,950
And I actually think these views
are either wrong… so I think
51
00:03:01,030 --> 00:03:04,870
that good debaters are often
poor reasoners… or incomplete.
52
00:03:05,040 --> 00:03:07,940
They capture part of good
reasoning, but not everything.
53
00:03:08,179 --> 00:03:13,459
So basically, what I am trying to
teach in my classes is teaching
54
00:03:13,460 --> 00:03:16,149
people to ask good questions.
55
00:03:16,190 --> 00:03:21,889
Ask what I call fruitful questions,
questions that help us get better answers,
56
00:03:22,350 --> 00:03:26,549
not killer questions that end debates,
but questions that, if you're trying to
57
00:03:26,549 --> 00:03:28,840
answer them, help get us better answers.
58
00:03:29,090 --> 00:03:33,540
So when you view that as the goal,
how you go about teaching critical
59
00:03:33,549 --> 00:03:35,160
thinking becomes much easier.
60
00:03:35,770 --> 00:03:39,110
JOHN: And as you suggested, people
have many different descriptions
61
00:03:39,119 --> 00:03:40,480
of what critical thinking is.
62
00:03:40,480 --> 00:03:44,059
I know whenever we talked about the
importance of this in our Gen Ed program,
63
00:03:44,340 --> 00:03:48,159
everyone agrees that critical thinking
is extremely important, but they all
64
00:03:48,160 --> 00:03:50,459
disagree as to what exactly it is.
65
00:03:50,459 --> 00:03:53,150
So it's helpful, I think, to
start with that statement.
66
00:03:53,639 --> 00:03:57,539
In this experiment, you're
focused on five common biases.
67
00:03:57,799 --> 00:04:01,900
Can you describe the biases that
were selected for this study and
68
00:04:01,939 --> 00:04:05,639
provide some brief examples of
those biases for our listeners?
69
00:04:05,869 --> 00:04:06,229
MICHAEL: Sure.
70
00:04:06,580 --> 00:04:10,420
The way psychologists think about
these is in terms of biases.
71
00:04:10,430 --> 00:04:14,120
The way I think about them
is in terms of cheat codes.
72
00:04:14,290 --> 00:04:19,029
So there are certain cheat codes
people use in debating or in
73
00:04:19,040 --> 00:04:23,210
arguments, and cheat code arguments
are arguments that you can use to
74
00:04:23,210 --> 00:04:25,749
defend pretty much any view you want.
75
00:04:25,959 --> 00:04:31,529
And so cheat codes are really tools
of manipulation, more than tools
76
00:04:31,540 --> 00:04:34,950
of trying to figure out what the
truth is or what a good decision is.
77
00:04:35,329 --> 00:04:40,679
And so when you think about good critical
thinkers as thinkers who can ask good
78
00:04:40,680 --> 00:04:46,110
questions, the way I think about it is
when you detect a cheat code, when you
79
00:04:46,110 --> 00:04:49,439
detect somebody using a cheat code on
you, you should be able to ask them a
80
00:04:49,439 --> 00:04:53,329
question that diffuses the cheat code
and points them in better directions.
81
00:04:53,339 --> 00:04:55,739
So let me tell you some
of these cheat codes.
82
00:04:55,939 --> 00:05:00,530
One I call the focalizing cheat code,
and it might go something like this:
83
00:05:00,530 --> 00:05:06,449
We should go to war because it has
certain benefits, so, if an argument
84
00:05:06,459 --> 00:05:10,529
like that, or we should not go to
war because it has certain costs.
85
00:05:10,569 --> 00:05:14,379
If you're convinced by an argument like
that, I think that you've fallen for
86
00:05:14,379 --> 00:05:20,129
a cheat code, and the best way to see
this is to consider an example where
87
00:05:20,129 --> 00:05:21,319
where it's clear that it's a cheat code.
88
00:05:21,470 --> 00:05:26,359
So, suppose I gave you two envelopes, and
I said you can look in these envelopes,
89
00:05:26,479 --> 00:05:30,129
and you can take all the money that's
in the envelope that you choose, but
90
00:05:30,129 --> 00:05:31,429
you can only choose one envelope.
91
00:05:31,889 --> 00:05:36,739
Suppose I open one envelope, see
that there's $5 and I take it without
92
00:05:36,739 --> 00:05:38,059
looking in the second envelope.
93
00:05:38,779 --> 00:05:43,859
In that case, you'd think that I've made
a mistake, that I haven't reasoned wisely
94
00:05:43,860 --> 00:05:48,050
about this particular decision because
I didn't look in the second envelope.
95
00:05:48,440 --> 00:05:53,170
If I say, "Look, the reason I chose the
first envelope is that having $5 is better
96
00:05:53,170 --> 00:05:58,780
than not having $5.” It's true that having
$5 is better than not having $5, so there
97
00:05:58,780 --> 00:06:00,860
is an advantage to choosing envelope one.
98
00:06:01,140 --> 00:06:05,760
But when you think about it, what
makes a choice a good one isn't that it
99
00:06:05,770 --> 00:06:07,829
makes you better off than you are now.
100
00:06:08,179 --> 00:06:11,559
What makes a choice a good one is
that it makes you better off than
101
00:06:12,020 --> 00:06:13,330
what else you might have done.
102
00:06:13,770 --> 00:06:17,900
Now, economists call this opportunity
cost, as I suspect you guys know.
103
00:06:18,340 --> 00:06:23,050
And so, one thing I'm trying to do is
get my students to, when they see someone
104
00:06:23,050 --> 00:06:27,469
argue for, “Hey, we should go to war
because it's going to bring us benefits.”
105
00:06:27,760 --> 00:06:31,660
I want them to at least stop and be able
to ask the question, wait a minute, what
106
00:06:31,660 --> 00:06:33,769
are some alternatives to going to war?
107
00:06:34,049 --> 00:06:37,630
Because even if it's true that going
to war brings us benefits, and we can
108
00:06:37,630 --> 00:06:41,749
argue about that, but even if it's true,
it might not be the best thing to do,
109
00:06:41,750 --> 00:06:45,449
in the same way that choosing envelope
one, even though it brings us benefits,
110
00:06:45,459 --> 00:06:47,079
might not be the smart thing to do.
111
00:06:47,149 --> 00:06:49,500
So I call that the focalizing cheat code.
112
00:06:50,059 --> 00:06:55,259
Another cheat code that I teach is what
you might call the correlation cheat code.
113
00:06:55,259 --> 00:07:00,440
So I say, "Hey, a new healthcare plan
is passed, and healthcare costs have
114
00:07:00,440 --> 00:07:04,580
gone up 7% a year since this plan is
passed. Boy, this plan really stinks.
115
00:07:04,650 --> 00:07:09,809
It's caused our healthcare costs to
go up a lot.” This again, the question
116
00:07:09,809 --> 00:07:14,129
isn't is the conclusion right in the
war case or in this case, the question
117
00:07:14,129 --> 00:07:15,870
is: Is this a cheat code argument?
118
00:07:15,870 --> 00:07:16,619
And it is.
119
00:07:16,680 --> 00:07:20,489
And the reason is, what you don't know,
and what you really need to know, is
120
00:07:20,489 --> 00:07:25,189
how high would have costs been had
this healthcare plan not been in place.
121
00:07:25,509 --> 00:07:30,599
So again, I say this healthcare plan
is great or it's bad, and I point to
122
00:07:30,599 --> 00:07:33,219
something that happened afterwards that.
123
00:07:33,220 --> 00:07:36,289
That's what I call the correlation
cheat code because you're taking this
124
00:07:36,320 --> 00:07:40,629
correlation and you're taking it as
evidence for thinking that this policy
125
00:07:40,639 --> 00:07:43,349
has caused this particular result.
126
00:07:43,869 --> 00:07:47,189
And so again, what I want people to do
is just stop and ask the question: “Well,
127
00:07:47,189 --> 00:07:51,210
wait a minute. Maybe without the health
care plan, costs would have gone up
128
00:07:51,210 --> 00:07:54,579
even more, or maybe they would have gone
up just the same, or maybe they'd have
129
00:07:54,579 --> 00:07:58,670
gone up less.” But unless you ask the
question, it's easy to get manipulated
130
00:07:59,000 --> 00:08:01,359
into accepting views that aren't true.
131
00:08:01,669 --> 00:08:04,069
Another cheat code is the
regression cheat code.
132
00:08:04,500 --> 00:08:05,719
This is a very common one.
133
00:08:05,729 --> 00:08:10,019
So lots of coaches and parents
think that punishment is
134
00:08:10,020 --> 00:08:12,359
effective at molding behavior.
135
00:08:12,360 --> 00:08:16,509
I know that when I played sports as a
youth, my coaches thought that after
136
00:08:16,509 --> 00:08:21,709
a poor performance, if you punish the
players by making them work really hard
137
00:08:21,709 --> 00:08:23,729
in practice, they'll do better next time.
138
00:08:23,729 --> 00:08:26,649
And sure enough, after a poor
performance and after we were punished,
139
00:08:26,929 --> 00:08:28,219
we did tend to perform better.
140
00:08:28,639 --> 00:08:32,489
But again, the question is: “Did
the punishment do that, or was it
141
00:08:32,500 --> 00:08:37,319
just a return to our normal level of
performance?” So that's the regression
142
00:08:37,339 --> 00:08:40,420
cheat code, rather than just saying,
"Hey, maybe you're returning to your
143
00:08:40,429 --> 00:08:44,109
normal level of performance,” saying that
“it's the result of a punishment or the
144
00:08:44,110 --> 00:08:49,109
result of a reward.” The third one is
sunk cost cheat code, another one that's
145
00:08:49,110 --> 00:08:53,580
a favorite of economists, defending a
decision because of the resources already
146
00:08:53,580 --> 00:08:58,680
spent on it, so let's stick with this
policy or this war because of sunk costs.
147
00:08:59,040 --> 00:09:01,870
Those sunk costs are gone; they're
gone no matter what you do.
148
00:09:02,170 --> 00:09:04,719
So you might as well do
what's best for your future.
149
00:09:05,090 --> 00:09:09,920
And the last one that we looked at was the
gambler's fallacy, and that is basically
150
00:09:09,929 --> 00:09:14,510
treating statistically independent events,
as if they were statistically dependent.
151
00:09:14,539 --> 00:09:18,670
So, if a coin comes up five times
heads, and it's really a fair
152
00:09:18,709 --> 00:09:21,399
coin, thinking that it's more
likely to come up tails next time.
153
00:09:21,740 --> 00:09:24,369
So, those were the five
cheat codes we looked at.
154
00:09:24,730 --> 00:09:27,930
REBECCA: I think the soundtrack that we
have going on with the thunder is really
155
00:09:28,000 --> 00:09:29,999
emphasizing that these are cheat codes.
156
00:09:30,989 --> 00:09:31,049
[LAUGHTER]
157
00:09:31,049 --> 00:09:31,776
MICHAEL: Indeed, indeed.
158
00:09:31,776 --> 00:09:32,139
[LAUGHTER]
159
00:09:32,139 --> 00:09:33,050
REBECCA: It's very ominous.
160
00:09:34,689 --> 00:09:36,909
[LAUGHTER] Can you talk a
little bit about how you chose
161
00:09:36,910 --> 00:09:38,889
these particular cheat codes?
162
00:09:39,219 --> 00:09:39,719
MICHAEL: Sure.
163
00:09:39,919 --> 00:09:44,249
So this goes back to my
view about good reasoning.
164
00:09:44,250 --> 00:09:49,260
So 20 years ago or so, me and a friend
defended a theory of what rational
165
00:09:49,260 --> 00:09:55,120
reasoning was, and what we said was
that the good reasoning is reasoning
166
00:09:55,120 --> 00:09:59,409
that's cheap, it's accurate, and
it's focused on significant problems.
167
00:09:59,720 --> 00:10:06,759
And so the way I chose these
cheat codes is because I thought
168
00:10:06,779 --> 00:10:08,600
that they were significant.
169
00:10:08,650 --> 00:10:14,989
That is, people use them to manipulate you
when it comes to really important issues.
170
00:10:15,250 --> 00:10:19,389
So I see these cheat codes a lot
out there being used in the world
171
00:10:19,400 --> 00:10:20,959
for really important issues.
172
00:10:21,279 --> 00:10:27,099
And the cheap part came in because I
think that in thinking about how to
173
00:10:27,099 --> 00:10:32,419
teach critical thinking, I think one
thing you want to do as a critical
174
00:10:32,430 --> 00:10:34,899
thinker is to diagnose the problem.
175
00:10:34,970 --> 00:10:39,059
Why are my students thinking
badly when they do think badly?
176
00:10:39,629 --> 00:10:43,519
And I think there's three general
answers that people tend to
177
00:10:43,629 --> 00:10:47,449
think about, although often this
doesn't get talked about enough.
178
00:10:47,649 --> 00:10:52,369
So one reason students might be thinking
badly is that they don't know these rules,
179
00:10:52,379 --> 00:10:54,109
so it might be an ignorance problem.
180
00:10:54,540 --> 00:10:59,049
And if it's an ignorance problem,
then it's going to be really hard
181
00:10:59,059 --> 00:11:02,929
to teach them to use these rules
in a way that they keep using the
182
00:11:02,929 --> 00:11:04,969
rules after they leave the class.
183
00:11:05,319 --> 00:11:10,499
Because not only do they have to learn
the rules and learn to apply the rules,
184
00:11:10,690 --> 00:11:15,379
but when they leave your class, they need
to stop thinking in their old ways and
185
00:11:15,379 --> 00:11:20,900
use these rules, which tend to be hard to
remember, hard to apply in the real world.
186
00:11:20,900 --> 00:11:26,529
And so, one reason that I think studies
have not been successful or have had
187
00:11:26,539 --> 00:11:31,780
limited success in teaching people to
reason well in a way that sticks once
188
00:11:31,780 --> 00:11:36,479
they leave the classroom, is that too
often we teach critical thinking as if
189
00:11:36,639 --> 00:11:38,460
the problem is an ignorance problem.
190
00:11:38,910 --> 00:11:41,509
So we need to teach
our students new rules.
191
00:11:42,069 --> 00:11:46,389
So one reason I chose these rules
is that I speculated that they
192
00:11:46,389 --> 00:11:50,269
already knew these rules and they
sometimes applied them correctly.
193
00:11:50,660 --> 00:11:55,480
And if that's right, if they sometimes
apply these rules correctly, then I don't
194
00:11:55,500 --> 00:12:01,060
need to teach them a new rule, and I don't
need to teach them how to apply the rule.
195
00:12:01,080 --> 00:12:02,410
They're already doing that.
196
00:12:02,760 --> 00:12:08,080
All I need to do is get them to see that
it's the right rule, and then ask them,
197
00:12:08,080 --> 00:12:12,959
"Hey, why aren't you using it in these
cases over here, where it seems like you
198
00:12:12,959 --> 00:12:16,589
should still be using the rules.” So,
for example, think about the two envelope
199
00:12:16,600 --> 00:12:21,049
problem that I talked about before.
None of my students think, "Oh, yeah,
200
00:12:21,090 --> 00:12:23,740
choose the $5, that's the way to go.
201
00:12:24,090 --> 00:12:25,280
None of them think that.
202
00:12:25,590 --> 00:12:30,619
They know perfectly well that the way you
evaluate that choice is not in terms of
203
00:12:30,619 --> 00:12:36,180
“Hey I'm $5 better off than I am now,”
it's “Hey if there's $1,000 in the other
204
00:12:36,180 --> 00:12:41,449
envelope, then I'm 995 dollars worse
off than if I'd have chosen envelope
205
00:12:41,450 --> 00:12:47,400
number two.” They all see that, so I
don't have to teach them any new rules.
206
00:12:47,939 --> 00:12:52,670
All I have to do is when I say “Hey
let's go to war, it has this advantage,”
207
00:12:53,039 --> 00:12:56,100
and if they say, "Yeah, that sounds
good,” I can say, "Wait a minute.
208
00:12:56,139 --> 00:13:01,129
How is this different from the two
envelope problem?” So the main reason I
209
00:13:01,159 --> 00:13:04,810
chose these, and these aren't the only
rules I teach in my critical thinking
210
00:13:04,810 --> 00:13:09,399
class, but the main reason I choose
the rules that I do is that I think
211
00:13:09,619 --> 00:13:14,269
that my students already know these
rules and at least sometimes apply
212
00:13:14,269 --> 00:13:20,369
them, and so when I introduce them
to these rules in cases to which they
213
00:13:20,369 --> 00:13:25,080
are inclined to apply them, they give
me what I call the old bovine stare.
214
00:13:25,089 --> 00:13:27,860
They look at me like, "Well,
that's obvious. You're not teaching
215
00:13:27,860 --> 00:13:31,039
me anything I didn't already
know.” And that's what I want.
216
00:13:31,089 --> 00:13:34,830
I want them when I introduce these
rules to say, "I already knew that.
217
00:13:34,830 --> 00:13:39,690
Why are you teaching me something so
obvious?” Because then the next class
218
00:13:39,690 --> 00:13:44,559
period, or a week from now, or a month
from now, when I introduce an argument
219
00:13:44,619 --> 00:13:48,049
where they ought to be using this
rule that they said was so obvious
220
00:13:48,049 --> 00:13:51,969
and they're not, I don't have to say,
"Hey, you guys have made a mistake.”
221
00:13:52,089 --> 00:13:55,839
I can just ask them, "Okay, wait a
minute, you said this rule was obvious,
222
00:13:55,959 --> 00:13:57,559
why aren't you applying it here?
223
00:13:57,869 --> 00:14:03,249
Either you were wrong on day one about
that rule being the right rule, or
224
00:14:03,250 --> 00:14:07,469
you're wrong right now in not applying
that rule.” And so there's a real
225
00:14:07,470 --> 00:14:12,249
sense in which the reason I chose these
rules is because I think I'm really
226
00:14:12,250 --> 00:14:14,729
not teaching my students anything new.
227
00:14:15,119 --> 00:14:19,800
So here's one way to think about it:
rather than teaching them like calculus,
228
00:14:19,989 --> 00:14:25,560
like a chain rule, like a new formula that
they don't know, what I'm really trying
229
00:14:25,560 --> 00:14:29,330
to do is get them to recognize cardinals.
230
00:14:29,569 --> 00:14:32,570
They already know what a cardinal looks
like, so I'm just trying to get them to
231
00:14:32,570 --> 00:14:38,290
recognize cardinals, and so that's why
I think that the way I teach critical
232
00:14:38,300 --> 00:14:44,539
thinking tends to work and is way easier
than starting off with the assumption that
233
00:14:44,559 --> 00:14:48,289
“Hey, my students don't know anything,
and I need to teach them a whole bunch of
234
00:14:48,300 --> 00:14:52,860
new rules.” So that's why I chose these
rules, and that's why I choose the rules
235
00:14:52,870 --> 00:14:55,050
that I use in my critical thinking class.
236
00:14:55,670 --> 00:14:59,059
JOHN: And these are rules that,
when you think about them as you
237
00:14:59,059 --> 00:15:01,120
framed it, they're pretty obvious.
238
00:15:01,300 --> 00:15:05,340
But in practice, they're rules
that people violate all of the
239
00:15:05,340 --> 00:15:06,959
time in their decision making.
240
00:15:07,209 --> 00:15:10,929
What really impressed me about the study
was that you did get some [LAUGHTER]
241
00:15:10,939 --> 00:15:15,649
results out of this because so many
studies of behavior modification have
242
00:15:15,650 --> 00:15:20,599
been found to be ineffective beyond a
very short period of time, often not
243
00:15:20,599 --> 00:15:22,279
lasting through the end of the class.
244
00:15:22,549 --> 00:15:23,410
MICHAEL: That's absolutely right.
245
00:15:23,410 --> 00:15:27,799
In fact, one of my old professors, he said
this about logic classes, but I think it
246
00:15:27,849 --> 00:15:31,899
unfortunately is often true of critical
thinking classes, but he said that logic
247
00:15:31,899 --> 00:15:35,129
classes teach you to do one and only
one thing, and that's take logic tests.
248
00:15:35,379 --> 00:15:39,760
And I sometimes worry that that's
the case with a lot of our attempts
249
00:15:39,770 --> 00:15:43,939
to teach people to think better,
is that what we're really doing is
250
00:15:44,000 --> 00:15:48,399
teaching them to take tests because
they will use the rules on the test.
251
00:15:48,809 --> 00:15:52,809
But if you want them to think about
this afterwards, I really do think
252
00:15:52,820 --> 00:15:57,640
that the main thing is to get them to
commit to the rule, and then get them to
253
00:15:57,640 --> 00:16:02,399
recognize, “Hey, wait a minute,” to be
sensitive to when they see it in the real
254
00:16:02,400 --> 00:16:04,279
world because it's all over the place.
255
00:16:04,290 --> 00:16:07,130
Once you get sensitive to it,
you start seeing it everywhere.
256
00:16:07,130 --> 00:16:11,249
And indeed, one danger of my class, of
my critical thinking class that I talk
257
00:16:11,250 --> 00:16:15,429
to my students is what I call “the turd
in the punch bowl problem,” and that
258
00:16:15,430 --> 00:16:19,589
is being a good critical thinker isn't
necessarily going to make you popular, and
259
00:16:19,589 --> 00:16:24,989
so you need to be careful about when you
point out to people that their arguments
260
00:16:25,040 --> 00:16:29,489
aren't very good, because if you do it
too much, you will end up being like
261
00:16:29,659 --> 00:16:31,239
the turd in the punch bowl at a party.
262
00:16:31,400 --> 00:16:34,109
People will look at you and say,
"Oh, I'm going to stay away from
263
00:16:34,110 --> 00:16:37,939
that person.” And so, at the end of
the semester, I will ask them, "Have
264
00:16:37,939 --> 00:16:42,139
you had any turd in the punch bowl
experiences as a result of this class?
265
00:16:42,199 --> 00:16:45,569
And at the end of the semester, a lot
of the students – and I don't know if
266
00:16:45,569 --> 00:16:51,569
this is good or bad – but about half
of the cases involve the sunk cost
267
00:16:51,569 --> 00:16:58,019
principle, and usually it involves
the destruction of a relationship.
268
00:16:58,429 --> 00:17:01,960
So people will end a relationship
because the only reason they've been
269
00:17:01,960 --> 00:17:06,769
sticking with the relationship is
all the pain and effort and heartache
270
00:17:06,809 --> 00:17:08,299
they've put into the relationship.
271
00:17:09,020 --> 00:17:12,000
JOHN: And I know behavioral economists
have noted that quite a bit.
272
00:17:12,020 --> 00:17:15,750
That once they really get this message
through, it leads to a breakup of so
273
00:17:15,750 --> 00:17:17,629
many relationships of their students.
274
00:17:18,149 --> 00:17:23,120
But even people who know these things
sometimes fall into these traps.
275
00:17:23,139 --> 00:17:26,320
And one example I think I mentioned
on the podcast before, and I know I've
276
00:17:26,320 --> 00:17:30,299
mentioned it to Rebecca, is one time
I was at a conference with a few other
277
00:17:30,300 --> 00:17:34,439
people, and we went over to Universal
park, and it was just really hot that
278
00:17:34,440 --> 00:17:39,110
day, it was like 97 degrees, and we had
spent quite a bit for the tickets for the
279
00:17:39,110 --> 00:17:41,670
day, and we were just kind of miserable.
280
00:17:41,690 --> 00:17:44,749
And one of the other people
said we should just go.
281
00:17:44,830 --> 00:17:53,230
And my response to a historian was that
we spent $230 and we've gone on two rides.
282
00:17:53,610 --> 00:17:55,810
We really should stay and
get our money's worth.
283
00:17:56,080 --> 00:18:01,459
And her response was, "But that's a
sunk cost. It shouldn't matter,” and
284
00:18:01,549 --> 00:18:03,610
it's easy for us to fall into that.
285
00:18:04,000 --> 00:18:06,710
MICHAEL: Absolutely, and indeed,
this is one of the reasons I choose
286
00:18:06,710 --> 00:18:09,429
the rules that I do, is that I
think many of them are significant.
287
00:18:09,430 --> 00:18:13,249
They really can make a difference
to the quality of your decisions
288
00:18:13,510 --> 00:18:15,119
and the quality of your judgments.
289
00:18:15,639 --> 00:18:18,410
REBECCA: Can you talk a little
bit about the two-stage training
290
00:18:18,410 --> 00:18:20,350
program that was used in class?
291
00:18:20,899 --> 00:18:21,459
MICHAEL: Sure.
292
00:18:21,549 --> 00:18:24,970
So I gave you a little bit of an
example of it already, but let me
293
00:18:24,970 --> 00:18:28,040
give you an example with another
case with the correlation cheat code.
294
00:18:28,419 --> 00:18:34,720
So my go-to example here is I'll tell the
students, “Well, you know, yesterday I ate
295
00:18:34,730 --> 00:18:36,950
chocolate and today I broke out with zits.
296
00:18:37,250 --> 00:18:41,459
Boy, that means that eating chocolate
causes zits,” and they immediately say,
297
00:18:41,459 --> 00:18:45,889
“No, that's terrible.” So the first stage,
what I'm doing is I'm introducing examples
298
00:18:45,889 --> 00:18:49,179
where they are disposed to use the rule.
299
00:18:49,529 --> 00:18:53,079
So nobody says, "Yeah, that's great
evidence for thinking that eating
300
00:18:53,239 --> 00:18:56,929
chocolate causes zits.” And then we
talk a little bit about, “Okay, well,
301
00:18:56,979 --> 00:19:01,230
what would count as good evidence?”
And they come up pretty quickly with
302
00:19:01,230 --> 00:19:04,450
maybe just a little bit of help, “Well,
you want to know what happens when you
303
00:19:04,450 --> 00:19:07,480
eat chocolate and when you don't eat
chocolate, because what you really want
304
00:19:07,480 --> 00:19:12,749
to know is what difference does eating
chocolate make to your complexion, and
305
00:19:12,750 --> 00:19:15,240
so what you want to know is what happens
when you eat chocolate and you don't eat
306
00:19:15,240 --> 00:19:18,889
chocolate, and everything else relevant
to your complexion is the same, and
307
00:19:18,889 --> 00:19:23,120
what difference is there in these two
cases,” and so that's the first stage.
308
00:19:23,120 --> 00:19:28,440
The first stage is just introducing
them to these rules, these cheat codes.
309
00:19:28,440 --> 00:19:31,839
So I give them the cheat code,
but really a cheat code which they
310
00:19:31,839 --> 00:19:35,229
detect right away as a cheat code,
and that they know as a cheat code.
311
00:19:35,250 --> 00:19:40,599
And then I spend the rest of the semester
giving them arguments that use that cheat
312
00:19:40,599 --> 00:19:43,779
code, but are not as easy to detect.
313
00:19:44,089 --> 00:19:47,839
And often these are political arguments,
and one reason I don't get into trouble
314
00:19:47,939 --> 00:19:51,649
– at least I haven't so far – in giving
political arguments is that these cheat
315
00:19:51,649 --> 00:19:55,979
codes you can use them to defend views
on the left, on the right, in the middle.
316
00:19:56,099 --> 00:19:57,789
They're tools of manipulation.
317
00:19:58,039 --> 00:19:59,829
It doesn't matter what the truth is.
318
00:20:00,080 --> 00:20:04,170
You can use these cheat codes to
defend the true view, the false view,
319
00:20:04,449 --> 00:20:06,830
smart decisions, dumb decisions.
320
00:20:07,190 --> 00:20:11,979
And so, what I'm really interested
in doing is getting them to be
321
00:20:11,979 --> 00:20:13,559
able to detect these cheat codes.
322
00:20:13,559 --> 00:20:16,940
So a first step is just introduce
them to these cheat codes in a
323
00:20:16,940 --> 00:20:21,359
way that they will be disposed
to accept and think is obvious.
324
00:20:21,720 --> 00:20:24,479
And then after that, I'll give
them lots of examples in which
325
00:20:24,519 --> 00:20:27,359
they're, at least many of them, are
not disposed to think is obvious.
326
00:20:27,500 --> 00:20:32,060
And then rather than me sort of coming
from on high and saying, "Well, you know,
327
00:20:32,389 --> 00:20:36,539
you made a mistake there, dummy.” I just
step back and I say, "Wait a minute.
328
00:20:36,989 --> 00:20:38,799
We seem to have an inconsistency here.
329
00:20:39,399 --> 00:20:44,000
You insisted that this rule was
right on Monday, and on Friday you're
330
00:20:44,000 --> 00:20:46,049
not using the rule in this case.
331
00:20:46,269 --> 00:20:50,540
What's going on?” And I have yet
to have a student, and I've taught
332
00:20:50,570 --> 00:20:54,459
1000s of students, I've yet to have
a student say, "Okay, I'm going to
333
00:20:54,460 --> 00:20:58,080
rethink that rule.” They all say,
"Oh, well, I made a mistake here.
334
00:20:58,150 --> 00:21:02,820
I reasoned badly,” and by the end of the
semester, they're not reasoning badly.
335
00:21:03,129 --> 00:21:05,930
REBECCA: Sounds like you spend a very
small amount of time on stage one
336
00:21:05,930 --> 00:21:08,749
and significant time on stage two.
337
00:21:09,250 --> 00:21:12,420
Do you do all of the cheat codes at
the beginning of the semester and
338
00:21:12,420 --> 00:21:16,440
then work on utilizing them throughout
the semester, or do you kind of
339
00:21:16,440 --> 00:21:18,550
do units on different cheat codes?
340
00:21:18,550 --> 00:21:19,579
How do you organize?
341
00:21:19,959 --> 00:21:23,519
MICHAEL: So I will introduce a
cheat code and then I'll give them
342
00:21:23,520 --> 00:21:27,079
examples using that cheat code and
all the cheat codes we've covered.
343
00:21:27,540 --> 00:21:33,600
So once a cheat code is introduced, they
might see it at any point in the future,
344
00:21:33,860 --> 00:21:37,409
and I think it's really important they
keep practicing it and keep being able to
345
00:21:37,419 --> 00:21:42,189
recognize it, and not just think, "Hey, I
know this is the cheat code we're working
346
00:21:42,189 --> 00:21:45,759
on today, so I just need to be on the
lookout for that one.” Another thing I
347
00:21:45,759 --> 00:21:51,100
do, and I think this is really important,
is I will sometimes give them arguments
348
00:21:51,110 --> 00:21:56,070
that are promising, that if you grant the
truth of the premises, they at least give
349
00:21:56,070 --> 00:21:59,369
you some reason to accept the conclusion.
350
00:21:59,709 --> 00:22:04,870
Because I don't want the students
just to be skeptical of all arguments.
351
00:22:04,879 --> 00:22:08,900
I want them to be able to distinguish
between “Okay, that's a cheat code.
352
00:22:09,180 --> 00:22:10,740
That's definitely a bad argument.
353
00:22:10,740 --> 00:22:13,809
The conclusion might be true, the
conclusion might be false, but that's
354
00:22:13,809 --> 00:22:17,779
not a good reason to accept anything”
and then “Okay, well, what would count
355
00:22:17,779 --> 00:22:22,629
as good evidence for thinking that
this is a smart decision or that this
356
00:22:22,650 --> 00:22:27,739
policy has led to increased costs or
decreased costs?” And I want them to be
357
00:22:27,739 --> 00:22:31,319
able to say, "Okay, look, that argument
right there, that's pretty promising.
358
00:22:31,409 --> 00:22:33,899
This argument over here
definitely uses a cheat code.”
359
00:22:34,610 --> 00:22:37,479
JOHN: So you start off with a little
bit of retrieval practice with some
360
00:22:37,479 --> 00:22:41,279
applications of the same concept, and
then you move into some interleaved
361
00:22:41,279 --> 00:22:45,029
practice, which raises the cognitive
difficulty just a little bit.
362
00:22:45,339 --> 00:22:49,780
But it's one of those desirable
difficulties where the more cognitive
363
00:22:49,809 --> 00:22:54,369
effort that students have to place
in identifying what is at issue, the
364
00:22:54,369 --> 00:22:57,740
more deeply they tend to learn it,
and the more likely it is that those
365
00:22:57,740 --> 00:23:02,559
concepts will stick and also transfer
into other applications, which is what
366
00:23:02,559 --> 00:23:04,489
you're building to in this program.
367
00:23:04,500 --> 00:23:08,969
MICHAEL: Absolutely, absolutely,
the interleaved practice is really
368
00:23:08,970 --> 00:23:14,979
important, and I wouldn't go back
to sort of the blocking practice.
369
00:23:15,539 --> 00:23:17,819
REBECCA: So we've talked a little
bit about what the intervention is.
370
00:23:17,819 --> 00:23:21,379
Can you talk a little bit about how the
study was conducted, like the size of the
371
00:23:21,379 --> 00:23:23,409
treatment group, the control group, etc.
372
00:23:23,529 --> 00:23:23,919
MICHAEL: Sure.
373
00:23:24,399 --> 00:23:29,359
So in the paper, we really describe
just the second study we did, we
374
00:23:29,360 --> 00:23:33,479
actually did two studies, and in both
studies we had almost 400 students.
375
00:23:34,209 --> 00:23:39,839
In the first study we had 395 students,
and the second study we had 397 students.
376
00:23:40,550 --> 00:23:44,639
And some of the students were taking
a critical thinking class from me,
377
00:23:44,740 --> 00:23:48,980
and the rest of the students were
taking other philosophy classes.
378
00:23:49,410 --> 00:23:55,039
So in the study that we report
on, we had 113 critical thinking
379
00:23:55,040 --> 00:23:58,459
students and 284 control students.
380
00:23:58,920 --> 00:24:04,469
Then what we did is, in the first week
of class, we gave everyone a pretest,
381
00:24:04,740 --> 00:24:10,259
and this involved the five cheat
codes, and then two good arguments,
382
00:24:10,449 --> 00:24:11,769
or at least promising arguments.
383
00:24:11,779 --> 00:24:15,709
They weren't necessarily great, but
they didn't use any cheat codes.
384
00:24:16,070 --> 00:24:20,150
And the reason we included these, what
we called skepticism checks, was just to
385
00:24:20,169 --> 00:24:25,480
check if all we were doing was creating
skeptical students rather than students
386
00:24:25,480 --> 00:24:29,939
who could distinguish between promising
arguments and cheat code arguments.
387
00:24:30,730 --> 00:24:34,619
Then, at the end of the semester,
we gave them the same survey,
388
00:24:34,689 --> 00:24:40,239
same seven questions, and not
surprisingly, in both of the studies,
389
00:24:40,799 --> 00:24:45,209
the training group students did much
better than the control students.
390
00:24:46,079 --> 00:24:48,509
In the first study, we
waited three months.
391
00:24:48,879 --> 00:24:51,469
So basically, the way to think
about it is that these classes
392
00:24:51,470 --> 00:24:52,550
all happened in the fall.
393
00:24:52,860 --> 00:24:54,779
We waited till the end
of the spring semester.
394
00:24:55,139 --> 00:24:59,029
It might have been four months,
and gave them same questions
395
00:24:59,029 --> 00:25:00,589
again in the first study.
396
00:25:00,589 --> 00:25:02,069
We gave them the same questions again.
397
00:25:02,379 --> 00:25:05,869
The trained students did much better, and
then we gave them different questions,
398
00:25:05,959 --> 00:25:07,209
and they still did much better.
399
00:25:07,439 --> 00:25:12,080
So in the second study we did, which is
really the only study that we report on
400
00:25:12,189 --> 00:25:16,970
in the article, we gave them pre-test at
the beginning of the semester, post-test.
401
00:25:17,000 --> 00:25:23,800
Then we waited a full academic year plus
the spring, so we waited 16 months and
402
00:25:23,809 --> 00:25:26,339
only asked them different questions.
403
00:25:26,949 --> 00:25:31,199
So the pretest and posttest were
the same questions, and again my
404
00:25:31,199 --> 00:25:32,959
students did much much better.
405
00:25:33,269 --> 00:25:39,379
But then we waited 16 months and asked
them seven different questions, and
406
00:25:39,459 --> 00:25:43,810
the trained students were still doing
much better, detecting the cheat codes
407
00:25:44,049 --> 00:25:47,709
and recognizing that the good arguments
were still pretty good arguments.
408
00:25:48,309 --> 00:25:50,959
JOHN: DId the trained students
do significantly better with
409
00:25:50,969 --> 00:25:52,970
all 5 of these cheat codes?
410
00:25:53,369 --> 00:25:57,160
MICHAEL: No, they did better
on 4 of the 5 cheat codes.
411
00:25:57,190 --> 00:26:01,880
They did better on the correlation,
regression, sunk cost, and opportunity
412
00:26:01,880 --> 00:26:08,089
cost cheat codes, but in all the
studies that we did, on all the surveys
413
00:26:08,089 --> 00:26:12,179
that we asked, we just couldn’t get
the trained students to do better
414
00:26:12,199 --> 00:26:14,210
on the gambler’s fallacy questions.
415
00:26:14,690 --> 00:26:18,629
JOHN: But still, having students do
better on 4 out of those 5 cheat codes
416
00:26:18,940 --> 00:26:23,449
so long after the initial intervention
is a really impressive result.
417
00:26:23,970 --> 00:26:27,179
One of the things you talk about
near the end of the paper, though,
418
00:26:27,220 --> 00:26:31,600
is that you note that there are some
possible sample selection issues.
419
00:26:32,039 --> 00:26:36,979
Could you just talk a little bit about
what source of sample selection might have
420
00:26:36,979 --> 00:26:39,099
affected the magnitude of those results?
421
00:26:39,789 --> 00:26:43,490
MICHAEL: So one worry that one
might have is that the students
422
00:26:43,490 --> 00:26:45,230
were not distributed randomly.
423
00:26:45,420 --> 00:26:49,290
So the students chose to take the
critical thinking class, the students
424
00:26:49,320 --> 00:26:53,409
chose to take other philosophy classes,
and so there might be something about
425
00:26:53,410 --> 00:26:58,259
the students who chose critical thinking
classes that would incline them to be
426
00:26:58,270 --> 00:27:02,899
better critical thinkers, despite the
fact that on the pre-tests, all the
427
00:27:02,900 --> 00:27:04,930
students in both groups did equally badly.
428
00:27:05,139 --> 00:27:09,730
So that's one worry one might have,
and the thing to do there is, although
429
00:27:09,730 --> 00:27:15,039
it's not really easy given the
institutions of academia, is to randomly
430
00:27:15,040 --> 00:27:18,919
distribute students into critical
thinking classes and other classes.
431
00:27:19,639 --> 00:27:22,109
So that's the main selection worry.
432
00:27:22,739 --> 00:27:26,029
Another reason why we might have
gotten good results is that I
433
00:27:26,029 --> 00:27:27,920
had those students for 15 weeks.
434
00:27:28,259 --> 00:27:32,149
Some interventions talk about full-
semester interventions, but often
435
00:27:32,549 --> 00:27:34,339
they're fairly short interventions.
436
00:27:34,660 --> 00:27:40,679
I was able to have the students for
15 weeks and made them practice a lot.
437
00:27:41,020 --> 00:27:44,199
REBECCA: It sounds like in the
design of the experiment in your
438
00:27:44,199 --> 00:27:47,379
class, just generally, there's lots
of opportunities for feedback and
439
00:27:47,379 --> 00:27:49,419
correction throughout the semester.
440
00:27:49,650 --> 00:27:50,870
MICHAEL: Yes, absolutely.
441
00:27:50,979 --> 00:27:57,279
There's a lot of feedback, a lot of
correction, and I have a big bank
442
00:27:57,549 --> 00:28:01,339
of good and bad arguments that I
can give them over the course of the
443
00:28:01,339 --> 00:28:05,119
semester that use these cheat codes
or that don't use the cheat codes.
444
00:28:05,339 --> 00:28:10,729
And one of the things I do for an
essay in this class is ask the students
445
00:28:11,129 --> 00:28:16,549
to put themselves in the shoes of
someone who's trying to come up with
446
00:28:16,549 --> 00:28:21,399
a cheat code argument and ask them
to give me bad arguments that use
447
00:28:21,400 --> 00:28:24,339
these cheat codes, and they actually
turn out to be pretty good at it.
448
00:28:24,430 --> 00:28:28,969
And many of the examples that I use
on later students are examples that
449
00:28:28,970 --> 00:28:30,879
prior students have come up with.
450
00:28:31,369 --> 00:28:32,940
REBECCA: Sounds like it makes
it a little bit fun too.
451
00:28:33,540 --> 00:28:37,339
MICHAEL: Yes, I do ask them
for any humorous examples or
452
00:28:37,339 --> 00:28:40,899
examples that us boomers would
have a hard time understanding.
453
00:28:42,489 --> 00:28:42,539
[LAUGHTER]
454
00:28:42,930 --> 00:28:47,329
JOHN: One of the things we appreciated
about this study is that the data, the
455
00:28:47,369 --> 00:28:52,069
results, and the analyses that were
done are all reported in more detail
456
00:28:52,410 --> 00:28:54,860
on the Open Science Framework website.
457
00:28:55,190 --> 00:28:59,289
We'll include a link to that website
in the show notes for this episode.
458
00:29:00,250 --> 00:29:03,209
REBECCA: So we always wrap
up by asking, "What's next?
459
00:29:04,129 --> 00:29:05,259
MICHAEL: A few things are next.
460
00:29:05,289 --> 00:29:11,769
One is I'm writing a textbook that
uses the materials that I teach.
461
00:29:12,169 --> 00:29:15,599
The way I teach critical
thinking is different from
462
00:29:15,779 --> 00:29:16,919
the way other people teach it.
463
00:29:17,229 --> 00:29:20,429
I haven't found a good textbook
that teaches critical thinking
464
00:29:20,429 --> 00:29:21,599
the way I like to teach it.
465
00:29:21,619 --> 00:29:24,909
So I'm currently writing a
textbook that hopefully will help
466
00:29:24,910 --> 00:29:26,599
other people get good results.
467
00:29:26,910 --> 00:29:32,739
I'm also planning to build, if I can,
a game that anybody can play to see
468
00:29:32,739 --> 00:29:38,360
whether they can avoid being bamboozled
[LAUGHTER] by cheat code arguments, this
469
00:29:38,360 --> 00:29:43,490
is sort of a longer term project because
writing a book is going to be useful,
470
00:29:43,490 --> 00:29:45,410
hopefully, for people to teach this.
471
00:29:45,510 --> 00:29:50,170
But what's really important, in
my view, what makes this stick is
472
00:29:50,170 --> 00:29:51,880
practicing on lots of examples.
473
00:29:52,240 --> 00:29:57,159
And so, in a game where people see an
argument and you say, “Well, is this
474
00:29:57,209 --> 00:30:01,360
promising argument, or is somebody trying
to manipulate you?” we'll hopefully
475
00:30:01,360 --> 00:30:05,019
give people practice at being able
to detect those cheat code arguments.
476
00:30:05,350 --> 00:30:07,560
So those are two of the
projects that I'm working on.
477
00:30:08,019 --> 00:30:10,820
JOHN: On the latter project,
will it be a board game?
478
00:30:10,830 --> 00:30:12,649
Will it be an electronic game?
479
00:30:12,649 --> 00:30:16,589
Will it be an online game, or
is that something that'll be
480
00:30:16,590 --> 00:30:18,050
decided further in the future?
481
00:30:18,110 --> 00:30:19,999
MICHAEL: No, it'll be an online game.
482
00:30:20,049 --> 00:30:21,989
Is what I'm hoping to do.
483
00:30:22,529 --> 00:30:26,139
I want to do something that can
be used not just by my students
484
00:30:26,139 --> 00:30:27,369
and that's easily accessible.
485
00:30:27,590 --> 00:30:28,969
REBECCA: Sounds like some great projects.
486
00:30:28,969 --> 00:30:30,510
Thanks for joining us today.
487
00:30:30,779 --> 00:30:31,469
MICHAEL: Oh, you're welcome.
488
00:30:31,469 --> 00:30:32,259
It's been a pleasure.
489
00:30:32,570 --> 00:30:33,059
JOHN: Thank you.
490
00:30:33,160 --> 00:30:36,229
I was really impressed by this study
[LAUGHTER] because, as I mentioned
491
00:30:36,239 --> 00:30:41,280
before, it's so rare to see effects
that are so significant and persistent.
492
00:30:41,520 --> 00:30:45,879
It's, I think, something that people can
learn from in terms of other types of
493
00:30:45,879 --> 00:30:50,879
issues where it's hard to convey this
sort of long-term impact on students.
494
00:30:51,299 --> 00:30:52,159
MICHAEL: Well, thank you so much.
495
00:31:00,729 --> 00:31:00,829
[MUSIC]
496
00:31:03,669 --> 00:31:06,440
JOHN: If you've enjoyed this
podcast, please subscribe and
497
00:31:06,440 --> 00:31:10,500
leave a review on Apple Podcasts
or your favorite podcast service.
498
00:31:11,059 --> 00:31:15,589
To continue the conversation, join us
on our Tea for Teaching Facebook page.
499
00:31:16,540 --> 00:31:19,149
REBECCA: You can find show
notes, transcripts and other
500
00:31:19,150 --> 00:31:21,480
materials on teaforteaching.com.
501
00:31:22,509 --> 00:31:24,479
Music by Michael Gary Brewer.
502
00:31:24,479 --> 00:31:28,479
[MUSIC]
00:00:00,520 --> 00:00:03,580
JOHN: Students with strong
critical thinking skills can make
2
00:00:03,590 --> 00:00:07,870
better decisions that have real
consequences on their quality of life.
3
00:00:08,260 --> 00:00:12,769
In this episode, we explore an approach
to developing these skills that appears
4
00:00:12,780 --> 00:00:18,859
to have a persistent effect on students’
ability to avoid common cognitive biases.
5
00:00:24,139 --> 00:00:24,259
[MUSIC]
6
00:00:24,490 --> 00:00:28,810
Thanks for joining us for Tea for
Teaching, an informal discussion
7
00:00:28,810 --> 00:00:31,940
of innovative and effective
practices in teaching and learning.
8
00:00:32,529 --> 00:00:35,920
REBECCA: This podcast series is
hosted by John Kane, an economist…
9
00:00:36,389 --> 00:00:38,519
JOHN: …and Rebecca Mushtare,
a graphic designer…
10
00:00:38,869 --> 00:00:42,169
REBECCA: …and features guests doing
important research and advocacy work
11
00:00:42,309 --> 00:00:46,060
to make higher education more inclusive
and supportive of all learners.
12
00:00:49,149 --> 00:00:49,269
[MUSIC]
13
00:00:55,090 --> 00:00:56,579
Our guest today is Michael Bishop.
14
00:00:56,799 --> 00:00:59,809
He is professor of philosophy
at Florida State University.
15
00:01:00,020 --> 00:01:04,669
His research interests focus on two areas:
rationality [how should we think about
16
00:01:04,669 --> 00:01:08,779
important things] and happiness [what is
lasting happiness, and how can we get it].
17
00:01:09,010 --> 00:01:12,459
He is also a co-author of a recent
study investigating the impact
18
00:01:12,459 --> 00:01:16,029
of critical thinking classes
on common cognitive biases.
19
00:01:16,500 --> 00:01:17,120
Welcome, Michael.
20
00:01:17,470 --> 00:01:17,890
MICHAEL: Thank you.
21
00:01:17,890 --> 00:01:18,839
Glad to be here.
22
00:01:19,159 --> 00:01:20,209
JOHN: Thank you for joining us.
23
00:01:20,779 --> 00:01:23,979
Today's teas are:… Michael,
do you have tea by any chance?
24
00:01:24,389 --> 00:01:25,629
MICHAEL: No, not today.
25
00:01:25,629 --> 00:01:31,639
I usually drink tea in my office, and when
I do, it's usually a Chinese black tea.
26
00:01:31,939 --> 00:01:33,219
REBECCA: Always a nice choice.
27
00:01:33,470 --> 00:01:37,139
Today I have an English breakfast
tea, nothing that exciting,
28
00:01:37,610 --> 00:01:41,220
JOHN: And I just have nothing that
exciting either, a Lady Grey today.
29
00:01:41,599 --> 00:01:43,020
REBECCA: We got to step up our game, John.
30
00:01:43,509 --> 00:01:46,859
So, Michael, we've invited here today
to discuss the field experiment that
31
00:01:46,860 --> 00:01:51,129
is described in your co-authored paper
entitled "Critical Thinking Classes Can
32
00:01:51,129 --> 00:01:55,429
Reduce Common Biases: Results from a
Field Experiment.” But before we discuss
33
00:01:55,430 --> 00:01:59,449
the study, can we start a little bit
with the concept of critical thinking?
34
00:01:59,519 --> 00:02:00,919
You teach critical thinking.
35
00:02:00,930 --> 00:02:01,939
What are you trying to teach?
36
00:02:01,939 --> 00:02:04,029
And what makes someone
a good critical thinker?
37
00:02:04,489 --> 00:02:05,979
MICHAEL: Yeah, I think
that's a great question.
38
00:02:06,219 --> 00:02:11,450
One of my favorite sayings is the
philosopher George Santayana said that a
39
00:02:11,450 --> 00:02:15,349
fanatic is someone who, having lost sight
of their goals, redoubles their efforts.
40
00:02:15,649 --> 00:02:20,759
And so, in teaching critical thinking,
I think it's important to start with
41
00:02:20,759 --> 00:02:23,069
an idea of what you want to teach.
42
00:02:23,069 --> 00:02:25,239
And so there's a lot of possible answers.
43
00:02:25,569 --> 00:02:29,669
Some people think that a good critical
thinker is someone who uses certain
44
00:02:29,750 --> 00:02:35,199
rational methods or rules like logic or
probability for the scientific method.
45
00:02:35,239 --> 00:02:38,759
Other people think that a rational
thinker would be an expert debater,
46
00:02:38,949 --> 00:02:44,369
someone who can detect fallacies and
build a coherent case for their views.
47
00:02:44,940 --> 00:02:48,819
Other people view good critical
thinkers as virtuous thinkers;
48
00:02:48,929 --> 00:02:53,769
someone who is careful,
open-minded, curious, intellectually
49
00:02:53,789 --> 00:02:55,479
honest, intellectually humble.
50
00:02:55,899 --> 00:03:00,950
And I actually think these views
are either wrong… so I think
51
00:03:01,030 --> 00:03:04,870
that good debaters are often
poor reasoners… or incomplete.
52
00:03:05,040 --> 00:03:07,940
They capture part of good
reasoning, but not everything.
53
00:03:08,179 --> 00:03:13,459
So basically, what I am trying to
teach in my classes is teaching
54
00:03:13,460 --> 00:03:16,149
people to ask good questions.
55
00:03:16,190 --> 00:03:21,889
Ask what I call fruitful questions,
questions that help us get better answers,
56
00:03:22,350 --> 00:03:26,549
not killer questions that end debates,
but questions that, if you're trying to
57
00:03:26,549 --> 00:03:28,840
answer them, help get us better answers.
58
00:03:29,090 --> 00:03:33,540
So when you view that as the goal,
how you go about teaching critical
59
00:03:33,549 --> 00:03:35,160
thinking becomes much easier.
60
00:03:35,770 --> 00:03:39,110
JOHN: And as you suggested, people
have many different descriptions
61
00:03:39,119 --> 00:03:40,480
of what critical thinking is.
62
00:03:40,480 --> 00:03:44,059
I know whenever we talked about the
importance of this in our Gen Ed program,
63
00:03:44,340 --> 00:03:48,159
everyone agrees that critical thinking
is extremely important, but they all
64
00:03:48,160 --> 00:03:50,459
disagree as to what exactly it is.
65
00:03:50,459 --> 00:03:53,150
So it's helpful, I think, to
start with that statement.
66
00:03:53,639 --> 00:03:57,539
In this experiment, you're
focused on five common biases.
67
00:03:57,799 --> 00:04:01,900
Can you describe the biases that
were selected for this study and
68
00:04:01,939 --> 00:04:05,639
provide some brief examples of
those biases for our listeners?
69
00:04:05,869 --> 00:04:06,229
MICHAEL: Sure.
70
00:04:06,580 --> 00:04:10,420
The way psychologists think about
these is in terms of biases.
71
00:04:10,430 --> 00:04:14,120
The way I think about them
is in terms of cheat codes.
72
00:04:14,290 --> 00:04:19,029
So there are certain cheat codes
people use in debating or in
73
00:04:19,040 --> 00:04:23,210
arguments, and cheat code arguments
are arguments that you can use to
74
00:04:23,210 --> 00:04:25,749
defend pretty much any view you want.
75
00:04:25,959 --> 00:04:31,529
And so cheat codes are really tools
of manipulation, more than tools
76
00:04:31,540 --> 00:04:34,950
of trying to figure out what the
truth is or what a good decision is.
77
00:04:35,329 --> 00:04:40,679
And so when you think about good critical
thinkers as thinkers who can ask good
78
00:04:40,680 --> 00:04:46,110
questions, the way I think about it is
when you detect a cheat code, when you
79
00:04:46,110 --> 00:04:49,439
detect somebody using a cheat code on
you, you should be able to ask them a
80
00:04:49,439 --> 00:04:53,329
question that diffuses the cheat code
and points them in better directions.
81
00:04:53,339 --> 00:04:55,739
So let me tell you some
of these cheat codes.
82
00:04:55,939 --> 00:05:00,530
One I call the focalizing cheat code,
and it might go something like this:
83
00:05:00,530 --> 00:05:06,449
We should go to war because it has
certain benefits, so, if an argument
84
00:05:06,459 --> 00:05:10,529
like that, or we should not go to
war because it has certain costs.
85
00:05:10,569 --> 00:05:14,379
If you're convinced by an argument like
that, I think that you've fallen for
86
00:05:14,379 --> 00:05:20,129
a cheat code, and the best way to see
this is to consider an example where
87
00:05:20,129 --> 00:05:21,319
where it's clear that it's a cheat code.
88
00:05:21,470 --> 00:05:26,359
So, suppose I gave you two envelopes, and
I said you can look in these envelopes,
89
00:05:26,479 --> 00:05:30,129
and you can take all the money that's
in the envelope that you choose, but
90
00:05:30,129 --> 00:05:31,429
you can only choose one envelope.
91
00:05:31,889 --> 00:05:36,739
Suppose I open one envelope, see
that there's $5 and I take it without
92
00:05:36,739 --> 00:05:38,059
looking in the second envelope.
93
00:05:38,779 --> 00:05:43,859
In that case, you'd think that I've made
a mistake, that I haven't reasoned wisely
94
00:05:43,860 --> 00:05:48,050
about this particular decision because
I didn't look in the second envelope.
95
00:05:48,440 --> 00:05:53,170
If I say, "Look, the reason I chose the
first envelope is that having $5 is better
96
00:05:53,170 --> 00:05:58,780
than not having $5.” It's true that having
$5 is better than not having $5, so there
97
00:05:58,780 --> 00:06:00,860
is an advantage to choosing envelope one.
98
00:06:01,140 --> 00:06:05,760
But when you think about it, what
makes a choice a good one isn't that it
99
00:06:05,770 --> 00:06:07,829
makes you better off than you are now.
100
00:06:08,179 --> 00:06:11,559
What makes a choice a good one is
that it makes you better off than
101
00:06:12,020 --> 00:06:13,330
what else you might have done.
102
00:06:13,770 --> 00:06:17,900
Now, economists call this opportunity
cost, as I suspect you guys know.
103
00:06:18,340 --> 00:06:23,050
And so, one thing I'm trying to do is
get my students to, when they see someone
104
00:06:23,050 --> 00:06:27,469
argue for, “Hey, we should go to war
because it's going to bring us benefits.”
105
00:06:27,760 --> 00:06:31,660
I want them to at least stop and be able
to ask the question, wait a minute, what
106
00:06:31,660 --> 00:06:33,769
are some alternatives to going to war?
107
00:06:34,049 --> 00:06:37,630
Because even if it's true that going
to war brings us benefits, and we can
108
00:06:37,630 --> 00:06:41,749
argue about that, but even if it's true,
it might not be the best thing to do,
109
00:06:41,750 --> 00:06:45,449
in the same way that choosing envelope
one, even though it brings us benefits,
110
00:06:45,459 --> 00:06:47,079
might not be the smart thing to do.
111
00:06:47,149 --> 00:06:49,500
So I call that the focalizing cheat code.
112
00:06:50,059 --> 00:06:55,259
Another cheat code that I teach is what
you might call the correlation cheat code.
113
00:06:55,259 --> 00:07:00,440
So I say, "Hey, a new healthcare plan
is passed, and healthcare costs have
114
00:07:00,440 --> 00:07:04,580
gone up 7% a year since this plan is
passed. Boy, this plan really stinks.
115
00:07:04,650 --> 00:07:09,809
It's caused our healthcare costs to
go up a lot.” This again, the question
116
00:07:09,809 --> 00:07:14,129
isn't is the conclusion right in the
war case or in this case, the question
117
00:07:14,129 --> 00:07:15,870
is: Is this a cheat code argument?
118
00:07:15,870 --> 00:07:16,619
And it is.
119
00:07:16,680 --> 00:07:20,489
And the reason is, what you don't know,
and what you really need to know, is
120
00:07:20,489 --> 00:07:25,189
how high would have costs been had
this healthcare plan not been in place.
121
00:07:25,509 --> 00:07:30,599
So again, I say this healthcare plan
is great or it's bad, and I point to
122
00:07:30,599 --> 00:07:33,219
something that happened afterwards that.
123
00:07:33,220 --> 00:07:36,289
That's what I call the correlation
cheat code because you're taking this
124
00:07:36,320 --> 00:07:40,629
correlation and you're taking it as
evidence for thinking that this policy
125
00:07:40,639 --> 00:07:43,349
has caused this particular result.
126
00:07:43,869 --> 00:07:47,189
And so again, what I want people to do
is just stop and ask the question: “Well,
127
00:07:47,189 --> 00:07:51,210
wait a minute. Maybe without the health
care plan, costs would have gone up
128
00:07:51,210 --> 00:07:54,579
even more, or maybe they would have gone
up just the same, or maybe they'd have
129
00:07:54,579 --> 00:07:58,670
gone up less.” But unless you ask the
question, it's easy to get manipulated
130
00:07:59,000 --> 00:08:01,359
into accepting views that aren't true.
131
00:08:01,669 --> 00:08:04,069
Another cheat code is the
regression cheat code.
132
00:08:04,500 --> 00:08:05,719
This is a very common one.
133
00:08:05,729 --> 00:08:10,019
So lots of coaches and parents
think that punishment is
134
00:08:10,020 --> 00:08:12,359
effective at molding behavior.
135
00:08:12,360 --> 00:08:16,509
I know that when I played sports as a
youth, my coaches thought that after
136
00:08:16,509 --> 00:08:21,709
a poor performance, if you punish the
players by making them work really hard
137
00:08:21,709 --> 00:08:23,729
in practice, they'll do better next time.
138
00:08:23,729 --> 00:08:26,649
And sure enough, after a poor
performance and after we were punished,
139
00:08:26,929 --> 00:08:28,219
we did tend to perform better.
140
00:08:28,639 --> 00:08:32,489
But again, the question is: “Did
the punishment do that, or was it
141
00:08:32,500 --> 00:08:37,319
just a return to our normal level of
performance?” So that's the regression
142
00:08:37,339 --> 00:08:40,420
cheat code, rather than just saying,
"Hey, maybe you're returning to your
143
00:08:40,429 --> 00:08:44,109
normal level of performance,” saying that
“it's the result of a punishment or the
144
00:08:44,110 --> 00:08:49,109
result of a reward.” The third one is
sunk cost cheat code, another one that's
145
00:08:49,110 --> 00:08:53,580
a favorite of economists, defending a
decision because of the resources already
146
00:08:53,580 --> 00:08:58,680
spent on it, so let's stick with this
policy or this war because of sunk costs.
147
00:08:59,040 --> 00:09:01,870
Those sunk costs are gone; they're
gone no matter what you do.
148
00:09:02,170 --> 00:09:04,719
So you might as well do
what's best for your future.
149
00:09:05,090 --> 00:09:09,920
And the last one that we looked at was the
gambler's fallacy, and that is basically
150
00:09:09,929 --> 00:09:14,510
treating statistically independent events,
as if they were statistically dependent.
151
00:09:14,539 --> 00:09:18,670
So, if a coin comes up five times
heads, and it's really a fair
152
00:09:18,709 --> 00:09:21,399
coin, thinking that it's more
likely to come up tails next time.
153
00:09:21,740 --> 00:09:24,369
So, those were the five
cheat codes we looked at.
154
00:09:24,730 --> 00:09:27,930
REBECCA: I think the soundtrack that we
have going on with the thunder is really
155
00:09:28,000 --> 00:09:29,999
emphasizing that these are cheat codes.
156
00:09:30,989 --> 00:09:31,049
[LAUGHTER]
157
00:09:31,049 --> 00:09:31,776
MICHAEL: Indeed, indeed.
158
00:09:31,776 --> 00:09:32,139
[LAUGHTER]
159
00:09:32,139 --> 00:09:33,050
REBECCA: It's very ominous.
160
00:09:34,689 --> 00:09:36,909
[LAUGHTER] Can you talk a
little bit about how you chose
161
00:09:36,910 --> 00:09:38,889
these particular cheat codes?
162
00:09:39,219 --> 00:09:39,719
MICHAEL: Sure.
163
00:09:39,919 --> 00:09:44,249
So this goes back to my
view about good reasoning.
164
00:09:44,250 --> 00:09:49,260
So 20 years ago or so, me and a friend
defended a theory of what rational
165
00:09:49,260 --> 00:09:55,120
reasoning was, and what we said was
that the good reasoning is reasoning
166
00:09:55,120 --> 00:09:59,409
that's cheap, it's accurate, and
it's focused on significant problems.
167
00:09:59,720 --> 00:10:06,759
And so the way I chose these
cheat codes is because I thought
168
00:10:06,779 --> 00:10:08,600
that they were significant.
169
00:10:08,650 --> 00:10:14,989
That is, people use them to manipulate you
when it comes to really important issues.
170
00:10:15,250 --> 00:10:19,389
So I see these cheat codes a lot
out there being used in the world
171
00:10:19,400 --> 00:10:20,959
for really important issues.
172
00:10:21,279 --> 00:10:27,099
And the cheap part came in because I
think that in thinking about how to
173
00:10:27,099 --> 00:10:32,419
teach critical thinking, I think one
thing you want to do as a critical
174
00:10:32,430 --> 00:10:34,899
thinker is to diagnose the problem.
175
00:10:34,970 --> 00:10:39,059
Why are my students thinking
badly when they do think badly?
176
00:10:39,629 --> 00:10:43,519
And I think there's three general
answers that people tend to
177
00:10:43,629 --> 00:10:47,449
think about, although often this
doesn't get talked about enough.
178
00:10:47,649 --> 00:10:52,369
So one reason students might be thinking
badly is that they don't know these rules,
179
00:10:52,379 --> 00:10:54,109
so it might be an ignorance problem.
180
00:10:54,540 --> 00:10:59,049
And if it's an ignorance problem,
then it's going to be really hard
181
00:10:59,059 --> 00:11:02,929
to teach them to use these rules
in a way that they keep using the
182
00:11:02,929 --> 00:11:04,969
rules after they leave the class.
183
00:11:05,319 --> 00:11:10,499
Because not only do they have to learn
the rules and learn to apply the rules,
184
00:11:10,690 --> 00:11:15,379
but when they leave your class, they need
to stop thinking in their old ways and
185
00:11:15,379 --> 00:11:20,900
use these rules, which tend to be hard to
remember, hard to apply in the real world.
186
00:11:20,900 --> 00:11:26,529
And so, one reason that I think studies
have not been successful or have had
187
00:11:26,539 --> 00:11:31,780
limited success in teaching people to
reason well in a way that sticks once
188
00:11:31,780 --> 00:11:36,479
they leave the classroom, is that too
often we teach critical thinking as if
189
00:11:36,639 --> 00:11:38,460
the problem is an ignorance problem.
190
00:11:38,910 --> 00:11:41,509
So we need to teach
our students new rules.
191
00:11:42,069 --> 00:11:46,389
So one reason I chose these rules
is that I speculated that they
192
00:11:46,389 --> 00:11:50,269
already knew these rules and they
sometimes applied them correctly.
193
00:11:50,660 --> 00:11:55,480
And if that's right, if they sometimes
apply these rules correctly, then I don't
194
00:11:55,500 --> 00:12:01,060
need to teach them a new rule, and I don't
need to teach them how to apply the rule.
195
00:12:01,080 --> 00:12:02,410
They're already doing that.
196
00:12:02,760 --> 00:12:08,080
All I need to do is get them to see that
it's the right rule, and then ask them,
197
00:12:08,080 --> 00:12:12,959
"Hey, why aren't you using it in these
cases over here, where it seems like you
198
00:12:12,959 --> 00:12:16,589
should still be using the rules.” So,
for example, think about the two envelope
199
00:12:16,600 --> 00:12:21,049
problem that I talked about before.
None of my students think, "Oh, yeah,
200
00:12:21,090 --> 00:12:23,740
choose the $5, that's the way to go.
201
00:12:24,090 --> 00:12:25,280
None of them think that.
202
00:12:25,590 --> 00:12:30,619
They know perfectly well that the way you
evaluate that choice is not in terms of
203
00:12:30,619 --> 00:12:36,180
“Hey I'm $5 better off than I am now,”
it's “Hey if there's $1,000 in the other
204
00:12:36,180 --> 00:12:41,449
envelope, then I'm 995 dollars worse
off than if I'd have chosen envelope
205
00:12:41,450 --> 00:12:47,400
number two.” They all see that, so I
don't have to teach them any new rules.
206
00:12:47,939 --> 00:12:52,670
All I have to do is when I say “Hey
let's go to war, it has this advantage,”
207
00:12:53,039 --> 00:12:56,100
and if they say, "Yeah, that sounds
good,” I can say, "Wait a minute.
208
00:12:56,139 --> 00:13:01,129
How is this different from the two
envelope problem?” So the main reason I
209
00:13:01,159 --> 00:13:04,810
chose these, and these aren't the only
rules I teach in my critical thinking
210
00:13:04,810 --> 00:13:09,399
class, but the main reason I choose
the rules that I do is that I think
211
00:13:09,619 --> 00:13:14,269
that my students already know these
rules and at least sometimes apply
212
00:13:14,269 --> 00:13:20,369
them, and so when I introduce them
to these rules in cases to which they
213
00:13:20,369 --> 00:13:25,080
are inclined to apply them, they give
me what I call the old bovine stare.
214
00:13:25,089 --> 00:13:27,860
They look at me like, "Well,
that's obvious. You're not teaching
215
00:13:27,860 --> 00:13:31,039
me anything I didn't already
know.” And that's what I want.
216
00:13:31,089 --> 00:13:34,830
I want them when I introduce these
rules to say, "I already knew that.
217
00:13:34,830 --> 00:13:39,690
Why are you teaching me something so
obvious?” Because then the next class
218
00:13:39,690 --> 00:13:44,559
period, or a week from now, or a month
from now, when I introduce an argument
219
00:13:44,619 --> 00:13:48,049
where they ought to be using this
rule that they said was so obvious
220
00:13:48,049 --> 00:13:51,969
and they're not, I don't have to say,
"Hey, you guys have made a mistake.”
221
00:13:52,089 --> 00:13:55,839
I can just ask them, "Okay, wait a
minute, you said this rule was obvious,
222
00:13:55,959 --> 00:13:57,559
why aren't you applying it here?
223
00:13:57,869 --> 00:14:03,249
Either you were wrong on day one about
that rule being the right rule, or
224
00:14:03,250 --> 00:14:07,469
you're wrong right now in not applying
that rule.” And so there's a real
225
00:14:07,470 --> 00:14:12,249
sense in which the reason I chose these
rules is because I think I'm really
226
00:14:12,250 --> 00:14:14,729
not teaching my students anything new.
227
00:14:15,119 --> 00:14:19,800
So here's one way to think about it:
rather than teaching them like calculus,
228
00:14:19,989 --> 00:14:25,560
like a chain rule, like a new formula that
they don't know, what I'm really trying
229
00:14:25,560 --> 00:14:29,330
to do is get them to recognize cardinals.
230
00:14:29,569 --> 00:14:32,570
They already know what a cardinal looks
like, so I'm just trying to get them to
231
00:14:32,570 --> 00:14:38,290
recognize cardinals, and so that's why
I think that the way I teach critical
232
00:14:38,300 --> 00:14:44,539
thinking tends to work and is way easier
than starting off with the assumption that
233
00:14:44,559 --> 00:14:48,289
“Hey, my students don't know anything,
and I need to teach them a whole bunch of
234
00:14:48,300 --> 00:14:52,860
new rules.” So that's why I chose these
rules, and that's why I choose the rules
235
00:14:52,870 --> 00:14:55,050
that I use in my critical thinking class.
236
00:14:55,670 --> 00:14:59,059
JOHN: And these are rules that,
when you think about them as you
237
00:14:59,059 --> 00:15:01,120
framed it, they're pretty obvious.
238
00:15:01,300 --> 00:15:05,340
But in practice, they're rules
that people violate all of the
239
00:15:05,340 --> 00:15:06,959
time in their decision making.
240
00:15:07,209 --> 00:15:10,929
What really impressed me about the study
was that you did get some [LAUGHTER]
241
00:15:10,939 --> 00:15:15,649
results out of this because so many
studies of behavior modification have
242
00:15:15,650 --> 00:15:20,599
been found to be ineffective beyond a
very short period of time, often not
243
00:15:20,599 --> 00:15:22,279
lasting through the end of the class.
244
00:15:22,549 --> 00:15:23,410
MICHAEL: That's absolutely right.
245
00:15:23,410 --> 00:15:27,799
In fact, one of my old professors, he said
this about logic classes, but I think it
246
00:15:27,849 --> 00:15:31,899
unfortunately is often true of critical
thinking classes, but he said that logic
247
00:15:31,899 --> 00:15:35,129
classes teach you to do one and only
one thing, and that's take logic tests.
248
00:15:35,379 --> 00:15:39,760
And I sometimes worry that that's
the case with a lot of our attempts
249
00:15:39,770 --> 00:15:43,939
to teach people to think better,
is that what we're really doing is
250
00:15:44,000 --> 00:15:48,399
teaching them to take tests because
they will use the rules on the test.
251
00:15:48,809 --> 00:15:52,809
But if you want them to think about
this afterwards, I really do think
252
00:15:52,820 --> 00:15:57,640
that the main thing is to get them to
commit to the rule, and then get them to
253
00:15:57,640 --> 00:16:02,399
recognize, “Hey, wait a minute,” to be
sensitive to when they see it in the real
254
00:16:02,400 --> 00:16:04,279
world because it's all over the place.
255
00:16:04,290 --> 00:16:07,130
Once you get sensitive to it,
you start seeing it everywhere.
256
00:16:07,130 --> 00:16:11,249
And indeed, one danger of my class, of
my critical thinking class that I talk
257
00:16:11,250 --> 00:16:15,429
to my students is what I call “the turd
in the punch bowl problem,” and that
258
00:16:15,430 --> 00:16:19,589
is being a good critical thinker isn't
necessarily going to make you popular, and
259
00:16:19,589 --> 00:16:24,989
so you need to be careful about when you
point out to people that their arguments
260
00:16:25,040 --> 00:16:29,489
aren't very good, because if you do it
too much, you will end up being like
261
00:16:29,659 --> 00:16:31,239
the turd in the punch bowl at a party.
262
00:16:31,400 --> 00:16:34,109
People will look at you and say,
"Oh, I'm going to stay away from
263
00:16:34,110 --> 00:16:37,939
that person.” And so, at the end of
the semester, I will ask them, "Have
264
00:16:37,939 --> 00:16:42,139
you had any turd in the punch bowl
experiences as a result of this class?
265
00:16:42,199 --> 00:16:45,569
And at the end of the semester, a lot
of the students – and I don't know if
266
00:16:45,569 --> 00:16:51,569
this is good or bad – but about half
of the cases involve the sunk cost
267
00:16:51,569 --> 00:16:58,019
principle, and usually it involves
the destruction of a relationship.
268
00:16:58,429 --> 00:17:01,960
So people will end a relationship
because the only reason they've been
269
00:17:01,960 --> 00:17:06,769
sticking with the relationship is
all the pain and effort and heartache
270
00:17:06,809 --> 00:17:08,299
they've put into the relationship.
271
00:17:09,020 --> 00:17:12,000
JOHN: And I know behavioral economists
have noted that quite a bit.
272
00:17:12,020 --> 00:17:15,750
That once they really get this message
through, it leads to a breakup of so
273
00:17:15,750 --> 00:17:17,629
many relationships of their students.
274
00:17:18,149 --> 00:17:23,120
But even people who know these things
sometimes fall into these traps.
275
00:17:23,139 --> 00:17:26,320
And one example I think I mentioned
on the podcast before, and I know I've
276
00:17:26,320 --> 00:17:30,299
mentioned it to Rebecca, is one time
I was at a conference with a few other
277
00:17:30,300 --> 00:17:34,439
people, and we went over to Universal
park, and it was just really hot that
278
00:17:34,440 --> 00:17:39,110
day, it was like 97 degrees, and we had
spent quite a bit for the tickets for the
279
00:17:39,110 --> 00:17:41,670
day, and we were just kind of miserable.
280
00:17:41,690 --> 00:17:44,749
And one of the other people
said we should just go.
281
00:17:44,830 --> 00:17:53,230
And my response to a historian was that
we spent $230 and we've gone on two rides.
282
00:17:53,610 --> 00:17:55,810
We really should stay and
get our money's worth.
283
00:17:56,080 --> 00:18:01,459
And her response was, "But that's a
sunk cost. It shouldn't matter,” and
284
00:18:01,549 --> 00:18:03,610
it's easy for us to fall into that.
285
00:18:04,000 --> 00:18:06,710
MICHAEL: Absolutely, and indeed,
this is one of the reasons I choose
286
00:18:06,710 --> 00:18:09,429
the rules that I do, is that I
think many of them are significant.
287
00:18:09,430 --> 00:18:13,249
They really can make a difference
to the quality of your decisions
288
00:18:13,510 --> 00:18:15,119
and the quality of your judgments.
289
00:18:15,639 --> 00:18:18,410
REBECCA: Can you talk a little
bit about the two-stage training
290
00:18:18,410 --> 00:18:20,350
program that was used in class?
291
00:18:20,899 --> 00:18:21,459
MICHAEL: Sure.
292
00:18:21,549 --> 00:18:24,970
So I gave you a little bit of an
example of it already, but let me
293
00:18:24,970 --> 00:18:28,040
give you an example with another
case with the correlation cheat code.
294
00:18:28,419 --> 00:18:34,720
So my go-to example here is I'll tell the
students, “Well, you know, yesterday I ate
295
00:18:34,730 --> 00:18:36,950
chocolate and today I broke out with zits.
296
00:18:37,250 --> 00:18:41,459
Boy, that means that eating chocolate
causes zits,” and they immediately say,
297
00:18:41,459 --> 00:18:45,889
“No, that's terrible.” So the first stage,
what I'm doing is I'm introducing examples
298
00:18:45,889 --> 00:18:49,179
where they are disposed to use the rule.
299
00:18:49,529 --> 00:18:53,079
So nobody says, "Yeah, that's great
evidence for thinking that eating
300
00:18:53,239 --> 00:18:56,929
chocolate causes zits.” And then we
talk a little bit about, “Okay, well,
301
00:18:56,979 --> 00:19:01,230
what would count as good evidence?”
And they come up pretty quickly with
302
00:19:01,230 --> 00:19:04,450
maybe just a little bit of help, “Well,
you want to know what happens when you
303
00:19:04,450 --> 00:19:07,480
eat chocolate and when you don't eat
chocolate, because what you really want
304
00:19:07,480 --> 00:19:12,749
to know is what difference does eating
chocolate make to your complexion, and
305
00:19:12,750 --> 00:19:15,240
so what you want to know is what happens
when you eat chocolate and you don't eat
306
00:19:15,240 --> 00:19:18,889
chocolate, and everything else relevant
to your complexion is the same, and
307
00:19:18,889 --> 00:19:23,120
what difference is there in these two
cases,” and so that's the first stage.
308
00:19:23,120 --> 00:19:28,440
The first stage is just introducing
them to these rules, these cheat codes.
309
00:19:28,440 --> 00:19:31,839
So I give them the cheat code,
but really a cheat code which they
310
00:19:31,839 --> 00:19:35,229
detect right away as a cheat code,
and that they know as a cheat code.
311
00:19:35,250 --> 00:19:40,599
And then I spend the rest of the semester
giving them arguments that use that cheat
312
00:19:40,599 --> 00:19:43,779
code, but are not as easy to detect.
313
00:19:44,089 --> 00:19:47,839
And often these are political arguments,
and one reason I don't get into trouble
314
00:19:47,939 --> 00:19:51,649
– at least I haven't so far – in giving
political arguments is that these cheat
315
00:19:51,649 --> 00:19:55,979
codes you can use them to defend views
on the left, on the right, in the middle.
316
00:19:56,099 --> 00:19:57,789
They're tools of manipulation.
317
00:19:58,039 --> 00:19:59,829
It doesn't matter what the truth is.
318
00:20:00,080 --> 00:20:04,170
You can use these cheat codes to
defend the true view, the false view,
319
00:20:04,449 --> 00:20:06,830
smart decisions, dumb decisions.
320
00:20:07,190 --> 00:20:11,979
And so, what I'm really interested
in doing is getting them to be
321
00:20:11,979 --> 00:20:13,559
able to detect these cheat codes.
322
00:20:13,559 --> 00:20:16,940
So a first step is just introduce
them to these cheat codes in a
323
00:20:16,940 --> 00:20:21,359
way that they will be disposed
to accept and think is obvious.
324
00:20:21,720 --> 00:20:24,479
And then after that, I'll give
them lots of examples in which
325
00:20:24,519 --> 00:20:27,359
they're, at least many of them, are
not disposed to think is obvious.
326
00:20:27,500 --> 00:20:32,060
And then rather than me sort of coming
from on high and saying, "Well, you know,
327
00:20:32,389 --> 00:20:36,539
you made a mistake there, dummy.” I just
step back and I say, "Wait a minute.
328
00:20:36,989 --> 00:20:38,799
We seem to have an inconsistency here.
329
00:20:39,399 --> 00:20:44,000
You insisted that this rule was
right on Monday, and on Friday you're
330
00:20:44,000 --> 00:20:46,049
not using the rule in this case.
331
00:20:46,269 --> 00:20:50,540
What's going on?” And I have yet
to have a student, and I've taught
332
00:20:50,570 --> 00:20:54,459
1000s of students, I've yet to have
a student say, "Okay, I'm going to
333
00:20:54,460 --> 00:20:58,080
rethink that rule.” They all say,
"Oh, well, I made a mistake here.
334
00:20:58,150 --> 00:21:02,820
I reasoned badly,” and by the end of the
semester, they're not reasoning badly.
335
00:21:03,129 --> 00:21:05,930
REBECCA: Sounds like you spend a very
small amount of time on stage one
336
00:21:05,930 --> 00:21:08,749
and significant time on stage two.
337
00:21:09,250 --> 00:21:12,420
Do you do all of the cheat codes at
the beginning of the semester and
338
00:21:12,420 --> 00:21:16,440
then work on utilizing them throughout
the semester, or do you kind of
339
00:21:16,440 --> 00:21:18,550
do units on different cheat codes?
340
00:21:18,550 --> 00:21:19,579
How do you organize?
341
00:21:19,959 --> 00:21:23,519
MICHAEL: So I will introduce a
cheat code and then I'll give them
342
00:21:23,520 --> 00:21:27,079
examples using that cheat code and
all the cheat codes we've covered.
343
00:21:27,540 --> 00:21:33,600
So once a cheat code is introduced, they
might see it at any point in the future,
344
00:21:33,860 --> 00:21:37,409
and I think it's really important they
keep practicing it and keep being able to
345
00:21:37,419 --> 00:21:42,189
recognize it, and not just think, "Hey, I
know this is the cheat code we're working
346
00:21:42,189 --> 00:21:45,759
on today, so I just need to be on the
lookout for that one.” Another thing I
347
00:21:45,759 --> 00:21:51,100
do, and I think this is really important,
is I will sometimes give them arguments
348
00:21:51,110 --> 00:21:56,070
that are promising, that if you grant the
truth of the premises, they at least give
349
00:21:56,070 --> 00:21:59,369
you some reason to accept the conclusion.
350
00:21:59,709 --> 00:22:04,870
Because I don't want the students
just to be skeptical of all arguments.
351
00:22:04,879 --> 00:22:08,900
I want them to be able to distinguish
between “Okay, that's a cheat code.
352
00:22:09,180 --> 00:22:10,740
That's definitely a bad argument.
353
00:22:10,740 --> 00:22:13,809
The conclusion might be true, the
conclusion might be false, but that's
354
00:22:13,809 --> 00:22:17,779
not a good reason to accept anything”
and then “Okay, well, what would count
355
00:22:17,779 --> 00:22:22,629
as good evidence for thinking that
this is a smart decision or that this
356
00:22:22,650 --> 00:22:27,739
policy has led to increased costs or
decreased costs?” And I want them to be
357
00:22:27,739 --> 00:22:31,319
able to say, "Okay, look, that argument
right there, that's pretty promising.
358
00:22:31,409 --> 00:22:33,899
This argument over here
definitely uses a cheat code.”
359
00:22:34,610 --> 00:22:37,479
JOHN: So you start off with a little
bit of retrieval practice with some
360
00:22:37,479 --> 00:22:41,279
applications of the same concept, and
then you move into some interleaved
361
00:22:41,279 --> 00:22:45,029
practice, which raises the cognitive
difficulty just a little bit.
362
00:22:45,339 --> 00:22:49,780
But it's one of those desirable
difficulties where the more cognitive
363
00:22:49,809 --> 00:22:54,369
effort that students have to place
in identifying what is at issue, the
364
00:22:54,369 --> 00:22:57,740
more deeply they tend to learn it,
and the more likely it is that those
365
00:22:57,740 --> 00:23:02,559
concepts will stick and also transfer
into other applications, which is what
366
00:23:02,559 --> 00:23:04,489
you're building to in this program.
367
00:23:04,500 --> 00:23:08,969
MICHAEL: Absolutely, absolutely,
the interleaved practice is really
368
00:23:08,970 --> 00:23:14,979
important, and I wouldn't go back
to sort of the blocking practice.
369
00:23:15,539 --> 00:23:17,819
REBECCA: So we've talked a little
bit about what the intervention is.
370
00:23:17,819 --> 00:23:21,379
Can you talk a little bit about how the
study was conducted, like the size of the
371
00:23:21,379 --> 00:23:23,409
treatment group, the control group, etc.
372
00:23:23,529 --> 00:23:23,919
MICHAEL: Sure.
373
00:23:24,399 --> 00:23:29,359
So in the paper, we really describe
just the second study we did, we
374
00:23:29,360 --> 00:23:33,479
actually did two studies, and in both
studies we had almost 400 students.
375
00:23:34,209 --> 00:23:39,839
In the first study we had 395 students,
and the second study we had 397 students.
376
00:23:40,550 --> 00:23:44,639
And some of the students were taking
a critical thinking class from me,
377
00:23:44,740 --> 00:23:48,980
and the rest of the students were
taking other philosophy classes.
378
00:23:49,410 --> 00:23:55,039
So in the study that we report
on, we had 113 critical thinking
379
00:23:55,040 --> 00:23:58,459
students and 284 control students.
380
00:23:58,920 --> 00:24:04,469
Then what we did is, in the first week
of class, we gave everyone a pretest,
381
00:24:04,740 --> 00:24:10,259
and this involved the five cheat
codes, and then two good arguments,
382
00:24:10,449 --> 00:24:11,769
or at least promising arguments.
383
00:24:11,779 --> 00:24:15,709
They weren't necessarily great, but
they didn't use any cheat codes.
384
00:24:16,070 --> 00:24:20,150
And the reason we included these, what
we called skepticism checks, was just to
385
00:24:20,169 --> 00:24:25,480
check if all we were doing was creating
skeptical students rather than students
386
00:24:25,480 --> 00:24:29,939
who could distinguish between promising
arguments and cheat code arguments.
387
00:24:30,730 --> 00:24:34,619
Then, at the end of the semester,
we gave them the same survey,
388
00:24:34,689 --> 00:24:40,239
same seven questions, and not
surprisingly, in both of the studies,
389
00:24:40,799 --> 00:24:45,209
the training group students did much
better than the control students.
390
00:24:46,079 --> 00:24:48,509
In the first study, we
waited three months.
391
00:24:48,879 --> 00:24:51,469
So basically, the way to think
about it is that these classes
392
00:24:51,470 --> 00:24:52,550
all happened in the fall.
393
00:24:52,860 --> 00:24:54,779
We waited till the end
of the spring semester.
394
00:24:55,139 --> 00:24:59,029
It might have been four months,
and gave them same questions
395
00:24:59,029 --> 00:25:00,589
again in the first study.
396
00:25:00,589 --> 00:25:02,069
We gave them the same questions again.
397
00:25:02,379 --> 00:25:05,869
The trained students did much better, and
then we gave them different questions,
398
00:25:05,959 --> 00:25:07,209
and they still did much better.
399
00:25:07,439 --> 00:25:12,080
So in the second study we did, which is
really the only study that we report on
400
00:25:12,189 --> 00:25:16,970
in the article, we gave them pre-test at
the beginning of the semester, post-test.
401
00:25:17,000 --> 00:25:23,800
Then we waited a full academic year plus
the spring, so we waited 16 months and
402
00:25:23,809 --> 00:25:26,339
only asked them different questions.
403
00:25:26,949 --> 00:25:31,199
So the pretest and posttest were
the same questions, and again my
404
00:25:31,199 --> 00:25:32,959
students did much much better.
405
00:25:33,269 --> 00:25:39,379
But then we waited 16 months and asked
them seven different questions, and
406
00:25:39,459 --> 00:25:43,810
the trained students were still doing
much better, detecting the cheat codes
407
00:25:44,049 --> 00:25:47,709
and recognizing that the good arguments
were still pretty good arguments.
408
00:25:48,309 --> 00:25:50,959
JOHN: DId the trained students
do significantly better with
409
00:25:50,969 --> 00:25:52,970
all 5 of these cheat codes?
410
00:25:53,369 --> 00:25:57,160
MICHAEL: No, they did better
on 4 of the 5 cheat codes.
411
00:25:57,190 --> 00:26:01,880
They did better on the correlation,
regression, sunk cost, and opportunity
412
00:26:01,880 --> 00:26:08,089
cost cheat codes, but in all the
studies that we did, on all the surveys
413
00:26:08,089 --> 00:26:12,179
that we asked, we just couldn’t get
the trained students to do better
414
00:26:12,199 --> 00:26:14,210
on the gambler’s fallacy questions.
415
00:26:14,690 --> 00:26:18,629
JOHN: But still, having students do
better on 4 out of those 5 cheat codes
416
00:26:18,940 --> 00:26:23,449
so long after the initial intervention
is a really impressive result.
417
00:26:23,970 --> 00:26:27,179
One of the things you talk about
near the end of the paper, though,
418
00:26:27,220 --> 00:26:31,600
is that you note that there are some
possible sample selection issues.
419
00:26:32,039 --> 00:26:36,979
Could you just talk a little bit about
what source of sample selection might have
420
00:26:36,979 --> 00:26:39,099
affected the magnitude of those results?
421
00:26:39,789 --> 00:26:43,490
MICHAEL: So one worry that one
might have is that the students
422
00:26:43,490 --> 00:26:45,230
were not distributed randomly.
423
00:26:45,420 --> 00:26:49,290
So the students chose to take the
critical thinking class, the students
424
00:26:49,320 --> 00:26:53,409
chose to take other philosophy classes,
and so there might be something about
425
00:26:53,410 --> 00:26:58,259
the students who chose critical thinking
classes that would incline them to be
426
00:26:58,270 --> 00:27:02,899
better critical thinkers, despite the
fact that on the pre-tests, all the
427
00:27:02,900 --> 00:27:04,930
students in both groups did equally badly.
428
00:27:05,139 --> 00:27:09,730
So that's one worry one might have,
and the thing to do there is, although
429
00:27:09,730 --> 00:27:15,039
it's not really easy given the
institutions of academia, is to randomly
430
00:27:15,040 --> 00:27:18,919
distribute students into critical
thinking classes and other classes.
431
00:27:19,639 --> 00:27:22,109
So that's the main selection worry.
432
00:27:22,739 --> 00:27:26,029
Another reason why we might have
gotten good results is that I
433
00:27:26,029 --> 00:27:27,920
had those students for 15 weeks.
434
00:27:28,259 --> 00:27:32,149
Some interventions talk about full-
semester interventions, but often
435
00:27:32,549 --> 00:27:34,339
they're fairly short interventions.
436
00:27:34,660 --> 00:27:40,679
I was able to have the students for
15 weeks and made them practice a lot.
437
00:27:41,020 --> 00:27:44,199
REBECCA: It sounds like in the
design of the experiment in your
438
00:27:44,199 --> 00:27:47,379
class, just generally, there's lots
of opportunities for feedback and
439
00:27:47,379 --> 00:27:49,419
correction throughout the semester.
440
00:27:49,650 --> 00:27:50,870
MICHAEL: Yes, absolutely.
441
00:27:50,979 --> 00:27:57,279
There's a lot of feedback, a lot of
correction, and I have a big bank
442
00:27:57,549 --> 00:28:01,339
of good and bad arguments that I
can give them over the course of the
443
00:28:01,339 --> 00:28:05,119
semester that use these cheat codes
or that don't use the cheat codes.
444
00:28:05,339 --> 00:28:10,729
And one of the things I do for an
essay in this class is ask the students
445
00:28:11,129 --> 00:28:16,549
to put themselves in the shoes of
someone who's trying to come up with
446
00:28:16,549 --> 00:28:21,399
a cheat code argument and ask them
to give me bad arguments that use
447
00:28:21,400 --> 00:28:24,339
these cheat codes, and they actually
turn out to be pretty good at it.
448
00:28:24,430 --> 00:28:28,969
And many of the examples that I use
on later students are examples that
449
00:28:28,970 --> 00:28:30,879
prior students have come up with.
450
00:28:31,369 --> 00:28:32,940
REBECCA: Sounds like it makes
it a little bit fun too.
451
00:28:33,540 --> 00:28:37,339
MICHAEL: Yes, I do ask them
for any humorous examples or
452
00:28:37,339 --> 00:28:40,899
examples that us boomers would
have a hard time understanding.
453
00:28:42,489 --> 00:28:42,539
[LAUGHTER]
454
00:28:42,930 --> 00:28:47,329
JOHN: One of the things we appreciated
about this study is that the data, the
455
00:28:47,369 --> 00:28:52,069
results, and the analyses that were
done are all reported in more detail
456
00:28:52,410 --> 00:28:54,860
on the Open Science Framework website.
457
00:28:55,190 --> 00:28:59,289
We'll include a link to that website
in the show notes for this episode.
458
00:29:00,250 --> 00:29:03,209
REBECCA: So we always wrap
up by asking, "What's next?
459
00:29:04,129 --> 00:29:05,259
MICHAEL: A few things are next.
460
00:29:05,289 --> 00:29:11,769
One is I'm writing a textbook that
uses the materials that I teach.
461
00:29:12,169 --> 00:29:15,599
The way I teach critical
thinking is different from
462
00:29:15,779 --> 00:29:16,919
the way other people teach it.
463
00:29:17,229 --> 00:29:20,429
I haven't found a good textbook
that teaches critical thinking
464
00:29:20,429 --> 00:29:21,599
the way I like to teach it.
465
00:29:21,619 --> 00:29:24,909
So I'm currently writing a
textbook that hopefully will help
466
00:29:24,910 --> 00:29:26,599
other people get good results.
467
00:29:26,910 --> 00:29:32,739
I'm also planning to build, if I can,
a game that anybody can play to see
468
00:29:32,739 --> 00:29:38,360
whether they can avoid being bamboozled
[LAUGHTER] by cheat code arguments, this
469
00:29:38,360 --> 00:29:43,490
is sort of a longer term project because
writing a book is going to be useful,
470
00:29:43,490 --> 00:29:45,410
hopefully, for people to teach this.
471
00:29:45,510 --> 00:29:50,170
But what's really important, in
my view, what makes this stick is
472
00:29:50,170 --> 00:29:51,880
practicing on lots of examples.
473
00:29:52,240 --> 00:29:57,159
And so, in a game where people see an
argument and you say, “Well, is this
474
00:29:57,209 --> 00:30:01,360
promising argument, or is somebody trying
to manipulate you?” we'll hopefully
475
00:30:01,360 --> 00:30:05,019
give people practice at being able
to detect those cheat code arguments.
476
00:30:05,350 --> 00:30:07,560
So those are two of the
projects that I'm working on.
477
00:30:08,019 --> 00:30:10,820
JOHN: On the latter project,
will it be a board game?
478
00:30:10,830 --> 00:30:12,649
Will it be an electronic game?
479
00:30:12,649 --> 00:30:16,589
Will it be an online game, or
is that something that'll be
480
00:30:16,590 --> 00:30:18,050
decided further in the future?
481
00:30:18,110 --> 00:30:19,999
MICHAEL: No, it'll be an online game.
482
00:30:20,049 --> 00:30:21,989
Is what I'm hoping to do.
483
00:30:22,529 --> 00:30:26,139
I want to do something that can
be used not just by my students
484
00:30:26,139 --> 00:30:27,369
and that's easily accessible.
485
00:30:27,590 --> 00:30:28,969
REBECCA: Sounds like some great projects.
486
00:30:28,969 --> 00:30:30,510
Thanks for joining us today.
487
00:30:30,779 --> 00:30:31,469
MICHAEL: Oh, you're welcome.
488
00:30:31,469 --> 00:30:32,259
It's been a pleasure.
489
00:30:32,570 --> 00:30:33,059
JOHN: Thank you.
490
00:30:33,160 --> 00:30:36,229
I was really impressed by this study
[LAUGHTER] because, as I mentioned
491
00:30:36,239 --> 00:30:41,280
before, it's so rare to see effects
that are so significant and persistent.
492
00:30:41,520 --> 00:30:45,879
It's, I think, something that people can
learn from in terms of other types of
493
00:30:45,879 --> 00:30:50,879
issues where it's hard to convey this
sort of long-term impact on students.
494
00:30:51,299 --> 00:30:52,159
MICHAEL: Well, thank you so much.
495
00:31:00,729 --> 00:31:00,829
[MUSIC]
496
00:31:03,669 --> 00:31:06,440
JOHN: If you've enjoyed this
podcast, please subscribe and
497
00:31:06,440 --> 00:31:10,500
leave a review on Apple Podcasts
or your favorite podcast service.
498
00:31:11,059 --> 00:31:15,589
To continue the conversation, join us
on our Tea for Teaching Facebook page.
499
00:31:16,540 --> 00:31:19,149
REBECCA: You can find show
notes, transcripts and other
500
00:31:19,150 --> 00:31:21,480
materials on teaforteaching.com.
501
00:31:22,509 --> 00:31:24,479
Music by Michael Gary Brewer.
502
00:31:24,479 --> 00:31:28,479
[MUSIC]
