#73 - Does first impression ‘matter’ -and does it matter if I think about it?

1
00:00:17,895 --> 00:00:19,756
LARA VARPIO: Hi, everybody, and
welcome to another episode of
2
00:00:19,736 --> 00:00:24,200
the Papers Podcast. It is our
joy to come into your phone and
3
00:00:24,200 --> 00:00:27,903
your earphones every week and
share with you a little summary,
4
00:00:27,923 --> 00:00:30,185
a discussion of some of the
medical education literature
5
00:00:30,185 --> 00:00:32,647
that we think is actually worth
a little bit of time and
6
00:00:32,667 --> 00:00:33,127
reflection.
7
00:00:33,507 --> 00:00:35,248
And so with that, of course, Of
course, it wouldn't be a
8
00:00:35,289 --> 00:00:37,530
conversation without some
friends. So first, let me say,
9
00:00:37,590 --> 00:00:38,711
hey, John, how you doing?
10
00:00:39,691 --> 00:00:43,614
JONATHAN SHERBINO: Good, Laura.
I like your ballsy Canadian
11
00:00:43,654 --> 00:00:46,996
tuxedo wear. Yeah. Do you want
to explain for the audio segment
12
00:00:46,996 --> 00:00:47,497
what you have on?
13
00:00:47,857 --> 00:00:49,518
LARA VARPIO: So for those of you
who aren't in the know, a
14
00:00:49,538 --> 00:00:53,381
Canadian tuxedo is a flannel
shirt with checks. And I have
15
00:00:53,421 --> 00:00:56,123
one of my favorites on today
because I forgot we were doing
16
00:00:56,163 --> 00:01:00,265
recordings. And no makeup, no
hair, and Canadian tuxedo. I'm
17
00:01:00,285 --> 00:01:00,706
just saying.
18
00:01:00,706 --> 00:01:03,007
JONATHAN SHERBINO: It's my
favorite version of you.
19
00:01:03,387 --> 00:01:04,008
LARA VARPIO: Oh, bless you.
20
00:01:05,309 --> 00:01:07,210
And Jason, where's your tuxedo,
dude?
21
00:01:07,630 --> 00:01:09,932
JASON FRANK: These are my
jammies. So this is where I'm
22
00:01:09,952 --> 00:01:12,994
always in. Yeah. Hi, everybody.
It's great to be here.
23
00:01:13,695 --> 00:01:16,116
LARA VARPIO: And today, my
friend, Linda, it's always good
24
00:01:16,116 --> 00:01:19,759
to see you. You have picked a
selection for us about first
25
00:01:19,819 --> 00:01:22,240
impressions. So I'm really
excited because my Canadian
26
00:01:22,300 --> 00:01:25,002
tuxedo is definitely making a
first impression.
27
00:01:25,903 --> 00:01:30,346
LINDA SNELL: Definitely. It is
plaid shirts and all. So I'll
28
00:01:30,526 --> 00:01:36,710
start by saying this article is.
Called Implicit Versus Explicit
29
00:01:36,851 --> 00:01:40,033
First Impressions in
Performance-Based Assessment,
30
00:01:40,713 --> 00:01:44,436
colon, Will Raiders Overcome
Their First Impressions When
31
00:01:44,516 --> 00:01:47,719
Learner Performance Changes?
It's in 2024.
32
00:01:47,719 --> 00:01:52,182
It advances in health sciences
education with a cast of stars,
33
00:01:52,642 --> 00:01:57,526
including Tim Woods, Vijay
Daniels, Deborah Pugh, Claire
34
00:01:57,586 --> 00:02:01,609
Touchie, Samantha Halman, Susan
Humphrey-Murto. All colleagues,
35
00:02:02,129 --> 00:02:07,734
most of them are colleagues of
Jason. So my title for this is,
36
00:02:08,174 --> 00:02:12,457
Does My First Impression Matter?
And Does It Matter If I Think
37
00:02:12,497 --> 00:02:12,897
About It?
38
00:02:14,198 --> 00:02:20,163
So let me ask you very briefly
to describe a first impression
39
00:02:20,383 --> 00:02:24,086
where maybe it wasn't confirmed
as you got to know an
40
00:02:24,126 --> 00:02:26,828
individual. I'll go Lara, Jason,
John.
41
00:02:27,416 --> 00:02:29,697
LARA VARPIO: Linda, I don't have
one about a person, individual
42
00:02:29,697 --> 00:02:33,299
that I knew, but my favorite
example of first impressions
43
00:02:33,339 --> 00:02:37,261
gone awry. Did you all used to
watch Britain's Got Talent? One
44
00:02:37,261 --> 00:02:40,223
of the earliest seasons, there
was a woman who was a singer.
45
00:02:40,263 --> 00:02:42,624
And when they first showed her,
she just looked like a normal
46
00:02:42,664 --> 00:02:45,626
person, just like any of us.
47
00:02:46,066 --> 00:02:49,868
And then she sat up on stage and
you could tell that the Simon
48
00:02:49,928 --> 00:02:52,650
Cowell, one of the judges was a
little bit unimpressed with how
49
00:02:52,690 --> 00:02:56,392
this was already going. And then
she opened her mouth and like
50
00:02:56,432 --> 00:02:59,974
a... Voice of an angel just
like, bam, just stopped
51
00:03:00,035 --> 00:03:03,377
everybody dead. I love it when
that happens. So that's my
52
00:03:03,457 --> 00:03:05,659
favorite example of first
impressions gone awry.
53
00:03:06,139 --> 00:03:07,620
JONATHAN SHERBINO: Her name was
Susan something, I think.
54
00:03:07,720 --> 00:03:08,481
LARA VARPIO: Yeah, it was.
55
00:03:08,561 --> 00:03:09,201
JONATHAN SHERBINO: Susan
something.
56
00:03:09,482 --> 00:03:11,663
LINDA SNELL: Something. She got
a recording out of it too, I
57
00:03:11,683 --> 00:03:11,883
think.
58
00:03:12,484 --> 00:03:14,025
LARA VARPIO: And while you
discuss it, I will find her
59
00:03:14,045 --> 00:03:14,205
name.
60
00:03:15,306 --> 00:03:16,967
JASON FRANK: Didn't she perform
in sweatpants the first time or
61
00:03:16,967 --> 00:03:18,969
something like that? Like
totally casual.
62
00:03:20,010 --> 00:03:21,991
LINDA SNELL: Going back to first
impressions, Jason.
63
00:03:23,412 --> 00:03:26,313
JASON FRANK: This happens every
week with learners. You can meet
64
00:03:26,373 --> 00:03:30,475
someone and hear the beginning
of some case or watch some piece
65
00:03:30,475 --> 00:03:33,436
of something, and then over the
course of working with them, you
66
00:03:33,456 --> 00:03:36,757
realize that they're actually in
some other place or they're just
67
00:03:37,098 --> 00:03:37,958
growing in front of you.
68
00:03:39,018 --> 00:03:41,000
Just recently, I worked with
somebody, and it was actually
69
00:03:41,000 --> 00:03:45,161
their first day in Ottawa with
us in the emergency department,
70
00:03:45,681 --> 00:03:49,262
and they were struggling with
the electronic medical record
71
00:03:49,262 --> 00:03:51,864
and they were struggling with
how things work here, and that
72
00:03:51,904 --> 00:03:52,784
made them look okay.
73
00:03:53,184 --> 00:03:55,405
But not superstar. And then I
worked with them again a month
74
00:03:55,465 --> 00:03:58,225
later and they were superstars.
So this happens all the time.
75
00:03:58,946 --> 00:04:01,466
All of us who are raters have to
be really careful about
76
00:04:02,467 --> 00:04:05,047
documenting what we see and not
comparing them to some other
77
00:04:05,127 --> 00:04:05,528
archetype.
78
00:04:06,548 --> 00:04:09,129
LARA VARPIO: Her name was Susan
Boyle. Her name was Susan Boyle
79
00:04:09,289 --> 00:04:13,710
and Boyle made over 5 million
British pounds. And by today's
80
00:04:13,750 --> 00:04:17,251
money, that's 8.5 million with
the release of I Dreamed a Dream
81
00:04:18,331 --> 00:04:21,472
and her success continued her
second album. Was one of the
82
00:04:21,472 --> 00:04:24,874
first to top both the UK and US
album charts with two different
83
00:04:24,934 --> 00:04:25,835
albums in the same year.
84
00:04:26,475 --> 00:04:27,396
LINDA SNELL: Boom.
85
00:04:27,476 --> 00:04:29,737
JASON FRANK: Lava, you can sing
too. Go ahead.
86
00:04:30,737 --> 00:04:32,318
LARA VARPIO: No, that's not me.
No, no.
87
00:04:32,939 --> 00:04:35,420
JONATHAN SHERBINO: Let's just
keep going. So my first
88
00:04:35,460 --> 00:04:38,202
impression was of the three of
you, I thought, wow, such
89
00:04:38,322 --> 00:04:40,543
beautiful, smart, intelligent
people.
90
00:04:40,583 --> 00:04:43,785
LARA VARPIO: Oh, no. Careful, he
wants money or he wants a favor.
91
00:04:43,785 --> 00:04:44,925
JONATHAN SHERBINO: Look, the
reader decided that first
92
00:04:44,945 --> 00:04:47,547
impression held true 20 years
later.
93
00:04:48,007 --> 00:04:48,888
JASON FRANK: That's the before.
94
00:04:50,048 --> 00:04:52,453
JONATHAN SHERBINO: Here's my
first impression story, Linda,
95
00:04:52,634 --> 00:04:55,620
is I've done some work with Matt
Sebold at McMaster University
96
00:04:55,740 --> 00:04:59,380
where we showed. Clinicians in
practice with lots of
97
00:04:59,380 --> 00:05:03,662
experience, a five-second clip
of people in our resuscitation
98
00:05:03,702 --> 00:05:04,022
rooms.
99
00:05:04,622 --> 00:05:08,784
And with that five-second clip,
without any other information,
100
00:05:09,645 --> 00:05:13,066
they had an accuracy of about
two-thirds to predict whether
101
00:05:13,066 --> 00:05:16,328
the patient got admitted to the
ICU, to the ward, or went home.
102
00:05:16,848 --> 00:05:19,269
And so I think first impressions
are important in our clinical
103
00:05:19,290 --> 00:05:21,611
world. I'm interested to hear
what you talked about in our
104
00:05:21,671 --> 00:05:22,251
educational world.
105
00:05:23,031 --> 00:05:25,732
LINDA SNELL: Thank you. Well, my
first impression story, I'm
106
00:05:25,752 --> 00:05:29,493
always reminded about it when I
started off as a resident
107
00:05:29,573 --> 00:05:32,414
running the ICU in a small
hospital. And one of the nurses
108
00:05:32,474 --> 00:05:38,455
came to me and said, you know,
you have about one nanosecond
109
00:05:38,575 --> 00:05:42,536
before we actually make up our
minds about you. And they said,
110
00:05:42,556 --> 00:05:45,557
so the nurses are making up
their minds about the residents
111
00:05:45,597 --> 00:05:47,298
who are rotating through their
ICU.
112
00:05:48,058 --> 00:05:52,359
And then she said, and we often
don't change our opinion on it.
113
00:05:52,683 --> 00:05:56,105
So you better make a good first
impression. Scared the hell out
114
00:05:56,105 --> 00:06:01,629
of me. All right. So first
impressions are important. What
115
00:06:01,829 --> 00:06:05,792
is a first impression? If we
take it from the article, first
116
00:06:05,852 --> 00:06:11,076
impressions are also called thin
slices or zero acquaintance
117
00:06:11,276 --> 00:06:11,856
judgments.
118
00:06:12,577 --> 00:06:14,819
And these are judgments about
people who we don't know.
119
00:06:15,339 --> 00:06:19,182
They're formed within seconds.
And they're obviously based on
120
00:06:19,222 --> 00:06:22,704
incomplete data. And they're
made. Automatically with little
121
00:06:22,964 --> 00:06:26,627
consciousness and little
awareness. Why is this
122
00:06:26,707 --> 00:06:30,830
important? I must say the
authors give a really excellent
123
00:06:30,890 --> 00:06:33,512
two-page summary of the
literature in this area, and I'm
124
00:06:33,512 --> 00:06:35,573
going to try and summarize it in
five points.
125
00:06:36,133 --> 00:06:41,037
First impressions are at times
predictive of subsequent
126
00:06:41,077 --> 00:06:44,000
judgments. For instance, in
OSCEs, multiple mini-interviews,
127
00:06:45,020 --> 00:06:50,487
or teacher evaluations. First
impressions may impact if there
128
00:06:50,507 --> 00:06:54,048
is a change in performance. So
in other words, if you think
129
00:06:54,108 --> 00:06:57,369
somebody's really good, you may
end up overrating them later on
130
00:06:57,709 --> 00:07:00,550
or vice versa if your first
impression is not so hot.
131
00:07:01,490 --> 00:07:05,691
But most of the studies on first
impressions have been done in
132
00:07:06,072 --> 00:07:09,873
medical education in exam
contexts and not in real life.
133
00:07:10,853 --> 00:07:15,414
Fourth, in competency-based
education, first impressions
134
00:07:15,674 --> 00:07:19,607
might influence entrustment
decisions. The initial
135
00:07:19,648 --> 00:07:24,250
entrustment is like a thin slice
or a presumptive entrustment, or
136
00:07:24,470 --> 00:07:27,952
what could be called a
non-acquaintance judgment.
137
00:07:28,913 --> 00:07:33,315
And finally, there is a thought
and some literature suggesting
138
00:07:33,375 --> 00:07:37,658
that the act of making a first
impression explicit, when you
139
00:07:37,658 --> 00:07:41,020
have to actually say what your
impression is, influences
140
00:07:41,140 --> 00:07:45,342
subsequent ratings compared to
situations where the rating is
141
00:07:45,362 --> 00:07:49,664
done implicitly. But again, most
of these have not been done in
142
00:07:49,644 --> 00:07:51,064
the real-life clinical setting.
143
00:07:51,824 --> 00:07:56,606
So our authors set out to first
determine the degree that raters
144
00:07:56,746 --> 00:07:59,466
are influenced by first
impressions when the learner
145
00:07:59,506 --> 00:08:04,328
performance changes in a more
naturalistic context. And
146
00:08:04,448 --> 00:08:08,629
second, to determine if making a
first impression judgment
147
00:08:09,009 --> 00:08:13,090
explicit affects subsequent
ratings compared to a more
148
00:08:13,150 --> 00:08:16,607
natural, which we would think
would be an implicit. Rating
149
00:08:16,667 --> 00:08:20,890
process. So what do you think?
Have I hooked you on Jason John
150
00:08:20,950 --> 00:08:21,170
Lara?
151
00:08:22,370 --> 00:08:24,411
JASON FRANK: I think this topic
is a fantastic topic, right?
152
00:08:24,431 --> 00:08:28,133
Because it is a important
component of all of our
153
00:08:28,173 --> 00:08:31,675
assessments, whether they're
structured and high stakes or
154
00:08:31,755 --> 00:08:35,437
they're semi-structured and
they're workplace based or just
155
00:08:35,477 --> 00:08:38,479
our informal assessments all day
long. So this is a really
156
00:08:38,519 --> 00:08:40,040
important topic about that first
impression.
157
00:08:40,380 --> 00:08:43,722
Is there something we need to
manage here or is it going to be
158
00:08:43,802 --> 00:08:46,900
okay? So you got me at that
premise. Premise of this paper
159
00:08:46,960 --> 00:08:48,981
is we're going to move away from
the OSCE setting. We're going to
160
00:08:48,981 --> 00:08:52,463
do something a little bit closer
to authenticity. And I'll have
161
00:08:52,503 --> 00:08:53,184
more about that later.
162
00:08:54,105 --> 00:08:55,666
JONATHAN SHERBINO: I think it's
an important topic. I really
163
00:08:55,686 --> 00:08:58,508
like the cognitive psychology of
so-called first impressions,
164
00:08:58,528 --> 00:09:02,951
which is, can I retrieve from
long-term memory an exemplar
165
00:09:03,031 --> 00:09:06,113
that maps? It's a pattern
recognition task. If the
166
00:09:06,593 --> 00:09:10,276
correlation is strong, then yes,
I have a diagnosis, clinical
167
00:09:10,296 --> 00:09:13,178
diagnosis. But presumably, it's
going to work similar in an
168
00:09:13,218 --> 00:09:14,359
educational diagnosis learner.
169
00:09:15,007 --> 00:09:17,488
And if there's not a strong
correlation between what I can
170
00:09:17,528 --> 00:09:21,609
retrieve, then I might need to
use a much more thoughtful,
171
00:09:21,829 --> 00:09:24,910
energy intensive, resource
intensive type of analysis and
172
00:09:24,910 --> 00:09:27,691
work through first principles to
understand what's happening with
173
00:09:27,671 --> 00:09:30,712
our learners. But it's never
either or. And so when we get
174
00:09:30,712 --> 00:09:33,813
into these either or
dichotomies, we get away from
175
00:09:33,833 --> 00:09:35,573
the messiness of a real world.
176
00:09:36,373 --> 00:09:38,354
Like Jason, I'm going to
foreshadow that this is not an
177
00:09:38,434 --> 00:09:43,316
authentic type of study. It'S...
One step removed from OSCE, but
178
00:09:43,396 --> 00:09:47,039
not into the realm of anything
that looks authentic. So I think
179
00:09:47,039 --> 00:09:49,841
there's a bit of sleight of hand
in how it gets set up, but we'll
180
00:09:49,901 --> 00:09:51,522
leave that for when you describe
the methods a bit more.
181
00:09:52,390 --> 00:09:54,611
LARA VARPIO: Some of these
authors are some of the people I
182
00:09:54,711 --> 00:09:58,152
cherish in my world. So when
people I adore write, I write, I
183
00:09:58,172 --> 00:10:03,013
read. It's going to happen. Tim
Wood and I have been friends and
184
00:10:03,053 --> 00:10:08,454
colleagues for many, many years.
When I worked back in Ottawa,
185
00:10:08,895 --> 00:10:12,996
same with Claire and Deb and
Sue. So when they're writing,
186
00:10:13,036 --> 00:10:15,416
I'm reading. Just going to
happen. So yeah, I'm in.
187
00:10:16,917 --> 00:10:19,757
LINDA SNELL: Great. So let's
move on to the methods. And I'm
188
00:10:19,777 --> 00:10:21,678
going to summarize the methods
for you.
189
00:10:22,954 --> 00:10:29,160
2, 2, 6, 3, 1. Okay. That's the
summary.
190
00:10:29,280 --> 00:10:31,762
JONATHAN SHERBINO: 42 is the
answer. That's the answer.
191
00:10:31,802 --> 00:10:33,883
LARA VARPIO: Dude, that's like
the first time I think I
192
00:10:33,883 --> 00:10:36,405
understood what was happening.
We agree. The answer, 42.
193
00:10:36,465 --> 00:10:37,046
Results. Of course.
194
00:10:37,166 --> 00:10:39,288
LINDA SNELL: Life, the universe,
and everything. Right?
195
00:10:39,608 --> 00:10:39,788
LARA VARPIO: Yeah.
196
00:10:40,108 --> 00:10:42,590
LINDA SNELL: Two Canadian
universities, University Of
197
00:10:42,610 --> 00:10:44,952
Ottawa and University Of
Alberta, so they've got East and
198
00:10:44,992 --> 00:10:49,696
West. Clinical supervisors from
internal medicine, emergency
199
00:10:49,776 --> 00:10:52,899
medicine, family medicine,
divided into two. Two groups,
200
00:10:53,779 --> 00:10:59,862
the implicit and the explicit
expression of first impressions,
201
00:11:01,223 --> 00:11:05,466
six videos of trainee patient
interactions of common clinical
202
00:11:05,526 --> 00:11:06,026
problems.
203
00:11:06,806 --> 00:11:10,608
And I must say, they spent
several paragraphs describing
204
00:11:12,710 --> 00:11:16,031
how they produced the videos.
And I really liked it. It was
205
00:11:16,091 --> 00:11:19,433
very well done. You could really
understand what they were doing
206
00:11:19,453 --> 00:11:20,534
and why they were doing it.
207
00:11:21,262 --> 00:11:23,323
JONATHAN SHERBINO: Could I just
jump in here and say what they
208
00:11:23,363 --> 00:11:27,746
do is they staged these cases.
These are not filmed cases of
209
00:11:27,786 --> 00:11:31,588
residents acting. So it's not
authentic. And that's what I was
210
00:11:31,928 --> 00:11:32,748
referring back to before.
211
00:11:33,569 --> 00:11:36,530
LINDA SNELL: So what it is, is
it's staged in that they have
212
00:11:36,630 --> 00:11:41,393
residents who are sort of given
a case and told to act in a
213
00:11:41,433 --> 00:11:46,696
certain way, weak, strong or
indifferent, basically. And the
214
00:11:46,776 --> 00:11:50,918
patient is actually acted by
another staff, a staff member.
215
00:11:52,266 --> 00:11:56,829
Back to the numbers, three
experimental conditions of
216
00:11:56,910 --> 00:12:01,012
trainee performance, weak to
strong, strong to weak, and
217
00:12:01,072 --> 00:12:05,035
consistent, the same in either
one. So the performance may
218
00:12:05,155 --> 00:12:08,998
start off as weak and then
improve during the video, or it
219
00:12:09,018 --> 00:12:11,760
might be the other way around,
or it might stay the same. And
220
00:12:11,760 --> 00:12:15,502
they used one rating instrument,
a modified mini CEX.
221
00:12:17,283 --> 00:12:22,194
And for the explicit supervisor
group. Remember, there's two
222
00:12:22,214 --> 00:12:25,357
groups of supervisors, explicit
and implicit. The explicit one,
223
00:12:25,817 --> 00:12:30,800
they stop the video at 60
seconds and say, please provide
224
00:12:30,820 --> 00:12:34,743
a first impression global
rating.
225
00:12:35,804 --> 00:12:38,666
And then they go on with the
video. And then for both groups,
226
00:12:38,766 --> 00:12:42,308
explicit and implicit, not only
do they complete the full mini
227
00:12:42,388 --> 00:12:46,431
CEX and all the parts of it, but
they have a final global rating.
228
00:12:46,991 --> 00:12:50,654
So really the only difference in
the two groups is...
229
00:12:52,042 --> 00:12:57,325
The explicit group has one extra
step, which is to provide what's
230
00:12:57,425 --> 00:13:00,547
called a first impression,
although it really isn't 60
231
00:13:00,627 --> 00:13:04,309
seconds. The analysis is done
with many correlations. I'm not
232
00:13:04,329 --> 00:13:08,431
going to go into the details of
it. There's an internal
233
00:13:08,491 --> 00:13:12,914
consistency of the six mini CEX
item ratings. There's the item
234
00:13:12,974 --> 00:13:14,935
ratings versus the global
ratings.
235
00:13:14,955 --> 00:13:18,417
There's first impression global
ratings versus the regular
236
00:13:18,457 --> 00:13:23,180
global rating at the end. And
there's mini-CEX IDA ratings for
237
00:13:23,220 --> 00:13:26,902
explicit and implicit. And
there's also between the three
238
00:13:26,962 --> 00:13:29,824
conditions, weak to strong,
strong to weak, and consistent.
239
00:13:30,265 --> 00:13:35,308
And they did calculate a sample
size as 25 per group. So
240
00:13:35,408 --> 00:13:39,271
comments on the methods,
anybody? We'll go Jason John
241
00:13:39,331 --> 00:13:39,591
Lara.
242
00:13:40,532 --> 00:13:42,273
JASON FRANK: Okay, this is where
I got to declare that I work
243
00:13:42,293 --> 00:13:45,495
with all these folks and I adore
them all. These are amazing
244
00:13:45,795 --> 00:13:48,997
medical educators that I get to
play with. I was not involved in
245
00:13:48,997 --> 00:13:52,724
this paper. So if I just with
fresh eyes reading this paper, I
246
00:13:52,764 --> 00:13:55,707
follow the logic. They're
basically saying, yeah, yeah,
247
00:13:55,727 --> 00:13:57,128
yeah. We did this with an OSCE
before.
248
00:13:57,228 --> 00:13:59,810
Now let's do this with some
videos that are kind of like
249
00:14:00,251 --> 00:14:04,294
what might happen on some wards.
Remember that there's a little
250
00:14:04,354 --> 00:14:07,917
caveat here that they had people
from multiple specialties and
251
00:14:07,897 --> 00:14:09,919
then they had they tried to pick
some acute what they called
252
00:14:09,959 --> 00:14:14,022
acute problems and said, hey,
all of these specialties should
253
00:14:14,022 --> 00:14:16,965
be able to rate how this
resident tackles this problem.
254
00:14:17,405 --> 00:14:20,868
That made me pause because. I'm
not sure I would look at a
255
00:14:20,888 --> 00:14:24,891
resident performance the same as
somebody in internal medicine,
256
00:14:25,191 --> 00:14:27,814
because I'm looking for
different things. I'm looking
257
00:14:27,814 --> 00:14:30,576
for, for example, making
diagnosis versus maybe
258
00:14:30,776 --> 00:14:33,559
definitive management. And I'm
wondering how that went with the
259
00:14:33,579 --> 00:14:36,821
study, because I didn't see
their checklist and their global
260
00:14:36,861 --> 00:14:37,842
rating and how it was worded.
261
00:14:38,463 --> 00:14:41,145
Nevertheless, the logic's pretty
straightforward. I understand
262
00:14:41,165 --> 00:14:44,308
what they did with two examples
of each case, make six, and then
263
00:14:44,308 --> 00:14:47,527
they had people rate. And then
that whole... Does your first
264
00:14:47,547 --> 00:14:49,543
impression making it explicit
make a difference from the
265
00:14:49,563 --> 00:14:52,461
literature? I thought that was
clever enough. Good questions.
266
00:14:53,422 --> 00:14:56,804
JONATHAN SHERBINO: I don't have
anything of serious critique to
267
00:14:56,824 --> 00:15:01,648
offer for the method section. I
think it's a nice, I was going
268
00:15:01,648 --> 00:15:04,130
to use the word elegant, but it
was a little bit hard to read
269
00:15:04,130 --> 00:15:07,313
with all the acronyms and the
explicit implicit. And the
270
00:15:07,653 --> 00:15:10,175
assumption or the research
question, I had to go back to a
271
00:15:10,195 --> 00:15:10,675
couple of times.
272
00:15:11,136 --> 00:15:14,358
But overall, it's kind of an
elegant little experiment in a
273
00:15:14,438 --> 00:15:16,760
simulated fashion. There's a
couple of things I just kind of
274
00:15:16,760 --> 00:15:19,042
scratched my head at is why did
they unmask their research
275
00:15:19,082 --> 00:15:21,765
question? They told everybody,
hey, we're trying to see if
276
00:15:21,785 --> 00:15:23,386
first impressions mess you up.
277
00:15:23,486 --> 00:15:27,610
So I wonder if they incepted
people to say, oh, go against
278
00:15:27,610 --> 00:15:29,712
what you really do or be
attentive to what's happening
279
00:15:29,732 --> 00:15:33,816
here. So I don't understand the
purpose of that. The second is
280
00:15:33,816 --> 00:15:36,839
they use the mini CEX, which is
really old technology. It's got
281
00:15:36,839 --> 00:15:40,382
to be like 25, 30 years old. And
apologies to John Norcini if he
282
00:15:40,402 --> 00:15:40,723
hears me.
283
00:15:41,975 --> 00:15:45,738
It uses a one to nine scale that
has unsatisfactory,
284
00:15:46,079 --> 00:15:49,982
satisfactory, and superior as
the anchors. And if you talk to
285
00:15:49,982 --> 00:15:53,005
anybody who knows about scales
and measurement, that's not what
286
00:15:53,045 --> 00:15:54,266
we think about as contemporary.
287
00:15:54,486 --> 00:15:57,649
And more importantly, their
version of the Mini CX has seven
288
00:15:57,729 --> 00:16:01,372
questions, which they actually
show have a high correlation one
289
00:16:01,372 --> 00:16:03,975
to another, which means there's
some massive halo or millstone
290
00:16:03,995 --> 00:16:08,418
effect, and you're just going on
the overall global rating
291
00:16:08,479 --> 00:16:11,364
anyway. And so they could have
got that cognitive burden out of
292
00:16:11,364 --> 00:16:13,444
the way and say, what's your
impression at the start?
293
00:16:13,544 --> 00:16:16,125
What's your impression at the
end? Yeah. Because all the other
294
00:16:16,265 --> 00:16:19,086
subdomains don't actually
differentiate at all, which
295
00:16:19,146 --> 00:16:20,686
tells you that you have
something wrong with your scale.
296
00:16:20,986 --> 00:16:22,187
But that's been the problem with
scales.
297
00:16:22,287 --> 00:16:25,268
That's been the problem with
Mini CX forever. They chose
298
00:16:25,288 --> 00:16:29,909
what's common, but maybe with a
little bit more of an eye, they
299
00:16:29,949 --> 00:16:31,970
might have thought about using a
different scale. It's a minor
300
00:16:32,030 --> 00:16:34,910
point, but I guess worthy of
just pointing out to our
301
00:16:34,950 --> 00:16:35,291
audience.
302
00:16:36,071 --> 00:16:36,751
JASON FRANK: Really good point,
John.
303
00:16:37,763 --> 00:16:39,504
LARA VARPIO: So I have very
little to add to that, Linda,
304
00:16:39,504 --> 00:16:42,807
because one of the things I do
when I read quantitative Papers
305
00:16:42,807 --> 00:16:45,189
and I'm trying to figure out how
I feel about the methods is I
306
00:16:45,209 --> 00:16:51,094
draw this picture. And the goal
for me, a good paper is a paper
307
00:16:51,134 --> 00:16:53,216
where I can draw this picture
without thinking. You know, I
308
00:16:53,256 --> 00:16:55,958
write down the research question
at the top. I write down the
309
00:16:56,618 --> 00:16:59,200
factors that they're considering
and where the intervention
310
00:16:59,240 --> 00:16:59,541
happens.
311
00:16:59,561 --> 00:17:02,743
It's like a little flow diagram.
And then I look at their
312
00:17:02,763 --> 00:17:05,425
analyses. I understand the
analyses on the abstract. And I
313
00:17:05,445 --> 00:17:08,723
see if that maps back to my
picture. And then. If it works,
314
00:17:08,723 --> 00:17:13,226
it works. And when I mapped the
picture for this study, there
315
00:17:13,246 --> 00:17:16,888
was nothing there that I thought
was incongruent. So from my
316
00:17:16,948 --> 00:17:21,030
perspective, I could draw this.
So yeah, works for me.
317
00:17:21,991 --> 00:17:25,933
LINDA SNELL: Good. Well, the
only thing I'll add to what John
318
00:17:26,073 --> 00:17:30,535
said is I agree with you, but
they didn't need to use the mini
319
00:17:30,635 --> 00:17:35,878
CEX, except that some of the
calculations were made after
320
00:17:35,918 --> 00:17:36,899
they'd done the mini CEX.
321
00:17:37,019 --> 00:17:38,960
When they found out that they
could just use the global
322
00:17:39,040 --> 00:17:42,243
rating. Now, is that new? No,
that's not new. People do know
323
00:17:42,303 --> 00:17:46,366
that. So let's go to the
results. And John, I have a
324
00:17:46,406 --> 00:17:51,389
question for you specifically.
The sample size was 25. True,
325
00:17:51,389 --> 00:17:51,629
false.
326
00:17:52,270 --> 00:17:53,831
JONATHAN SHERBINO: Oh, I'm
sorry, I didn't answer my
327
00:17:53,831 --> 00:17:54,331
question.
328
00:17:54,331 --> 00:17:55,972
LINDA SNELL: I just know what
the question's going to be. The
329
00:17:56,232 --> 00:18:02,517
sample size was to have 25 per
group. They came up with,
330
00:18:03,197 --> 00:18:07,845
instead of 50 raters, 45 raters.
Does it matter that they did not
331
00:18:08,025 --> 00:18:10,066
reach the sample size that they
wanted?
332
00:18:11,067 --> 00:18:12,869
JONATHAN SHERBINO: I guess it's
a philosophical question. And
333
00:18:12,869 --> 00:18:15,331
when you're talking about, you
know, less than 10% of the
334
00:18:15,371 --> 00:18:18,593
overall sample, I'm not sure if
that's critical, but you do a
335
00:18:18,613 --> 00:18:21,576
sample size to say, okay, can we
hit a threshold after which we
336
00:18:21,616 --> 00:18:25,239
will be able to determine some
educationally significant
337
00:18:25,299 --> 00:18:25,699
difference?
338
00:18:26,960 --> 00:18:29,122
That's not true. Over which
we'll be able to determine some
339
00:18:29,182 --> 00:18:31,984
statistical difference, the
difference which you need to
340
00:18:32,004 --> 00:18:34,366
subsequently adjudicate to say,
does that actually, do you care
341
00:18:34,366 --> 00:18:36,784
about it as an educator? And so
because they don't hit their
342
00:18:36,824 --> 00:18:40,046
sample size, there is that
theoretical risk of a type two
343
00:18:40,106 --> 00:18:42,187
error, which says we found no
difference.
344
00:18:42,628 --> 00:18:44,429
But maybe that's because we just
didn't have enough people to
345
00:18:44,449 --> 00:18:47,451
show the difference. So if they
said we need 50 people, we
346
00:18:47,471 --> 00:18:51,374
didn't get 50 people. We found
no statistical difference. Then
347
00:18:52,334 --> 00:18:54,916
in the back of your mind, you
could say, but if you recruited
348
00:18:55,496 --> 00:18:58,659
five more people, what would
have happened there? And there's
349
00:18:58,699 --> 00:19:01,020
something called a sensitivity
analysis where you can simply
350
00:19:01,621 --> 00:19:04,423
try to make the people that are
missing.
351
00:19:04,643 --> 00:19:08,766
And. Put them to the extreme of
the outcome and see what happens
352
00:19:08,806 --> 00:19:12,709
to the overall analysis. Does if
you make five people have some
353
00:19:13,470 --> 00:19:17,774
wildly terrible scoring, does
that unbalance everything? Was
354
00:19:17,774 --> 00:19:20,136
it kind of balanced? And now
suddenly you tip it off balance.
355
00:19:20,356 --> 00:19:23,138
They didn't do that type of
sensitivity analysis. It's okay.
356
00:19:23,619 --> 00:19:25,220
We're talking about 10%
difference.
357
00:19:25,901 --> 00:19:28,863
Eyeballing it, it feels all
right. But if you are a hardcore
358
00:19:28,923 --> 00:19:32,927
math nerd right now, you're
saying, ah, or whatever they
359
00:19:32,967 --> 00:19:38,190
say. 1 0 0 0 1 1 1 0 no no they
go ah euclid and then off they
360
00:19:38,190 --> 00:19:42,253
go well i was trying to do like
a like a computer programming
361
00:19:42,293 --> 00:19:47,977
joke which is even more lost on
the phd English major right like
362
00:19:48,037 --> 00:19:50,339
i think so foreign language gone
yeah.
363
00:20:01,716 --> 00:20:03,317
LINDA SNELL: Let's get to some
of the other results.
364
00:20:04,737 --> 00:20:09,739
There's a lot of data, and most
of the data reinforces prior
365
00:20:09,819 --> 00:20:14,822
studies, which is nice. But
what's really new here? In the
366
00:20:14,902 --> 00:20:18,423
explicit group, when the
performance was consistent, it
367
00:20:18,463 --> 00:20:21,761
started good, they ended good,
whatever. The first impression
368
00:20:21,801 --> 00:20:25,583
didn't make a difference. For
the other two conditions, good
369
00:20:25,583 --> 00:20:28,806
to bad or bad to good, the
global rating moved in the
370
00:20:28,906 --> 00:20:30,067
expected direction.
371
00:20:30,247 --> 00:20:33,489
In other words, it suggests that
the first impression didn't make
372
00:20:33,489 --> 00:20:37,492
a difference. The raters changed
judgment. And there were no
373
00:20:37,612 --> 00:20:41,234
significant differences
comparing implicit and explicit
374
00:20:41,294 --> 00:20:45,017
at the end of Mini-CX global
ratings. So the authors
375
00:20:45,097 --> 00:20:49,040
summarize this as physician
raters adjusted their judgment
376
00:20:49,160 --> 00:20:51,921
based on changes in the
learner's performance within a
377
00:20:52,041 --> 00:20:55,762
single patient encounter in a
workplace context.
378
00:20:56,422 --> 00:21:00,203
Now, I recognize, Jason, you're
going to say, this is still not
379
00:21:00,203 --> 00:21:04,144
a real workplace context, and I
agree with you on that. So, in a
380
00:21:04,524 --> 00:21:08,525
simulated workplace context. And
there was no difference in the
381
00:21:08,625 --> 00:21:11,706
global rating as a function of a
first impression being made
382
00:21:11,826 --> 00:21:14,187
explicitly or formed implicitly.
383
00:21:14,627 --> 00:21:19,069
So, those are the two relatively
new things. In this study. The
384
00:21:19,089 --> 00:21:22,250
rest of the stuff just sort of
reinforces things. However,
385
00:21:22,890 --> 00:21:26,692
there is one statement here that
I'm a little uncomfortable with,
386
00:21:26,832 --> 00:21:30,373
and I'll read it to you. It
says, it would require a
387
00:21:30,453 --> 00:21:33,434
considerable number of
participants to achieve
388
00:21:33,574 --> 00:21:36,516
statistical significance in
these comparisons.
389
00:21:37,256 --> 00:21:40,677
Therefore, we are comfortable
concluding that making a first
390
00:21:40,737 --> 00:21:44,979
impression explicit, which is an
artificial task, does not
391
00:21:45,119 --> 00:21:48,601
influence subsequent ratings.
Any more than what would occur
392
00:21:49,021 --> 00:21:52,402
when ratings form their first
impressions naturally, which
393
00:21:52,442 --> 00:21:57,204
would be implicit. This finding
supports the generalizability of
394
00:21:57,204 --> 00:21:57,885
the results.
395
00:22:00,086 --> 00:22:03,167
I don't know what they're trying
to work around there. Maybe one
396
00:22:03,167 --> 00:22:05,108
or two of you can comment on
that, John?
397
00:22:06,148 --> 00:22:08,729
JONATHAN SHERBINO: The
difference between them that
398
00:22:08,749 --> 00:22:11,870
doesn't achieve statistical
significance is so small that
399
00:22:11,910 --> 00:22:14,992
you would need to recruit
hundreds of people into both
400
00:22:15,032 --> 00:22:18,998
groups. And because we're
looking at 25 readers looking at
401
00:22:19,839 --> 00:22:24,882
six so-called residents, that's
far more than the feasibility or
402
00:22:24,922 --> 00:22:28,745
the real world operational
challenges that we have in our
403
00:22:28,765 --> 00:22:29,025
system.
404
00:22:29,085 --> 00:22:34,009
So we are never going to have
observation of practice exceed
405
00:22:34,289 --> 00:22:37,351
their totals. And so we probably
won't see some kind of change.
406
00:22:37,691 --> 00:22:40,513
But moreover, the question is,
right now, we don't ask people
407
00:22:40,513 --> 00:22:44,800
to say, form an explicit
judgment at 60 seconds. We
408
00:22:44,840 --> 00:22:48,022
probably want to see that having
that judgment doesn't actually
409
00:22:48,042 --> 00:22:49,142
affect downstream.
410
00:22:49,623 --> 00:22:52,945
So we don't want to introduce
this intervention for any
411
00:22:53,385 --> 00:22:56,807
practical reason into our
assessment programs. It's okay
412
00:22:57,207 --> 00:23:00,449
to have a snap judgment
unconsciously because the
413
00:23:00,449 --> 00:23:04,791
results show that things work
out okay anyway. The human brain
414
00:23:04,811 --> 00:23:07,793
is more complex than what we
give it credit for sometimes and
415
00:23:08,253 --> 00:23:09,093
we adjust.
416
00:23:09,373 --> 00:23:13,516
And that's what every analysis
in this study shows. We anchor
417
00:23:13,516 --> 00:23:16,522
and adjust. And that's how
humans get through this complex
418
00:23:16,582 --> 00:23:21,086
world. So yeah, I agree with her
statement for two reasons. We'd
419
00:23:21,126 --> 00:23:24,609
never changed the
operationalizing of our system
420
00:23:24,689 --> 00:23:27,491
either for the number of raters.
And we would never ask people to
421
00:23:27,511 --> 00:23:29,893
say, hey, explicitly give your
gestalt up front.
422
00:23:30,854 --> 00:23:34,457
LINDA SNELL: I guess the thing
that concerned me was not
423
00:23:34,577 --> 00:23:36,879
disagreeing with what you said,
John, but that sentence, this
424
00:23:36,959 --> 00:23:39,841
finding supports the
generalizability of the results.
425
00:23:40,341 --> 00:23:44,020
It just makes me a little
skeptical about this. The
426
00:23:44,080 --> 00:23:48,242
discussion I thought was really
interesting. However, it
427
00:23:48,282 --> 00:23:52,484
describes the two findings and
then asks four questions that
428
00:23:52,484 --> 00:23:56,205
the authors say they could not
explore. It doesn't really talk
429
00:23:56,245 --> 00:23:57,186
about their results.
430
00:23:57,226 --> 00:24:00,707
It talks about what their
results didn't do or couldn't
431
00:24:00,767 --> 00:24:05,369
do, related to rate or
variability, magnitude of
432
00:24:05,409 --> 00:24:08,031
change, differs with the
valence, self-fulfilling
433
00:24:08,091 --> 00:24:11,632
prophecies, and what happens
with multiple interactions. All
434
00:24:11,632 --> 00:24:13,694
of which are really good
questions, but then they say,
435
00:24:13,914 --> 00:24:16,856
here, these are some really good
questions, but we can't answer
436
00:24:16,896 --> 00:24:17,016
them.
437
00:24:17,716 --> 00:24:21,899
And the author's conclusions are
that we can apply these findings
438
00:24:22,340 --> 00:24:26,322
to, number one, entrustment
decisions, because they may be
439
00:24:26,442 --> 00:24:31,066
influenced by context and may
lead to first impressions being
440
00:24:31,186 --> 00:24:33,467
used because you're entrusting
when you don't have all that
441
00:24:33,507 --> 00:24:36,730
much time. And two, and this is
maybe a little bit more
442
00:24:36,790 --> 00:24:41,194
important, learner handover. If
you are, in fact, given an
443
00:24:41,234 --> 00:24:46,444
impression of a learner, it may
act as a first impression.
444
00:24:47,225 --> 00:24:51,034
And that's a whole area that
needs exploring. In terms of
445
00:24:51,094 --> 00:24:55,135
conclusions, it replicates the
past research on the influence
446
00:24:55,135 --> 00:24:58,896
of first impressions, but it
extends this research, according
447
00:24:58,936 --> 00:25:03,198
to the authors, to a
workplace-based assessment tool
448
00:25:03,398 --> 00:25:07,019
and setting. And I think we've
already discussed that this is a
449
00:25:07,199 --> 00:25:10,880
relatively old-fashioned tool,
and it's not really a workplace
450
00:25:11,200 --> 00:25:16,341
setting. So, thoughts on the
results, Lara, Jason, John?
451
00:25:17,122 --> 00:25:19,462
LARA VARPIO: I found the results
really quite reassuring. I find
452
00:25:19,462 --> 00:25:22,083
it reassuring that if we have a
first impression, we can change
453
00:25:22,083 --> 00:25:24,964
our first impression. To me,
that's nothing but good news.
454
00:25:25,364 --> 00:25:27,324
There's one piece of this,
though, that I find, you know,
455
00:25:27,424 --> 00:25:29,585
they talk, the four questions
they leave us with saying they
456
00:25:29,585 --> 00:25:30,625
couldn't find them for great.
457
00:25:30,645 --> 00:25:34,346
You know what? Every good, if
honest to God, if you're a real
458
00:25:34,406 --> 00:25:38,267
researcher in my heart, the more
research you do, the more humble
459
00:25:38,307 --> 00:25:41,828
you should be because the depth
of things you don't know just
460
00:25:41,848 --> 00:25:44,829
becomes more clear. So the fact
that they left with four good
461
00:25:44,869 --> 00:25:47,590
questions to me is just like,
yes, total respect.
462
00:25:47,730 --> 00:25:50,712
That's how I feel about research
too. But there's, I want to add
463
00:25:50,752 --> 00:25:52,794
another piece to that, to their
list. I want to put a number
464
00:25:52,854 --> 00:25:56,937
five or six. I've lost track
now. Because for me, what I find
465
00:25:57,937 --> 00:26:01,340
is, and I think they do note it
as a limitation, is that they,
466
00:26:01,560 --> 00:26:03,982
you know, we know our
assessments can be impacted by
467
00:26:04,102 --> 00:26:04,922
implicit bias.
468
00:26:05,743 --> 00:26:09,926
So we know that depending on the
way the person presents, on all
469
00:26:09,986 --> 00:26:12,668
kinds of factors and
considerations that individuals
470
00:26:12,748 --> 00:26:15,986
are under no control of sort of
in their own ways. That we can
471
00:26:16,006 --> 00:26:20,250
be biased against individuals.
And what I want to know is, can
472
00:26:20,330 --> 00:26:23,412
those first impression biases
change?
473
00:26:25,754 --> 00:26:31,359
Can we make a difference at that
moment of first impressions? Can
474
00:26:31,379 --> 00:26:36,463
we do something about the bias
of assessment? That's something
475
00:26:36,683 --> 00:26:39,966
I would love to know more about.
But again, all good research
476
00:26:40,086 --> 00:26:41,787
leaves you asking more
questions. So that's a question
477
00:26:41,827 --> 00:26:43,429
I would throw down as a possible
next step.
478
00:26:43,889 --> 00:26:47,592
JONATHAN SHERBINO: Do you mean
social bias? Yep. Okay. Yeah. A
479
00:26:47,632 --> 00:26:49,894
very, from a cognitive
psychology point of view, a very
480
00:26:49,994 --> 00:26:53,036
different thing, but equally
important, but different kinds
481
00:26:53,036 --> 00:26:53,497
of bias, right?
482
00:26:53,817 --> 00:26:55,639
LARA VARPIO: Yeah, totally.
Yeah, exactly. Different kind of
483
00:26:55,639 --> 00:26:58,281
bias, but it's still there,
right? So can we do something
484
00:26:58,301 --> 00:27:00,803
about that? Can we, like, are we
able to do it? Or is this
485
00:27:00,843 --> 00:27:03,605
something that is going to need
real structural change? I don't
486
00:27:03,605 --> 00:27:03,725
know.
487
00:27:04,786 --> 00:27:07,488
JASON FRANK: These results are
super reassuring. So they should
488
00:27:07,508 --> 00:27:09,930
make all of us in medical
education and health
489
00:27:09,930 --> 00:27:13,053
professionals education around
the world feel good that at
490
00:27:13,093 --> 00:27:14,234
least in these simulated.
491
00:27:14,606 --> 00:27:17,348
Control conditions with these
people who we don't know a lot
492
00:27:17,368 --> 00:27:19,509
about because there's not a lot
of description in the paper
493
00:27:19,889 --> 00:27:23,871
about who participated that they
were able to change opinions
494
00:27:23,931 --> 00:27:28,614
based on what they observe hey
that's great because we're
495
00:27:28,614 --> 00:27:32,676
talking a lot about implicit
bias these days that that people
496
00:27:33,516 --> 00:27:36,878
have have some things hardwired
into their brain from past
497
00:27:36,938 --> 00:27:40,860
experiences so this says
thoughtful preceptors can change
498
00:27:40,880 --> 00:27:45,114
their mind based on what they
see that is super helpful We
499
00:27:45,114 --> 00:27:46,555
should emphasize this is
simulated.
500
00:27:47,056 --> 00:27:51,599
I'm not sure how generalizable
it is to other settings. I'm not
501
00:27:51,659 --> 00:27:55,882
sure that this really informs
entrustment decisions, which is
502
00:27:56,182 --> 00:27:59,364
a different assessment scenario.
I'm not sure it informs things
503
00:27:59,444 --> 00:28:03,987
beyond a Mini CX and these
problems. But nevertheless, it
504
00:28:04,448 --> 00:28:07,350
does help me sleep at night when
I think about assessment. And I
505
00:28:07,330 --> 00:28:07,450
do.
506
00:28:09,031 --> 00:28:13,154
JONATHAN SHERBINO: I like the
results. It confirms a whole...
507
00:28:14,006 --> 00:28:16,648
In another setting where the
literature has gone, and so
508
00:28:16,728 --> 00:28:18,649
shout out to John Brush, who's
at the University Of Virginia.
509
00:28:18,709 --> 00:28:21,411
He was the first author on a
study that we did that got
510
00:28:21,451 --> 00:28:24,113
published in JAMA that looked at
this phenomenon. Can you anchor
511
00:28:24,133 --> 00:28:27,496
on a place and then adjust to
it? So this is a phenomenon that
512
00:28:27,556 --> 00:28:28,356
happens all the time.
513
00:28:29,537 --> 00:28:32,860
Interestingly, if there's
nothing in the data, the authors
514
00:28:32,920 --> 00:28:37,703
point to it. They don't
standardize for how strong to
515
00:28:37,763 --> 00:28:40,785
weak versus how weak to strong
their cases are. But it seems if
516
00:28:40,765 --> 00:28:44,303
you start out good. You go down
less than if you start up poor,
517
00:28:44,364 --> 00:28:47,366
you don't go up as high. So
where you start from has an
518
00:28:47,406 --> 00:28:49,507
impact about what's the, where
you end up.
519
00:28:50,168 --> 00:28:54,230
And maybe we have, and certainly
from a biological point of view,
520
00:28:54,771 --> 00:28:58,854
we have this scarcity mindset.
We are wired for fear. We always
521
00:28:58,854 --> 00:29:00,815
are looking for the negative. So
if we see something negative,
522
00:29:01,135 --> 00:29:04,838
it's hard to us to adjust from
that initial anchoring. Their
523
00:29:05,018 --> 00:29:08,780
data shows that trend. It
doesn't actually support from an
524
00:29:08,900 --> 00:29:11,022
experimental model, that trend,
but I think that's one little.
525
00:29:11,242 --> 00:29:12,002
Interesting piece here.
526
00:29:12,643 --> 00:29:14,724
JASON FRANK: So don't be late
for a shift. There you go.
527
00:29:15,504 --> 00:29:18,846
LINDA SNELL: Right. Yeah,
exactly. Or for a morning report
528
00:29:18,906 --> 00:29:22,668
in my setting. One of you
mentioned, we don't know a lot
529
00:29:22,708 --> 00:29:26,750
about the raters. It's true. The
authors didn't collect
530
00:29:26,790 --> 00:29:29,771
demographic data. One of my
paperclips would be, it really
531
00:29:29,831 --> 00:29:33,353
would be useful to know if there
was a difference in any of the
532
00:29:33,433 --> 00:29:37,651
usual variables. If somebody
who'd been... A supervisor for
533
00:29:37,711 --> 00:29:41,494
20 years was different than
somebody who supervised for a
534
00:29:41,534 --> 00:29:42,936
year or two, for instance.
535
00:29:43,396 --> 00:29:47,099
And collecting demographic data
is easy. They're doing a quant
536
00:29:47,199 --> 00:29:51,381
study. I would have... Like to
have seen that. I also think it
537
00:29:51,381 --> 00:29:54,441
would have been interesting in a
future study, back to what Lara
538
00:29:54,541 --> 00:29:58,383
says about there's always more
questions, to do some sort of a
539
00:29:58,423 --> 00:30:03,404
mixed method study to answer
some of our why questions, which
540
00:30:03,464 --> 00:30:06,265
came up as part of this. But
that's just future research.
541
00:30:12,666 --> 00:30:17,328
Let's go to our rating. First of
all, on the methods section,
542
00:30:17,628 --> 00:30:20,810
we'll start with the most
quantitative person here, Lara,
543
00:30:20,870 --> 00:30:24,052
Jason, John.
544
00:30:24,052 --> 00:30:26,334
LARA VARPIO: Yeah, the most
quantitative person here. I
545
00:30:26,354 --> 00:30:27,235
could draw my picture.
546
00:30:27,775 --> 00:30:29,777
JONATHAN SHERBINO: Did you say
constipated?
547
00:30:30,858 --> 00:30:36,402
LARA VARPIO: I did not say
constipated. Well, you know,
548
00:30:38,424 --> 00:30:39,485
it's been a bang up day.
549
00:30:42,067 --> 00:30:46,931
I am hilarious. Honest to God.
Anyway, I could draw my picture
550
00:30:46,931 --> 00:30:49,572
for this paper. I could figure
out the variables. It was all
551
00:30:49,592 --> 00:30:50,313
those sorts of things.
552
00:30:51,533 --> 00:30:53,774
What do you mean, what? You
didn't get my joke? You didn't
553
00:30:53,794 --> 00:30:55,394
get it? I'll draw you a picture
later.
554
00:30:55,974 --> 00:30:59,095
LINDA SNELL: That's agenda
number three.
555
00:30:59,855 --> 00:31:01,696
LARA VARPIO: On the car ride
home. We have a lot of ground to
556
00:31:01,716 --> 00:31:05,437
cover, Jason. Just saying. But
they didn't tell me their
557
00:31:05,457 --> 00:31:08,017
paradigmatic orientation, which
of course is my pet peeve. So
558
00:31:08,117 --> 00:31:09,298
I'm going to give it a four.
559
00:31:10,378 --> 00:31:12,199
JONATHAN SHERBINO: This is, for
those of you playing at home.
560
00:31:12,199 --> 00:31:13,639
JASON FRANK: Straightforward.
Why are you laughing?
561
00:31:13,679 --> 00:31:15,019
JONATHAN SHERBINO: Jason has
missed where he's supposed to
562
00:31:15,059 --> 00:31:15,860
talk in this episode.
563
00:31:15,860 --> 00:31:17,380
LARA VARPIO: Almost every time.
564
00:31:17,620 --> 00:31:20,802
JASON FRANK: Yeah i'm sorry it's
only when i follow lava because
565
00:31:20,802 --> 00:31:23,843
she's distracted i don't know
the bingo square like i don't
566
00:31:23,843 --> 00:31:27,125
know the i'll try to play bingo
throw me under the bus for your
567
00:31:27,245 --> 00:31:32,188
issue but whatever i'm stuck i'm
constipated like that's awesome
568
00:31:32,228 --> 00:31:34,950
term like if you have too many
experimental Papers in a row are
569
00:31:34,950 --> 00:31:39,872
you constipated i think you are
honestly there you go anyway
570
00:31:40,293 --> 00:31:44,075
this paper is i like how it
walks you through a logic i love
571
00:31:44,095 --> 00:31:48,661
the result it's so reassuring as
we said And I'm not sure that
572
00:31:48,801 --> 00:31:52,604
it's as generalizable to real
life as the authors have put
573
00:31:52,644 --> 00:31:55,786
forward, but it's one step
closer to authenticity in a
574
00:31:55,806 --> 00:31:57,067
workplace-based assessment.
575
00:31:57,467 --> 00:31:58,728
So I'm going to give it a four.
576
00:31:59,789 --> 00:32:03,031
LINDA SNELL: And there, I must
say you mixed up methods and
577
00:32:03,071 --> 00:32:04,172
outcomes, but that's okay.
578
00:32:04,533 --> 00:32:04,913
JASON FRANK: It's true.
579
00:32:05,633 --> 00:32:05,893
LINDA SNELL: John.
580
00:32:06,794 --> 00:32:08,115
JONATHAN SHERBINO: You've heard
my critiques. I'm going to give
581
00:32:08,115 --> 00:32:08,976
it a four for methods.
582
00:32:10,096 --> 00:32:13,679
LINDA SNELL: So we're all into
four. Me too. I thought it was
583
00:32:13,919 --> 00:32:14,880
very well explained.
584
00:32:20,720 --> 00:32:26,362
Now, how about impact? We'll go
the reverse. Can you remember
585
00:32:26,423 --> 00:32:27,783
which is the reverse, Jason?
586
00:32:27,823 --> 00:32:29,644
LARA VARPIO: Why do this to us?
Why, Liz? Easy.
587
00:32:30,424 --> 00:32:31,645
JONATHAN SHERBINO: You go last,
Marlis.
588
00:32:31,725 --> 00:32:34,086
LARA VARPIO: I know when I'm
going, but Jason doesn't. Go on,
589
00:32:34,086 --> 00:32:34,166
John.
590
00:32:34,166 --> 00:32:35,426
JONATHAN SHERBINO: It doesn't
matter. He's still being
591
00:32:35,466 --> 00:32:38,228
confused. Big silent pause, Jay.
That's you.
592
00:32:39,468 --> 00:32:44,390
I'm going to give this paper a
three. It builds on the
593
00:32:44,410 --> 00:32:50,653
literature of how readers...
Process assessments of trainees.
594
00:32:51,533 --> 00:32:54,835
It has an outcome that's
reassuring to the design. The
595
00:32:55,255 --> 00:32:57,476
experimentalists and the
educational psychologists and
596
00:32:57,476 --> 00:32:59,577
the cognitive psychologists will
be very pleased of this data.
597
00:33:00,237 --> 00:33:02,798
If you are a program director,
you'd be very pleased that you
598
00:33:02,818 --> 00:33:05,099
don't need to do any kind of
faculty development to say, hey,
599
00:33:05,479 --> 00:33:08,840
keep an open mind because in
fact, humans can keep an open
600
00:33:08,880 --> 00:33:10,961
mind. It's how we get through
the world. So a three for me.
601
00:33:11,962 --> 00:33:13,042
LINDA SNELL: Long pause.
602
00:33:15,063 --> 00:33:17,784
JASON FRANK: That's not nice.
That's using a previous bias.
603
00:33:18,996 --> 00:33:23,439
Oh this is a our first
impression i should say this is
604
00:33:23,439 --> 00:33:27,341
a three because i'll use it when
we're if i'm talking to anybody
605
00:33:27,421 --> 00:33:31,324
about concerns about first
impressions so this is a paper i
606
00:33:31,324 --> 00:33:34,866
would pull not johnson i'm just
kidding this isn't another paper
607
00:33:34,886 --> 00:33:38,408
i would pull i'm not sure it
changes my management with
608
00:33:38,448 --> 00:33:41,790
respect to design of
programmatic assessment using
609
00:33:42,330 --> 00:33:45,092
authentic assessment i'm not
sure yet it's a three you.
610
00:33:46,268 --> 00:33:47,749
LARA VARPIO: As someone who
doesn't play in the assessment
611
00:33:47,849 --> 00:33:50,231
space very often and those sorts
of things, this paper is going
612
00:33:50,231 --> 00:33:53,254
to be a solid three because it
gives me exactly the argument I
613
00:33:53,274 --> 00:33:56,897
need. It's perfect for helping
me to reassure people about
614
00:33:56,997 --> 00:33:59,759
immediate biases, first
impressions. So yeah, no, three.
615
00:34:00,500 --> 00:34:03,342
LINDA SNELL: Well, here we are.
Our concordance was very good on
616
00:34:03,382 --> 00:34:07,065
methods and now it's very good
on impact. I too will give it a
617
00:34:07,105 --> 00:34:11,989
three, reassuring, relevant, a
little bit of new information
618
00:34:12,510 --> 00:34:14,131
and nicely put together.
619
00:34:14,812 --> 00:34:15,232
JONATHAN SHERBINO: Could I
just...
620
00:34:15,472 --> 00:34:18,413
LARA VARPIO: Jump in for one
second and say both times lava
621
00:34:18,793 --> 00:34:24,134
gave us whole numbers i know you
know why i love this so much
622
00:34:24,174 --> 00:34:27,835
more i want to give it whoops it
was my cause it changed
623
00:34:27,915 --> 00:34:30,436
everybody's cognition when you
do the pause like that i got all
624
00:34:30,456 --> 00:34:32,717
this i've got a whole list of
things i got to keep track of to
625
00:34:32,737 --> 00:34:35,738
talk about with Jason later so i
like gotta keep all that in my
626
00:34:35,758 --> 00:34:40,319
brain i'm feeling constipated
all right okay.
627
00:34:40,599 --> 00:34:44,500
LINDA SNELL: For those of you
out there in Papers Podcast land
628
00:34:45,389 --> 00:34:48,773
You can stop listening to this
nonsense and write and tell us
629
00:34:48,853 --> 00:34:52,958
what you think. You can write to
us at thepeeperspodcast at
630
00:34:52,978 --> 00:34:54,040
gmail.com.
631
00:34:54,552 --> 00:34:59,895
Or in our now repaired message
link in the website at
632
00:35:00,095 --> 00:35:04,738
paperspodcast.com. It's been
great chatting with you three
633
00:35:05,178 --> 00:35:08,480
and recognizing that you now
recognize when I choose a good
634
00:35:08,520 --> 00:35:13,042
paper because you seem to agree
with me exactly, both implicitly
635
00:35:13,362 --> 00:35:14,583
and explicitly.
636
00:35:15,223 --> 00:35:15,403
JASON FRANK: Boo.
637
00:35:17,064 --> 00:35:19,846
LINDA SNELL: Boo? Is that what
you're saying instead of take
638
00:35:19,886 --> 00:35:20,726
care, everybody, now?
639
00:35:21,627 --> 00:35:23,087
LARA VARPIO: No, you got to say
it like Jason says it.
640
00:35:23,147 --> 00:35:23,808
LINDA SNELL: Take care,
everybody.
641
00:35:24,676 --> 00:35:25,016
JASON FRANK: Exactly.
642
00:35:25,996 --> 00:35:27,657
JONATHAN SHERBINO: That's a
great impersonation. Thanks for
643
00:35:27,657 --> 00:35:27,917
listening.
644
00:35:30,318 --> 00:35:31,298
LARA VARPIO: You say talk to you
later.
645
00:35:31,398 --> 00:35:32,578
LINDA SNELL: Then Jason, see how
it feels.
646
00:35:33,919 --> 00:35:34,659
JASON FRANK: Talk to you later.
647
00:35:35,399 --> 00:35:39,800
LINDA SNELL: It's not the same.
Okay. So bye-bye. Now it's your
648
00:35:39,860 --> 00:35:41,521
turn. Talk to you later.
649
00:35:41,661 --> 00:35:41,901
JASON FRANK: No, no.
650
00:35:41,901 --> 00:35:42,881
JONATHAN SHERBINO: Thanks for
listening.
651
00:35:44,162 --> 00:35:45,442
Take care, everybody.
652
00:35:52,668 --> 00:35:53,968
JASON FRANK: Take care, little
buddy.
653
00:35:54,529 --> 00:35:56,129
LARA VARPIO: I did not say
constipated, did I?
654
00:35:56,709 --> 00:35:57,349
JASON FRANK: Yes, you did.
655
00:35:57,850 --> 00:36:00,270
LINDA SNELL: Yes, you did.
Everybody, stop listening. This
656
00:36:00,290 --> 00:36:02,611
is deteriorating.
657
00:36:02,611 --> 00:36:04,552
LARA VARPIO: It went downhill
early, so anyway.
658
00:36:04,872 --> 00:36:06,472
JASON FRANK: You've been
listening to the Papers Podcast.
659
00:36:06,852 --> 00:36:10,354
We hope we made you just
slightly smarter. The podcast is
660
00:36:10,334 --> 00:36:12,694
a production of the Unit For
Teaching And Learning at the
661
00:36:12,734 --> 00:36:16,335
Karolinska Institute. The
executive producer today was my
662
00:36:16,395 --> 00:36:17,396
friend Teresa Sorrow.
663
00:36:18,504 --> 00:36:22,109
The technical producer today was
Samuel Lundberg. You can learn
664
00:36:22,169 --> 00:36:25,233
more about the Papers Podcast
and contact us at
665
00:36:25,474 --> 00:36:29,479
www.thepaperspodcast.com. Thank
you for listening, everybody,
666
00:36:29,760 --> 00:36:32,003
and thank you for all you do.
Take care.
667
00:36:39,676 --> 00:36:41,479
JONATHAN SHERBINO: Welcome back
to the Papers Podcast. I think
668
00:36:41,479 --> 00:36:43,683
we should go to kind of an
abbreviated version. I was
669
00:36:43,683 --> 00:36:45,826
thinking the PP cast, but I...
670
00:36:46,607 --> 00:36:47,249
JASON FRANK: Let us die.
671
00:36:47,289 --> 00:36:47,809
JONATHAN SHERBINO: Let us die.
672
00:36:48,230 --> 00:36:49,532
JASON FRANK: We could not do
that, John.
673
00:36:50,213 --> 00:36:50,874
JONATHAN SHERBINO: No, okay, all
right.