1
00:00:15,750 --> 00:00:20,010
So today I'm going to start with a little bit of history here.
2
00:00:20,330 --> 00:00:33,640
the University of Iowa, which is where I and my next guest did our graduate studies, and that's where we met, they're a global leader in measurement, anchored by the Iowa testing programs, and even the College of Education is named after E.F.
3
00:00:33,640 --> 00:00:37,640
Lindquist, who has a very deep history in measurement.
4
00:00:38,59.999 --> 00:00:55,639.998
he started the Iowa Testing Programs, which many listeners maybe have heard of, contributed to helping to establish the American College Testing program, or ACT, which is one of the largest college entry testing programs in the nat- in the United States.
5
00:00:56,250 --> 00:01:01,279.998
And more than 70 years later, my next guest, Liz and I, Dr.
6
00:01:01,279.999 --> 00:01:10,430
Hollingworth, went, met in graduate school in a statistics class and took so many, so many statistics, measurement.
7
00:01:10,630 --> 00:01:20,340.003
I've never read so much about, ever knew that there was so much to read about validity in my life until we were in graduate school, and then commiserated over coffees and lunches, et cetera.
8
00:01:21,10.001 --> 00:01:24,279.998
So I'm so excited to introduce Dr.
9
00:01:24,279.998 --> 00:01:32,639.999
Liz Hollingworth, professor and director of the Center for Evaluation and Assessment, which is in the College of Education at the University of Iowa.
10
00:01:33,179.999 --> 00:01:36,800
You are listening to Measure What Matters by Elevate What Matters.
11
00:01:37,889.998 --> 00:01:44,879.999
Here's where we pull back the curtain on how funders, evaluators, and communities measure what works and what doesn't.
12
00:01:45,190.001 --> 00:01:47,149.999
I'm Melissa, the host of your podcast.
13
00:01:48,339.995 --> 00:01:51,270
So Liz, we met in graduate school.
14
00:01:51,330.001 --> 00:01:52,169.995
We- Hi, Melissa.
15
00:01:52,169.995 --> 00:01:52,829.995
Hi.
16
00:01:52,849.997 --> 00:01:54,851.42557143
It's so good to meet you again.
17
00:01:54,851.42557143 --> 00:01:56,280
It's lovely to see you.
18
00:01:56,859.999 --> 00:02:01,729.999
I think our first class together was, Intermediate statistics fall of, yep.
19
00:02:02,119.999 --> 00:02:04,209.999
And it was in the first year of graduate school.
20
00:02:04,209.999 --> 00:02:06,179.999
So we were both in a PhD program.
21
00:02:06,599.999 --> 00:02:11,89.999
I was studying educational psychology, and you were in a doctoral program for ed policy and leadership.
22
00:02:11,89.999 --> 00:02:11,799.998
Is that right? Yeah.
23
00:02:12,529.999 --> 00:02:18,179.998
And then we both, because of the storied history at, in the College of Ed, took a lot of measurements.
24
00:02:18,179.999 --> 00:02:27,339.998
So I was gonna start by hearing from you on what are some highlights to your journey of when we met through school to becoming a professor.
25
00:02:27,640 --> 00:02:35,409.998
I will let you talk about all the things that you do and the many jobs that you do, including director of a university-based evaluation center.
26
00:02:36,329.998 --> 00:02:36,879.998
Okay.
27
00:02:36,939.997 --> 00:02:45,289.998
after I gradu- during, graduate school, I had an assistantship with Iowa testing programs, and when I graduated, I got a postdoc with them.
28
00:02:45,770 --> 00:02:56,880
And, I helped write test items, I helped, expand the Iowa Test of Basic Skills to, US Virgin Islands, Ohio, Arkansas.
29
00:02:57,359.996 --> 00:02:59,249.993
It was during No Child Left Behind.
30
00:02:59,329.994 --> 00:03:09,549.987
And every school district in every state needed to have some sort of test for third grade, fifth grade, eighth grade, and I helped write those tests.
31
00:03:09,859.992 --> 00:03:10,789.987
It was really fun.
32
00:03:10,789.992 --> 00:03:12,319.992
It was super interesting.
33
00:03:12,739.992 --> 00:03:22,239.989
when I finished my postdoc, I got a job, at the University of Iowa, which is really unusual, to get a job at the place where you graduated from.
34
00:03:22,469.992 --> 00:03:35,9.986
And I am in the educational leadership program, and part of the reason why that was such a good fit was, again, because No Child Left Behind was really changing how people were being trained to lead schools.
35
00:03:35,109.991 --> 00:03:51,589.987
So I created a bunch of classes on standards-based education, on, social advocacy for students who had special needs, because all of that testing would shed light on kids who needed extra help.
36
00:03:51,839.991 --> 00:03:58,429.981
So I worked to train people who were gonna be principals and superintendents how to do measurement.
37
00:03:59,619.99 --> 00:04:08,679.99
And, And I have to interject to say, just because listeners might not know this about you, you were a teacher before you went to graduate school for your, PhD.
38
00:04:08,999.99 --> 00:04:09,439.99
I was.
39
00:04:09,439.99 --> 00:04:17,79.989
And so I think- I taught- Yeah I taught in, I taught third grade in Chicago public schools, and then I taught sixth and eighth grade, not at the same time.
40
00:04:17,249.989 --> 00:04:21,399.988
First I taught sixth grade in Michigan, and then, I got moved to eighth grade.
41
00:04:21,419.987 --> 00:04:22,809.987
So I had the same kids.
42
00:04:22,829.987 --> 00:04:23,959.987
It was really fun.
43
00:04:24,179.988 --> 00:04:28,839.988
So I had the same kids in sixth grade, and then two years later I joined up with them again in eighth grade.
44
00:04:29,309.988 --> 00:04:30,689.986
so I taught there for five years.
45
00:04:30,739.988 --> 00:04:34,159.987
And, yeah, that was my K12 teaching background.
46
00:04:34,159.989 --> 00:04:39,39.992
I was also the administrator for gifted and talented education- in that district.
47
00:04:39,879.994 --> 00:05:02,509.992
so- And sorry to interrupt, I had wanted to mention that because you're talking about the, these things that you were doing in your jobs after graduate school, and it's the accumulation of the experience and training that you had had at that point that put you at that intersection at such a interesting time that we were both in a program like the one that, the ones we were at at the University of Iowa during No Child Left Behind.
48
00:05:02,519.99 --> 00:05:07,539.992
And we're gonna get into policies and how that affects the work that we do at federal, state, or local levels.
49
00:05:07,939.988 --> 00:05:08,419.995
You can continue.
50
00:05:08,579.995 --> 00:05:10,9.994
Yeah, it was such a weird time.
51
00:05:10,309.994 --> 00:05:10,519.99
It was such a, yeah.
52
00:05:11,149.994 --> 00:05:11,289.994
Yep.
53
00:05:11,329.994 --> 00:05:22,409.989
And then, yeah, and then I started doing program evaluation work with Don Yarbrough ugh who helped write the program evaluation standards and was directing the Center for Evaluation and Assessment.
54
00:05:22,809.99 --> 00:05:25,489.991
And he came to me and he said well, I think it was the other way around.
55
00:05:25,509.988 --> 00:05:30,179.991
I came to him and I said, "I'm being asked to do some program evaluation work.
56
00:05:30,769.99 --> 00:05:39,239.988
How, how can we collaborate on this?" And he said, "Well, if you run it through my center, then you can make use of the graduate students."
57
00:05:39,529.992 --> 00:05:40,769.987
All right, that works out great.
58
00:05:41,149.988 --> 00:05:54,469.987
and so I was doing stuff for the Iowa Department of Education on formative assessment, and helping them figure out how to do formative assessment in the schools that was aligned to the state standards.
59
00:05:55,249.987 --> 00:05:56,449.984
Totally great project.
60
00:05:56,459.986 --> 00:05:59,209.987
I had three graduate students I was able to fund on it.
61
00:05:59,539.988 --> 00:06:02,189.988
We worked through the Center for Evaluation and Assessment.
62
00:06:02,739.992 --> 00:06:09,39.984
And, when Don Yarbrough was ready to retire, I became the new director.
63
00:06:09,849.978 --> 00:06:18,813.984
And I have been the director, I think- 11 years I've been the director for the Center for Evaluation and Assessment.
64
00:06:19,3.984 --> 00:06:19,463.984
Time flies.
65
00:06:19,693.984 --> 00:06:20,603.984
And I know.
66
00:06:20,603.984 --> 00:06:33,543.985
And our center does, we provide high quality program evaluation for clients, we train the next generation of graduate students, and we advance, evaluation theory and practice.
67
00:06:33,893.985 --> 00:06:39,133.984
So we do that by attending the American Evaluation Association annual meeting.
68
00:06:39,503.984 --> 00:06:47,583.985
this next coming one I've got a bunch of graduate students who are international, who put together a panel, and I'm gonna be the moderator of the panel.
69
00:06:47,873.983 --> 00:06:54,663.983
And they're gonna talk about their experiences as international students doing program evaluation in the United States.
70
00:06:54,953.984 --> 00:06:55,723.984
Fascinating.
71
00:06:55,803.983 --> 00:06:57,163.985
Oh, it's so interesting.
72
00:06:57,293.984 --> 00:06:58,583.985
'cause for a lot of them- I'll look that up.
73
00:06:59,13.985 --> 00:06:59,633.985
Yeah.
74
00:06:59,633.985 --> 00:07:13,23.982
So I have a, a Korean student, a Nigerian student, and a student from Ghana, and the three of them, have just brought this wealth of evaluation experience to the center.
75
00:07:13,303.982 --> 00:07:20,33.982
But for them, like learning this whole new context in the U- in the United States has been like diving into the deep end.
76
00:07:20,933.979 --> 00:07:22,453.981
So really fun.
77
00:07:22,633.983 --> 00:07:23,213.983
I bet.
78
00:07:23,243.976 --> 00:07:29,503.976
And there's one thing that I have to share with you since you have been director of the CEA now for 11 years.
79
00:07:30,43.979 --> 00:07:40,673.982
as I had grown up as a graduate student there, that's really where I learned the core from Don Yarbrough and from that mentor.
80
00:07:40,693.98 --> 00:07:54,303.983
I call it a cognitive apprenticeship because I'm an educational psychologist, right? So I started not ever having done a qualitative interview, and I learned from gr- more advanced graduate students and staff how to do that.
81
00:07:54,363.983 --> 00:08:05,573.984
And then I built up my skills until I was able to be kind of a baby evaluator on a project with Don, who gave me more and more as I proved I could do it, and he mentored me, and we reflected.
82
00:08:05,933.981 --> 00:08:12,93.981
It was, it's how I have taught and mentored every single person that I've taught and mentored since I left the University of Iowa.
83
00:08:12,133.982 --> 00:08:13,573.978
So I do think it's very special.
84
00:08:14,173.982 --> 00:08:14,403.982
Same.
85
00:08:14,733.982 --> 00:08:14,793.982
Right? Same.
86
00:08:15,313.982 --> 00:08:16,53.982
Absolutely.
87
00:08:16,53.982 --> 00:08:22,3.975
So and then what happens is when I get great graduate students, I steal them and I make them staff.
88
00:08:22,273.981 --> 00:08:22,523.981
Yes.
89
00:08:22,833.979 --> 00:08:23,53.979
Yes.
90
00:08:23,463.983 --> 00:08:24,343.983
Absolutely.
91
00:08:24,723.978 --> 00:08:32,333.976
So all of my current staff members have been graduate students in the CEA, and they wanted to stay.
92
00:08:34,689.984 --> 00:08:34,839.984
Kudos.
93
00:08:35,179.984 --> 00:08:35,629.984
Win-win.
94
00:08:35,759.984 --> 00:08:37,669.984
Win-win for everyone It's a huge win-win, yeah.
95
00:08:37,709.984 --> 00:08:38,29.984
Yeah.
96
00:08:38,489.984 --> 00:08:38,889.984
Yeah.
97
00:08:39,409.984 --> 00:08:56,509.984
I wondered if you could talk more about, because the CEA does work across states, and you can pick a project or you can talk more broadly, the intersection of policy, especially because you are a policy person, and how that impacts the work that you do, the evaluations.
98
00:08:56,519.985 --> 00:09:05,599.982
So whether you're- doing it in a, two different states with different policy environments or two different states with a federal, requirement, but it's implemented differently.
99
00:09:05,989.983 --> 00:09:12,489.983
what does that look like in your experience, and how do you navigate that? we had some programs canceled.
100
00:09:12,949.984 --> 00:09:27,499.985
one of them was this really cool project in the University of Iowa ca- the Holden Cancer Center, and they were working with people who had graduated from a college that didn't have a medical school.
101
00:09:27,559.985 --> 00:09:54,169.978
And they were training them and helping them figure out, they would work in a lab and spend the summer learning how to do medical research, and then they would decide what they wanna be when they grow up, if they wanted to go to graduate school, if they wanted to be medical school, if they wanted to be an MD-PhD, if they wanted to keep working in a lab or if they wanted to be a practitioner, they would figure that out over that summer and then go into the health sciences.
102
00:09:55,189.978 --> 00:10:08,389.978
The reason the program got canceled was because the program very much privileged students who were from minoritized populations, and the government cut the funding for it.
103
00:10:08,699.98 --> 00:10:12,939.978
So they're in the process of looking for private funding to continue the program.
104
00:10:13,349.977 --> 00:10:24,89.979
but we were doing the evaluation for it, and it's, that's a really interesting point where policy, federal policy, has affected a program, that we were working on.
105
00:10:25,181.98 --> 00:10:40,131.98
some of the other programs that we're doing right now, I've got one that is looking at interventions for students who don't read at grade level, and we're looking at the third graders specifically.
106
00:10:40,531.979 --> 00:10:53,221.98
And the way that works is there's an intervention that is computer-based, and the computer-based intervention, is supposed to elevate the student's ability to read at grade level.
107
00:10:53,741.981 --> 00:11:04,641.981
And so we're doing that project in Wyoming, Illinois, Oklahoma, and a couple of districts in Iowa, where we're looking at how well that intervention works.
108
00:11:05,161.981 --> 00:11:07,351.981
And it's this really cool design.
109
00:11:07,351.981 --> 00:11:09,151.979
It's a, an RCT.
110
00:11:09,181.978 --> 00:11:11,91.978
It's really interesting.
111
00:11:11,471.977 --> 00:11:16,661.977
And we're in collaboration with the Iowa Reading Research Center for it, and we're the evaluators.
112
00:11:17,581.975 --> 00:11:20,651.978
So that's one of our big projects that we've got right now.
113
00:11:20,651.978 --> 00:11:27,631.979
That was a- That was a, yeah, that was a, a huge $10 million grant from the federal government- Oh to do that reading research.
114
00:11:28,41.983 --> 00:11:33,941.981
And to have a methodology like an RCT, a randomized control trial, that is, Gold standard if you're talking about rigor, gold standard.
115
00:11:33,961.983 --> 00:11:34,381.983
Yes.
116
00:11:34,411.98 --> 00:11:34,661.98
Yeah.
117
00:11:35,71.984 --> 00:11:35,341.984
Yeah.
118
00:11:37,59.983 --> 00:11:38,679.983
Really looking forward to that project.
119
00:11:38,679.983 --> 00:11:39,699.983
It just started.
120
00:11:39,799.983 --> 00:11:41,589.983
we begin the intervention in August.
121
00:11:41,589.983 --> 00:11:42,209.983
Very cool.
122
00:11:42,209.983 --> 00:11:42,769.983
Very cool.
123
00:11:43,59.983 --> 00:11:56,679.984
some of our other projects that we're working on, we've got this project through, the policy center here at the University of Iowa for health sciences, and, it's the College of Public Health.
124
00:11:57,69.984 --> 00:12:04,349.981
And they're looking at interventions for, people who are women, who run farms.
125
00:12:04,939.982 --> 00:12:07,639.982
And it's really interesting.
126
00:12:07,649.981 --> 00:12:07,929.981
Yeah.
127
00:12:07,929.981 --> 00:12:08,869.982
And they're bring- You got my attention.
128
00:12:09,409.979 --> 00:12:15,429.981
Yeah, they've got, mini grants that they're doing to help them, with their farming practice.
129
00:12:15,729.982 --> 00:12:19,749.981
And we're doing the program evaluation for each of the little mini grants.
130
00:12:20,159.98 --> 00:12:28,939.979
And, and in fact, in two weeks we're gonna go to Nebraska City, Nebraska for an opportunity to bring everybody together, because these people aren't just Iowa farmers.
131
00:12:28,939.98 --> 00:12:30,959.979
They're also from Nebraska and Missouri.
132
00:12:31,269.982 --> 00:12:36,439.977
And, so that's another interes- I get to do all kinds of fun stuff as an evaluator.
133
00:12:36,989.983 --> 00:12:40,879.978
We've had projects through- the College of Engineering, through the College of Pharmacy.
134
00:12:41,329.974 --> 00:12:43,299.976
it's a, it's a really fun job.
135
00:12:44,519.977 --> 00:12:51,659.969
It sounds really fun, and I love the different content areas, the different geographies, the different, the name it.
136
00:12:52,809.974 --> 00:13:09,889.965
So how, especially thinking about different methodologies you might use from qualitative to a randomized control trial, mixed method studies- Mm-hmm um, do you have any powerful examples of, who typically decides what gets measured? Sometimes we're told by funders- and you do it.
137
00:13:10,589.97 --> 00:13:29,649.975
And other times, like maybe with the, the medical school project, what I found with some projects that were canceled, including, SNAP-Ed, which was canceled federally or defunded, there were cost benefit numbers- Mm qualitative data, quantitative data, and yet it was canceled.
138
00:13:30,439.974 --> 00:13:30,759.974
Yeah.
139
00:13:30,759.974 --> 00:13:31,439.967
Not funded.
140
00:13:31,439.974 --> 00:13:38,959.963
So it just brings up the question of, Yeah, like who decides what gets measured and then what's done with that and how decisions are made? It's a big question.
141
00:13:38,989.963 --> 00:13:43,629.963
I don't know if an example comes to mind though that like sticks out when you think about the crux of that issue.
142
00:13:43,869.963 --> 00:13:44,799.963
it changes.
143
00:13:44,999.963 --> 00:13:55,589.963
So if it's somebody who is looking to have their funding continued, they want some sort of summative evaluation that demonstrates the efficacy of the program.
144
00:13:55,979.962 --> 00:14:06,609.961
And so that's a completely different strategy than a program that's just starting, where they're looking for formative feedback to continue to improve the program over time.
145
00:14:07,289.962 --> 00:14:10,289.961
So some of that is negotiation with the client.
146
00:14:10,589.961 --> 00:14:14,959.961
A lot of times they defer to us as the experts in the evaluation.
147
00:14:15,479.96 --> 00:14:25,579.963
The best thing that can happen is when a client comes to us before they've even written the grant, and they say, "We're, we're looking to get funding for this.
148
00:14:26,39.962 --> 00:14:37,569.963
We need an evaluator because the grant says 10% of our, points come from the evaluation, and they're wanting something called a logic model, and we don't know what that is."
149
00:14:37,999.961 --> 00:14:38,919.96
And I'm like, "I'm your girl.
150
00:14:39,99.959 --> 00:14:39,749.965
I'm your girl."
151
00:14:39,889.96 --> 00:14:42,679.962
So I will help them- We can speak logic model language.
152
00:14:43,9.963 --> 00:14:44,339.965
We can talk about language.
153
00:14:44,349.959 --> 00:14:44,369.959
Yeah.
154
00:14:44,619.963 --> 00:14:52,569.96
So we write the evaluation section of the grant, in the best case scenario, and we say "There's gonna be interviews.
155
00:14:52,569.96 --> 00:14:53,719.961
There's gonna be focus groups.
156
00:14:53,719.961 --> 00:15:02,199.96
There's gonna be, some sort of evaluation methodology that we decide in advance based on the aims of the grant."
157
00:15:03,203.963 --> 00:15:15,783.963
And then we write their logic model, we work back and forth with the client once they understand what the logic model's for, and we go through the outputs and the outcomes, and, we end up with this grant proposal.
158
00:15:15,783.963 --> 00:15:18,213.963
And sometimes I only get three-quarters of a page.
159
00:15:18,473.963 --> 00:15:21,363.961
sometimes I get two pages plus an appendix.
160
00:15:21,563.961 --> 00:15:21,713.961
Oh.
161
00:15:21,813.962 --> 00:15:28,653.963
so, so we are very good at making our plan and fitting it into the grant proposal.
162
00:15:28,723.964 --> 00:15:39,453.962
And then when the grant is accepted and funded, then we're able to follow the plan that we all agreed on at the beginning, and we get a memorandum of understanding, and we start the work.
163
00:15:39,943.964 --> 00:15:55,133.963
the worst case scenario is when somebody gets a grant, they realize after the fact that they're going to need some kind of formative evaluation every year to see how their program is going, and then they contact us.
164
00:15:55,373.964 --> 00:16:09,513.957
it's gonna be complicated, isn't it? Then I have to work with whatever they wrote in the grant proposal and try to fix it, because oftentimes people say, I, I've had a client say before, "Well, I'm a social scientist, so I knew what to write."
165
00:16:09,523.958 --> 00:16:09,533.958
Oh.
166
00:16:10,503.961 --> 00:16:18,133.962
And I'm like, "Social science has the same methods as evaluation, but there's a huge difference."
167
00:16:18,183.962 --> 00:16:18,203.962
Yeah.
168
00:16:18,243.959 --> 00:16:21,513.964
The purpose for why we're doing this is not the same.
169
00:16:22,303.957 --> 00:16:26,663.963
So I've had to do some- As in it's not knowledge generation necessarily.
170
00:16:26,673.96 --> 00:16:33,263.961
It can be, but it's- But it's not in service to a program, a policy, whatever you're eval- a curriculum, whatever you're evaluating.
171
00:16:33,763.964 --> 00:16:34,573.964
Precisely.
172
00:16:34,913.957 --> 00:16:41,883.952
And, I teach a program evaluation class, and what I try to hammer home to the students is that the client is the boss.
173
00:16:41,943.955 --> 00:16:48,473.957
And so the client tells us what matters, and the client tells us what they want evaluated.
174
00:16:48,873.955 --> 00:16:52,253.952
What we do is we tell them how we're going to go about doing it.
175
00:16:52,533.959 --> 00:16:59,313.964
So that's part of the, providing high-quality program evaluation services to clients.
176
00:17:00,905.964 --> 00:17:02,205.964
There's a couple things that come up there.
177
00:17:02,305.964 --> 00:17:11,215.964
A, when you're teaching new evaluators in a class or in a professional setting, that ultimately we're not the decision-makers.
178
00:17:11,335.963 --> 00:17:20,745.964
We are the ones who facilitate and do the kind of in the background whatever the data collection is, synthesis, meaning making, and then elevate it.
179
00:17:20,755.964 --> 00:17:23,905.963
We can reflect mirrors to them, but we are not the decision-makers.
180
00:17:24,205.963 --> 00:17:24,325.963
That's right.
181
00:17:24,405.964 --> 00:17:29,795.963
And I periodically had a, challenges with people who are upset 'cause they're like, "That's not right," or, "You didn't look at the data."
182
00:17:29,795.963 --> 00:17:33,825.964
And it's like at the end of the day, it's not our decision, and that's tough sometimes.
183
00:17:34,65.963 --> 00:17:36,405.962
So I have only been fired once.
184
00:17:37,245.964 --> 00:17:42,925.961
And I was fired by a client who did not like the results of the evaluation.
185
00:17:43,415.962 --> 00:17:43,435.962
Yeah.
186
00:17:43,775.963 --> 00:17:50,405.96
And it was for a training of superintendents in a different state, so I won't even tell you what the state was.
187
00:17:50,595.962 --> 00:17:50,815.962
Yeah.
188
00:17:50,875.961 --> 00:17:53,225.964
and this was, like, 10 years ago.
189
00:17:53,265.965 --> 00:17:54,505.964
This was a long time ago.
190
00:17:54,895.962 --> 00:17:58,645.962
And she said, "This intervention is helping superintendents do X."
191
00:17:59,305.962 --> 00:18:04,685.961
And so I did individual interviews with each superintendent because there were only 10 of them.
192
00:18:04,695.965 --> 00:18:09,475.962
And, they weren't all available at the same time to do a focus group, so I was like, "That's fine.
193
00:18:09,475.962 --> 00:18:10,805.962
I'll just do individual interviews."
194
00:18:10,805.962 --> 00:18:18,535.96
And I did them in person, and every single one of them said, "This intervention is a waste of time."
195
00:18:18,885.962 --> 00:18:19,465.961
Oh, no.
196
00:18:19,875.96 --> 00:18:30,95.964
"This intervention isn't aligned to the standards for us in this state, and the only reason we're doing this is 'cause we're getting paid, and we thought it was gonna be useful, but it wasn't."
197
00:18:32,219.962 --> 00:18:36,319.962
So I met with the client Which is why you hire an external evaluator, so they can say that.
198
00:18:36,319.962 --> 00:18:44,319.961
I met with the client- That's true and I was like, "You know, these are the ways that the, the participants are saying that this program isn't working," and she fired me on the spot.
199
00:18:45,239.962 --> 00:18:46,349.961
Seriously? Wow.
200
00:18:46,549.962 --> 00:18:47,879.962
She said, "This is it.
201
00:18:47,889.962 --> 00:18:48,359.962
You're...
202
00:18:48,379.961 --> 00:18:49,799.961
You know, I don't believe you.
203
00:18:50,59.961 --> 00:18:51,449.961
this isn't working for me.
204
00:18:51,749.961 --> 00:18:59,469.96
I know that this program actually works- Oh and I don't believe your feedback, and, thank you for everything, and you can go."
205
00:19:00,329.961 --> 00:19:01,59.96
So I flew home.
206
00:19:01,829.961 --> 00:19:02,79.961
Yeah.
207
00:19:02,459.96 --> 00:19:02,689.96
Yeah.
208
00:19:02,689.96 --> 00:19:03,429.961
So I flew home.
209
00:19:04,209.96 --> 00:19:06,269.959
it was- You don't forget an experience like that.
210
00:19:06,269.959 --> 00:19:06,819.958
Uh-uh.
211
00:19:06,869.961 --> 00:19:09,649.9593333
at the, in the moment, I was humiliated, right? Of course.
212
00:19:09,649.9593333 --> 00:19:19,299.955
And I was just like, "This is ter- I'm fired? What do you mean I'm fired?" And then, now as time's gone by, I'm just like, I, maybe I didn't give her the feedback that she wanted.
213
00:19:19,659.956 --> 00:19:22,409.957
Maybe I wasn't careful with my words.
214
00:19:22,499.955 --> 00:19:25,549.953
Right? maybe it was too hot to touch.
215
00:19:26,269.952 --> 00:19:28,129.955
Oh, speaking of too hot to touch.
216
00:19:28,409.955 --> 00:19:44,419.95
So I had an evaluation that I was doing, in a school district where I was working with school leaders, and I sat them down for the focus group, and within a minute, someone started crying.
217
00:19:45,209.95 --> 00:19:45,499.95
Oh.
218
00:19:45,889.95 --> 00:19:47,539.954
And then another person started crying.
219
00:19:47,769.955 --> 00:19:48,119.952
Oh, my.
220
00:19:48,119.954 --> 00:19:50,99.953
And I was like, "All I did was ask you how it's going."
221
00:19:50,519.955 --> 00:19:52,209.956
And so I'm looking around for Kleenex.
222
00:19:52,219.952 --> 00:19:54,899.952
We were in a library and, in the school district.
223
00:19:55,139.95 --> 00:19:58,149.951
And and it was an empty library 'cause they had arranged for us to meet there.
224
00:19:58,459.946 --> 00:19:59,749.953
I was like, "Tell me what's going on.
225
00:19:59,759.949 --> 00:20:02,139.949
Why are you guys crying?" And they're like, "We can't do our program.
226
00:20:02,569.947 --> 00:20:04,99.948
We can't implement this program.
227
00:20:04,419.953 --> 00:20:10,909.953
We can't get somebody in central office to do what we need to do to get this program going."
228
00:20:10,919.953 --> 00:20:17,449.96
And it was a really cool program where students were l- doing internships in businesses around the community.
229
00:20:17,509.957 --> 00:20:19,899.956
And they needed transportation permissions.
230
00:20:20,209.955 --> 00:20:31,129.954
They needed, the central office to sign off on the computer things that the kids were taking out into the school district, and they were frustrated, which is why they were crying.
231
00:20:32,79.958 --> 00:20:33,859.958
And I got to the bottom of it.
232
00:20:34,49.958 --> 00:20:44,909.958
Eventually, they told me the reason it wasn't working is because the two people in central office they needed to do the work were having an extramarital affair with each other.
233
00:20:45,19.957 --> 00:20:45,629.958
Oh, yeah.
234
00:20:45,839.958 --> 00:20:46,389.958
Oh, my.
235
00:20:46,939.957 --> 00:20:57,639.958
And that was why the work wasn't getting done, because they were too busy to do the work, and that's why the teachers and the pr- the principals- Oh were crying, was because they're so frustrated and there's like, "But there's nothing we can do.
236
00:20:57,769.957 --> 00:20:58,949.957
we can't tell anybody."
237
00:20:59,59.958 --> 00:21:00,599.958
And I was like, "Well, I can.
238
00:21:01,369.957 --> 00:21:01,949.958
I'm your outside evaluator."
239
00:21:02,579.957 --> 00:21:02,599.957
Yeah.
240
00:21:03,329.958 --> 00:21:06,589.959
So I asked- And then- the superintendent for a private meeting.
241
00:21:07,219.957 --> 00:21:07,359.958
Mm-hmm.
242
00:21:07,389.956 --> 00:21:11,149.957
And I told the superintendent, I was like, "There are two people in central office."
243
00:21:11,169.958 --> 00:21:13,219.957
I didn't say, you know, what was going on.
244
00:21:13,469.957 --> 00:21:19,849.956
I said, "There's two people in central office who are blocking this program from being implemented," and I s- I named them.
245
00:21:20,49.955 --> 00:21:28,149.958
I said, this is, I'm not gonna tell you who told me this, but I'm telling you that this program isn't gonna get off the ground if these two people continue to be the barrier."
246
00:21:28,969.959 --> 00:21:38,489.957
And the superintendent kind of looked at me like he wasn't surprised, and, you know, we n- I ne- so what I say is what's important.
247
00:21:38,549.955 --> 00:21:48,589.951
I deal with hot things that happen- Yeah in gracious ways, but the superintendent needed to know that there was a barrier to implementation.
248
00:21:49,869.955 --> 00:21:49,889.955
Yeah.
249
00:21:49,939.955 --> 00:21:51,609.952
So we're always careful.
250
00:21:51,809.95 --> 00:21:58,109.955
But what a powerful example of how this is not research.
251
00:21:58,679.953 --> 00:22:10,629.947
We use research methodologies, but also all sorts, I don't know, human resources, counseling, facilitation, sensitivity training, All of it.
252
00:22:10,629.949 --> 00:22:15,909.948
It's, it's, Judgment, lots of judgment on so m- on every level of, you know- Oh, I felt like being their therapist.
253
00:22:15,949.943 --> 00:22:18,279.947
They were crying and telling me what was going wrong.
254
00:22:18,289.947 --> 00:22:18,309.947
Yeah.
255
00:22:18,499.946 --> 00:22:21,489.944
And they were frustrated, and I'm just like, "Why are you crying? Stop crying."
256
00:22:21,489.944 --> 00:22:32,629.942
It's as much a human experience as it is data, and- Mm-hmm it's putting those in action which makes the work that we do, I think, really cool and- Yeah hard and memorable.
257
00:22:32,629.943 --> 00:22:35,549.938
and I love how passionate people are about their programs.
258
00:22:35,599.937 --> 00:22:37,69.944
That's the other really fun thing.
259
00:22:37,259.941 --> 00:22:37,529.941
Yeah.
260
00:22:38,89.942 --> 00:22:38,389.942
Yeah.
261
00:22:39,579.933 --> 00:22:49,969.945
So when it comes to, y- you've been doing this work for a long time, Liz, and you've seen it from, you know, program side, evaluator side, leadership levels.
262
00:22:50,579.933 --> 00:23:12,999.94
if there was one thing you wish organizations understood better about how funders think about data and evaluation- Hmm yeah, what do you wish they better understood? I think I go back to my comment about, clients not understanding that evaluation is different.
263
00:23:13,9.94 --> 00:23:23,289.94
And that when a funding agency says, "10% of your grant proposal is evaluation," that needs to also be reflected in the budget.
264
00:23:24,59.939 --> 00:23:29,999.94
And the industry standard is, you know, 10 to 15% of the budget gets e- gets evaluation.
265
00:23:30,449.939 --> 00:23:35,869.937
And a lot of times there's a negotiation with clients where they say, "Well, we don't have that much in the budget."
266
00:23:36,229.939 --> 00:23:37,909.941
And I say, "You're not gonna get funded."
267
00:23:38,119.938 --> 00:23:48,479.938
because if the funding agency is telling you 10% of your points come from the evaluation, they're signaling to you that they want 10% of it in the budget.
268
00:23:49,269.935 --> 00:23:57,99.935
And so that's a really frustrating thing for me sometimes is talking to clients about, um, the importance of this work.
269
00:23:57,759.936 --> 00:24:01,999.935
And then once we get into it and we're telling them, this is how the program's going.
270
00:24:02,9.936 --> 00:24:03,839.934
These are the things that are going really well.
271
00:24:03,939.936 --> 00:24:05,699.936
Here are some tweaks that you could do.
272
00:24:05,799.933 --> 00:24:09,399.936
Here are the implementation barriers that you didn't even know were happening."
273
00:24:09,639.933 --> 00:24:09,659.935
Mm-hmm.
274
00:24:09,669.932 --> 00:24:15,729.936
Then they come to a place where they appreciate the power of program evaluation.
275
00:24:16,479.937 --> 00:24:19,199.935
So it's a, it's a process.
276
00:24:19,469.935 --> 00:24:19,629.934
Oh, yeah.
277
00:24:19,629.934 --> 00:24:21,509.936
Some clients have the experience.
278
00:24:21,669.932 --> 00:24:28,999.935
You know, sometimes they have the experience with evaluators where they know, you know, like a repeat client, they'll say, "Hey, I know we can't do this without you."
279
00:24:29,579.936 --> 00:24:30,949.932
And those are my favorite.
280
00:24:32,99.932 --> 00:24:34,219.931
Well, the highest compliment, right? Yes.
281
00:24:35,409.938 --> 00:24:37,619.941
The, yeah, it's a true partnership.
282
00:24:37,689.937 --> 00:24:46,989.938
And when you were talking, I just had this memory of when I was at the University of Minnesota, we would have a lot of organizations call in and say, "We need an evaluator," because that was part of my job there.
283
00:24:47,429.935 --> 00:24:56,589.94
And this program was giving, I think, homeless youth, and this is probably, like, a decade ago, iPads, and- putting a lot of money into the programmatic aspect.
284
00:24:56,659.938 --> 00:25:16,829.939
And, Had thought through a lot of that, but they had, you'll never guess, for the evaluation where they wanted a literature review- What? all the data collection, instruments created- What? implemented, analyzed, and they had Do you wanna take a guess at, they had the budget for the evaluation? 3%.
285
00:25:16,969.939 --> 00:25:20,939.939
I- I don't even know the percentage to be honest with you, but it was $500.
286
00:25:22,739.94 --> 00:25:25,129.939
You could try to see if a student can volunteer.
287
00:25:25,969.94 --> 00:25:26,929.939
That's all I can say.
288
00:25:26,989.94 --> 00:25:31,99.939
I can't help you 'cause they're just like- I can't help you with that "What do you mean?" That's not gonna work.
289
00:25:31,339.938 --> 00:25:31,689.938
Yeah.
290
00:25:31,689.941 --> 00:25:40,129.94
So I think also understanding the work that goes into the work that we do can be a, a whole process to educate folks on over time.
291
00:25:40,319.938 --> 00:25:40,939.939
Takes some time.
292
00:25:40,959.937 --> 00:25:41,489.937
Absolutely.
293
00:25:41,859.939 --> 00:25:44,519.94
'Cause it's not always, materialized as a deliverable.
294
00:25:45,589.94 --> 00:25:52,89.941
And I don't think people, normal people, don't take program evaluation classes in grad school.
295
00:25:52,339.94 --> 00:25:53,129.941
No, probably not.
296
00:25:53,689.938 --> 00:25:54,759.939
It's pretty niche.
297
00:25:55,49.939 --> 00:25:55,469.939
yeah.
298
00:25:55,469.941 --> 00:26:00,759.938
So as professors start writing grants, they're like, "Uh, we don't even know what this means."
299
00:26:01,29.939 --> 00:26:03,799.942
And they contact, you know, the- Your center.
300
00:26:04,409.944 --> 00:26:08,69.941
they, they will randomly call us and say, "We don't know what we need."
301
00:26:08,129.941 --> 00:26:09,559.938
But- You have evaluation in your name.
302
00:26:09,979.94 --> 00:26:12,459.943
But you have evaluation in your name, so maybe you can help us.
303
00:26:12,519.941 --> 00:26:12,889.938
yeah.
304
00:26:13,19.938 --> 00:26:13,379.938
Help.
305
00:26:14,279.94 --> 00:26:14,699.943
Mm-hmm.
306
00:26:15,989.939 --> 00:26:51,109.946
as we're, you know, in a pretty tenuous situation for our fields, for funding, for a lot of the programs that, you and I and many others have been evaluating for many years, what If you had a magic wand, what is one thing that you would put out into the universe, whether it's people better understood what we do, or, data's used differently, or something else? what would you hope might change in the next one to three years? What I really wish is that there were more opportunities for government funding.
307
00:26:51,629.944 --> 00:27:01,889.946
a lot of that has been frozen, and so people with really cool ideas that have grants under review are being told that there are no reviews happening.
308
00:27:02,749.942 --> 00:27:11,179.953
Yeah and so that's been sad to watch because, you know, I'll write the evaluation section, I'll read the whole grant proposal, and I'll say, this is definitely getting funded.
309
00:27:11,209.944 --> 00:27:12,109.945
This is cool.
310
00:27:12,119.948 --> 00:27:13,689.948
This is a really cool program."
311
00:27:14,229.948 --> 00:27:30,639.952
and then I would say I have at least four clients who said the government funding is up in the air, and so it's not The review has been canceled, and they're telling us, "We'll hold onto it if the funding comes back through.
312
00:27:30,959.952 --> 00:27:38,149.955
you know, nothing for you to do now, but I know you said you would start in September, but we're not gonna have the reviews done in time for you to start in September."
313
00:27:39,671.96 --> 00:27:52,281.96
So that's, I think if I could wave a magic wand, I would want the federal government to be back to a place where those federal reviews were happening again, and, that funding would get released.
314
00:27:53,231.96 --> 00:27:53,741.96
Y- yes.
315
00:27:53,771.96 --> 00:27:57,801.96
And I just, I wish more people understood the ecosystem that it impacts.
316
00:27:57,821.96 --> 00:28:06,521.957
It impacts innovation, professors' livelihoods, researchers' livelihoods, the data that's out there, and everyone else, you know, who makes that work happen.
317
00:28:06,731.958 --> 00:28:10,681.956
It is a whole ecosys- the admin, that are supported by that work happening.
318
00:28:10,681.957 --> 00:28:16,467.6698571
So it's at every level that there's a whole ecosystem that's been severely- The graduate students um, disruptive.
319
00:28:16,467.6698571 --> 00:28:16,560.5272857
yeah.
320
00:28:16,560.5272857 --> 00:28:19,129.0975714
It's been hard, you know, finding funding for graduate students.
321
00:28:19,199.1005714 --> 00:28:19,459.0995714
Oh, I bet.
322
00:28:19,469.0965714 --> 00:28:27,639.0975714
And I've, I, I get emails at least once a week from potential new graduate students asking if I've got funding.
323
00:28:28,59.0975714 --> 00:28:28,359.0975714
Oof.
324
00:28:29,769.0995714 --> 00:28:30,289.0995714
And I don't.
325
00:28:30,769.0995714 --> 00:28:31,269.0965714
I don't.
326
00:28:31,469.0965714 --> 00:28:34,189.0945714
I mean, I have six graduate students who I'm funding now.
327
00:28:34,679.0965714 --> 00:28:35,369.0955714
but I- Which is amazing.
328
00:28:35,719.0955714 --> 00:28:36,219.0955714
Yeah.
329
00:28:36,359.0955714 --> 00:28:37,849.0955714
But I don't have room for new ones.
330
00:28:37,949.0955714 --> 00:28:38,239.0955714
yeah.
331
00:28:38,899.0955714 --> 00:28:51,849.0945714
I'm very heartened to hear that you are continuing on the, I'll just call it cognitive apprenticeship, you might call it something else, but that, that development of evaluators in a professional environment as they're learning.
332
00:28:51,859.0915714 --> 00:29:02,389.1015714
And it's bidirectional too, right? every time I mentor someone, I'm learning something from that person, like your graduate students that will present at the annual American Evaluation Association Conference.
333
00:29:02,889.0985714 --> 00:29:05,479.1015714
And Liz, I'd love to have you back in the future.
334
00:29:05,499.0995714 --> 00:29:12,154.6555714
I know that there's a variety of topics that we could dive into and talk about, but thank you today for your time and for sharing your insights.
335
00:29:12,154.6555714 --> 00:29:13,709.1025714
It was so good to see you again, Melissa.
336
00:29:14,229.1025714 --> 00:29:14,889.1025714
Likewise.
337
00:29:17,139.0985714 --> 00:29:17,529.1025714
Take care.
338
00:29:18,79.1015714 --> 00:29:18,759.0955714
Thank you.
339
00:29:18,759.0955714 --> 00:29:19,309.0965714
Thank you.
340
00:29:19,359.0965714 --> 00:29:25,109.0955714
so I would love to hear for our listeners what stood out to you.
341
00:29:25,539.0995714 --> 00:29:27,589.0995714
please subscribe, get in touch with us.
342
00:29:27,589.0995714 --> 00:29:33,169.0935714
Let us know what you wanna hear, and thank you for joining this episode of Measure What Matters.
343
00:29:33,449.0955714 --> 00:29:33,859.0965714
Take care.
00:00:15,750 --> 00:00:20,010
So today I'm going to start with a little bit of history here.
2
00:00:20,330 --> 00:00:33,640
the University of Iowa, which is where I and my next guest did our graduate studies, and that's where we met, they're a global leader in measurement, anchored by the Iowa testing programs, and even the College of Education is named after E.F.
3
00:00:33,640 --> 00:00:37,640
Lindquist, who has a very deep history in measurement.
4
00:00:38,59.999 --> 00:00:55,639.998
he started the Iowa Testing Programs, which many listeners maybe have heard of, contributed to helping to establish the American College Testing program, or ACT, which is one of the largest college entry testing programs in the nat- in the United States.
5
00:00:56,250 --> 00:01:01,279.998
And more than 70 years later, my next guest, Liz and I, Dr.
6
00:01:01,279.999 --> 00:01:10,430
Hollingworth, went, met in graduate school in a statistics class and took so many, so many statistics, measurement.
7
00:01:10,630 --> 00:01:20,340.003
I've never read so much about, ever knew that there was so much to read about validity in my life until we were in graduate school, and then commiserated over coffees and lunches, et cetera.
8
00:01:21,10.001 --> 00:01:24,279.998
So I'm so excited to introduce Dr.
9
00:01:24,279.998 --> 00:01:32,639.999
Liz Hollingworth, professor and director of the Center for Evaluation and Assessment, which is in the College of Education at the University of Iowa.
10
00:01:33,179.999 --> 00:01:36,800
You are listening to Measure What Matters by Elevate What Matters.
11
00:01:37,889.998 --> 00:01:44,879.999
Here's where we pull back the curtain on how funders, evaluators, and communities measure what works and what doesn't.
12
00:01:45,190.001 --> 00:01:47,149.999
I'm Melissa, the host of your podcast.
13
00:01:48,339.995 --> 00:01:51,270
So Liz, we met in graduate school.
14
00:01:51,330.001 --> 00:01:52,169.995
We- Hi, Melissa.
15
00:01:52,169.995 --> 00:01:52,829.995
Hi.
16
00:01:52,849.997 --> 00:01:54,851.42557143
It's so good to meet you again.
17
00:01:54,851.42557143 --> 00:01:56,280
It's lovely to see you.
18
00:01:56,859.999 --> 00:02:01,729.999
I think our first class together was, Intermediate statistics fall of, yep.
19
00:02:02,119.999 --> 00:02:04,209.999
And it was in the first year of graduate school.
20
00:02:04,209.999 --> 00:02:06,179.999
So we were both in a PhD program.
21
00:02:06,599.999 --> 00:02:11,89.999
I was studying educational psychology, and you were in a doctoral program for ed policy and leadership.
22
00:02:11,89.999 --> 00:02:11,799.998
Is that right? Yeah.
23
00:02:12,529.999 --> 00:02:18,179.998
And then we both, because of the storied history at, in the College of Ed, took a lot of measurements.
24
00:02:18,179.999 --> 00:02:27,339.998
So I was gonna start by hearing from you on what are some highlights to your journey of when we met through school to becoming a professor.
25
00:02:27,640 --> 00:02:35,409.998
I will let you talk about all the things that you do and the many jobs that you do, including director of a university-based evaluation center.
26
00:02:36,329.998 --> 00:02:36,879.998
Okay.
27
00:02:36,939.997 --> 00:02:45,289.998
after I gradu- during, graduate school, I had an assistantship with Iowa testing programs, and when I graduated, I got a postdoc with them.
28
00:02:45,770 --> 00:02:56,880
And, I helped write test items, I helped, expand the Iowa Test of Basic Skills to, US Virgin Islands, Ohio, Arkansas.
29
00:02:57,359.996 --> 00:02:59,249.993
It was during No Child Left Behind.
30
00:02:59,329.994 --> 00:03:09,549.987
And every school district in every state needed to have some sort of test for third grade, fifth grade, eighth grade, and I helped write those tests.
31
00:03:09,859.992 --> 00:03:10,789.987
It was really fun.
32
00:03:10,789.992 --> 00:03:12,319.992
It was super interesting.
33
00:03:12,739.992 --> 00:03:22,239.989
when I finished my postdoc, I got a job, at the University of Iowa, which is really unusual, to get a job at the place where you graduated from.
34
00:03:22,469.992 --> 00:03:35,9.986
And I am in the educational leadership program, and part of the reason why that was such a good fit was, again, because No Child Left Behind was really changing how people were being trained to lead schools.
35
00:03:35,109.991 --> 00:03:51,589.987
So I created a bunch of classes on standards-based education, on, social advocacy for students who had special needs, because all of that testing would shed light on kids who needed extra help.
36
00:03:51,839.991 --> 00:03:58,429.981
So I worked to train people who were gonna be principals and superintendents how to do measurement.
37
00:03:59,619.99 --> 00:04:08,679.99
And, And I have to interject to say, just because listeners might not know this about you, you were a teacher before you went to graduate school for your, PhD.
38
00:04:08,999.99 --> 00:04:09,439.99
I was.
39
00:04:09,439.99 --> 00:04:17,79.989
And so I think- I taught- Yeah I taught in, I taught third grade in Chicago public schools, and then I taught sixth and eighth grade, not at the same time.
40
00:04:17,249.989 --> 00:04:21,399.988
First I taught sixth grade in Michigan, and then, I got moved to eighth grade.
41
00:04:21,419.987 --> 00:04:22,809.987
So I had the same kids.
42
00:04:22,829.987 --> 00:04:23,959.987
It was really fun.
43
00:04:24,179.988 --> 00:04:28,839.988
So I had the same kids in sixth grade, and then two years later I joined up with them again in eighth grade.
44
00:04:29,309.988 --> 00:04:30,689.986
so I taught there for five years.
45
00:04:30,739.988 --> 00:04:34,159.987
And, yeah, that was my K12 teaching background.
46
00:04:34,159.989 --> 00:04:39,39.992
I was also the administrator for gifted and talented education- in that district.
47
00:04:39,879.994 --> 00:05:02,509.992
so- And sorry to interrupt, I had wanted to mention that because you're talking about the, these things that you were doing in your jobs after graduate school, and it's the accumulation of the experience and training that you had had at that point that put you at that intersection at such a interesting time that we were both in a program like the one that, the ones we were at at the University of Iowa during No Child Left Behind.
48
00:05:02,519.99 --> 00:05:07,539.992
And we're gonna get into policies and how that affects the work that we do at federal, state, or local levels.
49
00:05:07,939.988 --> 00:05:08,419.995
You can continue.
50
00:05:08,579.995 --> 00:05:10,9.994
Yeah, it was such a weird time.
51
00:05:10,309.994 --> 00:05:10,519.99
It was such a, yeah.
52
00:05:11,149.994 --> 00:05:11,289.994
Yep.
53
00:05:11,329.994 --> 00:05:22,409.989
And then, yeah, and then I started doing program evaluation work with Don Yarbrough ugh who helped write the program evaluation standards and was directing the Center for Evaluation and Assessment.
54
00:05:22,809.99 --> 00:05:25,489.991
And he came to me and he said well, I think it was the other way around.
55
00:05:25,509.988 --> 00:05:30,179.991
I came to him and I said, "I'm being asked to do some program evaluation work.
56
00:05:30,769.99 --> 00:05:39,239.988
How, how can we collaborate on this?" And he said, "Well, if you run it through my center, then you can make use of the graduate students."
57
00:05:39,529.992 --> 00:05:40,769.987
All right, that works out great.
58
00:05:41,149.988 --> 00:05:54,469.987
and so I was doing stuff for the Iowa Department of Education on formative assessment, and helping them figure out how to do formative assessment in the schools that was aligned to the state standards.
59
00:05:55,249.987 --> 00:05:56,449.984
Totally great project.
60
00:05:56,459.986 --> 00:05:59,209.987
I had three graduate students I was able to fund on it.
61
00:05:59,539.988 --> 00:06:02,189.988
We worked through the Center for Evaluation and Assessment.
62
00:06:02,739.992 --> 00:06:09,39.984
And, when Don Yarbrough was ready to retire, I became the new director.
63
00:06:09,849.978 --> 00:06:18,813.984
And I have been the director, I think- 11 years I've been the director for the Center for Evaluation and Assessment.
64
00:06:19,3.984 --> 00:06:19,463.984
Time flies.
65
00:06:19,693.984 --> 00:06:20,603.984
And I know.
66
00:06:20,603.984 --> 00:06:33,543.985
And our center does, we provide high quality program evaluation for clients, we train the next generation of graduate students, and we advance, evaluation theory and practice.
67
00:06:33,893.985 --> 00:06:39,133.984
So we do that by attending the American Evaluation Association annual meeting.
68
00:06:39,503.984 --> 00:06:47,583.985
this next coming one I've got a bunch of graduate students who are international, who put together a panel, and I'm gonna be the moderator of the panel.
69
00:06:47,873.983 --> 00:06:54,663.983
And they're gonna talk about their experiences as international students doing program evaluation in the United States.
70
00:06:54,953.984 --> 00:06:55,723.984
Fascinating.
71
00:06:55,803.983 --> 00:06:57,163.985
Oh, it's so interesting.
72
00:06:57,293.984 --> 00:06:58,583.985
'cause for a lot of them- I'll look that up.
73
00:06:59,13.985 --> 00:06:59,633.985
Yeah.
74
00:06:59,633.985 --> 00:07:13,23.982
So I have a, a Korean student, a Nigerian student, and a student from Ghana, and the three of them, have just brought this wealth of evaluation experience to the center.
75
00:07:13,303.982 --> 00:07:20,33.982
But for them, like learning this whole new context in the U- in the United States has been like diving into the deep end.
76
00:07:20,933.979 --> 00:07:22,453.981
So really fun.
77
00:07:22,633.983 --> 00:07:23,213.983
I bet.
78
00:07:23,243.976 --> 00:07:29,503.976
And there's one thing that I have to share with you since you have been director of the CEA now for 11 years.
79
00:07:30,43.979 --> 00:07:40,673.982
as I had grown up as a graduate student there, that's really where I learned the core from Don Yarbrough and from that mentor.
80
00:07:40,693.98 --> 00:07:54,303.983
I call it a cognitive apprenticeship because I'm an educational psychologist, right? So I started not ever having done a qualitative interview, and I learned from gr- more advanced graduate students and staff how to do that.
81
00:07:54,363.983 --> 00:08:05,573.984
And then I built up my skills until I was able to be kind of a baby evaluator on a project with Don, who gave me more and more as I proved I could do it, and he mentored me, and we reflected.
82
00:08:05,933.981 --> 00:08:12,93.981
It was, it's how I have taught and mentored every single person that I've taught and mentored since I left the University of Iowa.
83
00:08:12,133.982 --> 00:08:13,573.978
So I do think it's very special.
84
00:08:14,173.982 --> 00:08:14,403.982
Same.
85
00:08:14,733.982 --> 00:08:14,793.982
Right? Same.
86
00:08:15,313.982 --> 00:08:16,53.982
Absolutely.
87
00:08:16,53.982 --> 00:08:22,3.975
So and then what happens is when I get great graduate students, I steal them and I make them staff.
88
00:08:22,273.981 --> 00:08:22,523.981
Yes.
89
00:08:22,833.979 --> 00:08:23,53.979
Yes.
90
00:08:23,463.983 --> 00:08:24,343.983
Absolutely.
91
00:08:24,723.978 --> 00:08:32,333.976
So all of my current staff members have been graduate students in the CEA, and they wanted to stay.
92
00:08:34,689.984 --> 00:08:34,839.984
Kudos.
93
00:08:35,179.984 --> 00:08:35,629.984
Win-win.
94
00:08:35,759.984 --> 00:08:37,669.984
Win-win for everyone It's a huge win-win, yeah.
95
00:08:37,709.984 --> 00:08:38,29.984
Yeah.
96
00:08:38,489.984 --> 00:08:38,889.984
Yeah.
97
00:08:39,409.984 --> 00:08:56,509.984
I wondered if you could talk more about, because the CEA does work across states, and you can pick a project or you can talk more broadly, the intersection of policy, especially because you are a policy person, and how that impacts the work that you do, the evaluations.
98
00:08:56,519.985 --> 00:09:05,599.982
So whether you're- doing it in a, two different states with different policy environments or two different states with a federal, requirement, but it's implemented differently.
99
00:09:05,989.983 --> 00:09:12,489.983
what does that look like in your experience, and how do you navigate that? we had some programs canceled.
100
00:09:12,949.984 --> 00:09:27,499.985
one of them was this really cool project in the University of Iowa ca- the Holden Cancer Center, and they were working with people who had graduated from a college that didn't have a medical school.
101
00:09:27,559.985 --> 00:09:54,169.978
And they were training them and helping them figure out, they would work in a lab and spend the summer learning how to do medical research, and then they would decide what they wanna be when they grow up, if they wanted to go to graduate school, if they wanted to be medical school, if they wanted to be an MD-PhD, if they wanted to keep working in a lab or if they wanted to be a practitioner, they would figure that out over that summer and then go into the health sciences.
102
00:09:55,189.978 --> 00:10:08,389.978
The reason the program got canceled was because the program very much privileged students who were from minoritized populations, and the government cut the funding for it.
103
00:10:08,699.98 --> 00:10:12,939.978
So they're in the process of looking for private funding to continue the program.
104
00:10:13,349.977 --> 00:10:24,89.979
but we were doing the evaluation for it, and it's, that's a really interesting point where policy, federal policy, has affected a program, that we were working on.
105
00:10:25,181.98 --> 00:10:40,131.98
some of the other programs that we're doing right now, I've got one that is looking at interventions for students who don't read at grade level, and we're looking at the third graders specifically.
106
00:10:40,531.979 --> 00:10:53,221.98
And the way that works is there's an intervention that is computer-based, and the computer-based intervention, is supposed to elevate the student's ability to read at grade level.
107
00:10:53,741.981 --> 00:11:04,641.981
And so we're doing that project in Wyoming, Illinois, Oklahoma, and a couple of districts in Iowa, where we're looking at how well that intervention works.
108
00:11:05,161.981 --> 00:11:07,351.981
And it's this really cool design.
109
00:11:07,351.981 --> 00:11:09,151.979
It's a, an RCT.
110
00:11:09,181.978 --> 00:11:11,91.978
It's really interesting.
111
00:11:11,471.977 --> 00:11:16,661.977
And we're in collaboration with the Iowa Reading Research Center for it, and we're the evaluators.
112
00:11:17,581.975 --> 00:11:20,651.978
So that's one of our big projects that we've got right now.
113
00:11:20,651.978 --> 00:11:27,631.979
That was a- That was a, yeah, that was a, a huge $10 million grant from the federal government- Oh to do that reading research.
114
00:11:28,41.983 --> 00:11:33,941.981
And to have a methodology like an RCT, a randomized control trial, that is, Gold standard if you're talking about rigor, gold standard.
115
00:11:33,961.983 --> 00:11:34,381.983
Yes.
116
00:11:34,411.98 --> 00:11:34,661.98
Yeah.
117
00:11:35,71.984 --> 00:11:35,341.984
Yeah.
118
00:11:37,59.983 --> 00:11:38,679.983
Really looking forward to that project.
119
00:11:38,679.983 --> 00:11:39,699.983
It just started.
120
00:11:39,799.983 --> 00:11:41,589.983
we begin the intervention in August.
121
00:11:41,589.983 --> 00:11:42,209.983
Very cool.
122
00:11:42,209.983 --> 00:11:42,769.983
Very cool.
123
00:11:43,59.983 --> 00:11:56,679.984
some of our other projects that we're working on, we've got this project through, the policy center here at the University of Iowa for health sciences, and, it's the College of Public Health.
124
00:11:57,69.984 --> 00:12:04,349.981
And they're looking at interventions for, people who are women, who run farms.
125
00:12:04,939.982 --> 00:12:07,639.982
And it's really interesting.
126
00:12:07,649.981 --> 00:12:07,929.981
Yeah.
127
00:12:07,929.981 --> 00:12:08,869.982
And they're bring- You got my attention.
128
00:12:09,409.979 --> 00:12:15,429.981
Yeah, they've got, mini grants that they're doing to help them, with their farming practice.
129
00:12:15,729.982 --> 00:12:19,749.981
And we're doing the program evaluation for each of the little mini grants.
130
00:12:20,159.98 --> 00:12:28,939.979
And, and in fact, in two weeks we're gonna go to Nebraska City, Nebraska for an opportunity to bring everybody together, because these people aren't just Iowa farmers.
131
00:12:28,939.98 --> 00:12:30,959.979
They're also from Nebraska and Missouri.
132
00:12:31,269.982 --> 00:12:36,439.977
And, so that's another interes- I get to do all kinds of fun stuff as an evaluator.
133
00:12:36,989.983 --> 00:12:40,879.978
We've had projects through- the College of Engineering, through the College of Pharmacy.
134
00:12:41,329.974 --> 00:12:43,299.976
it's a, it's a really fun job.
135
00:12:44,519.977 --> 00:12:51,659.969
It sounds really fun, and I love the different content areas, the different geographies, the different, the name it.
136
00:12:52,809.974 --> 00:13:09,889.965
So how, especially thinking about different methodologies you might use from qualitative to a randomized control trial, mixed method studies- Mm-hmm um, do you have any powerful examples of, who typically decides what gets measured? Sometimes we're told by funders- and you do it.
137
00:13:10,589.97 --> 00:13:29,649.975
And other times, like maybe with the, the medical school project, what I found with some projects that were canceled, including, SNAP-Ed, which was canceled federally or defunded, there were cost benefit numbers- Mm qualitative data, quantitative data, and yet it was canceled.
138
00:13:30,439.974 --> 00:13:30,759.974
Yeah.
139
00:13:30,759.974 --> 00:13:31,439.967
Not funded.
140
00:13:31,439.974 --> 00:13:38,959.963
So it just brings up the question of, Yeah, like who decides what gets measured and then what's done with that and how decisions are made? It's a big question.
141
00:13:38,989.963 --> 00:13:43,629.963
I don't know if an example comes to mind though that like sticks out when you think about the crux of that issue.
142
00:13:43,869.963 --> 00:13:44,799.963
it changes.
143
00:13:44,999.963 --> 00:13:55,589.963
So if it's somebody who is looking to have their funding continued, they want some sort of summative evaluation that demonstrates the efficacy of the program.
144
00:13:55,979.962 --> 00:14:06,609.961
And so that's a completely different strategy than a program that's just starting, where they're looking for formative feedback to continue to improve the program over time.
145
00:14:07,289.962 --> 00:14:10,289.961
So some of that is negotiation with the client.
146
00:14:10,589.961 --> 00:14:14,959.961
A lot of times they defer to us as the experts in the evaluation.
147
00:14:15,479.96 --> 00:14:25,579.963
The best thing that can happen is when a client comes to us before they've even written the grant, and they say, "We're, we're looking to get funding for this.
148
00:14:26,39.962 --> 00:14:37,569.963
We need an evaluator because the grant says 10% of our, points come from the evaluation, and they're wanting something called a logic model, and we don't know what that is."
149
00:14:37,999.961 --> 00:14:38,919.96
And I'm like, "I'm your girl.
150
00:14:39,99.959 --> 00:14:39,749.965
I'm your girl."
151
00:14:39,889.96 --> 00:14:42,679.962
So I will help them- We can speak logic model language.
152
00:14:43,9.963 --> 00:14:44,339.965
We can talk about language.
153
00:14:44,349.959 --> 00:14:44,369.959
Yeah.
154
00:14:44,619.963 --> 00:14:52,569.96
So we write the evaluation section of the grant, in the best case scenario, and we say "There's gonna be interviews.
155
00:14:52,569.96 --> 00:14:53,719.961
There's gonna be focus groups.
156
00:14:53,719.961 --> 00:15:02,199.96
There's gonna be, some sort of evaluation methodology that we decide in advance based on the aims of the grant."
157
00:15:03,203.963 --> 00:15:15,783.963
And then we write their logic model, we work back and forth with the client once they understand what the logic model's for, and we go through the outputs and the outcomes, and, we end up with this grant proposal.
158
00:15:15,783.963 --> 00:15:18,213.963
And sometimes I only get three-quarters of a page.
159
00:15:18,473.963 --> 00:15:21,363.961
sometimes I get two pages plus an appendix.
160
00:15:21,563.961 --> 00:15:21,713.961
Oh.
161
00:15:21,813.962 --> 00:15:28,653.963
so, so we are very good at making our plan and fitting it into the grant proposal.
162
00:15:28,723.964 --> 00:15:39,453.962
And then when the grant is accepted and funded, then we're able to follow the plan that we all agreed on at the beginning, and we get a memorandum of understanding, and we start the work.
163
00:15:39,943.964 --> 00:15:55,133.963
the worst case scenario is when somebody gets a grant, they realize after the fact that they're going to need some kind of formative evaluation every year to see how their program is going, and then they contact us.
164
00:15:55,373.964 --> 00:16:09,513.957
it's gonna be complicated, isn't it? Then I have to work with whatever they wrote in the grant proposal and try to fix it, because oftentimes people say, I, I've had a client say before, "Well, I'm a social scientist, so I knew what to write."
165
00:16:09,523.958 --> 00:16:09,533.958
Oh.
166
00:16:10,503.961 --> 00:16:18,133.962
And I'm like, "Social science has the same methods as evaluation, but there's a huge difference."
167
00:16:18,183.962 --> 00:16:18,203.962
Yeah.
168
00:16:18,243.959 --> 00:16:21,513.964
The purpose for why we're doing this is not the same.
169
00:16:22,303.957 --> 00:16:26,663.963
So I've had to do some- As in it's not knowledge generation necessarily.
170
00:16:26,673.96 --> 00:16:33,263.961
It can be, but it's- But it's not in service to a program, a policy, whatever you're eval- a curriculum, whatever you're evaluating.
171
00:16:33,763.964 --> 00:16:34,573.964
Precisely.
172
00:16:34,913.957 --> 00:16:41,883.952
And, I teach a program evaluation class, and what I try to hammer home to the students is that the client is the boss.
173
00:16:41,943.955 --> 00:16:48,473.957
And so the client tells us what matters, and the client tells us what they want evaluated.
174
00:16:48,873.955 --> 00:16:52,253.952
What we do is we tell them how we're going to go about doing it.
175
00:16:52,533.959 --> 00:16:59,313.964
So that's part of the, providing high-quality program evaluation services to clients.
176
00:17:00,905.964 --> 00:17:02,205.964
There's a couple things that come up there.
177
00:17:02,305.964 --> 00:17:11,215.964
A, when you're teaching new evaluators in a class or in a professional setting, that ultimately we're not the decision-makers.
178
00:17:11,335.963 --> 00:17:20,745.964
We are the ones who facilitate and do the kind of in the background whatever the data collection is, synthesis, meaning making, and then elevate it.
179
00:17:20,755.964 --> 00:17:23,905.963
We can reflect mirrors to them, but we are not the decision-makers.
180
00:17:24,205.963 --> 00:17:24,325.963
That's right.
181
00:17:24,405.964 --> 00:17:29,795.963
And I periodically had a, challenges with people who are upset 'cause they're like, "That's not right," or, "You didn't look at the data."
182
00:17:29,795.963 --> 00:17:33,825.964
And it's like at the end of the day, it's not our decision, and that's tough sometimes.
183
00:17:34,65.963 --> 00:17:36,405.962
So I have only been fired once.
184
00:17:37,245.964 --> 00:17:42,925.961
And I was fired by a client who did not like the results of the evaluation.
185
00:17:43,415.962 --> 00:17:43,435.962
Yeah.
186
00:17:43,775.963 --> 00:17:50,405.96
And it was for a training of superintendents in a different state, so I won't even tell you what the state was.
187
00:17:50,595.962 --> 00:17:50,815.962
Yeah.
188
00:17:50,875.961 --> 00:17:53,225.964
and this was, like, 10 years ago.
189
00:17:53,265.965 --> 00:17:54,505.964
This was a long time ago.
190
00:17:54,895.962 --> 00:17:58,645.962
And she said, "This intervention is helping superintendents do X."
191
00:17:59,305.962 --> 00:18:04,685.961
And so I did individual interviews with each superintendent because there were only 10 of them.
192
00:18:04,695.965 --> 00:18:09,475.962
And, they weren't all available at the same time to do a focus group, so I was like, "That's fine.
193
00:18:09,475.962 --> 00:18:10,805.962
I'll just do individual interviews."
194
00:18:10,805.962 --> 00:18:18,535.96
And I did them in person, and every single one of them said, "This intervention is a waste of time."
195
00:18:18,885.962 --> 00:18:19,465.961
Oh, no.
196
00:18:19,875.96 --> 00:18:30,95.964
"This intervention isn't aligned to the standards for us in this state, and the only reason we're doing this is 'cause we're getting paid, and we thought it was gonna be useful, but it wasn't."
197
00:18:32,219.962 --> 00:18:36,319.962
So I met with the client Which is why you hire an external evaluator, so they can say that.
198
00:18:36,319.962 --> 00:18:44,319.961
I met with the client- That's true and I was like, "You know, these are the ways that the, the participants are saying that this program isn't working," and she fired me on the spot.
199
00:18:45,239.962 --> 00:18:46,349.961
Seriously? Wow.
200
00:18:46,549.962 --> 00:18:47,879.962
She said, "This is it.
201
00:18:47,889.962 --> 00:18:48,359.962
You're...
202
00:18:48,379.961 --> 00:18:49,799.961
You know, I don't believe you.
203
00:18:50,59.961 --> 00:18:51,449.961
this isn't working for me.
204
00:18:51,749.961 --> 00:18:59,469.96
I know that this program actually works- Oh and I don't believe your feedback, and, thank you for everything, and you can go."
205
00:19:00,329.961 --> 00:19:01,59.96
So I flew home.
206
00:19:01,829.961 --> 00:19:02,79.961
Yeah.
207
00:19:02,459.96 --> 00:19:02,689.96
Yeah.
208
00:19:02,689.96 --> 00:19:03,429.961
So I flew home.
209
00:19:04,209.96 --> 00:19:06,269.959
it was- You don't forget an experience like that.
210
00:19:06,269.959 --> 00:19:06,819.958
Uh-uh.
211
00:19:06,869.961 --> 00:19:09,649.9593333
at the, in the moment, I was humiliated, right? Of course.
212
00:19:09,649.9593333 --> 00:19:19,299.955
And I was just like, "This is ter- I'm fired? What do you mean I'm fired?" And then, now as time's gone by, I'm just like, I, maybe I didn't give her the feedback that she wanted.
213
00:19:19,659.956 --> 00:19:22,409.957
Maybe I wasn't careful with my words.
214
00:19:22,499.955 --> 00:19:25,549.953
Right? maybe it was too hot to touch.
215
00:19:26,269.952 --> 00:19:28,129.955
Oh, speaking of too hot to touch.
216
00:19:28,409.955 --> 00:19:44,419.95
So I had an evaluation that I was doing, in a school district where I was working with school leaders, and I sat them down for the focus group, and within a minute, someone started crying.
217
00:19:45,209.95 --> 00:19:45,499.95
Oh.
218
00:19:45,889.95 --> 00:19:47,539.954
And then another person started crying.
219
00:19:47,769.955 --> 00:19:48,119.952
Oh, my.
220
00:19:48,119.954 --> 00:19:50,99.953
And I was like, "All I did was ask you how it's going."
221
00:19:50,519.955 --> 00:19:52,209.956
And so I'm looking around for Kleenex.
222
00:19:52,219.952 --> 00:19:54,899.952
We were in a library and, in the school district.
223
00:19:55,139.95 --> 00:19:58,149.951
And and it was an empty library 'cause they had arranged for us to meet there.
224
00:19:58,459.946 --> 00:19:59,749.953
I was like, "Tell me what's going on.
225
00:19:59,759.949 --> 00:20:02,139.949
Why are you guys crying?" And they're like, "We can't do our program.
226
00:20:02,569.947 --> 00:20:04,99.948
We can't implement this program.
227
00:20:04,419.953 --> 00:20:10,909.953
We can't get somebody in central office to do what we need to do to get this program going."
228
00:20:10,919.953 --> 00:20:17,449.96
And it was a really cool program where students were l- doing internships in businesses around the community.
229
00:20:17,509.957 --> 00:20:19,899.956
And they needed transportation permissions.
230
00:20:20,209.955 --> 00:20:31,129.954
They needed, the central office to sign off on the computer things that the kids were taking out into the school district, and they were frustrated, which is why they were crying.
231
00:20:32,79.958 --> 00:20:33,859.958
And I got to the bottom of it.
232
00:20:34,49.958 --> 00:20:44,909.958
Eventually, they told me the reason it wasn't working is because the two people in central office they needed to do the work were having an extramarital affair with each other.
233
00:20:45,19.957 --> 00:20:45,629.958
Oh, yeah.
234
00:20:45,839.958 --> 00:20:46,389.958
Oh, my.
235
00:20:46,939.957 --> 00:20:57,639.958
And that was why the work wasn't getting done, because they were too busy to do the work, and that's why the teachers and the pr- the principals- Oh were crying, was because they're so frustrated and there's like, "But there's nothing we can do.
236
00:20:57,769.957 --> 00:20:58,949.957
we can't tell anybody."
237
00:20:59,59.958 --> 00:21:00,599.958
And I was like, "Well, I can.
238
00:21:01,369.957 --> 00:21:01,949.958
I'm your outside evaluator."
239
00:21:02,579.957 --> 00:21:02,599.957
Yeah.
240
00:21:03,329.958 --> 00:21:06,589.959
So I asked- And then- the superintendent for a private meeting.
241
00:21:07,219.957 --> 00:21:07,359.958
Mm-hmm.
242
00:21:07,389.956 --> 00:21:11,149.957
And I told the superintendent, I was like, "There are two people in central office."
243
00:21:11,169.958 --> 00:21:13,219.957
I didn't say, you know, what was going on.
244
00:21:13,469.957 --> 00:21:19,849.956
I said, "There's two people in central office who are blocking this program from being implemented," and I s- I named them.
245
00:21:20,49.955 --> 00:21:28,149.958
I said, this is, I'm not gonna tell you who told me this, but I'm telling you that this program isn't gonna get off the ground if these two people continue to be the barrier."
246
00:21:28,969.959 --> 00:21:38,489.957
And the superintendent kind of looked at me like he wasn't surprised, and, you know, we n- I ne- so what I say is what's important.
247
00:21:38,549.955 --> 00:21:48,589.951
I deal with hot things that happen- Yeah in gracious ways, but the superintendent needed to know that there was a barrier to implementation.
248
00:21:49,869.955 --> 00:21:49,889.955
Yeah.
249
00:21:49,939.955 --> 00:21:51,609.952
So we're always careful.
250
00:21:51,809.95 --> 00:21:58,109.955
But what a powerful example of how this is not research.
251
00:21:58,679.953 --> 00:22:10,629.947
We use research methodologies, but also all sorts, I don't know, human resources, counseling, facilitation, sensitivity training, All of it.
252
00:22:10,629.949 --> 00:22:15,909.948
It's, it's, Judgment, lots of judgment on so m- on every level of, you know- Oh, I felt like being their therapist.
253
00:22:15,949.943 --> 00:22:18,279.947
They were crying and telling me what was going wrong.
254
00:22:18,289.947 --> 00:22:18,309.947
Yeah.
255
00:22:18,499.946 --> 00:22:21,489.944
And they were frustrated, and I'm just like, "Why are you crying? Stop crying."
256
00:22:21,489.944 --> 00:22:32,629.942
It's as much a human experience as it is data, and- Mm-hmm it's putting those in action which makes the work that we do, I think, really cool and- Yeah hard and memorable.
257
00:22:32,629.943 --> 00:22:35,549.938
and I love how passionate people are about their programs.
258
00:22:35,599.937 --> 00:22:37,69.944
That's the other really fun thing.
259
00:22:37,259.941 --> 00:22:37,529.941
Yeah.
260
00:22:38,89.942 --> 00:22:38,389.942
Yeah.
261
00:22:39,579.933 --> 00:22:49,969.945
So when it comes to, y- you've been doing this work for a long time, Liz, and you've seen it from, you know, program side, evaluator side, leadership levels.
262
00:22:50,579.933 --> 00:23:12,999.94
if there was one thing you wish organizations understood better about how funders think about data and evaluation- Hmm yeah, what do you wish they better understood? I think I go back to my comment about, clients not understanding that evaluation is different.
263
00:23:13,9.94 --> 00:23:23,289.94
And that when a funding agency says, "10% of your grant proposal is evaluation," that needs to also be reflected in the budget.
264
00:23:24,59.939 --> 00:23:29,999.94
And the industry standard is, you know, 10 to 15% of the budget gets e- gets evaluation.
265
00:23:30,449.939 --> 00:23:35,869.937
And a lot of times there's a negotiation with clients where they say, "Well, we don't have that much in the budget."
266
00:23:36,229.939 --> 00:23:37,909.941
And I say, "You're not gonna get funded."
267
00:23:38,119.938 --> 00:23:48,479.938
because if the funding agency is telling you 10% of your points come from the evaluation, they're signaling to you that they want 10% of it in the budget.
268
00:23:49,269.935 --> 00:23:57,99.935
And so that's a really frustrating thing for me sometimes is talking to clients about, um, the importance of this work.
269
00:23:57,759.936 --> 00:24:01,999.935
And then once we get into it and we're telling them, this is how the program's going.
270
00:24:02,9.936 --> 00:24:03,839.934
These are the things that are going really well.
271
00:24:03,939.936 --> 00:24:05,699.936
Here are some tweaks that you could do.
272
00:24:05,799.933 --> 00:24:09,399.936
Here are the implementation barriers that you didn't even know were happening."
273
00:24:09,639.933 --> 00:24:09,659.935
Mm-hmm.
274
00:24:09,669.932 --> 00:24:15,729.936
Then they come to a place where they appreciate the power of program evaluation.
275
00:24:16,479.937 --> 00:24:19,199.935
So it's a, it's a process.
276
00:24:19,469.935 --> 00:24:19,629.934
Oh, yeah.
277
00:24:19,629.934 --> 00:24:21,509.936
Some clients have the experience.
278
00:24:21,669.932 --> 00:24:28,999.935
You know, sometimes they have the experience with evaluators where they know, you know, like a repeat client, they'll say, "Hey, I know we can't do this without you."
279
00:24:29,579.936 --> 00:24:30,949.932
And those are my favorite.
280
00:24:32,99.932 --> 00:24:34,219.931
Well, the highest compliment, right? Yes.
281
00:24:35,409.938 --> 00:24:37,619.941
The, yeah, it's a true partnership.
282
00:24:37,689.937 --> 00:24:46,989.938
And when you were talking, I just had this memory of when I was at the University of Minnesota, we would have a lot of organizations call in and say, "We need an evaluator," because that was part of my job there.
283
00:24:47,429.935 --> 00:24:56,589.94
And this program was giving, I think, homeless youth, and this is probably, like, a decade ago, iPads, and- putting a lot of money into the programmatic aspect.
284
00:24:56,659.938 --> 00:25:16,829.939
And, Had thought through a lot of that, but they had, you'll never guess, for the evaluation where they wanted a literature review- What? all the data collection, instruments created- What? implemented, analyzed, and they had Do you wanna take a guess at, they had the budget for the evaluation? 3%.
285
00:25:16,969.939 --> 00:25:20,939.939
I- I don't even know the percentage to be honest with you, but it was $500.
286
00:25:22,739.94 --> 00:25:25,129.939
You could try to see if a student can volunteer.
287
00:25:25,969.94 --> 00:25:26,929.939
That's all I can say.
288
00:25:26,989.94 --> 00:25:31,99.939
I can't help you 'cause they're just like- I can't help you with that "What do you mean?" That's not gonna work.
289
00:25:31,339.938 --> 00:25:31,689.938
Yeah.
290
00:25:31,689.941 --> 00:25:40,129.94
So I think also understanding the work that goes into the work that we do can be a, a whole process to educate folks on over time.
291
00:25:40,319.938 --> 00:25:40,939.939
Takes some time.
292
00:25:40,959.937 --> 00:25:41,489.937
Absolutely.
293
00:25:41,859.939 --> 00:25:44,519.94
'Cause it's not always, materialized as a deliverable.
294
00:25:45,589.94 --> 00:25:52,89.941
And I don't think people, normal people, don't take program evaluation classes in grad school.
295
00:25:52,339.94 --> 00:25:53,129.941
No, probably not.
296
00:25:53,689.938 --> 00:25:54,759.939
It's pretty niche.
297
00:25:55,49.939 --> 00:25:55,469.939
yeah.
298
00:25:55,469.941 --> 00:26:00,759.938
So as professors start writing grants, they're like, "Uh, we don't even know what this means."
299
00:26:01,29.939 --> 00:26:03,799.942
And they contact, you know, the- Your center.
300
00:26:04,409.944 --> 00:26:08,69.941
they, they will randomly call us and say, "We don't know what we need."
301
00:26:08,129.941 --> 00:26:09,559.938
But- You have evaluation in your name.
302
00:26:09,979.94 --> 00:26:12,459.943
But you have evaluation in your name, so maybe you can help us.
303
00:26:12,519.941 --> 00:26:12,889.938
yeah.
304
00:26:13,19.938 --> 00:26:13,379.938
Help.
305
00:26:14,279.94 --> 00:26:14,699.943
Mm-hmm.
306
00:26:15,989.939 --> 00:26:51,109.946
as we're, you know, in a pretty tenuous situation for our fields, for funding, for a lot of the programs that, you and I and many others have been evaluating for many years, what If you had a magic wand, what is one thing that you would put out into the universe, whether it's people better understood what we do, or, data's used differently, or something else? what would you hope might change in the next one to three years? What I really wish is that there were more opportunities for government funding.
307
00:26:51,629.944 --> 00:27:01,889.946
a lot of that has been frozen, and so people with really cool ideas that have grants under review are being told that there are no reviews happening.
308
00:27:02,749.942 --> 00:27:11,179.953
Yeah and so that's been sad to watch because, you know, I'll write the evaluation section, I'll read the whole grant proposal, and I'll say, this is definitely getting funded.
309
00:27:11,209.944 --> 00:27:12,109.945
This is cool.
310
00:27:12,119.948 --> 00:27:13,689.948
This is a really cool program."
311
00:27:14,229.948 --> 00:27:30,639.952
and then I would say I have at least four clients who said the government funding is up in the air, and so it's not The review has been canceled, and they're telling us, "We'll hold onto it if the funding comes back through.
312
00:27:30,959.952 --> 00:27:38,149.955
you know, nothing for you to do now, but I know you said you would start in September, but we're not gonna have the reviews done in time for you to start in September."
313
00:27:39,671.96 --> 00:27:52,281.96
So that's, I think if I could wave a magic wand, I would want the federal government to be back to a place where those federal reviews were happening again, and, that funding would get released.
314
00:27:53,231.96 --> 00:27:53,741.96
Y- yes.
315
00:27:53,771.96 --> 00:27:57,801.96
And I just, I wish more people understood the ecosystem that it impacts.
316
00:27:57,821.96 --> 00:28:06,521.957
It impacts innovation, professors' livelihoods, researchers' livelihoods, the data that's out there, and everyone else, you know, who makes that work happen.
317
00:28:06,731.958 --> 00:28:10,681.956
It is a whole ecosys- the admin, that are supported by that work happening.
318
00:28:10,681.957 --> 00:28:16,467.6698571
So it's at every level that there's a whole ecosystem that's been severely- The graduate students um, disruptive.
319
00:28:16,467.6698571 --> 00:28:16,560.5272857
yeah.
320
00:28:16,560.5272857 --> 00:28:19,129.0975714
It's been hard, you know, finding funding for graduate students.
321
00:28:19,199.1005714 --> 00:28:19,459.0995714
Oh, I bet.
322
00:28:19,469.0965714 --> 00:28:27,639.0975714
And I've, I, I get emails at least once a week from potential new graduate students asking if I've got funding.
323
00:28:28,59.0975714 --> 00:28:28,359.0975714
Oof.
324
00:28:29,769.0995714 --> 00:28:30,289.0995714
And I don't.
325
00:28:30,769.0995714 --> 00:28:31,269.0965714
I don't.
326
00:28:31,469.0965714 --> 00:28:34,189.0945714
I mean, I have six graduate students who I'm funding now.
327
00:28:34,679.0965714 --> 00:28:35,369.0955714
but I- Which is amazing.
328
00:28:35,719.0955714 --> 00:28:36,219.0955714
Yeah.
329
00:28:36,359.0955714 --> 00:28:37,849.0955714
But I don't have room for new ones.
330
00:28:37,949.0955714 --> 00:28:38,239.0955714
yeah.
331
00:28:38,899.0955714 --> 00:28:51,849.0945714
I'm very heartened to hear that you are continuing on the, I'll just call it cognitive apprenticeship, you might call it something else, but that, that development of evaluators in a professional environment as they're learning.
332
00:28:51,859.0915714 --> 00:29:02,389.1015714
And it's bidirectional too, right? every time I mentor someone, I'm learning something from that person, like your graduate students that will present at the annual American Evaluation Association Conference.
333
00:29:02,889.0985714 --> 00:29:05,479.1015714
And Liz, I'd love to have you back in the future.
334
00:29:05,499.0995714 --> 00:29:12,154.6555714
I know that there's a variety of topics that we could dive into and talk about, but thank you today for your time and for sharing your insights.
335
00:29:12,154.6555714 --> 00:29:13,709.1025714
It was so good to see you again, Melissa.
336
00:29:14,229.1025714 --> 00:29:14,889.1025714
Likewise.
337
00:29:17,139.0985714 --> 00:29:17,529.1025714
Take care.
338
00:29:18,79.1015714 --> 00:29:18,759.0955714
Thank you.
339
00:29:18,759.0955714 --> 00:29:19,309.0965714
Thank you.
340
00:29:19,359.0965714 --> 00:29:25,109.0955714
so I would love to hear for our listeners what stood out to you.
341
00:29:25,539.0995714 --> 00:29:27,589.0995714
please subscribe, get in touch with us.
342
00:29:27,589.0995714 --> 00:29:33,169.0935714
Let us know what you wanna hear, and thank you for joining this episode of Measure What Matters.
343
00:29:33,449.0955714 --> 00:29:33,859.0965714
Take care.
