1
00:00:00,000 --> 00:00:07,600
Welcome to The Brain Architects, a podcast from the Center on the Developing Child at
2
00:00:07,600 --> 00:00:08,600
Harvard University.
3
00:00:08,600 --> 00:00:12,440
I'm Corrie Zimmerman, the Center's Chief Program Officer.
4
00:00:12,440 --> 00:00:16,880
Our Center believes that advances in the science of child development provide a powerful source
5
00:00:16,880 --> 00:00:20,920
of new ideas that can improve outcomes for children and their caregivers.
6
00:00:20,920 --> 00:00:25,440
By sharing the latest science from the field, we hope to help you make that science actionable
7
00:00:25,440 --> 00:00:29,400
and apply it in your work in ways that can increase your impact.
8
00:00:29,400 --> 00:00:33,160
With that goal in mind, the Center recently released the Ideas Impact Framework Toolkit.
9
00:00:33,160 --> 00:00:38,480
It's a free, online resource designed to help innovators in the field of early childhood
10
00:00:38,480 --> 00:00:42,600
build improved programs and products that are positioned to achieve greater impact in
11
00:00:42,600 --> 00:00:43,600
their communities.
12
00:00:43,600 --> 00:00:48,640
This toolkit is self-guided, self-paced, and provides a structured and flexible approach
13
00:00:48,640 --> 00:00:53,160
that facilitates program development, evaluation, and fast-cycle iteration.
14
00:00:53,160 --> 00:00:57,000
It includes resources to help teams develop and investigate a clear and precise theory
15
00:00:57,000 --> 00:00:58,000
of change.
16
00:00:58,000 --> 00:01:03,160
In April, we hosted a webinar about the toolkit, where we provided an overview of the site
17
00:01:03,160 --> 00:01:06,960
and had the opportunity to hear from teams at several organizations in the field about
18
00:01:06,960 --> 00:01:10,560
how they leverage the toolkit and its resources to shape their work.
19
00:01:10,560 --> 00:01:14,040
We're excited to share those discussions with you here on this episode of the Brain
20
00:01:14,040 --> 00:01:16,000
Architects podcast.
21
00:01:16,000 --> 00:01:19,880
If you're interested in hearing a full walkthrough of the toolkit by the director of our Pediatric
22
00:01:19,880 --> 00:01:24,400
Innovation Initiative, Dr. Melanie Berry, please head over to our YouTube channel to view the
23
00:01:24,400 --> 00:01:25,920
full webinar recording.
24
00:01:25,920 --> 00:01:30,200
You'll also hear from Dr. Melanie Berry during the Q&A portion.
25
00:01:30,200 --> 00:01:38,080
The full ideas toolkit we'll be talking about today can be found at ideas.developingchild.harvard.edu.
26
00:01:38,080 --> 00:01:42,800
And now, without further ado, here's Dr. Eshna Badruzman, the center's senior project manager
27
00:01:42,800 --> 00:01:47,000
for instructional design and the moderator for our panel discussion.
28
00:01:47,000 --> 00:01:50,120
Hello, everyone.
29
00:01:50,120 --> 00:01:52,000
Welcome.
30
00:01:52,000 --> 00:01:54,160
My name is Dr. Eshna Badruzman.
31
00:01:54,160 --> 00:01:59,320
I am a senior project manager for instructional design at the Center on the Developing Child
32
00:01:59,320 --> 00:02:02,560
at Harvard University, or HCDC.
33
00:02:02,560 --> 00:02:07,520
And I'm part of the development team of the Ideas Impact Framework Toolkit.
34
00:02:07,520 --> 00:02:09,240
And today, I'll be your host.
35
00:02:09,240 --> 00:02:13,720
So you may hear me come off mute and help guide presenters.
36
00:02:13,720 --> 00:02:17,500
And I'll be facilitating our question and answer period.
37
00:02:17,500 --> 00:02:22,480
So we are so pleased to be talking to you today about this resource.
38
00:02:22,480 --> 00:02:27,560
The Ideas Impact Framework was born out of more than a decade of the Frontiers of Innovation
39
00:02:27,560 --> 00:02:30,200
Initiative, or FOI.
40
00:02:30,200 --> 00:02:33,860
And some of you may have been partners in that effort.
41
00:02:33,860 --> 00:02:39,600
So while our team no longer offers live training on the framework, we are so excited to be
42
00:02:39,600 --> 00:02:43,600
introducing it to you as a free open access resource.
43
00:02:43,600 --> 00:02:49,320
And we really hope that this format is going to help make ideas accessible to innovators
44
00:02:49,320 --> 00:02:53,120
in the field of early childhood development moving forward.
45
00:02:53,120 --> 00:02:57,440
The framework was developed in partnership with the University of Washington College
46
00:02:57,440 --> 00:03:02,240
of Education and the University of Oregon Center for Translational Neuroscience with
47
00:03:02,240 --> 00:03:08,560
support from the Gates Foundation, the LEGO Foundation, Horticus, and the Hemera Foundation.
48
00:03:08,560 --> 00:03:15,240
I encourage you to check out our history and acknowledgments page of the toolkit for more
49
00:03:15,240 --> 00:03:21,920
information about our various collaborations and supporters throughout time as well.
50
00:03:21,920 --> 00:03:29,080
Now I'll go ahead and introduce our first set of speakers from folks at Valley Settlement.
51
00:03:29,080 --> 00:03:34,720
We have with us Carla Reyes, who is the program manager of the El Paso Cito Mobile Preschool
52
00:03:34,720 --> 00:03:41,120
Program at Valley Settlement, which is a nonprofit that works to create opportunities for the
53
00:03:41,120 --> 00:03:46,440
Latino community in the Aspen to Parishute region of Colorado. Carla joined Valley Settlement
54
00:03:46,440 --> 00:03:53,680
in March 2015 as a preschool teacher for El Paso Cito until June 2021 when she took on
55
00:03:53,680 --> 00:03:59,880
a leadership role. And we also have Sally Boatten, who is the director of development
56
00:03:59,880 --> 00:04:04,840
and communications at Valley Settlement, a nonprofit again serving the rural Aspen to
57
00:04:04,840 --> 00:04:12,040
Parishute region of Colorado with six two-generation programs designed by and for local Latino
58
00:04:12,040 --> 00:04:17,480
immigrant families. And Sally has been with Valley Settlement for over five years and
59
00:04:17,480 --> 00:04:23,640
began managing the organization's evaluation function in 2021. Thank you so much Carla
60
00:04:23,640 --> 00:04:28,200
and Sally, look forward to hearing from you.
61
00:04:28,200 --> 00:04:36,320
Good afternoon everyone. Thank you for inviting us to share our work with you all and how
62
00:04:36,320 --> 00:04:41,040
we have used the framework. I'm going to talk a little bit about the El Paso Cito program
63
00:04:41,040 --> 00:04:49,520
and how it kind of started. The idea of El Paso Cito began in 2011. We had two bilingual
64
00:04:49,520 --> 00:04:57,240
and bicultural community organizers who met one-on-one with about 300 families from the
65
00:04:57,240 --> 00:05:04,720
Aspen to Parishute region of Colorado and they learned about their lives and the barriers
66
00:05:04,720 --> 00:05:09,760
that they faced within our community. One of the findings from the initial listening
67
00:05:09,760 --> 00:05:16,640
tour was that only 1% of Latino children in our community were enrolled in preschool.
68
00:05:16,640 --> 00:05:22,100
We also learned that three of the biggest barriers for families to participate in preschool
69
00:05:22,100 --> 00:05:28,640
programs were language, cost, and transportation resulting in lack of access. Now we have all
70
00:05:28,640 --> 00:05:34,200
this information and we started thinking creatively of different ways that we could bring more
71
00:05:34,200 --> 00:05:41,760
access to preschool education to our community. El Paso Cito was one of the first two-generation
72
00:05:41,760 --> 00:05:48,320
programs that we launched in Valley Settlement to address the needs of preschool education.
73
00:05:48,320 --> 00:05:53,520
And throughout the years, Valley Settlement has continued to learn, evolve, and co-design
74
00:05:53,520 --> 00:06:00,160
programming to respond to community needs. Now our program has four mobile preschool
75
00:06:00,160 --> 00:06:06,360
buses that have been retrofitted into small preschool classrooms. We have two teachers
76
00:06:06,360 --> 00:06:12,520
on the bus and we serve eight children at a time. We provide families with about five
77
00:06:12,520 --> 00:06:20,240
to ten hours of free preschool education. We have about 96 children that we serve annually,
78
00:06:20,240 --> 00:06:27,040
between 40 to 50 children graduating at the end of the school year and moving on to kindergarten.
79
00:06:27,040 --> 00:06:33,320
And currently right now we serve five different neighborhoods within our community and we
80
00:06:33,320 --> 00:06:41,320
strive to build close relationships with families. So our program really is designed to meet
81
00:06:41,320 --> 00:06:49,440
families at where they are at and start breaking down those barriers. We host family nights,
82
00:06:49,440 --> 00:06:55,160
we have home visits with our families, we have parent-teacher conferences, we have different
83
00:06:55,160 --> 00:07:01,360
ways that families can volunteer within our program. We provide a lot of materials for
84
00:07:01,360 --> 00:07:06,120
families to use at home so that they can do home activities and homework packets with
85
00:07:06,120 --> 00:07:12,240
their students and we really try to engage with the families. So each one of our teachers
86
00:07:12,240 --> 00:07:19,200
speaks Spanish, is bilingual and bicultural, so this really allows that bond and that relationship
87
00:07:19,200 --> 00:07:24,880
to build with each one of our families. I'm going to hand it off to Sally Bowen and she's
88
00:07:24,880 --> 00:07:29,440
going to talk a little bit more about how we've used the framework.
89
00:07:29,440 --> 00:07:34,760
Thanks Carla. So several years ago we started working with the team at Frontiers of Innovation
90
00:07:34,760 --> 00:07:40,880
to refine and evolve our evaluation practices. This work included creating theories of change
91
00:07:40,880 --> 00:07:45,520
for each of our programs, researching and recommending observational assessments to
92
00:07:45,520 --> 00:07:51,700
measure participant progress towards our program targets and outcomes, and creating implementation
93
00:07:51,700 --> 00:07:56,560
guides for our programs to detail the kind of critical components of our work and ensure
94
00:07:56,560 --> 00:08:01,760
that future staff can implement programs with fidelity while still continuing to listen
95
00:08:01,760 --> 00:08:08,000
to and evolve alongside families. Since the early days of our programming, Valley Settlement
96
00:08:08,000 --> 00:08:13,160
has invested in evaluation to measure, understand, and strengthen the changes that children and
97
00:08:13,160 --> 00:08:18,520
families create in their lives through our programming. Working with the team at FOI
98
00:08:18,520 --> 00:08:23,080
really brought this to the next level. Over the last few years, we've been working to
99
00:08:23,080 --> 00:08:28,880
be more inclusive and participatory in our evaluation process. So now our entire staff
100
00:08:28,880 --> 00:08:34,800
gathers for three days every summer in what I call an evaluation retreat where we review
101
00:08:34,800 --> 00:08:40,400
our annual program data as a team and then try to answer those questions. Que, por que,
102
00:08:40,400 --> 00:08:46,920
y ahora que? Or as the toolkit outlines, what? What does the data say? Why? Why might the
103
00:08:46,920 --> 00:08:53,120
data say this? Or what does it mean? And finally, now what? What do we do to tweak or change
104
00:08:53,120 --> 00:08:58,840
in our evaluation approach or in our programming based on what we see in the data?
105
00:08:58,840 --> 00:09:04,960
Our teams then create action plans to outline those changes that they want to make. We're
106
00:09:04,960 --> 00:09:11,640
usually tweaking one or more program components for the upcoming year. On day two of the retreat,
107
00:09:11,640 --> 00:09:17,240
teams then go in and refine and evolve their theories of change. So we really see the theory
108
00:09:17,240 --> 00:09:24,240
of change as a living document that breathes and grows alongside our programming. They
109
00:09:24,240 --> 00:09:28,800
identify what targets and outcomes they're interested in measuring for the coming program
110
00:09:28,800 --> 00:09:34,720
year. And then after that evaluation retreat, we work together with an evaluation consultant
111
00:09:34,720 --> 00:09:40,760
to refine our measurement tools. And then I always try to call out and highlight that
112
00:09:40,760 --> 00:09:46,440
I am in the minority at Valley Settlements. Our staff are largely of the community that
113
00:09:46,440 --> 00:09:51,160
we work with. Most of our staff are immigrants or children of immigrants. Many have grown
114
00:09:51,160 --> 00:09:56,560
up in this community or immigrated to the community as adults with young children. And
115
00:09:56,560 --> 00:10:01,280
so they have those shared lived experiences with the participants in our programs. And
116
00:10:01,280 --> 00:10:07,360
many of our staff have actually been former participants in our programs. Having the entire
117
00:10:07,360 --> 00:10:13,560
team participate in this process is incredibly valuable. It really places the experts in
118
00:10:13,560 --> 00:10:20,520
the work, our staff in the evaluator seat. And we gain so much more by having that inclusive
119
00:10:20,520 --> 00:10:26,380
participatory process. And we're really so grateful to have our work shared in the online
120
00:10:26,380 --> 00:10:32,560
toolkit because, you know, I am not an expert in the ideas framework by any means. And that's
121
00:10:32,560 --> 00:10:37,320
kind of the whole point is that it's very usable. You can go in, you can click through
122
00:10:37,320 --> 00:10:42,160
this toolkit, you can see how it all is structured and works. And it just makes for a really
123
00:10:42,160 --> 00:10:51,560
kind of manageable, useful process that you can engage in. Thanks so much.
124
00:10:51,560 --> 00:10:56,620
Thank you, Carla and Sally. Really appreciate you taking the time to share your experiences
125
00:10:56,620 --> 00:11:03,880
with us. Now we'll hear from folks from Raising a Reader. So Raising a Reader supports families
126
00:11:03,880 --> 00:11:08,960
to build, practice and grow reading routines at home. Their award-winning evidence-based
127
00:11:08,960 --> 00:11:14,560
program helps caring adults set their children up for success by creating shared reading
128
00:11:14,560 --> 00:11:19,280
routines, fostering social-emotional learning, healthy family relationships and learning
129
00:11:19,280 --> 00:11:25,360
skills needed to thrive in school and beyond. And first, we'll hear from Michelle Siosenheiman,
130
00:11:25,360 --> 00:11:31,120
who's Senior Vice President, Program and Partnerships. And in her role, Michelle is responsible for
131
00:11:31,120 --> 00:11:36,520
overseeing program development, growth and impact. And then we'll hear from Andres Garcia
132
00:11:36,520 --> 00:11:41,840
Lopez, who's a Senior Project Manager at the Center on the Developing Child. And in his
133
00:11:41,840 --> 00:11:48,360
role, he's coached many early childhood development entrepreneurs, including Raising a Reader,
134
00:11:48,360 --> 00:11:53,560
in developing strategies to maintain their science-based impact while scaling their ventures.
135
00:11:53,560 --> 00:11:59,240
Welcome, Michelle and Andres. Thank you so much for having me. I'll start
136
00:11:59,240 --> 00:12:04,200
with a brief overview of Raising a Reader and how we've used the framework. And then
137
00:12:04,200 --> 00:12:09,640
Andres and I will engage in some conversation. So Raising a Reader is a national family engagement
138
00:12:09,640 --> 00:12:13,680
and early literacy organization through our network of affiliates and partners across
139
00:12:13,680 --> 00:12:18,920
34 states in both rural and urban communities. We engage and support parents and the other
140
00:12:18,920 --> 00:12:24,120
caring adults in children's lives, help strengthen the bonds with their children while building
141
00:12:24,120 --> 00:12:30,120
critical early reading and social-emotional skills. So along with our award-winning multicultural
142
00:12:30,120 --> 00:12:34,840
and multilingual book collection, we provide easy-to-use materials and guides that are
143
00:12:34,840 --> 00:12:39,920
really designed to make the most of that shared reading time in the home. So our work really
144
00:12:39,920 --> 00:12:44,200
does begin, though, with partnering with local agencies who become members of our affiliate
145
00:12:44,200 --> 00:12:49,920
network, a community of practice in which we can share best practices and build connections.
146
00:12:49,920 --> 00:12:54,600
And we provide professional development, technical assistance, and capacity-building support
147
00:12:54,600 --> 00:12:59,760
to this network of affiliates and partners who really work across the intersection of
148
00:12:59,760 --> 00:13:04,520
systems, supporting children and families at the various points throughout their educational
149
00:13:04,520 --> 00:13:11,480
and developmental journey. So that's, you know, in ECE, K-12, Health and Human Services.
150
00:13:11,480 --> 00:13:17,120
And we're really able to meet families in the spaces and places where they are involved.
151
00:13:17,120 --> 00:13:21,840
And how the framework has really impacted our work was that we were introduced to Andres
152
00:13:21,840 --> 00:13:28,760
and the framework at a really critical inflection point in our history. So we're over 23 years
153
00:13:28,760 --> 00:13:33,800
old, and Raising Reader had 39 independent evaluations that proved the success of our
154
00:13:33,800 --> 00:13:38,800
classic Red Book Bag program and its impact on improving and sustaining home literacy
155
00:13:38,800 --> 00:13:43,440
environments. But one thing that we realized through our work with Andres was that there
156
00:13:43,440 --> 00:13:47,720
were critical aspects to our work that we weren't capturing in our theory of change.
157
00:13:47,720 --> 00:13:53,640
Andres, maybe I'll stop there and then we can chat. Does that work?
158
00:13:53,640 --> 00:13:58,920
Sounds good. That works, Michelle. Thanks so much for that overview. And I'm so excited
159
00:13:58,920 --> 00:14:04,600
to be part of this panel. And it's an honor to share it with you, Michelle, and with the
160
00:14:04,600 --> 00:14:09,840
Valley Settlement team. So I'll just add a few things. I was working with Michelle as
161
00:14:09,840 --> 00:14:21,000
part of a fellowship that the Center partnered with the Promise Venture Studio. And as it
162
00:14:21,000 --> 00:14:26,920
was mentioned before, the theory of change on the ideas framework really helps you think
163
00:14:26,920 --> 00:14:34,440
about what are the key ingredients that my organization, my program, works on or provides
164
00:14:34,440 --> 00:14:41,380
to families and or maybe to partners that get to the targets that move the needle towards
165
00:14:41,380 --> 00:14:46,000
my outcomes. One thing that was different about Raising Reader was that they work with
166
00:14:46,000 --> 00:14:51,440
partners. So I wanted to mention that sometimes the ideas framework can be and the theory
167
00:14:51,440 --> 00:14:58,080
of change could be flexible and adaptable to meet your needs. Originally, there are
168
00:14:58,080 --> 00:15:02,520
three columns in the theory of change. But we're working with Michelle, we thought we
169
00:15:02,520 --> 00:15:07,960
should have an extra column because they wanted to look at how working with partners and affiliate
170
00:15:07,960 --> 00:15:12,560
organizations, what the strategies that we're doing raising a reader was getting to the
171
00:15:12,560 --> 00:15:19,440
targets in partners and affiliates and how that was getting to the outcomes with families.
172
00:15:19,440 --> 00:15:25,440
And that was a key component on identifying precisely the actions that get to the targets
173
00:15:25,440 --> 00:15:29,480
and to the outcomes. And I'll pause there so that Michelle can share more about the
174
00:15:29,480 --> 00:15:35,640
specifics of what some of those strategies were and how that helped the organization.
175
00:15:35,640 --> 00:15:41,360
Thanks, Andres. Yeah, so one of the things, including that additional column that Andres
176
00:15:41,360 --> 00:15:46,680
was talking about, it really helped us think through how are we really building that educator
177
00:15:46,680 --> 00:15:52,760
capacity and how are we really providing professional development around early childhood development.
178
00:15:52,760 --> 00:15:57,680
And another aspect to it is that we knew we did it all the time and we had stories about
179
00:15:57,680 --> 00:16:02,380
how we did it all the time. But using the theory of change, the framework to really
180
00:16:02,380 --> 00:16:06,560
make it much more precise is really helping us think through how are we doing it. And
181
00:16:06,560 --> 00:16:12,040
so it also helped us think about how our programming is impacting early relational health through
182
00:16:12,040 --> 00:16:17,680
strengthening healthy family bonds. And so it really has made us to be better poised
183
00:16:17,680 --> 00:16:22,920
to effectively test and evaluate how we are doing this work and what is and what is not
184
00:16:22,920 --> 00:16:26,960
working. Thank you, Michelle. And one comment that I
185
00:16:26,960 --> 00:16:32,680
add as we have about 90% of the participants that are now in the webinar are new to the
186
00:16:32,680 --> 00:16:40,480
framework. Sometimes you may use the framework as a program developer or somebody who is
187
00:16:40,480 --> 00:16:46,360
implementing a program like Michelle, but you could also use it to help other organizations
188
00:16:46,360 --> 00:16:51,360
like the way I have used it as part of the Center on the Developing Child or as Promise
189
00:16:51,360 --> 00:16:56,560
Venture Studio has also used it with social entrepreneurs in their organization. The framework
190
00:16:56,560 --> 00:17:02,600
is really helpful in helping you think through your impact strategy. I'll mention a brief
191
00:17:02,600 --> 00:17:08,520
comment if you're familiar with other frameworks that innovators use like the Lean Startup
192
00:17:08,520 --> 00:17:15,800
or Business Model Canvas or other ones. These helps you think through in a very clear way,
193
00:17:15,800 --> 00:17:22,880
in a simplified way, what are your strategies and how am I getting to the outcomes. And
194
00:17:22,880 --> 00:17:28,760
because it's simple and it can fit in one page, it also facilitates communication, communicating
195
00:17:28,760 --> 00:17:34,840
internally and externally with the families you work with, with the potential funders
196
00:17:34,840 --> 00:17:40,680
and with potential partners. But I'll pause there.
197
00:17:40,680 --> 00:17:44,840
One more thing I just wanted to add about the framework is how it really helped us think
198
00:17:44,840 --> 00:17:50,560
about our innovations and new programming too into our theory of change and help facilitate
199
00:17:50,560 --> 00:17:55,560
that fast cycle iteration because over the last few years, we've developed two new programs
200
00:17:55,560 --> 00:18:01,160
and explored how we were successful in exploring how we can integrate technology into our programming.
201
00:18:01,160 --> 00:18:05,520
And we didn't have that in our previous theory of change. And so being able to build that
202
00:18:05,520 --> 00:18:10,600
into using the framework to build that into our theory of change, thinking about the evaluation,
203
00:18:10,600 --> 00:18:17,160
how do we get that feedback loop? It was really beneficial and helpful for us as we're continuing
204
00:18:17,160 --> 00:18:20,680
to innovate and develop new programs too.
205
00:18:20,680 --> 00:18:24,420
Thank you, Michelle.
206
00:18:24,420 --> 00:18:29,200
Thank you so much for sharing your experiences. And we're going to go ahead and pull some
207
00:18:29,200 --> 00:18:34,600
questions off chat. And I thought maybe we'll start with kind of a somewhat broad question
208
00:18:34,600 --> 00:18:41,220
and that came from Karen Soto is that are there any prerequisites for an organization
209
00:18:41,220 --> 00:18:46,520
to implement the ideas framework? And I know Melanie might want to speak to this.
210
00:18:46,520 --> 00:18:55,360
Sure. I would say there aren't necessarily any standard prerequisites, but I do think
211
00:18:55,360 --> 00:19:00,920
having worked with a lot of different organizations around this framework, there are some conditions
212
00:19:00,920 --> 00:19:07,680
that set you up to be more or less successful or effective using the framework. One thing
213
00:19:07,680 --> 00:19:13,560
I would say is it's important to have all the right people at the table. So I mentioned
214
00:19:13,560 --> 00:19:18,720
that one of the principles of the framework is co-creation and this idea that bringing
215
00:19:18,720 --> 00:19:22,840
together a group of people who have multiple perspectives on the program can be really
216
00:19:22,840 --> 00:19:28,660
valuable. And that might include leadership, people who are responsible for developing
217
00:19:28,660 --> 00:19:35,760
or implementing the program, people who will lead on the evaluation or research efforts.
218
00:19:35,760 --> 00:19:43,840
But equally importantly, you might invite people to contribute who have a role in actually
219
00:19:43,840 --> 00:19:49,200
delivering the service, working directly with kids, families and caregivers, and better
220
00:19:49,200 --> 00:19:55,020
yet invite a representative from the community that you serve to be part of this process.
221
00:19:55,020 --> 00:19:59,120
So that's the first piece is just having the right people at the table. And then the second
222
00:19:59,120 --> 00:20:04,240
thing I would say is timing can be important. So this framework is really designed to help
223
00:20:04,240 --> 00:20:12,400
you prepare for a fast cycle iteration process. So to prepare for a round of collecting data,
224
00:20:12,400 --> 00:20:17,420
reviewing that data, interpreting and analyzing it, making sense of it, and putting what you
225
00:20:17,420 --> 00:20:23,520
learn into practice. So the timing there can be important. Are you set up and prepared
226
00:20:23,520 --> 00:20:27,120
to actually put this plan into action? Do you have the resources you need as everyone
227
00:20:27,120 --> 00:20:34,080
bought in, et cetera? Yeah. And having the authority to put what you learn into practice.
228
00:20:34,080 --> 00:20:40,360
So if you're implementing a program that was developed by someone else, do you have sort
229
00:20:40,360 --> 00:20:44,960
of the leeway to make changes to how you're implementing that program based on what you
230
00:20:44,960 --> 00:20:51,000
learned? Or are there kind of more strict parameters around how you implement that program?
231
00:20:51,000 --> 00:20:58,400
Thanks, Melanie. The next question we have came from Nicholas, and it says, this is a
232
00:20:58,400 --> 00:21:04,720
question from Ms. Reyes from Albucieto. Were there any outcomes or benefits that happened
233
00:21:04,720 --> 00:21:10,880
unexpectedly from developing this program, i.e. unintended consequences that happened
234
00:21:10,880 --> 00:21:17,880
which you did not expect? Yes. So there, it's definitely been a learning curve. We've definitely
235
00:21:17,880 --> 00:21:24,140
had to modify and just evolve the program. One of the biggest changes that we've made
236
00:21:24,140 --> 00:21:30,600
just recently is changing the program from a five-hour week model where children receive
237
00:21:30,600 --> 00:21:38,240
two and a half hours of preschool twice a week to offering five hours of preschool twice
238
00:21:38,240 --> 00:21:44,720
a week. So in total, they're receiving 10 hours a week. And this really came from listening
239
00:21:44,720 --> 00:21:52,040
and taking the time to listen to parents and hear what their needs were. For years, our
240
00:21:52,040 --> 00:21:57,400
parents have been asking for more time on the bus. We're really trying to make an impact
241
00:21:57,400 --> 00:22:04,320
on how many children we served. Like I said, we have a capacity to serve 96 children in
242
00:22:04,320 --> 00:22:10,000
our valley. So that's 96 children that otherwise wouldn't be receiving preschool, you know,
243
00:22:10,000 --> 00:22:15,360
in a traditional preschool setting. And we've noticed recently we've had a decrease in our
244
00:22:15,360 --> 00:22:21,180
enrollment. So it's been a little bit harder to enroll children into our shorter classes.
245
00:22:21,180 --> 00:22:27,180
And I think that has now impacted our school district and our other centers that have grown
246
00:22:27,180 --> 00:22:33,380
their capacity in their centers, which was the ultimate goal to get more children into
247
00:22:33,380 --> 00:22:40,560
preschool and enrolled. So we've now looked at how we can, because we're mobile, we can
248
00:22:40,560 --> 00:22:47,880
now take our program and start serving communities that don't have that access. So it's been
249
00:22:47,880 --> 00:22:51,920
playing out lately that we've noticed these trends.
250
00:22:51,920 --> 00:22:58,520
Thank you. And actually, I just realized that this question is the way it got segmented
251
00:22:58,520 --> 00:23:03,400
in the question answer section. I didn't realize that the Nicholas who asked the question has
252
00:23:03,400 --> 00:23:08,380
the same question for folks at Raising a Reader. So were there any outcomes or benefits that
253
00:23:08,380 --> 00:23:12,360
happened unexpectedly from developing the program?
254
00:23:12,360 --> 00:23:19,760
Sure. So I'll say that we have had 39 independent evaluations that showed, and we knew that
255
00:23:19,760 --> 00:23:24,760
Raising a Reader helped improve home literacy environments, and which is like increased
256
00:23:24,760 --> 00:23:31,400
shared reading time, increased duration, and frequency, improved the number of books in
257
00:23:31,400 --> 00:23:36,600
the home. But one thing that we were hearing from folks was, oh, well, it's helping me
258
00:23:36,600 --> 00:23:42,080
build confidence in supporting my child's early learning in the home. It's really providing
259
00:23:42,080 --> 00:23:48,440
a sense of comfort and support for our families, this daily reading routine. And so creating
260
00:23:48,440 --> 00:23:53,200
a new theory of change with coaching support from Andres to really make much more precise.
261
00:23:53,200 --> 00:23:58,420
We were able to build those kinds of things into our theory of change, which then led
262
00:23:58,420 --> 00:24:03,640
us to improving our measurement tools. So asking specific questions so that we could
263
00:24:03,640 --> 00:24:08,360
actually get some more data around, well, the stories are great. It's great. It's also
264
00:24:08,360 --> 00:24:14,920
helpful to have our surveys also reflect some of that more quantitatively as well.
265
00:24:14,920 --> 00:24:23,240
Thank you, Michelle. And we have one question here that asks if someone could speak to how
266
00:24:23,240 --> 00:24:29,080
this theory of change framework can inform logic model use and development, ensuring
267
00:24:29,080 --> 00:24:34,540
that the information is complementary and not duplicative for programs who choose to
268
00:24:34,540 --> 00:24:37,720
create both types of resources.
269
00:24:37,720 --> 00:24:45,360
I'm happy to feel that, get asked that question a lot. From what I understand, those two terms,
270
00:24:45,360 --> 00:24:50,820
theory of change and logic model are often used actually interchangeably in the field.
271
00:24:50,820 --> 00:24:58,000
And there really isn't solid consensus on how the two are similar and different. The
272
00:24:58,000 --> 00:25:06,200
best guidance on that that I've found is that logic models tend to be more standardized.
273
00:25:06,200 --> 00:25:13,960
They often include inputs, activities, outputs, and then short term and long term outcomes
274
00:25:13,960 --> 00:25:21,880
or variations on that theme. And they're really, they're descriptive. Theories of change are
275
00:25:21,880 --> 00:25:28,000
a bit less standardized. So if you Google the term theory of change, you'll find many,
276
00:25:28,000 --> 00:25:34,200
many, many different approaches to theory of change. But in general, they're intended
277
00:25:34,200 --> 00:25:43,080
to be causal models that really explain how and why the expected changes come about. That's
278
00:25:43,080 --> 00:25:47,480
one way of thinking about that. Then logic models are more descriptive and theories of
279
00:25:47,480 --> 00:25:53,040
change are really intended to be explanatory causal models. Honestly, in practice, I find
280
00:25:53,040 --> 00:25:57,400
that they're, you know, when someone asks you for a logic model or a theory of change,
281
00:25:57,400 --> 00:26:02,160
you really have to follow up to ask what they mean specifically, what they're looking for
282
00:26:02,160 --> 00:26:08,360
there because I think expectations vary widely. Our approach to theory of change is really,
283
00:26:08,360 --> 00:26:16,440
it zooms in on the point of service delivery or the point of contact with kids, families,
284
00:26:16,440 --> 00:26:22,040
caregivers, and follows that through to the ultimate outcomes that you're hoping to see,
285
00:26:22,040 --> 00:26:26,720
which are typically child level in the field of early childhood. And it can be really helpful
286
00:26:26,720 --> 00:26:32,600
to set you up to make a plan for gathering data to better understand whether you're having
287
00:26:32,600 --> 00:26:37,080
the impact that you're hoping to achieve. I hope that's helpful.
288
00:26:37,080 --> 00:26:39,960
Sally, do you want to add to that?
289
00:26:39,960 --> 00:26:45,920
Yeah, so I actually have a real life example of how we've used both at the same time. So
290
00:26:45,920 --> 00:26:52,280
we recently worked with the team at Mathematica to create a two-gen logic model. They did
291
00:26:52,280 --> 00:26:57,840
a project with different two-gen organizations across the country. And so we have been using
292
00:26:57,840 --> 00:27:03,680
theories of change for years in our programs, in our six different programs, to really identify,
293
00:27:03,680 --> 00:27:08,200
you know, what is the, what are our strategies, what are the targets, what are the outcomes
294
00:27:08,200 --> 00:27:12,840
we're trying to have in each of our programs? So it's kind of granular. And then we worked
295
00:27:12,840 --> 00:27:20,040
with the team at Mathematica to create this, like, what does our whole organization do?
296
00:27:20,040 --> 00:27:24,760
And what is the whole change we're trying to make in the community and in children and
297
00:27:24,760 --> 00:27:31,840
families? And so having that overarching logic model that our theories of change then kind
298
00:27:31,840 --> 00:27:39,200
of feed up into, you can see how they interact, how they're incorporated in that larger logic
299
00:27:39,200 --> 00:27:44,320
model has been just really interesting. But we're not duplicating. So we don't have like
300
00:27:44,320 --> 00:27:48,240
a logic model and a theory of change for each of our programs.
301
00:27:48,240 --> 00:27:54,880
Does anybody else want to speak to that before I move to the next question? Okay. So we have
302
00:27:54,880 --> 00:28:00,280
a question here from Eric Marlowe and it asks, in your experience, I think this could be
303
00:28:00,280 --> 00:28:07,340
open to anyone here, how long are the typical iteration cycles? How long do you recommend
304
00:28:07,340 --> 00:28:11,560
evaluating and adapting elements of a given program so that changes are made neither too
305
00:28:11,560 --> 00:28:14,680
soon nor too late?
306
00:28:14,680 --> 00:28:19,180
I can take a first pass, but then I'd be curious to hear from our colleagues at Raising a Reader
307
00:28:19,180 --> 00:28:24,240
and Valley Settlement. If I understood correctly, Eshna, was the question like how often or
308
00:28:24,240 --> 00:28:27,560
how long does it take or maybe a little bit of both?
309
00:28:27,560 --> 00:28:30,040
My interpretation was a little bit of both.
310
00:28:30,040 --> 00:28:36,920
Okay. I think so the way this question gets asked to us often is how fast is fast cycle?
311
00:28:36,920 --> 00:28:42,360
Like, are we talking something you can do in days, weeks, months, years? And the answer
312
00:28:42,360 --> 00:28:49,760
there I think is, unfortunately, it depends. It really depends on the nature of the program
313
00:28:49,760 --> 00:28:56,120
or service or product that you're looking to evaluate and improve. If it's, let's say,
314
00:28:56,120 --> 00:29:04,340
a 10 week parenting group, then a single cycle could take, you know, you'd want a couple
315
00:29:04,340 --> 00:29:09,840
of months to plan and prepare to identify, to develop your theory of change, to identify
316
00:29:09,840 --> 00:29:15,120
your questions, to figure out your study design, figure out the tools you want to use, prepare
317
00:29:15,120 --> 00:29:21,920
for data collection. Then obviously you need the 10 weeks to go by where you're actually
318
00:29:21,920 --> 00:29:27,840
delivering the program to kids and families. And then you'll need some time afterwards
319
00:29:27,840 --> 00:29:33,680
to analyze, interpret, and make sense of that data. But that timeframe is obviously going
320
00:29:33,680 --> 00:29:37,760
to be really different if it's a program that's implemented on a school year calendar, for
321
00:29:37,760 --> 00:29:43,360
instance, or if you've developed something like an app or a website that families can
322
00:29:43,360 --> 00:29:50,400
engage with as they choose. And maybe dosage and engagement looks really different from
323
00:29:50,400 --> 00:29:57,040
parent to parent. So there's no right answer for how long a cycle can take. You really
324
00:29:57,040 --> 00:30:01,720
just need to be thoughtful about what you're hoping to learn. And then in terms of how
325
00:30:01,720 --> 00:30:07,880
often, I think that really varies again from organization to organization and what the
326
00:30:07,880 --> 00:30:13,320
appetite and bandwidth is to engage in this kind of iterative learning. I know that Valley
327
00:30:13,320 --> 00:30:18,800
Settlement, for instance, has really built this into your kind of culture and your routines
328
00:30:18,800 --> 00:30:23,840
as an organization. And it seems like you've developed a really nice kind of annual rhythm.
329
00:30:23,840 --> 00:30:28,040
So maybe I could pass the baton to Sally and Carla to talk more about that.
330
00:30:28,040 --> 00:30:34,180
Yeah. So as you say, Melanie, we really do our evaluation on an annual basis. So most
331
00:30:34,180 --> 00:30:40,280
of our programs happen during the school year. We do pre and post surveys. We also do pre,
332
00:30:40,280 --> 00:30:46,920
mid and post TS Gold assessments for our Boosecito preschool. And then we really, we do the bulk
333
00:30:46,920 --> 00:30:53,440
of data analysis in June and July. And then every July we gather together and do that
334
00:30:53,440 --> 00:30:58,880
evaluation retreat with our entire team. And then in August, we're kind of refining our
335
00:30:58,880 --> 00:31:04,880
evaluation and planning the next evaluation cycle. But what I will say is that when we're
336
00:31:04,880 --> 00:31:12,200
piloting new initiatives, we are trying to be a little more like eyes on as the initiative
337
00:31:12,200 --> 00:31:17,360
is happening. So for example, we worked a few years ago to implement a child development
338
00:31:17,360 --> 00:31:22,580
associate course for family, friends and neighbors providers, and also for high school students
339
00:31:22,580 --> 00:31:28,240
who are Spanish speaking. And we were, we were doing little pre and post assessments
340
00:31:28,240 --> 00:31:33,880
throughout at the beginning and end of kind of each section of the course or module, just
341
00:31:33,880 --> 00:31:38,320
to understand like, was our approach working? How could we pivot and adjust? So as we're
342
00:31:38,320 --> 00:31:44,560
piloting new things, we do try to be a little more rapid, if you will.
343
00:31:44,560 --> 00:31:49,440
I would just echo the same thing that we do have an annual, like an annual cycle, I guess,
344
00:31:49,440 --> 00:31:53,920
where we do an annual evaluation, annual check in with our affiliates is what we call it.
345
00:31:53,920 --> 00:31:57,880
And then we do have a couple of pilots right now where we are calling them mini learning
346
00:31:57,880 --> 00:32:03,560
phases. After each learning phase, then we'll take a look at the feedback and then see what
347
00:32:03,560 --> 00:32:07,840
tweaks or what modifications we need to do to improve the program. And then we have our
348
00:32:07,840 --> 00:32:15,160
second learning phase and things like that. A quick follow up from Nicholas to this question
349
00:32:15,160 --> 00:32:19,520
was, so does that mean that the theory of change is different with each iteration?
350
00:32:19,520 --> 00:32:25,800
So we definitely like, we can evolve our theory of change every year, we go in as a team and
351
00:32:25,800 --> 00:32:30,120
look at the strategies and say like, you know, are you still doing these things? How are
352
00:32:30,120 --> 00:32:35,280
you doing them? You know, we definitely change and evolve in our programming. We're not doing
353
00:32:35,280 --> 00:32:40,720
the same thing every year because programs, neighborhoods, communities change and evolve
354
00:32:40,720 --> 00:32:46,480
and we learn new things every year. So we do go in and tweak our theory of change every
355
00:32:46,480 --> 00:32:49,160
year.
356
00:32:49,160 --> 00:32:55,600
We just revised ours, so we haven't changed it. But one thing I think it is helping us
357
00:32:55,600 --> 00:33:00,640
think about though, as we are developing our program, how to stay focused on what it is
358
00:33:00,640 --> 00:33:05,480
that we really want to do, because there are so many needs and we serve so many different
359
00:33:05,480 --> 00:33:12,400
communities that have different needs. How do we stay true to this, the theory of change
360
00:33:12,400 --> 00:33:17,280
that for our mission and things like that. So it really helps us identify our clear lanes,
361
00:33:17,280 --> 00:33:22,440
like where is our most unique impact and helps us stay there and not kind of stray just because
362
00:33:22,440 --> 00:33:26,400
there might be a funding source over there or something like that.
363
00:33:26,400 --> 00:33:34,420
I'll just add that it's a great question and it gives us a chance to underscore that absolutely
364
00:33:34,420 --> 00:33:40,320
a theory of change can and should evolve over time as you learn. So we call them living
365
00:33:40,320 --> 00:33:47,160
documents. It's not a one-time exercise that you do and you make a PDF of it and it's done.
366
00:33:47,160 --> 00:33:53,000
It's a living document that you come back to after every round of learning and you say,
367
00:33:53,000 --> 00:33:57,960
what did we learn? How can we refine our theory of change? How can we refine our actual program
368
00:33:57,960 --> 00:34:04,160
or product or service? And how could we refine how we're gathering data and learning going
369
00:34:04,160 --> 00:34:09,880
forward? Thank you. So now we have a question from
370
00:34:09,880 --> 00:34:18,400
Megan Crystal asking if we have any examples, if we know of any state level policy or programs
371
00:34:18,400 --> 00:34:26,880
that have used this framework. I have a couple that come to mind. So a while
372
00:34:26,880 --> 00:34:33,020
back as part of the Frontiers of Innovation initiative that Ashna mentioned, there was
373
00:34:33,020 --> 00:34:41,320
a project team who implemented a video coaching program to support childcare and early learning
374
00:34:41,320 --> 00:34:47,920
professionals and that project was done in partnership with the State Department of Early
375
00:34:47,920 --> 00:34:55,400
Learning and this framework was used to sort of articulate theory of change for that approach
376
00:34:55,400 --> 00:35:01,060
and to actually work with partners at the University of Washington to craft the evaluation
377
00:35:01,060 --> 00:35:05,920
plan. That's one example. And then another more recent example that
378
00:35:05,920 --> 00:35:13,260
comes to mind is our center worked with partners at the Massachusetts Department of Early Education
379
00:35:13,260 --> 00:35:21,120
and Care to create an initiative wide theory of change, actually similarly for an effort
380
00:35:21,120 --> 00:35:29,040
underway to bring early care and support organizations to build the capacity of actually childcare
381
00:35:29,040 --> 00:35:33,280
directors across the state. So we worked with them to create an initiative
382
00:35:33,280 --> 00:35:40,200
wide theory of change and then each organization who is providing that service used that initiative
383
00:35:40,200 --> 00:35:45,340
wide theory of change as a template and kind of tailored it for their particular approach.
384
00:35:45,340 --> 00:35:49,520
And I think that's still being used right now as the initiative continues to kind of
385
00:35:49,520 --> 00:35:55,980
facilitate learning and improvement over time. There might be other examples, but those are
386
00:35:55,980 --> 00:35:59,400
two that come to mind. Thanks, Melanie. Does anybody else?
387
00:35:59,400 --> 00:36:04,160
Oh, can I squeeze in one more just in case folks are looking for examples? We worked
388
00:36:04,160 --> 00:36:10,560
with an organization called TOPS, which is based in the Netherlands and they provide,
389
00:36:10,560 --> 00:36:17,040
if I'm remembering correctly, services to families with newborn children. And I think
390
00:36:17,040 --> 00:36:24,480
that's actually like a nationwide program that's used ideas to drive towards greater
391
00:36:24,480 --> 00:36:30,840
impact. And there are resources or references about the TOPS program in the resources section
392
00:36:30,840 --> 00:36:36,080
of the toolkit. I think there's a research article there that talks about their experience.
393
00:36:36,080 --> 00:36:40,480
Great. Thanks, everyone. I was hoping we might have time for one more question, but I actually
394
00:36:40,480 --> 00:36:45,560
see that we're pretty close to the end there. So I just wanted to thank you all for joining
395
00:36:45,560 --> 00:36:53,120
us. Thank our panelists for sharing your experiences and your learning. We really appreciate it.
396
00:36:53,120 --> 00:36:58,500
And thank you all so much for joining. We really, really hope that this resource is
397
00:36:58,500 --> 00:37:03,560
useful for you all. And we wish you the best in your continued work supporting kids and
398
00:37:03,560 --> 00:37:08,560
families.
399
00:37:08,560 --> 00:37:13,040
The Brain Architects is a product of the Center on the Developing Child at Harvard University.
400
00:37:13,040 --> 00:37:17,660
You can find us at developingchild.harvard.edu, where we'll post any resources that were
401
00:37:17,660 --> 00:37:23,080
discussed in this episode. We're also on Twitter at Harvard Center, Facebook at Center on the
402
00:37:23,080 --> 00:37:26,040
Developing Child, and Instagram at Developing Child Harvard.
00:00:00,000 --> 00:00:07,600
Welcome to The Brain Architects, a podcast from the Center on the Developing Child at
2
00:00:07,600 --> 00:00:08,600
Harvard University.
3
00:00:08,600 --> 00:00:12,440
I'm Corrie Zimmerman, the Center's Chief Program Officer.
4
00:00:12,440 --> 00:00:16,880
Our Center believes that advances in the science of child development provide a powerful source
5
00:00:16,880 --> 00:00:20,920
of new ideas that can improve outcomes for children and their caregivers.
6
00:00:20,920 --> 00:00:25,440
By sharing the latest science from the field, we hope to help you make that science actionable
7
00:00:25,440 --> 00:00:29,400
and apply it in your work in ways that can increase your impact.
8
00:00:29,400 --> 00:00:33,160
With that goal in mind, the Center recently released the Ideas Impact Framework Toolkit.
9
00:00:33,160 --> 00:00:38,480
It's a free, online resource designed to help innovators in the field of early childhood
10
00:00:38,480 --> 00:00:42,600
build improved programs and products that are positioned to achieve greater impact in
11
00:00:42,600 --> 00:00:43,600
their communities.
12
00:00:43,600 --> 00:00:48,640
This toolkit is self-guided, self-paced, and provides a structured and flexible approach
13
00:00:48,640 --> 00:00:53,160
that facilitates program development, evaluation, and fast-cycle iteration.
14
00:00:53,160 --> 00:00:57,000
It includes resources to help teams develop and investigate a clear and precise theory
15
00:00:57,000 --> 00:00:58,000
of change.
16
00:00:58,000 --> 00:01:03,160
In April, we hosted a webinar about the toolkit, where we provided an overview of the site
17
00:01:03,160 --> 00:01:06,960
and had the opportunity to hear from teams at several organizations in the field about
18
00:01:06,960 --> 00:01:10,560
how they leverage the toolkit and its resources to shape their work.
19
00:01:10,560 --> 00:01:14,040
We're excited to share those discussions with you here on this episode of the Brain
20
00:01:14,040 --> 00:01:16,000
Architects podcast.
21
00:01:16,000 --> 00:01:19,880
If you're interested in hearing a full walkthrough of the toolkit by the director of our Pediatric
22
00:01:19,880 --> 00:01:24,400
Innovation Initiative, Dr. Melanie Berry, please head over to our YouTube channel to view the
23
00:01:24,400 --> 00:01:25,920
full webinar recording.
24
00:01:25,920 --> 00:01:30,200
You'll also hear from Dr. Melanie Berry during the Q&A portion.
25
00:01:30,200 --> 00:01:38,080
The full ideas toolkit we'll be talking about today can be found at ideas.developingchild.harvard.edu.
26
00:01:38,080 --> 00:01:42,800
And now, without further ado, here's Dr. Eshna Badruzman, the center's senior project manager
27
00:01:42,800 --> 00:01:47,000
for instructional design and the moderator for our panel discussion.
28
00:01:47,000 --> 00:01:50,120
Hello, everyone.
29
00:01:50,120 --> 00:01:52,000
Welcome.
30
00:01:52,000 --> 00:01:54,160
My name is Dr. Eshna Badruzman.
31
00:01:54,160 --> 00:01:59,320
I am a senior project manager for instructional design at the Center on the Developing Child
32
00:01:59,320 --> 00:02:02,560
at Harvard University, or HCDC.
33
00:02:02,560 --> 00:02:07,520
And I'm part of the development team of the Ideas Impact Framework Toolkit.
34
00:02:07,520 --> 00:02:09,240
And today, I'll be your host.
35
00:02:09,240 --> 00:02:13,720
So you may hear me come off mute and help guide presenters.
36
00:02:13,720 --> 00:02:17,500
And I'll be facilitating our question and answer period.
37
00:02:17,500 --> 00:02:22,480
So we are so pleased to be talking to you today about this resource.
38
00:02:22,480 --> 00:02:27,560
The Ideas Impact Framework was born out of more than a decade of the Frontiers of Innovation
39
00:02:27,560 --> 00:02:30,200
Initiative, or FOI.
40
00:02:30,200 --> 00:02:33,860
And some of you may have been partners in that effort.
41
00:02:33,860 --> 00:02:39,600
So while our team no longer offers live training on the framework, we are so excited to be
42
00:02:39,600 --> 00:02:43,600
introducing it to you as a free open access resource.
43
00:02:43,600 --> 00:02:49,320
And we really hope that this format is going to help make ideas accessible to innovators
44
00:02:49,320 --> 00:02:53,120
in the field of early childhood development moving forward.
45
00:02:53,120 --> 00:02:57,440
The framework was developed in partnership with the University of Washington College
46
00:02:57,440 --> 00:03:02,240
of Education and the University of Oregon Center for Translational Neuroscience with
47
00:03:02,240 --> 00:03:08,560
support from the Gates Foundation, the LEGO Foundation, Horticus, and the Hemera Foundation.
48
00:03:08,560 --> 00:03:15,240
I encourage you to check out our history and acknowledgments page of the toolkit for more
49
00:03:15,240 --> 00:03:21,920
information about our various collaborations and supporters throughout time as well.
50
00:03:21,920 --> 00:03:29,080
Now I'll go ahead and introduce our first set of speakers from folks at Valley Settlement.
51
00:03:29,080 --> 00:03:34,720
We have with us Carla Reyes, who is the program manager of the El Paso Cito Mobile Preschool
52
00:03:34,720 --> 00:03:41,120
Program at Valley Settlement, which is a nonprofit that works to create opportunities for the
53
00:03:41,120 --> 00:03:46,440
Latino community in the Aspen to Parishute region of Colorado. Carla joined Valley Settlement
54
00:03:46,440 --> 00:03:53,680
in March 2015 as a preschool teacher for El Paso Cito until June 2021 when she took on
55
00:03:53,680 --> 00:03:59,880
a leadership role. And we also have Sally Boatten, who is the director of development
56
00:03:59,880 --> 00:04:04,840
and communications at Valley Settlement, a nonprofit again serving the rural Aspen to
57
00:04:04,840 --> 00:04:12,040
Parishute region of Colorado with six two-generation programs designed by and for local Latino
58
00:04:12,040 --> 00:04:17,480
immigrant families. And Sally has been with Valley Settlement for over five years and
59
00:04:17,480 --> 00:04:23,640
began managing the organization's evaluation function in 2021. Thank you so much Carla
60
00:04:23,640 --> 00:04:28,200
and Sally, look forward to hearing from you.
61
00:04:28,200 --> 00:04:36,320
Good afternoon everyone. Thank you for inviting us to share our work with you all and how
62
00:04:36,320 --> 00:04:41,040
we have used the framework. I'm going to talk a little bit about the El Paso Cito program
63
00:04:41,040 --> 00:04:49,520
and how it kind of started. The idea of El Paso Cito began in 2011. We had two bilingual
64
00:04:49,520 --> 00:04:57,240
and bicultural community organizers who met one-on-one with about 300 families from the
65
00:04:57,240 --> 00:05:04,720
Aspen to Parishute region of Colorado and they learned about their lives and the barriers
66
00:05:04,720 --> 00:05:09,760
that they faced within our community. One of the findings from the initial listening
67
00:05:09,760 --> 00:05:16,640
tour was that only 1% of Latino children in our community were enrolled in preschool.
68
00:05:16,640 --> 00:05:22,100
We also learned that three of the biggest barriers for families to participate in preschool
69
00:05:22,100 --> 00:05:28,640
programs were language, cost, and transportation resulting in lack of access. Now we have all
70
00:05:28,640 --> 00:05:34,200
this information and we started thinking creatively of different ways that we could bring more
71
00:05:34,200 --> 00:05:41,760
access to preschool education to our community. El Paso Cito was one of the first two-generation
72
00:05:41,760 --> 00:05:48,320
programs that we launched in Valley Settlement to address the needs of preschool education.
73
00:05:48,320 --> 00:05:53,520
And throughout the years, Valley Settlement has continued to learn, evolve, and co-design
74
00:05:53,520 --> 00:06:00,160
programming to respond to community needs. Now our program has four mobile preschool
75
00:06:00,160 --> 00:06:06,360
buses that have been retrofitted into small preschool classrooms. We have two teachers
76
00:06:06,360 --> 00:06:12,520
on the bus and we serve eight children at a time. We provide families with about five
77
00:06:12,520 --> 00:06:20,240
to ten hours of free preschool education. We have about 96 children that we serve annually,
78
00:06:20,240 --> 00:06:27,040
between 40 to 50 children graduating at the end of the school year and moving on to kindergarten.
79
00:06:27,040 --> 00:06:33,320
And currently right now we serve five different neighborhoods within our community and we
80
00:06:33,320 --> 00:06:41,320
strive to build close relationships with families. So our program really is designed to meet
81
00:06:41,320 --> 00:06:49,440
families at where they are at and start breaking down those barriers. We host family nights,
82
00:06:49,440 --> 00:06:55,160
we have home visits with our families, we have parent-teacher conferences, we have different
83
00:06:55,160 --> 00:07:01,360
ways that families can volunteer within our program. We provide a lot of materials for
84
00:07:01,360 --> 00:07:06,120
families to use at home so that they can do home activities and homework packets with
85
00:07:06,120 --> 00:07:12,240
their students and we really try to engage with the families. So each one of our teachers
86
00:07:12,240 --> 00:07:19,200
speaks Spanish, is bilingual and bicultural, so this really allows that bond and that relationship
87
00:07:19,200 --> 00:07:24,880
to build with each one of our families. I'm going to hand it off to Sally Bowen and she's
88
00:07:24,880 --> 00:07:29,440
going to talk a little bit more about how we've used the framework.
89
00:07:29,440 --> 00:07:34,760
Thanks Carla. So several years ago we started working with the team at Frontiers of Innovation
90
00:07:34,760 --> 00:07:40,880
to refine and evolve our evaluation practices. This work included creating theories of change
91
00:07:40,880 --> 00:07:45,520
for each of our programs, researching and recommending observational assessments to
92
00:07:45,520 --> 00:07:51,700
measure participant progress towards our program targets and outcomes, and creating implementation
93
00:07:51,700 --> 00:07:56,560
guides for our programs to detail the kind of critical components of our work and ensure
94
00:07:56,560 --> 00:08:01,760
that future staff can implement programs with fidelity while still continuing to listen
95
00:08:01,760 --> 00:08:08,000
to and evolve alongside families. Since the early days of our programming, Valley Settlement
96
00:08:08,000 --> 00:08:13,160
has invested in evaluation to measure, understand, and strengthen the changes that children and
97
00:08:13,160 --> 00:08:18,520
families create in their lives through our programming. Working with the team at FOI
98
00:08:18,520 --> 00:08:23,080
really brought this to the next level. Over the last few years, we've been working to
99
00:08:23,080 --> 00:08:28,880
be more inclusive and participatory in our evaluation process. So now our entire staff
100
00:08:28,880 --> 00:08:34,800
gathers for three days every summer in what I call an evaluation retreat where we review
101
00:08:34,800 --> 00:08:40,400
our annual program data as a team and then try to answer those questions. Que, por que,
102
00:08:40,400 --> 00:08:46,920
y ahora que? Or as the toolkit outlines, what? What does the data say? Why? Why might the
103
00:08:46,920 --> 00:08:53,120
data say this? Or what does it mean? And finally, now what? What do we do to tweak or change
104
00:08:53,120 --> 00:08:58,840
in our evaluation approach or in our programming based on what we see in the data?
105
00:08:58,840 --> 00:09:04,960
Our teams then create action plans to outline those changes that they want to make. We're
106
00:09:04,960 --> 00:09:11,640
usually tweaking one or more program components for the upcoming year. On day two of the retreat,
107
00:09:11,640 --> 00:09:17,240
teams then go in and refine and evolve their theories of change. So we really see the theory
108
00:09:17,240 --> 00:09:24,240
of change as a living document that breathes and grows alongside our programming. They
109
00:09:24,240 --> 00:09:28,800
identify what targets and outcomes they're interested in measuring for the coming program
110
00:09:28,800 --> 00:09:34,720
year. And then after that evaluation retreat, we work together with an evaluation consultant
111
00:09:34,720 --> 00:09:40,760
to refine our measurement tools. And then I always try to call out and highlight that
112
00:09:40,760 --> 00:09:46,440
I am in the minority at Valley Settlements. Our staff are largely of the community that
113
00:09:46,440 --> 00:09:51,160
we work with. Most of our staff are immigrants or children of immigrants. Many have grown
114
00:09:51,160 --> 00:09:56,560
up in this community or immigrated to the community as adults with young children. And
115
00:09:56,560 --> 00:10:01,280
so they have those shared lived experiences with the participants in our programs. And
116
00:10:01,280 --> 00:10:07,360
many of our staff have actually been former participants in our programs. Having the entire
117
00:10:07,360 --> 00:10:13,560
team participate in this process is incredibly valuable. It really places the experts in
118
00:10:13,560 --> 00:10:20,520
the work, our staff in the evaluator seat. And we gain so much more by having that inclusive
119
00:10:20,520 --> 00:10:26,380
participatory process. And we're really so grateful to have our work shared in the online
120
00:10:26,380 --> 00:10:32,560
toolkit because, you know, I am not an expert in the ideas framework by any means. And that's
121
00:10:32,560 --> 00:10:37,320
kind of the whole point is that it's very usable. You can go in, you can click through
122
00:10:37,320 --> 00:10:42,160
this toolkit, you can see how it all is structured and works. And it just makes for a really
123
00:10:42,160 --> 00:10:51,560
kind of manageable, useful process that you can engage in. Thanks so much.
124
00:10:51,560 --> 00:10:56,620
Thank you, Carla and Sally. Really appreciate you taking the time to share your experiences
125
00:10:56,620 --> 00:11:03,880
with us. Now we'll hear from folks from Raising a Reader. So Raising a Reader supports families
126
00:11:03,880 --> 00:11:08,960
to build, practice and grow reading routines at home. Their award-winning evidence-based
127
00:11:08,960 --> 00:11:14,560
program helps caring adults set their children up for success by creating shared reading
128
00:11:14,560 --> 00:11:19,280
routines, fostering social-emotional learning, healthy family relationships and learning
129
00:11:19,280 --> 00:11:25,360
skills needed to thrive in school and beyond. And first, we'll hear from Michelle Siosenheiman,
130
00:11:25,360 --> 00:11:31,120
who's Senior Vice President, Program and Partnerships. And in her role, Michelle is responsible for
131
00:11:31,120 --> 00:11:36,520
overseeing program development, growth and impact. And then we'll hear from Andres Garcia
132
00:11:36,520 --> 00:11:41,840
Lopez, who's a Senior Project Manager at the Center on the Developing Child. And in his
133
00:11:41,840 --> 00:11:48,360
role, he's coached many early childhood development entrepreneurs, including Raising a Reader,
134
00:11:48,360 --> 00:11:53,560
in developing strategies to maintain their science-based impact while scaling their ventures.
135
00:11:53,560 --> 00:11:59,240
Welcome, Michelle and Andres. Thank you so much for having me. I'll start
136
00:11:59,240 --> 00:12:04,200
with a brief overview of Raising a Reader and how we've used the framework. And then
137
00:12:04,200 --> 00:12:09,640
Andres and I will engage in some conversation. So Raising a Reader is a national family engagement
138
00:12:09,640 --> 00:12:13,680
and early literacy organization through our network of affiliates and partners across
139
00:12:13,680 --> 00:12:18,920
34 states in both rural and urban communities. We engage and support parents and the other
140
00:12:18,920 --> 00:12:24,120
caring adults in children's lives, help strengthen the bonds with their children while building
141
00:12:24,120 --> 00:12:30,120
critical early reading and social-emotional skills. So along with our award-winning multicultural
142
00:12:30,120 --> 00:12:34,840
and multilingual book collection, we provide easy-to-use materials and guides that are
143
00:12:34,840 --> 00:12:39,920
really designed to make the most of that shared reading time in the home. So our work really
144
00:12:39,920 --> 00:12:44,200
does begin, though, with partnering with local agencies who become members of our affiliate
145
00:12:44,200 --> 00:12:49,920
network, a community of practice in which we can share best practices and build connections.
146
00:12:49,920 --> 00:12:54,600
And we provide professional development, technical assistance, and capacity-building support
147
00:12:54,600 --> 00:12:59,760
to this network of affiliates and partners who really work across the intersection of
148
00:12:59,760 --> 00:13:04,520
systems, supporting children and families at the various points throughout their educational
149
00:13:04,520 --> 00:13:11,480
and developmental journey. So that's, you know, in ECE, K-12, Health and Human Services.
150
00:13:11,480 --> 00:13:17,120
And we're really able to meet families in the spaces and places where they are involved.
151
00:13:17,120 --> 00:13:21,840
And how the framework has really impacted our work was that we were introduced to Andres
152
00:13:21,840 --> 00:13:28,760
and the framework at a really critical inflection point in our history. So we're over 23 years
153
00:13:28,760 --> 00:13:33,800
old, and Raising Reader had 39 independent evaluations that proved the success of our
154
00:13:33,800 --> 00:13:38,800
classic Red Book Bag program and its impact on improving and sustaining home literacy
155
00:13:38,800 --> 00:13:43,440
environments. But one thing that we realized through our work with Andres was that there
156
00:13:43,440 --> 00:13:47,720
were critical aspects to our work that we weren't capturing in our theory of change.
157
00:13:47,720 --> 00:13:53,640
Andres, maybe I'll stop there and then we can chat. Does that work?
158
00:13:53,640 --> 00:13:58,920
Sounds good. That works, Michelle. Thanks so much for that overview. And I'm so excited
159
00:13:58,920 --> 00:14:04,600
to be part of this panel. And it's an honor to share it with you, Michelle, and with the
160
00:14:04,600 --> 00:14:09,840
Valley Settlement team. So I'll just add a few things. I was working with Michelle as
161
00:14:09,840 --> 00:14:21,000
part of a fellowship that the Center partnered with the Promise Venture Studio. And as it
162
00:14:21,000 --> 00:14:26,920
was mentioned before, the theory of change on the ideas framework really helps you think
163
00:14:26,920 --> 00:14:34,440
about what are the key ingredients that my organization, my program, works on or provides
164
00:14:34,440 --> 00:14:41,380
to families and or maybe to partners that get to the targets that move the needle towards
165
00:14:41,380 --> 00:14:46,000
my outcomes. One thing that was different about Raising Reader was that they work with
166
00:14:46,000 --> 00:14:51,440
partners. So I wanted to mention that sometimes the ideas framework can be and the theory
167
00:14:51,440 --> 00:14:58,080
of change could be flexible and adaptable to meet your needs. Originally, there are
168
00:14:58,080 --> 00:15:02,520
three columns in the theory of change. But we're working with Michelle, we thought we
169
00:15:02,520 --> 00:15:07,960
should have an extra column because they wanted to look at how working with partners and affiliate
170
00:15:07,960 --> 00:15:12,560
organizations, what the strategies that we're doing raising a reader was getting to the
171
00:15:12,560 --> 00:15:19,440
targets in partners and affiliates and how that was getting to the outcomes with families.
172
00:15:19,440 --> 00:15:25,440
And that was a key component on identifying precisely the actions that get to the targets
173
00:15:25,440 --> 00:15:29,480
and to the outcomes. And I'll pause there so that Michelle can share more about the
174
00:15:29,480 --> 00:15:35,640
specifics of what some of those strategies were and how that helped the organization.
175
00:15:35,640 --> 00:15:41,360
Thanks, Andres. Yeah, so one of the things, including that additional column that Andres
176
00:15:41,360 --> 00:15:46,680
was talking about, it really helped us think through how are we really building that educator
177
00:15:46,680 --> 00:15:52,760
capacity and how are we really providing professional development around early childhood development.
178
00:15:52,760 --> 00:15:57,680
And another aspect to it is that we knew we did it all the time and we had stories about
179
00:15:57,680 --> 00:16:02,380
how we did it all the time. But using the theory of change, the framework to really
180
00:16:02,380 --> 00:16:06,560
make it much more precise is really helping us think through how are we doing it. And
181
00:16:06,560 --> 00:16:12,040
so it also helped us think about how our programming is impacting early relational health through
182
00:16:12,040 --> 00:16:17,680
strengthening healthy family bonds. And so it really has made us to be better poised
183
00:16:17,680 --> 00:16:22,920
to effectively test and evaluate how we are doing this work and what is and what is not
184
00:16:22,920 --> 00:16:26,960
working. Thank you, Michelle. And one comment that I
185
00:16:26,960 --> 00:16:32,680
add as we have about 90% of the participants that are now in the webinar are new to the
186
00:16:32,680 --> 00:16:40,480
framework. Sometimes you may use the framework as a program developer or somebody who is
187
00:16:40,480 --> 00:16:46,360
implementing a program like Michelle, but you could also use it to help other organizations
188
00:16:46,360 --> 00:16:51,360
like the way I have used it as part of the Center on the Developing Child or as Promise
189
00:16:51,360 --> 00:16:56,560
Venture Studio has also used it with social entrepreneurs in their organization. The framework
190
00:16:56,560 --> 00:17:02,600
is really helpful in helping you think through your impact strategy. I'll mention a brief
191
00:17:02,600 --> 00:17:08,520
comment if you're familiar with other frameworks that innovators use like the Lean Startup
192
00:17:08,520 --> 00:17:15,800
or Business Model Canvas or other ones. These helps you think through in a very clear way,
193
00:17:15,800 --> 00:17:22,880
in a simplified way, what are your strategies and how am I getting to the outcomes. And
194
00:17:22,880 --> 00:17:28,760
because it's simple and it can fit in one page, it also facilitates communication, communicating
195
00:17:28,760 --> 00:17:34,840
internally and externally with the families you work with, with the potential funders
196
00:17:34,840 --> 00:17:40,680
and with potential partners. But I'll pause there.
197
00:17:40,680 --> 00:17:44,840
One more thing I just wanted to add about the framework is how it really helped us think
198
00:17:44,840 --> 00:17:50,560
about our innovations and new programming too into our theory of change and help facilitate
199
00:17:50,560 --> 00:17:55,560
that fast cycle iteration because over the last few years, we've developed two new programs
200
00:17:55,560 --> 00:18:01,160
and explored how we were successful in exploring how we can integrate technology into our programming.
201
00:18:01,160 --> 00:18:05,520
And we didn't have that in our previous theory of change. And so being able to build that
202
00:18:05,520 --> 00:18:10,600
into using the framework to build that into our theory of change, thinking about the evaluation,
203
00:18:10,600 --> 00:18:17,160
how do we get that feedback loop? It was really beneficial and helpful for us as we're continuing
204
00:18:17,160 --> 00:18:20,680
to innovate and develop new programs too.
205
00:18:20,680 --> 00:18:24,420
Thank you, Michelle.
206
00:18:24,420 --> 00:18:29,200
Thank you so much for sharing your experiences. And we're going to go ahead and pull some
207
00:18:29,200 --> 00:18:34,600
questions off chat. And I thought maybe we'll start with kind of a somewhat broad question
208
00:18:34,600 --> 00:18:41,220
and that came from Karen Soto is that are there any prerequisites for an organization
209
00:18:41,220 --> 00:18:46,520
to implement the ideas framework? And I know Melanie might want to speak to this.
210
00:18:46,520 --> 00:18:55,360
Sure. I would say there aren't necessarily any standard prerequisites, but I do think
211
00:18:55,360 --> 00:19:00,920
having worked with a lot of different organizations around this framework, there are some conditions
212
00:19:00,920 --> 00:19:07,680
that set you up to be more or less successful or effective using the framework. One thing
213
00:19:07,680 --> 00:19:13,560
I would say is it's important to have all the right people at the table. So I mentioned
214
00:19:13,560 --> 00:19:18,720
that one of the principles of the framework is co-creation and this idea that bringing
215
00:19:18,720 --> 00:19:22,840
together a group of people who have multiple perspectives on the program can be really
216
00:19:22,840 --> 00:19:28,660
valuable. And that might include leadership, people who are responsible for developing
217
00:19:28,660 --> 00:19:35,760
or implementing the program, people who will lead on the evaluation or research efforts.
218
00:19:35,760 --> 00:19:43,840
But equally importantly, you might invite people to contribute who have a role in actually
219
00:19:43,840 --> 00:19:49,200
delivering the service, working directly with kids, families and caregivers, and better
220
00:19:49,200 --> 00:19:55,020
yet invite a representative from the community that you serve to be part of this process.
221
00:19:55,020 --> 00:19:59,120
So that's the first piece is just having the right people at the table. And then the second
222
00:19:59,120 --> 00:20:04,240
thing I would say is timing can be important. So this framework is really designed to help
223
00:20:04,240 --> 00:20:12,400
you prepare for a fast cycle iteration process. So to prepare for a round of collecting data,
224
00:20:12,400 --> 00:20:17,420
reviewing that data, interpreting and analyzing it, making sense of it, and putting what you
225
00:20:17,420 --> 00:20:23,520
learn into practice. So the timing there can be important. Are you set up and prepared
226
00:20:23,520 --> 00:20:27,120
to actually put this plan into action? Do you have the resources you need as everyone
227
00:20:27,120 --> 00:20:34,080
bought in, et cetera? Yeah. And having the authority to put what you learn into practice.
228
00:20:34,080 --> 00:20:40,360
So if you're implementing a program that was developed by someone else, do you have sort
229
00:20:40,360 --> 00:20:44,960
of the leeway to make changes to how you're implementing that program based on what you
230
00:20:44,960 --> 00:20:51,000
learned? Or are there kind of more strict parameters around how you implement that program?
231
00:20:51,000 --> 00:20:58,400
Thanks, Melanie. The next question we have came from Nicholas, and it says, this is a
232
00:20:58,400 --> 00:21:04,720
question from Ms. Reyes from Albucieto. Were there any outcomes or benefits that happened
233
00:21:04,720 --> 00:21:10,880
unexpectedly from developing this program, i.e. unintended consequences that happened
234
00:21:10,880 --> 00:21:17,880
which you did not expect? Yes. So there, it's definitely been a learning curve. We've definitely
235
00:21:17,880 --> 00:21:24,140
had to modify and just evolve the program. One of the biggest changes that we've made
236
00:21:24,140 --> 00:21:30,600
just recently is changing the program from a five-hour week model where children receive
237
00:21:30,600 --> 00:21:38,240
two and a half hours of preschool twice a week to offering five hours of preschool twice
238
00:21:38,240 --> 00:21:44,720
a week. So in total, they're receiving 10 hours a week. And this really came from listening
239
00:21:44,720 --> 00:21:52,040
and taking the time to listen to parents and hear what their needs were. For years, our
240
00:21:52,040 --> 00:21:57,400
parents have been asking for more time on the bus. We're really trying to make an impact
241
00:21:57,400 --> 00:22:04,320
on how many children we served. Like I said, we have a capacity to serve 96 children in
242
00:22:04,320 --> 00:22:10,000
our valley. So that's 96 children that otherwise wouldn't be receiving preschool, you know,
243
00:22:10,000 --> 00:22:15,360
in a traditional preschool setting. And we've noticed recently we've had a decrease in our
244
00:22:15,360 --> 00:22:21,180
enrollment. So it's been a little bit harder to enroll children into our shorter classes.
245
00:22:21,180 --> 00:22:27,180
And I think that has now impacted our school district and our other centers that have grown
246
00:22:27,180 --> 00:22:33,380
their capacity in their centers, which was the ultimate goal to get more children into
247
00:22:33,380 --> 00:22:40,560
preschool and enrolled. So we've now looked at how we can, because we're mobile, we can
248
00:22:40,560 --> 00:22:47,880
now take our program and start serving communities that don't have that access. So it's been
249
00:22:47,880 --> 00:22:51,920
playing out lately that we've noticed these trends.
250
00:22:51,920 --> 00:22:58,520
Thank you. And actually, I just realized that this question is the way it got segmented
251
00:22:58,520 --> 00:23:03,400
in the question answer section. I didn't realize that the Nicholas who asked the question has
252
00:23:03,400 --> 00:23:08,380
the same question for folks at Raising a Reader. So were there any outcomes or benefits that
253
00:23:08,380 --> 00:23:12,360
happened unexpectedly from developing the program?
254
00:23:12,360 --> 00:23:19,760
Sure. So I'll say that we have had 39 independent evaluations that showed, and we knew that
255
00:23:19,760 --> 00:23:24,760
Raising a Reader helped improve home literacy environments, and which is like increased
256
00:23:24,760 --> 00:23:31,400
shared reading time, increased duration, and frequency, improved the number of books in
257
00:23:31,400 --> 00:23:36,600
the home. But one thing that we were hearing from folks was, oh, well, it's helping me
258
00:23:36,600 --> 00:23:42,080
build confidence in supporting my child's early learning in the home. It's really providing
259
00:23:42,080 --> 00:23:48,440
a sense of comfort and support for our families, this daily reading routine. And so creating
260
00:23:48,440 --> 00:23:53,200
a new theory of change with coaching support from Andres to really make much more precise.
261
00:23:53,200 --> 00:23:58,420
We were able to build those kinds of things into our theory of change, which then led
262
00:23:58,420 --> 00:24:03,640
us to improving our measurement tools. So asking specific questions so that we could
263
00:24:03,640 --> 00:24:08,360
actually get some more data around, well, the stories are great. It's great. It's also
264
00:24:08,360 --> 00:24:14,920
helpful to have our surveys also reflect some of that more quantitatively as well.
265
00:24:14,920 --> 00:24:23,240
Thank you, Michelle. And we have one question here that asks if someone could speak to how
266
00:24:23,240 --> 00:24:29,080
this theory of change framework can inform logic model use and development, ensuring
267
00:24:29,080 --> 00:24:34,540
that the information is complementary and not duplicative for programs who choose to
268
00:24:34,540 --> 00:24:37,720
create both types of resources.
269
00:24:37,720 --> 00:24:45,360
I'm happy to feel that, get asked that question a lot. From what I understand, those two terms,
270
00:24:45,360 --> 00:24:50,820
theory of change and logic model are often used actually interchangeably in the field.
271
00:24:50,820 --> 00:24:58,000
And there really isn't solid consensus on how the two are similar and different. The
272
00:24:58,000 --> 00:25:06,200
best guidance on that that I've found is that logic models tend to be more standardized.
273
00:25:06,200 --> 00:25:13,960
They often include inputs, activities, outputs, and then short term and long term outcomes
274
00:25:13,960 --> 00:25:21,880
or variations on that theme. And they're really, they're descriptive. Theories of change are
275
00:25:21,880 --> 00:25:28,000
a bit less standardized. So if you Google the term theory of change, you'll find many,
276
00:25:28,000 --> 00:25:34,200
many, many different approaches to theory of change. But in general, they're intended
277
00:25:34,200 --> 00:25:43,080
to be causal models that really explain how and why the expected changes come about. That's
278
00:25:43,080 --> 00:25:47,480
one way of thinking about that. Then logic models are more descriptive and theories of
279
00:25:47,480 --> 00:25:53,040
change are really intended to be explanatory causal models. Honestly, in practice, I find
280
00:25:53,040 --> 00:25:57,400
that they're, you know, when someone asks you for a logic model or a theory of change,
281
00:25:57,400 --> 00:26:02,160
you really have to follow up to ask what they mean specifically, what they're looking for
282
00:26:02,160 --> 00:26:08,360
there because I think expectations vary widely. Our approach to theory of change is really,
283
00:26:08,360 --> 00:26:16,440
it zooms in on the point of service delivery or the point of contact with kids, families,
284
00:26:16,440 --> 00:26:22,040
caregivers, and follows that through to the ultimate outcomes that you're hoping to see,
285
00:26:22,040 --> 00:26:26,720
which are typically child level in the field of early childhood. And it can be really helpful
286
00:26:26,720 --> 00:26:32,600
to set you up to make a plan for gathering data to better understand whether you're having
287
00:26:32,600 --> 00:26:37,080
the impact that you're hoping to achieve. I hope that's helpful.
288
00:26:37,080 --> 00:26:39,960
Sally, do you want to add to that?
289
00:26:39,960 --> 00:26:45,920
Yeah, so I actually have a real life example of how we've used both at the same time. So
290
00:26:45,920 --> 00:26:52,280
we recently worked with the team at Mathematica to create a two-gen logic model. They did
291
00:26:52,280 --> 00:26:57,840
a project with different two-gen organizations across the country. And so we have been using
292
00:26:57,840 --> 00:27:03,680
theories of change for years in our programs, in our six different programs, to really identify,
293
00:27:03,680 --> 00:27:08,200
you know, what is the, what are our strategies, what are the targets, what are the outcomes
294
00:27:08,200 --> 00:27:12,840
we're trying to have in each of our programs? So it's kind of granular. And then we worked
295
00:27:12,840 --> 00:27:20,040
with the team at Mathematica to create this, like, what does our whole organization do?
296
00:27:20,040 --> 00:27:24,760
And what is the whole change we're trying to make in the community and in children and
297
00:27:24,760 --> 00:27:31,840
families? And so having that overarching logic model that our theories of change then kind
298
00:27:31,840 --> 00:27:39,200
of feed up into, you can see how they interact, how they're incorporated in that larger logic
299
00:27:39,200 --> 00:27:44,320
model has been just really interesting. But we're not duplicating. So we don't have like
300
00:27:44,320 --> 00:27:48,240
a logic model and a theory of change for each of our programs.
301
00:27:48,240 --> 00:27:54,880
Does anybody else want to speak to that before I move to the next question? Okay. So we have
302
00:27:54,880 --> 00:28:00,280
a question here from Eric Marlowe and it asks, in your experience, I think this could be
303
00:28:00,280 --> 00:28:07,340
open to anyone here, how long are the typical iteration cycles? How long do you recommend
304
00:28:07,340 --> 00:28:11,560
evaluating and adapting elements of a given program so that changes are made neither too
305
00:28:11,560 --> 00:28:14,680
soon nor too late?
306
00:28:14,680 --> 00:28:19,180
I can take a first pass, but then I'd be curious to hear from our colleagues at Raising a Reader
307
00:28:19,180 --> 00:28:24,240
and Valley Settlement. If I understood correctly, Eshna, was the question like how often or
308
00:28:24,240 --> 00:28:27,560
how long does it take or maybe a little bit of both?
309
00:28:27,560 --> 00:28:30,040
My interpretation was a little bit of both.
310
00:28:30,040 --> 00:28:36,920
Okay. I think so the way this question gets asked to us often is how fast is fast cycle?
311
00:28:36,920 --> 00:28:42,360
Like, are we talking something you can do in days, weeks, months, years? And the answer
312
00:28:42,360 --> 00:28:49,760
there I think is, unfortunately, it depends. It really depends on the nature of the program
313
00:28:49,760 --> 00:28:56,120
or service or product that you're looking to evaluate and improve. If it's, let's say,
314
00:28:56,120 --> 00:29:04,340
a 10 week parenting group, then a single cycle could take, you know, you'd want a couple
315
00:29:04,340 --> 00:29:09,840
of months to plan and prepare to identify, to develop your theory of change, to identify
316
00:29:09,840 --> 00:29:15,120
your questions, to figure out your study design, figure out the tools you want to use, prepare
317
00:29:15,120 --> 00:29:21,920
for data collection. Then obviously you need the 10 weeks to go by where you're actually
318
00:29:21,920 --> 00:29:27,840
delivering the program to kids and families. And then you'll need some time afterwards
319
00:29:27,840 --> 00:29:33,680
to analyze, interpret, and make sense of that data. But that timeframe is obviously going
320
00:29:33,680 --> 00:29:37,760
to be really different if it's a program that's implemented on a school year calendar, for
321
00:29:37,760 --> 00:29:43,360
instance, or if you've developed something like an app or a website that families can
322
00:29:43,360 --> 00:29:50,400
engage with as they choose. And maybe dosage and engagement looks really different from
323
00:29:50,400 --> 00:29:57,040
parent to parent. So there's no right answer for how long a cycle can take. You really
324
00:29:57,040 --> 00:30:01,720
just need to be thoughtful about what you're hoping to learn. And then in terms of how
325
00:30:01,720 --> 00:30:07,880
often, I think that really varies again from organization to organization and what the
326
00:30:07,880 --> 00:30:13,320
appetite and bandwidth is to engage in this kind of iterative learning. I know that Valley
327
00:30:13,320 --> 00:30:18,800
Settlement, for instance, has really built this into your kind of culture and your routines
328
00:30:18,800 --> 00:30:23,840
as an organization. And it seems like you've developed a really nice kind of annual rhythm.
329
00:30:23,840 --> 00:30:28,040
So maybe I could pass the baton to Sally and Carla to talk more about that.
330
00:30:28,040 --> 00:30:34,180
Yeah. So as you say, Melanie, we really do our evaluation on an annual basis. So most
331
00:30:34,180 --> 00:30:40,280
of our programs happen during the school year. We do pre and post surveys. We also do pre,
332
00:30:40,280 --> 00:30:46,920
mid and post TS Gold assessments for our Boosecito preschool. And then we really, we do the bulk
333
00:30:46,920 --> 00:30:53,440
of data analysis in June and July. And then every July we gather together and do that
334
00:30:53,440 --> 00:30:58,880
evaluation retreat with our entire team. And then in August, we're kind of refining our
335
00:30:58,880 --> 00:31:04,880
evaluation and planning the next evaluation cycle. But what I will say is that when we're
336
00:31:04,880 --> 00:31:12,200
piloting new initiatives, we are trying to be a little more like eyes on as the initiative
337
00:31:12,200 --> 00:31:17,360
is happening. So for example, we worked a few years ago to implement a child development
338
00:31:17,360 --> 00:31:22,580
associate course for family, friends and neighbors providers, and also for high school students
339
00:31:22,580 --> 00:31:28,240
who are Spanish speaking. And we were, we were doing little pre and post assessments
340
00:31:28,240 --> 00:31:33,880
throughout at the beginning and end of kind of each section of the course or module, just
341
00:31:33,880 --> 00:31:38,320
to understand like, was our approach working? How could we pivot and adjust? So as we're
342
00:31:38,320 --> 00:31:44,560
piloting new things, we do try to be a little more rapid, if you will.
343
00:31:44,560 --> 00:31:49,440
I would just echo the same thing that we do have an annual, like an annual cycle, I guess,
344
00:31:49,440 --> 00:31:53,920
where we do an annual evaluation, annual check in with our affiliates is what we call it.
345
00:31:53,920 --> 00:31:57,880
And then we do have a couple of pilots right now where we are calling them mini learning
346
00:31:57,880 --> 00:32:03,560
phases. After each learning phase, then we'll take a look at the feedback and then see what
347
00:32:03,560 --> 00:32:07,840
tweaks or what modifications we need to do to improve the program. And then we have our
348
00:32:07,840 --> 00:32:15,160
second learning phase and things like that. A quick follow up from Nicholas to this question
349
00:32:15,160 --> 00:32:19,520
was, so does that mean that the theory of change is different with each iteration?
350
00:32:19,520 --> 00:32:25,800
So we definitely like, we can evolve our theory of change every year, we go in as a team and
351
00:32:25,800 --> 00:32:30,120
look at the strategies and say like, you know, are you still doing these things? How are
352
00:32:30,120 --> 00:32:35,280
you doing them? You know, we definitely change and evolve in our programming. We're not doing
353
00:32:35,280 --> 00:32:40,720
the same thing every year because programs, neighborhoods, communities change and evolve
354
00:32:40,720 --> 00:32:46,480
and we learn new things every year. So we do go in and tweak our theory of change every
355
00:32:46,480 --> 00:32:49,160
year.
356
00:32:49,160 --> 00:32:55,600
We just revised ours, so we haven't changed it. But one thing I think it is helping us
357
00:32:55,600 --> 00:33:00,640
think about though, as we are developing our program, how to stay focused on what it is
358
00:33:00,640 --> 00:33:05,480
that we really want to do, because there are so many needs and we serve so many different
359
00:33:05,480 --> 00:33:12,400
communities that have different needs. How do we stay true to this, the theory of change
360
00:33:12,400 --> 00:33:17,280
that for our mission and things like that. So it really helps us identify our clear lanes,
361
00:33:17,280 --> 00:33:22,440
like where is our most unique impact and helps us stay there and not kind of stray just because
362
00:33:22,440 --> 00:33:26,400
there might be a funding source over there or something like that.
363
00:33:26,400 --> 00:33:34,420
I'll just add that it's a great question and it gives us a chance to underscore that absolutely
364
00:33:34,420 --> 00:33:40,320
a theory of change can and should evolve over time as you learn. So we call them living
365
00:33:40,320 --> 00:33:47,160
documents. It's not a one-time exercise that you do and you make a PDF of it and it's done.
366
00:33:47,160 --> 00:33:53,000
It's a living document that you come back to after every round of learning and you say,
367
00:33:53,000 --> 00:33:57,960
what did we learn? How can we refine our theory of change? How can we refine our actual program
368
00:33:57,960 --> 00:34:04,160
or product or service? And how could we refine how we're gathering data and learning going
369
00:34:04,160 --> 00:34:09,880
forward? Thank you. So now we have a question from
370
00:34:09,880 --> 00:34:18,400
Megan Crystal asking if we have any examples, if we know of any state level policy or programs
371
00:34:18,400 --> 00:34:26,880
that have used this framework. I have a couple that come to mind. So a while
372
00:34:26,880 --> 00:34:33,020
back as part of the Frontiers of Innovation initiative that Ashna mentioned, there was
373
00:34:33,020 --> 00:34:41,320
a project team who implemented a video coaching program to support childcare and early learning
374
00:34:41,320 --> 00:34:47,920
professionals and that project was done in partnership with the State Department of Early
375
00:34:47,920 --> 00:34:55,400
Learning and this framework was used to sort of articulate theory of change for that approach
376
00:34:55,400 --> 00:35:01,060
and to actually work with partners at the University of Washington to craft the evaluation
377
00:35:01,060 --> 00:35:05,920
plan. That's one example. And then another more recent example that
378
00:35:05,920 --> 00:35:13,260
comes to mind is our center worked with partners at the Massachusetts Department of Early Education
379
00:35:13,260 --> 00:35:21,120
and Care to create an initiative wide theory of change, actually similarly for an effort
380
00:35:21,120 --> 00:35:29,040
underway to bring early care and support organizations to build the capacity of actually childcare
381
00:35:29,040 --> 00:35:33,280
directors across the state. So we worked with them to create an initiative
382
00:35:33,280 --> 00:35:40,200
wide theory of change and then each organization who is providing that service used that initiative
383
00:35:40,200 --> 00:35:45,340
wide theory of change as a template and kind of tailored it for their particular approach.
384
00:35:45,340 --> 00:35:49,520
And I think that's still being used right now as the initiative continues to kind of
385
00:35:49,520 --> 00:35:55,980
facilitate learning and improvement over time. There might be other examples, but those are
386
00:35:55,980 --> 00:35:59,400
two that come to mind. Thanks, Melanie. Does anybody else?
387
00:35:59,400 --> 00:36:04,160
Oh, can I squeeze in one more just in case folks are looking for examples? We worked
388
00:36:04,160 --> 00:36:10,560
with an organization called TOPS, which is based in the Netherlands and they provide,
389
00:36:10,560 --> 00:36:17,040
if I'm remembering correctly, services to families with newborn children. And I think
390
00:36:17,040 --> 00:36:24,480
that's actually like a nationwide program that's used ideas to drive towards greater
391
00:36:24,480 --> 00:36:30,840
impact. And there are resources or references about the TOPS program in the resources section
392
00:36:30,840 --> 00:36:36,080
of the toolkit. I think there's a research article there that talks about their experience.
393
00:36:36,080 --> 00:36:40,480
Great. Thanks, everyone. I was hoping we might have time for one more question, but I actually
394
00:36:40,480 --> 00:36:45,560
see that we're pretty close to the end there. So I just wanted to thank you all for joining
395
00:36:45,560 --> 00:36:53,120
us. Thank our panelists for sharing your experiences and your learning. We really appreciate it.
396
00:36:53,120 --> 00:36:58,500
And thank you all so much for joining. We really, really hope that this resource is
397
00:36:58,500 --> 00:37:03,560
useful for you all. And we wish you the best in your continued work supporting kids and
398
00:37:03,560 --> 00:37:08,560
families.
399
00:37:08,560 --> 00:37:13,040
The Brain Architects is a product of the Center on the Developing Child at Harvard University.
400
00:37:13,040 --> 00:37:17,660
You can find us at developingchild.harvard.edu, where we'll post any resources that were
401
00:37:17,660 --> 00:37:23,080
discussed in this episode. We're also on Twitter at Harvard Center, Facebook at Center on the
402
00:37:23,080 --> 00:37:26,040
Developing Child, and Instagram at Developing Child Harvard.
