S7 E10 - How to Evaluate Spelling Instruction

WEBVTT
00:00:48.690 --> 00:00:49.650
Hello, hello.
00:00:49.650 --> 00:00:57.810
Welcome to this episode of the Structured Literacy Podcast recorded right here in Burnie, Tasmania, the lands of the Palawa people.
00:00:57.810 --> 00:01:02.930
I'm Jocelyn, I want to start this episode with a small warning.
00:01:02.930 --> 00:01:07.250
What I'm going to talk about today might feel a little bit confronting.
00:01:07.250 --> 00:01:13.570
And I do not mean the messaging in this episode to be harsh, I promise you that.
00:01:13.570 --> 00:01:15.810
But I do mean to be direct.
00:01:15.810 --> 00:01:17.890
I do mean to be no- nonsense.
00:01:18.609 --> 00:01:49.250
And I do mean to challenge some of what is happening in our classrooms, because the students I keep thinking about, the ones in year three, four, five, six, seven, eight, who are still struggling with spelling despite their previous years of what is claimed to be robust phonics instruction, do not have time for us to skirt around the edges of what is actually getting in the way of their learning.
00:01:49.250 --> 00:01:52.689
So with that in mind, let's get into it.
00:01:53.250 --> 00:02:11.250
Over the last couple of episodes of this podcast, I have been talking about spelling instruction in years three to eight, what the research tells us it should look like, what can get in the way, and the instructional design decisions that either serve or shortchange students.
00:02:11.250 --> 00:02:24.129
Today I want to take that a step further and give teachers and leaders a tool that you can use to audit what is happening in your school in this space.
00:02:24.129 --> 00:02:31.650
The reality is that most schools have a spelling program and most schools would claim that it's working.
00:02:31.650 --> 00:02:38.370
But most schools have not sat down and systematically asked whether it actually is.
00:02:38.370 --> 00:02:46.129
I think part of the reason for that is the way we tend to evaluate impact in education.
00:02:46.129 --> 00:02:53.409
So much of what we count as evidence is about what teachers, leaders, and systems do.
00:02:53.409 --> 00:03:04.449
We measure the inputs, how many professional learning hours were delivered, whether a program is being used, whether the curriculum scope and sequence has been mapped.
00:03:04.449 --> 00:03:12.050
All of these things matter, but they are not the same as asking what students are actually learning.
00:03:12.050 --> 00:03:21.569
We want to be really careful that we're not using the inputs as a proxy for the outcomes we're looking for.
00:03:21.569 --> 00:03:34.370
I'd like us to, whenever we make a claim that something is working in inverted commas, I really want us to get into the habit of immediately asking two questions.
00:03:34.370 --> 00:03:36.050
How do we know that?
00:03:36.050 --> 00:03:38.610
And what is our evidence?
00:03:39.410 --> 00:03:46.290
Before I go any further, I want to be clear about something for our foundation to year two teachers and leaders.
00:03:46.290 --> 00:03:50.850
In the early years, your spelling program is your phonics program.
00:03:50.850 --> 00:04:14.610
As long as you are giving encoding the same time and attention as decoding, and as long as every student is fully engaged in both, not copying off the board, not copying from someone next to them or watching someone do something at the front, as long as they're fully engaged, then reading and spelling genuinely enhance each other.
00:04:14.610 --> 00:04:23.730
So if you're in the early years, the audit that I'm about to walk through is not fully where your attention needs to go.
00:04:23.730 --> 00:04:32.290
You can use the steps of the process that I'm about to describe to consider and evaluate your phonics instruction for sure.
00:04:32.290 --> 00:04:37.730
But the specifics that I'm talking about today are for years three to eight.
00:04:37.730 --> 00:04:50.530
So this episode is for the leaders, curriculum coordinators, and teachers who are working with older students and asking, is our current approach actually achieving what we think it is?
00:04:51.410 --> 00:05:02.290
The tool that I'm going to introduce you to today comes from the work of Victoria Bernhardt, specifically from her book Data Analysis for Continuous School Improvement.
00:05:02.290 --> 00:05:04.930
And I have the third edition of that book.
00:05:04.930 --> 00:05:17.170
Bernhardt has spent decades developing frameworks and tools that help schools move from vague intentions about improvement to structured evidence-informed action.
00:05:17.170 --> 00:05:25.250
One of the tools she provides that we're going to talk through today is called Measuring Programs and Processes.
00:05:25.250 --> 00:05:34.050
It involves a simple but powerful matrix, and Bernhardt suggests giving your team about two hours to work through all of this.
00:05:34.050 --> 00:05:53.729
And that might sound like a lot, but when we consider that most of us are spending years implementing programs without ever systematically evaluating them, two hours is a small investment to help us get on and recognise that we are on the right track or the wrong track.
00:05:53.729 --> 00:06:03.009
So the matrix asks seven questions organised across three areas: purpose, participants, and implementation.
00:06:03.009 --> 00:06:05.409
And then it asks about results.
00:06:05.409 --> 00:06:10.370
Now what I love about this framework is that it forces us to be specific.
00:06:10.370 --> 00:06:13.889
We're not asking, do we think it's going well?
00:06:13.889 --> 00:06:21.009
We're asking, what's the evidence that we are making the impact we think we are?
00:06:21.009 --> 00:06:34.930
I've used this exact matrix in schools that I have led myself, and it's the sort of thinking that I help school leaders do in working with them on their school improvement processes and journey.
00:06:35.649 --> 00:06:46.209
What this sort of thinking reveals is that there's often a significant gap between what we believe is happening and what is actually happening.
00:06:46.209 --> 00:06:55.250
And in today's episode, we're going to apply this specifically to spelling, but as I said earlier, you can use it for any particular area of curriculum you like.
00:06:55.250 --> 00:07:02.769
So I'm going to walk you through each of the questions in Bernhardt's Matrix with a spelling focus.
00:07:02.769 --> 00:07:20.209
I also want to let you know that we've developed a free Spelling Instruction Evaluation tool drawing on the research that I've covered in the last few episodes and including this kind of thinking that we're going to see in the process that we're exploring today.
00:07:20.209 --> 00:07:29.329
You can download this tool from the show notes of this episode or from the Free Resources tab at jocelynseamereducation.com.
00:07:30.610 --> 00:07:37.649
So the first question that is asked is, what is the purpose of the program or the process we're evaluating?
00:07:37.649 --> 00:07:51.329
And it's important to know that this framework for thinking and reflection isn't only about curriculum, it's also about our attendance strategy or how we organise playground duty.
00:07:51.329 --> 00:07:58.289
So it works for everything and having a tool that we can use over and over, get used to that thinking, is really powerful.
00:07:58.289 --> 00:08:04.529
So this question of what is the purpose of the program or process is the starting point.
00:08:04.529 --> 00:08:09.810
So what is our spelling approach or our spelling program for?
00:08:09.810 --> 00:08:19.729
If the answer is something like, to teach students to spell the words on the list, then that is a program purpose, but it's not really a learning purpose.
00:08:19.729 --> 00:08:34.610
The learning purpose of spelling instruction is to build students' deep understanding of how words work so that they can apply that knowledge to words that they have never seen before in writing, they care about.
00:08:34.610 --> 00:08:48.690
Spelling instruction is about building automaticity and understanding so that cognitive energy can be devoted to those macro text- level elements that we know are so critical.
00:08:48.690 --> 00:09:02.930
If your program is producing students who can spell the words on the Friday test, but cannot transfer that knowledge to their own writing on Monday, then the learning purpose is not being met.
00:09:02.930 --> 00:09:05.649
And this is a common gap that we see.
00:09:05.970 --> 00:09:11.090
The second question is: how will you know the purpose is being met?
00:09:11.090 --> 00:09:28.450
And for spelling, the honest answer for most schools is either, well, we don't know, or we look at our NAPLAN, which NAPLAN is very limited in its scope of what it looks for in spelling, or we have a weekly test that we check for.
00:09:28.450 --> 00:09:34.690
Now, full disclosure, in our spelling program, Spelling Success in Action, there is a weekly test.
00:09:34.690 --> 00:09:44.210
We need to figure out whether students can recall specific words and use patterns immediately after studying them.
00:09:44.210 --> 00:09:55.410
But this on its own does not tell us whether they can apply the pattern to unfamiliar words after the formal instruction has ceased.
00:09:55.410 --> 00:10:04.450
It absolutely does not tell us whether their spelling is independent in the writing that they do and whether their writing overall is improving.
00:10:04.450 --> 00:10:14.210
So when you sit down with the team and ask what our data actually tell us, this is one of the critical questions.
00:10:14.210 --> 00:10:22.930
And spelling data is a little messy because there isn't really a spelling version of DIBELS.
00:10:22.930 --> 00:10:33.730
So the questions we're asking really are about have the students retained what we've taught and can they use it for their own independent purposes?
00:10:33.730 --> 00:10:42.930
So the first thing we have to acknowledge is we might not have the evidence to tell us what is happening, and we need to get on top of that.
00:10:43.330 --> 00:10:59.570
So this is probably the right time to mention that we have a set of free diagnostic spelling assessments that cover phonics, orthographic conventions, and suffixing conventions, the ones related to morphology, inside our website.
00:10:59.570 --> 00:11:13.490
And every school who has spelling success in action one has access to the training specifically designed to help you unpack that data and to use it to make active instructional decisions.
00:11:13.490 --> 00:11:22.850
But even if you're not using our programs, you can download the tools and there are spreadsheets and explanations there for you to refer to to help you make some decisions.
00:11:22.850 --> 00:11:31.250
Assessment, though, is only useful if we use it to determine whether we are changing things for the better in our classrooms.
00:11:31.250 --> 00:11:33.970
And that's where the training bit comes in.
00:11:34.370 --> 00:11:39.490
The next question is: who is the program intended to serve?
00:11:39.490 --> 00:11:44.770
So in theory, the answer should be: all students.
00:11:44.770 --> 00:11:53.009
In practice, the answer for many programs is basically the students who are already mostly fine.
00:11:53.009 --> 00:12:01.650
Because most programs just begin all students at the same point of instruction in the year, regardless of the diagnostic data.
00:12:01.650 --> 00:12:13.250
Instruction is then provided at a single pace based on what's on the page in front of us, with very little or no adjustment for students with gaps.
00:12:13.250 --> 00:12:32.690
So if that's the case, the program then tends to serve the middle because it doesn't stretch the students who really will benefit from applying knowledge in more complex, robust vocabulary, and it doesn't support those who are coming into the learning process with significant gaps.
00:12:32.690 --> 00:12:42.129
So programs at best then extend students who already have solid foundations, and then they leave the other students treading water.
00:12:43.250 --> 00:12:53.330
The next question, and it's one that is hard to really look at, is: who is being served and who is not?
00:12:53.330 --> 00:12:58.610
Answering this question requires us to look at the data and be honest.
00:12:58.610 --> 00:13:00.370
Who is making progress?
00:13:00.370 --> 00:13:02.370
Who is not making progress?
00:13:02.370 --> 00:13:05.490
Is there a pattern that we're seeing?
00:13:05.490 --> 00:13:10.290
Do the students who are not being served tend to share something in common?
00:13:10.290 --> 00:13:12.770
Have they come from a similar year level?
00:13:12.770 --> 00:13:14.290
Do they have the same gaps?
00:13:14.290 --> 00:13:16.450
Do they have English as an additional language?
00:13:16.450 --> 00:13:18.850
Is there a diagnosis of some sort?
00:13:18.850 --> 00:13:29.090
And none of these things are excuses that make a failure to learn acceptable, but they do help us understand the context in which we're working.
00:13:29.570 --> 00:13:41.490
And I have to be direct here, if we do not have the data, we cannot answer the question about who is being served and who is not being served, full stop.
00:13:41.490 --> 00:13:55.170
Because saying that students are doing really well or that a program is working really well without any data to back that up really means we feel like we are delivering instruction well.
00:13:55.170 --> 00:13:58.450
And those are two different things.
00:13:58.450 --> 00:14:04.129
Feeling like instruction is going well is not the same as knowing what students are learning.
00:14:04.129 --> 00:14:10.930
We have to be honest about the distinction because students' time is not something we can get back.
00:14:10.930 --> 00:14:15.009
This discussion is not about blame.
00:14:15.009 --> 00:14:22.129
Every teacher I have ever worked with cares about their students and every leader cares about their school.
00:14:22.129 --> 00:14:28.690
But caring is not a substitute for ensuring that learning is happening.
00:14:28.690 --> 00:14:39.730
If we want to know whether our students are being served, we have to move beyond a I feel it in my heart version of measuring outcomes.
00:14:40.450 --> 00:14:46.770
Next question: what will it look like when the approach or program is fully implemented?
00:14:46.770 --> 00:15:02.450
And this question asks us to describe concretely what strong implementation looks like, not what teachers' self-report is happening, what it actually looks like in a classroom when our approach is working.
00:15:02.450 --> 00:15:09.810
Teachers don't claim that they are doing the program well because they're trying to deceive leadership or they're lazy.
00:15:09.810 --> 00:15:32.850
When someone who is a novice at using a particular approach or a program, regardless of how experienced they are as a teacher, if someone who is new to something says, no, no, I'm doing it, I'm doing it, that claim is because they are evaluating their performance against their version of instruction.
00:15:32.850 --> 00:15:36.850
And so when people say, no, no, we're doing it, they're not lying to you.
00:15:36.850 --> 00:15:39.810
They are doing their version of instruction.
00:15:39.810 --> 00:15:47.009
But whether that instruction is robust enough to meet the needs of students may be a different thing.
00:15:47.330 --> 00:15:56.770
And this is where an evaluation tool becomes useful because this whole process is not about judging teachers.
00:15:56.770 --> 00:15:59.250
It's not performance management.
00:15:59.250 --> 00:16:11.810
It's asking have we done what is needed to create these systems and structures within our school that set everybody up for success?
00:16:11.810 --> 00:16:22.210
When you have a set of specific evidence markers, when you know what you're looking for, then everything becomes easier.
00:16:22.770 --> 00:16:29.250
So we're looking for things like is there one clear concept that's being explored per unit?
00:16:29.250 --> 00:16:37.890
Are students working with enough different words to genuinely build schema, not just one base word and its family?
00:16:37.890 --> 00:16:43.090
Are students reading and spelling the target pattern in every lesson?
00:16:43.090 --> 00:16:51.250
Is instruction reaching sentence and text level in a way that is optimised for the cognitive load of the students?
00:16:51.250 --> 00:16:56.530
Is the teacher explaining, modelling, and checking for understanding?
00:16:56.530 --> 00:17:05.009
Do we really have a shared vision for what strong, robust, effective instruction looks like in our school?
00:17:05.009 --> 00:17:11.170
How much of what we do in classrooms is based on opinion and a best guess?
00:17:11.170 --> 00:17:20.849
And how much is aligned to evidence and directly linked to using student data to evaluate impact?
00:17:22.289 --> 00:17:34.450
If we cannot describe what full implementation, effective implementation looks like in really concrete terms, we cannot evaluate what we're doing.
00:17:34.450 --> 00:17:46.849
So there's a difference here between evaluating teacher performance, if we want to call it that, and program components and the instructional design components of a program.
00:17:46.849 --> 00:17:50.049
So don't get those two things mixed up.
00:17:50.690 --> 00:17:56.450
Next question is: how is implementation being measured and should it be measured differently?
00:17:56.450 --> 00:18:02.850
So for most schools, implementation is not really measured and evaluated at all.
00:18:02.850 --> 00:18:18.450
There's often an assumption that because the program was purchased and teachers are using the materials, that the program is being implemented and that self-reporting about how well we think it's going is taken at face value by leadership.
00:18:18.450 --> 00:18:23.330
But implementation is not the same thing as effectiveness.
00:18:23.330 --> 00:18:27.809
What would it look like to actually measure implementation?
00:18:27.809 --> 00:18:31.410
Would it involve classroom walkthroughs with a specific focus?
00:18:31.410 --> 00:18:41.809
Would it involve collaborative review of lesson materials and re-watching snippets of training to compare what we're doing with what was actually in the training?
00:18:41.809 --> 00:18:54.210
And if you're a spelling success school, you have access to the teacher resources, the evaluation templates, and the meeting agendas that help you to do this.
00:18:54.210 --> 00:19:05.250
And yes, this process will include data showing that learning is occurring and that students are consolidating what is covered in class and what they're learning to use independently.
00:19:05.970 --> 00:19:12.850
So the next question is: to what degree is this process or program being implemented?
00:19:12.850 --> 00:19:16.370
And we have to have an honest reckoning.
00:19:16.370 --> 00:19:27.250
We have to go really from a scale of we have the resources and they sit on the shelf to every classroom is delivering this consistently and with genuine understanding.
00:19:27.250 --> 00:19:31.170
We have to acknowledge we may not be where we think we're up to.
00:19:31.170 --> 00:19:33.970
And again, this is not about blaming teachers.
00:19:33.970 --> 00:19:40.450
It is a question about how effective and embedded our systems of instruction really are.
00:19:40.450 --> 00:19:47.170
If implementation is patchy, the answer is rarely that teachers are not trying hard enough.
00:19:47.170 --> 00:20:05.330
You may be surprised at how often we hear from schools, "Oh we didn't have time to do the training." If we want implementation to be strong, we have to devote the time, and that's about creating those systems that set people up for success.
00:20:06.450 --> 00:20:07.890
Then what are the results?
00:20:07.890 --> 00:20:12.850
And we don't only ask that question once, the data is a part of everything that we're doing.
00:20:12.850 --> 00:20:20.690
We're not just looking at those basic test scores, though, we're looking at the full picture all the way through to application.
00:20:20.690 --> 00:20:34.930
So getting this full picture is about evaluating instruction against evidence and asking ourselves: is our approach fully aligned with evidence?
00:20:34.930 --> 00:20:41.890
Have we successfully connected the science of reading and writing with the science of learning?
00:20:41.890 --> 00:20:55.890
Or have we ticked boxes on the literacy elements, but we're not delivering in a way that reflects cognition and what we know about how human brains learn?
00:20:55.890 --> 00:21:05.569
Because there's two different things: are we implementing the program well, and is the program setting us up for success?
00:21:05.569 --> 00:21:11.569
Bernhardt's framework asks one final thing, and that is next steps.
00:21:11.569 --> 00:21:26.210
A specific plan, not to be perfect by next week, but to know where we're up to and how we're going to take steps to move us along to that next point in our development.
00:21:26.210 --> 00:21:44.850
So if we identify that we're at about a 6 out of 10, perhaps, when we look at evidence-based instruction, the goal isn't to get to 10 by next week, that's unrealistic, but it is to say, ok, this term, our goal is to get to 7, what is the practice that's going to get us there?
00:21:44.850 --> 00:21:47.250
And then do that well.
00:21:47.250 --> 00:22:09.809
Remember, evidence-based practice that is universally successful is not about doing one big thing, it's about 25 small things that are done consistently well and that are embedded in strong structures and processes so that everybody in the school community is set up for success.
00:22:10.130 --> 00:22:26.690
If you would like a practical tool to support exactly this kind of conversation around spelling, the Spelling Instruction Evaluation tool is available for free download in the show notes of this episode and in the Free Resources tab at jocelynseamereducation.com.
00:22:26.690 --> 00:22:39.330
It covers the key elements of strong spelling instruction drawn from research, both the science of reading and writing, if we want to call it that, and from the science of learning.
00:22:39.330 --> 00:22:45.490
And it's designed to be used by teams, not just by leaders sitting alone in a room with a spreadsheet.
00:22:45.490 --> 00:23:00.450
You can use this tool to evaluate, what's happening between classrooms, you can use it to evaluate different programs you might be considering purchasing or adopting, or you can use it for what's happening in general.
00:23:00.450 --> 00:23:02.370
Make it work for you.
00:23:02.850 --> 00:23:07.890
I said at the start of this episode that this discussion may feel a little confronting.
00:23:07.890 --> 00:23:09.730
I really hope it wasn't.
00:23:09.730 --> 00:23:11.650
I hope it was useful.
00:23:11.650 --> 00:23:28.210
The students in year three to eight who are still carrying spelling gaps, who are choosing simpler words in their writing, who are exhausted by the cognitive load of basic encoding, those students deserve instruction that meets them where they are.
00:23:28.210 --> 00:23:35.490
And teachers deserve the tools and the clarity to make that happen.
00:23:35.490 --> 00:23:53.569
We will only achieve universal success when we work collectively to make instruction reflect how humans learn and what the research that we have available tells us about effective literacy instruction.
00:23:53.569 --> 00:23:57.009
Until I see you in the next episode, happy teaching, everyone.
00:23:57.009 --> 00:23:57.410
Bye.