speaker-0: Hi and welcome to Well Measured, an interview series from Education Perfect, where we bring together leading thinkers in education and learning science to ask a deceptively simple question. What's actually working in EdTech and how do we know? I'm Jules Maguire, Group Head of Learning Innovation at Education Perfect. And with me here today is my co-host, James Sancher. who is a former teacher and leads impact research at here at EP. Hi James.
speaker-2: Hello, Jill. In previous episodes, we've heard from experts pushing us forward on evidence-based instruction and AI in education. Today, we're doing something a little bit different. We're talking to someone who's been building a company on those principles for nearly two decades and asking them how that has looked in practice.
speaker-0: So our guest today is none other than Shane Smith, co founder of Education Perfect. Hi Shane.
speaker-1: Hi there, Jill and James. It's a real pleasure to be here. ⁓ and congratulations on this really successful podcast series. There have been some really interesting ideas flowing through and looking forward to today's chat.
speaker-0: Well, we have been looking forward to this series for quite a while because for those of you who don't know, Shane co-founded EP with his brother Craig in 2007. And he did this remarkably while studying medicine at the University of Auckland. And from the very beginning, Shane and Craig built EP around principles of cognitive science. And that was way, way, way back before learning science became the movement that it is today. And after stepping away for a number of years, I think it has been quite a few. Shane is now back to lead EP's AI initiatives, including the development of our AI-powered feedback tool, which is being used by literally hundreds of thousands of students every year. Shane, welcome again to Well Measured.
speaker-1: Thank you so much, Jill.
speaker-0: Okay, well Shane, you started building Education Perfect almost twenty years ago, if you can believe that, and as a medical student, as you've told us. So clearly you didn't come from teaching. ⁓ I'm just wondering what made you want to change your game and innovate in education?
speaker-1: It's really interesting, Jill. ⁓ I I guess I had two entry points into EP. The first one was absolutely personal interest. And then the second one was very focused on impact. ⁓ so when I was going through university, ⁓ Craig had made the very first version of what was language perfect back then. ⁓ and I was just fascinated by ⁓ how our brains worked, ⁓ how ⁓ how we learned, ⁓ what good learning looked like. ⁓ and so I got pulled into some of those. really meaty, interesting challenges then as as a as a personal interest piece. But then I actually ⁓ finished my studies and and graduated and worked for a year as a junior doctor. and at the end of that, Craig called me and said, Shane, we're gonna go from teaching languages and teaching vocab in particular, ⁓ through to teaching ⁓ English and science and hit history and humanities. ⁓ and and at that stage, I guess I had a choice. ⁓ you know, I knew that this was something that I was passionate about and found really interesting. ⁓ but I also knew that as a doctor I could have quite a significant impact on each individual patient's life. And so I sort of did a bit of a a a weighted comparison and said, you know, I can either have a large impact on a very small number of people's lives as a doctor, or I can have a much smaller impact potentially, but on a much larger scale. And, you know, I've been really stoked that we've been able to scale EP ⁓ now to many millions of students who've been able to benefit from from the work we put in. and I think that that trade off ⁓ worked out pretty well in all in all.
speaker-2: That actually segues nicely into my into my question. So one of things I find really interesting about that early period is that you weren't just reading the research and applying it in a vacuum. You're also obsessively and I hope that word obsessively is OK. ⁓ Really close to what teachers and students were actually experiencing with the product. How did you balance what the science said should work?
speaker-1: It is accurate.
speaker-2: with what you're seeing and hearing from classrooms as you are going out there and seeing the product in action.
speaker-1: Yeah. So I guess we had a few ⁓ inputs to this. ⁓ the very first version of Language Perfect was obviously built ⁓ by a learner for a learner. ⁓ and and so ⁓ you know, that meant that we did a lot of testing on ourselves. ⁓ we we you know, Craig was using it to actively learn. ⁓ but then beyond that, when when he ⁓ graduated and went out to university himself, ⁓ I know that he spent many, many hours ⁓ in his student dorms. ⁓ pulling people aside and saying, Hey, ⁓ you're learning French or German or whatever language it was that the student was learning at the time. Can you sit with me for half an hour? I'd love to see you ⁓ to use this and and and see how you interact with it. ⁓ and he at that stage had an internal rule where he would sit silently no matter what the person did, ⁓ and just observe. ⁓ and if they got stuck, then then then he would observe that and then go away and and and and try and improve things. ⁓ But we were also fortunate enough that ⁓ right from the get-go, we're we were able to ⁓ get involved with some really supportive ⁓ schools. ⁓ at the time, I'm pretty sure Rangatoto College was one of those very supportive schools. ⁓ and we had a a great ⁓ relationship with our own school, St. Kenigan College. and with that, we were able to spend time in classrooms. ⁓ we were able to talk to teachers, we were able to talk to students, ⁓ and really overlay that over this ⁓ almost theoretical ⁓ basis for things ⁓ and and and mesh these these ideas and these concepts together because ⁓ it's all very well ⁓ to implement an idea that in a study has proven in a certain context ⁓ but what you really need to ensure that your is that your implementation of that idea in your context actually moves the dial and actually delivers what the research suggests that it should. ⁓ and and that very close ⁓ contact with students and teachers I guess was our it was our initial layer of validation there. Obviously since then we've we've gone and got a lot more formal validation which which ⁓ backs up ⁓ that that plan that this really does the ⁓ move the dial as well, ⁓ which I'm sure we'll be talking about today.
speaker-0: Yes, well we we will because fast forward twenty years and from those humble beginnings of language is perfect in New Zealand, ⁓ we're now covering all subjects across the curriculum, across New Zealand, Australia, Canada, internationally, ⁓ in in a lot of schools across the world. So so that has really expanded. And at the same time, the science of learning has become a much bigger conversation as well, especially in the last five or ten years. ⁓ it's now shaping public policy, it's in the mainstream, everyone's talking about it. So it must be nice watching the world catch up to where you were two decades ago. ⁓ but what I was thinking about is how has the evidence ⁓ around cognition, around the learning science changed anything in your mind about what EP should be doing right now?
speaker-1: It's a it's it's a great question. And I think, you know, watching ⁓ learning science become far more front and center has been a really healthy and I'd say really necessary evolution ⁓ in the space. ⁓ certainly when we ⁓ started building Language Perfect, it was a bit of a wild west. ⁓ and ⁓ I I I think you know ed tech itself has gone through a few phases during this time. You know, I think initially ⁓ there was this initial phase of excitement to say, hey, we have these devices in classrooms. ⁓ we want things that engage our students, we want things that are exciting. and and, you know, off the back of that, there were that were there were a series of tools that were, you know, engaging and exciting. ⁓ not necessarily so heavily learning paced. ⁓ there was also, you know, during COVID, there was this massive boost and rollout of of these tools. And since then we have seen some degree of disillusionment from from teachers saying, you know, a series of these tools which we really thought were going to move the dial for us, we have implemented, we've done our best, ⁓ and haven't have the haven't had the impact ⁓ that we thought. ⁓ and no, I think our journey ⁓ in parallel to that at EP has been ⁓ to ensure that as we've gone through this process, We've continued going back to the evidence. ⁓ but then also we have gone out ⁓ and more and more strengthened the evidence base ⁓ and and I guess the proof point around the product ⁓ to ensure that when we do get to this point of disillusionment, we can say, okay, well, it is true that some tools out there don't seem to move the dial. ⁓ but you can see here that that actually EP really cares about this. We have from from the ground up, it's been part of our DNA. ⁓ so why don't you give this a go? and I think, you know, if anything, it it it's it's more imp imperative than ever that we learn from the strong events base. ⁓ and ⁓ I've been really stoked ⁓ with the development of ⁓ our impact team at EP, ⁓ which which you, James and Jill are heavily involved with, ⁓ to see that ⁓ you know, on an ongoing basis. ⁓ every new thing we build as well as our historical things do have that really strong underpinning. ⁓ I think that's just becoming ⁓ table stakes as as we go forward now. It and which is fantastic for schools.
speaker-2: So let's now get into what it actually looks like to translate learning science into a product that works at scale because that's where it can get really hard. There's a real gap between knowing what the research says and building something that delivers on it in a real classroom with real constraints on a daily basis. Where have you found the transition from theory to product the hardest and what have you learned about getting it right?
speaker-1: I I think that one of the challenges in the classroom is that there is sometimes a a disjunct ⁓ between the things that feel good ⁓ to the learner ⁓ and the things that really move the dial. ⁓ and so, you know, we have a constant point attention to ensure that our learners not only feel good about their learning, but are genuinely challenged and pushed to the point that we're actually improving recall and and and and and building their ⁓ building their skills and their and their thinking capabilities. ⁓ you know, probably that the the classic example here is that it's so easy to go on YouTube and to watch a video and then, you know, a day or two later have completely forgotten all of the details. ⁓ I know that there are a number of ⁓ other ed tech firms out there that have that have you know lent very heavily heavily on passive video consumption.Feels great for the students. ⁓ it's something that we have veered away from ⁓ to be a lot more focused on actually really challenging our students on an ongoing basis because we know that the evidence ri really does support that that that approach. ⁓ that obviously in the classroom raises ⁓ challenges because you you you need to be able to ⁓ remind teachers of what the evidence says ⁓ in a in a head-to-head you know ⁓ comparison, ⁓ if it's purely the learner giving feedback. particularly if it's over a short time period where they haven't yet seen the the the results ⁓ of the effort that they've put in. ⁓ you know, sometimes the the edutainment ⁓ version ⁓ can can feel better and and and and get rated better. ⁓ but we also do know that teachers ⁓ also have ⁓ you know a a fantastic educ education themselves in this evidence, which I think is growing as well. ⁓ Probably another example is is differentiation. ⁓ and and I think this is an this is another ⁓ tug of war that we have ⁓ worked through over the years. ⁓ you know, an ideal learning system would be a hundred percent adaptive to each student. but the reality is that these students sit in a classroom with a teacher. ⁓ and one of our principles from day one is that we are a tool. that teachers deploy in their classrooms and out of class to support their teaching and to support students learning. And that means that sometimes we have to dial back the level of adaptivity to ensure that they have a classroom that they can keep track of. They have a classroom that that that they are in control of and in in in the most positive sense that they can guide and shepherd. There are ⁓ you know, mechanisms in EP where teachers can differentiate as much as they would would like to and many do use those. ⁓ and it allows them to move up ⁓ I I I guess the spectrum to that next level without forcing a hundred percent adaptivity and and differentiation on them when they're perhaps not ready for that yet.
speaker-0: Such a good point because it reminds me of that tug of war you've just talked about between research and practice and having, you know, the two work really well together. And it made me think you you've often talked about ⁓ how important teach and teacher and student feedback is to the product. ⁓ and that tug same tug of war between what teachers and students say they want and what the res what the research is is is saying. ⁓ so I wondered if you must have had a lot of moments along the journey where ⁓ users or teachers and students were saying that they wanted something and it made you rethink what the evidence says. were there examples you can share about that.
speaker-1: Yeah, I mean probably the the best example of that was when we shifted from the vocab tool language perfect out to education perfect. ⁓ so EP ⁓ is a set of structured lessons, whereas ⁓ our ⁓ the language perfect was was very much a targeted vocabulary acquisition tool. ⁓ and when we came ⁓ up with the idea of of of rolling ⁓ some of the best ideas from language perfect out to other subjects, ⁓ we knew that we had a strong evidence base behind us. We knew that it worked, at least we knew that it worked in languages. ⁓ and initially the very first version of ⁓ of of EP that we built was very much a quiz tool. It had it had a lot of these same mechanics of of our ⁓ language perfect tool. and but we built it out for history and geography and science and English. And we took it out into the classroom and spent quite a bit of time with with teachers and students. At that stage, to to be if I'm ⁓ honest with myself, we thought ⁓ that we kind of knew it all. ⁓ and we had this vision for how, you know, we were going to transform education with this with this new tool that we were ⁓ bringing forward, which, you know, was science-based and and did work. ⁓ But when we got into the classrooms, ⁓ we found that really a history classroom and a geography classroom and a science classroom all have different needs. They're trying to teach different shapes of knowledge. ⁓ and that actually, you know, vocabulary with with its very sort of atomic ⁓ facts ⁓ was, if anything, the exception, the outlier ⁓ rather than the rule. And and so we had to adapt. And I think rather than us saying the evidence was wrong, I think perhaps we misread the evidence. And and and when you look a little bit a little bit more broadly, ⁓ you know, there is a whole lot of ⁓ of evidence that then supports this transition that we made. ⁓ Bloom's Taxonomy probably is is the best example is saying actually we were catering incredibly well to one of the levels of Bloom's taxonomy, ⁓ but teachers were trying to teach the whole taxonomy, right? ⁓ and so to support them, to support their classrooms, to support the different shapes of knowledge and skills and to push students, we needed to build a platform that genuinely was able to to map ⁓ these these different shapes. And so EP's lessons ended up looking quite different to what what Language Perfect did. ⁓ we ended up building in very careful scaffolding. ⁓ we ended up ensuring that Anything we were testing students on. ⁓ We built in information up front so that they were actually able to get to the quiz having engaged with the information before they before they got there. And we also ⁓ made a more explicit ⁓ separation between between the learning and and and the quizzing side of things and our assessments, ⁓ and then closed that loop between that larger outer loop between assessments in the ⁓ the information we can gain from those ⁓ and tying those back into which lessons each student need needed to be focused on. So we ended up with a set of series of nested loops within loops ⁓ with with with EP. But you know that was very much based on ⁓ teachers and students saying, look, we see what you're building, ⁓ and it's interesting But it doesn't solve the problem that we need to have solved in our classroom at the moment. ⁓ and and and that forced us to ad adapt and and shift there.
speaker-2: Now Shane, we've talked a bit about evidence. We've talked about evidence in terms of product design and evidence sort of being gathered to make sure the product's working. And I to call out a bit of attention here because I think it's a fairly important tension to recognize. There can be a credibility question ⁓ that any tech company like UP faces when they claim to be evidence based because there's also the commercial reality of the incentive that they have to say that. Schools have been burned by products that dress up engagement metrics as learning outcomes and we've referred to that a little bit already. How do you think about that tension and what do you think EP still needs to do to keep earning that trust?
speaker-1: Jazz, I think this is a great question and and and it's a great call out. I I think when I reflect on this, ⁓ that there for are roughly three stages ⁓ that each ed tech firm goes through when they are grappling with evidence. ⁓ the first is really, you know, the marketing team cherry picking ideas from the research that sort of vaguely matches up ⁓ to concepts or mechanisms that you have in your product. obviously that this is the least valuable for students because ⁓ there's you know there there's not necessarily alignment there. ⁓ the second one, ⁓ which we do see quite a bit of is building in surface level implementations to kind of tick a box and to say we do Bloom's taxonomy because look, we have we have mapped it out here, therefore it's done. ⁓ and but I think that there is a third layer and and and to some extent. I would hope that all ed tech firms sort of move through these phases and get to this third point, the third layer, which is building an evidence base to demonstrate that not only have you understood ⁓ the ideas and the literature, ⁓ but you've crossed the chasm between a great idea and successfully implementing it in a way that moves the dial for students. ⁓ and this is an area that I think I'm incredibly proud to s to have seen EP mature in. ⁓ We have always had evidence as part of our DNA. ⁓ we probably have gone through these three phases ourselves to some extent. ⁓ I'd like to think much earlier than other ⁓ ed techs out there and and have got to this third phase a lot more quickly. but you know, there have definitely been moments in the past ⁓ where we where we've sort of looked around and said, ⁓ okay, well our our product obviously works. I wonder why. ⁓ And I think, you know, this is this is an area where ⁓ our impact team is doing a fantastic job ⁓ of ensuring that, you know, as we go forward, ⁓ every new feature that we that we build in has a very strong evidence space behind it. But not only that, that we're going out and after we build it, validating that the ideas that we know are good ideas actually work in our context and that we've implemented them in a way that actually moves the dial.
speaker-0: Yeah, look, I I think credibility and evidence are a perfect segue into this big topic of AI, ⁓ because we see a lot of players coming to that table trying to quickly spin up evidence to show that their AI tool is working. And I've heard you talk a lot about ⁓ making sure that it's AI for learning. ⁓ and ⁓ in recent years you've been heavily involved, and I mentioned in the introduction, coming back to be personally involved. in the use of AI in ⁓ to power the the formative feedback loop ⁓ and not just in a small way but at scale. ⁓ and it's led us to partner with Professor Rose Lucken and have that that impact study on it. So I thought it'd be really interesting to hear from your point of view what you are hoping ⁓ EP's AI feedback tool can ⁓ really create value for teachers and students and what's the problem it's trying to solve eventually.
speaker-1: Yeah, I mean, ⁓ certainly if I ⁓ think back gee, almost three years ago now, ⁓ when we saw these first ⁓ AI chatbots come out, ⁓ there, you know, obviously I'd been ⁓ in the ZedTech space for many years prior to that and had, I guess, an open set of challenges that I didn't yet know how to solve. ⁓ and ⁓ these AI tools came through and there was this light bulb moment that I think went out. off in many of our heads to say, hey, some of these unsolved challenges might actually be tractable now. One of the ones that ⁓ like really sprang to mind for us ⁓ was how ⁓ how to use AI to assist students as they engage with open-ended ⁓ responses and open-ended questioning. ⁓ so you know EP I I see EP as a as a series of nested feedback loops as as I might have mentioned. ⁓ and We've always had great feedback mechanisms for simple questions. so whether it's vocab, whether it's ⁓ multi-choice questions, for simple questions that computers could traditionally tractably gauge how well a student has done from. We were able to feed that into all of our repetition ⁓ algorithms and the space repetition, ⁓ and and achieve a fantastic recall with those. ⁓ however, As we move from the language perfect tool to EP, ⁓ and as we grappled with how this tool is going to work in ⁓ in a history classroom or an English classroom, we realized that these ⁓ these closed questions were actually really limiting. ⁓ and that a lesson that focused just on those ⁓ could never really open up ⁓ higher order thinking opportunities for students. ⁓ and so our EP lessons these days. have a whole bunch of open-ended textual responses in them. And we we did build in feedback mechanisms for these, but they they involved humans. So so we built, you know, initially a teacher feedback mechanism where teachers could give their students feedback on on their responses. We heard from teachers that this was a huge load on them ⁓ and also that they wanted to engage their students more on this process. And so we built in a peer review mechanism, which was a second attempt. At trying to trying to get feedback to the students about these sizable responses. We heard them from teachers to say theoretic is great, but we can't use it all the time. The overhead's too great for most of our lessons. It's fantastic from the occasional English essay or history essay where everyone's put a lot of time into the response and it's a great chance for reflection, but not all the time. ⁓ And so when when AI came through, ⁓ we saw that there was a fantastic opportunity ⁓ to really tighten that feedback loop that that we had there, the existing feedback loops that we had, ⁓ and to give students feedback in real time that was 100% personalized to their response and situation. ⁓ the other thing that we had seen prior to bringing AI into this was that when teachers were swamped, ⁓ which which we know that they are, ⁓ And when students knew that there was a chance that the work that they submitted may or may not get a review, ⁓ that there was a significant incentive for the students who wanted to take the path of least resistance ⁓ to put rubbish in, ⁓ to not really try. ⁓ and we really wanted also to solve ⁓ that that motivation piece ⁓ there ⁓ and and get all students up to a baseline level ⁓ so that actually they'd put in their best effort. ⁓ and had an opportunity to learn. ⁓ and so what we implemented was that now when students engage with ⁓ these open-ended r questions NEP, ⁓ we have an AI ⁓ tool in the background which analyzes the response. It's given as much context as we can get on on on on the background. You know, this is ⁓ year nine question, it's it's teaching a certain amount. This is how This is what a great answer looks like. This is what's expected of the student. And off the back of that, it gives a set of targeted feedback to the student within a few seconds. ⁓ And the student then has an opportunity to reflect on this feedback. And initially, when we brought this out, we stopped there. But we realized, well, hey, we've got an opportunity here to also address the motivation perspective of this as well. ⁓ and so we introduced something called the learning loop. And what the learning loop does is that answers that are either way off base because there's a significant misunderstanding, ⁓ or where the student hasn't put in the effort really required for this question, ⁓ the student is given that feedback, but then the system says, actually, we would really like you to give this another go. ⁓ and and there's a loop there where that where they need to reach a certain level ⁓ before they can proceed. ⁓ and what we found is that implementing that loop ⁓ had a significant impact on the style of responses that students ⁓ gave, the amount of effort that they put in, and off the back of that, ⁓ the quality of their responses and the amount of learning that they were able to do in the process.
speaker-2: That's great. Thank you, Shannon. Really, really good to get sort of the context and the problems that tool is solving as well as ⁓ hear about how it's working. And yeah, just to confirm the early signals from that AI feedback tool have been incredibly encouraging. So students are engaging with the feedback. Teachers are telling us it's very much filling that gap and that they couldn't fill themselves all the time in terms of giving the feedback. But we are still also very actively. developing that and tracking its performance and measuring its impact on learning. How do you think about that? That idea of shipping something that's showing real promise while we're still sort of building that evidence and iterating it.
speaker-1: Yeah, it's a it's a really great question. And I think it's particularly pertinent for AI at the moment because the technology itself is is evolving so rapidly ⁓ that it's really difficult for the evidence to keep up. Just literally the you know, the time that it takes to run the studies, ⁓ the models have shifted significantly from where they were when when you started. ⁓ I I think I've got a few ⁓ pieces to that answer. I mean, the first is that We had a sound set of principles to suggest that what we were looking at had a very high chance of working. ⁓ we knew ⁓ you know, from from the evidence that timely, personalized ⁓ feedback has a significant impact on students' ability to adapt and respond and to learn. ⁓ and so and and then the other thing is that we knew that, you know. This was an area of the product that we were quite dissatisfied with the status quo with and that we knew that there was quite a bit of room to to improve the the ESSM situation. So we were starting from what was an acceptable base point before, but actually a b a a baseline point that had lots of room for improvement. ⁓ and then applying some sensible principles around how we might improve that. The other thing to say is that we spent ⁓ over 12 months going from our first idea of this. ⁓ through to our official release. And and you know, the first that had a number of parts to it. The first is, you know, very consciously framing how the AI was going to ⁓ be integrated into the product. You know, there is ⁓ a temptation, ⁓ it's very easy to drop a chat bot anywhere ⁓ and say, right, I have AI on my product now. ⁓ but, you know, chatbots are open-ended. ⁓ and ⁓ there is some interesting ⁓ evidence that suggests that depending on who the student is, if you've got very, very motivated students, ⁓ they can get value out of a chat bot in isolation. ⁓ but there's also always a temptation to get the chatbot to do the heavy lifting for you. ⁓ at which stage ⁓ at which stage you've lost the learning opportunity. And so we were very conscious about how we integrated our AI feedback. It's not a chat bot. ⁓ it and it doesn't do the heavy lifting. It refuses to do the heavy lifting for for students. ⁓ but it does give them guidance around ⁓ where they can go to next. ⁓ the second piece was extensive dog fooding. So ⁓ we used ⁓ this tool for hundreds of hours ⁓ ourselves, ⁓ both in our own learning and and and and in simulated learning environments to see does this actually feel right? Does this that does it does this seem to work? ⁓ we also spent numerous hours, you know, doing interviews with both teachers and students, ⁓ and then went into classrooms with early versions of this ⁓ to get direct feedback from both teachers and students ⁓ about their experience of it up front. and so by the time we launched, we had every reason to believe ⁓ that this had a very high chance of working. And then upon launch, we immediately started tracking. ⁓ how people engaged with it and collecting ⁓ a first set of effica efficacy evidence ⁓ to validate that what we had built was actually out in the real world and working.
speaker-0: Thank you for sharing that because it it's so interesting reading the research about AI in the feedback loop. ⁓ I think everyone is excited about the possibilities of what it can do. ⁓ but we know that there can be a gap between, you know, what what what we're hoping for and what the reality is in practice. ⁓ that point around the feedback needing to be actionable is a big piece and that the AI needs to know exactly what, you know, needs to be close to the student to know exactly what comes next. And in concert with teachers. We know what an important role teachers play in in that feedback loop. And it made me think about you, James, because you're out there in classrooms with ⁓ hundreds of teachers all the time, especially through our impact study. So I just wondered if you're seeing firsthand what the feedback loop ⁓ you know, is is achieving in practice in classrooms.
speaker-2: I'm really confident and comfortable saying that the feedback loop is already doing something really, really meaningful. So students are getting responses to work that in a lot of cases would have gone unmarked and they're improving their answers. They're having another go. Teachers, they're building trust in it. They can see the feedback. They're increasingly comfortable with the quality. But the big open question is for us at the moment, sort of causation versus correlation almost, which is we're seeing improvement in the task in the moment. But we need to understand whether that's translating to deeper learning over time. That's where this really matters. We're also, and Shane, you refer to this, also when we spoke to Rose Lackin, we're also really watching for this risk of offloaded thinking and making sure that AI is sort of prompting students to think harder and not giving them an easy out. I'm interested though. So Shane, what's your ambition for where this goes next? not just as a feature, but in terms of the evidence that we build around it.
speaker-1: I I I think really I I think you hit the nail on the head saying that what we'd really like to see here is ⁓ is to track this from ⁓ implementation all the way through to long long term recall ⁓ and and prove at end to end ⁓ that having implemented this, ⁓ students have learnt more and and are better off in the process. ⁓ so far we've got some fantastic data to indicate that in the moment ⁓ they have responded more more effectively. ⁓ and I think ⁓ that it it's gonna be it's gonna be fascinating but also fantastic to follow this through and to see what that does their long-term recall ⁓ and so hopefully demonstrate that actually a more personalized adaptive set of feedback for them ⁓ does genuinely move the dial for them. ⁓ in terms of the feature itself, I I I think particularly as the technology continues to evolve. I'm really looking forward to being able to make this more and more personalized for each student, ⁓ and and being able to give ⁓ the AI as much context as possible, ⁓ so that when it gives feedback, it really is speaking to ⁓ this individual ⁓ and and really able to move them ⁓ through their journey ⁓ towards a deeper understanding, I guess.
speaker-0: I I'm really glad you called out long term memory because a previous guest, Nitty Stage Dava, ⁓ helped us understand that the definition of learning is a stable change in long term memory. And we know that the the interplay between short term memory ⁓ and long term re memory ⁓ and and a role that feedback has in that loop. So and AI can help in concert with the teacher and and it makes the impact study that that ⁓ I know you're leading, James, ⁓ so crucial for measuring that. ⁓ and it makes me think about the future and how we look forward to to where we're going with all of this. We've we've subtitled this episode, It's More Urgent Than Ever. And it does really feel like that with stakes are very high, schools are under a lot of pressure and it's coming at them from every direction. We've got workload issues for teachers, we've got budget constraints. We obviously have AI disrupting everything and and everyone ⁓ trying to work out how to make the most of that technology. We've also got this back to basics movement or or rather forward to fundamentals as we've called it ⁓ in in other conversations. But all of it is really asking for accountability, ⁓ which which is not a bad thing, but it's making me think about the approach that you've described, Shane. ⁓ evidence first, close to teachers, iterate honestly. does it matter more now? for you personally than when you started?
speaker-1: I I think it's you know, if I if I look back to where we started from and we think back to one of Cray's underlying principles, that underlying principle was all of our students have a limited amount of time in their lives. And so any time that they spend on our platform is precious. ⁓ and, you know, that applies to school environments too, you know, teachers and students of limited time in their day. And every time a teacher decides to ⁓ go to EP for ⁓ a a task to assist their teaching, there's a real opportunity cost for them choosing this rather than another style of teaching, whether it's ⁓ being up the front or going into small groups, for example. ⁓ and I think we have a duty of care ⁓ to ensure that they're spending this precious time in a way that's really gonna move the dial for their students. ⁓ and I think I think that is It has always been true, but it is very true today. and ⁓ you know, ⁓ it it is interesting to reflect on, you know, both schools ⁓ for both teachers and students, on on the value of an hour of of their day. and it's also then interesting to reflect on well, in in the scale of things, ⁓ when we're, you know. purchasing software or when we're when we're when we're thinking about what we're going to get our teachers and students to, do we really take into account just how valuable that time is? And do we take into account the opportunity cost of of applying a certain technique in a place as opposed to other options at that time?
speaker-2: Nice, it sort of pains me to say, but this is the last question for you Shane. So if another founder of an edtech firm came to you tomorrow and said that they genuinely want to put learning first, what would you tell them that they have to get right?
speaker-1: I think if there was one action that they could take that would move them in the best possible direction, it would be to spend as much time as possible with both teachers and students. ⁓ because ⁓ teachers have an amazing depth of knowledge themselves around both the evidence but also what works in practice. they ⁓ they tend to be very willing to share that. I know that we have always been warmly welcomed. ⁓ into ⁓ into classrooms ⁓ every everywhere that we've been and teachers have been jet very generous with their ideas and with their time with us. ⁓ and I you know of everything I think that puts you on the right pathway. ⁓ beyond that, obviously, you know, ⁓ go out and try and understand what the evidence actually says. ⁓ ensure that when you're implementing idea that once you've implemented you actually check that it works. ⁓ but I think the founding principle is that if you're spending a lot of time with teachers and students, ⁓ they're probably gonna tell you if it's not working. ⁓ and and that's gonna set you it set you back on track. So if I had one recommendation, it would be that.
speaker-0: Yep. It's ⁓ very hard to keep it simple i i in many ways, and there's many problems that we can solve, and you've certainly solved a lot of them over your time. ⁓ so thank you. Thank you for for ⁓ everyone who has used EP for you and and Craig starting this incredible company. where should listeners go to find out more about the journey and everything we've talked about today?
speaker-1: I think probably the the most interesting ⁓ place for them to go, we've just recently launched an impact page on our website, ⁓ which walks through a whole lot of ⁓ the work that we've recently been doing ⁓ in demonstrating this. So that's ⁓ w dot educationperfect.com slash impact. I think that's a fantastic starting point. ⁓ and it's a fantastic resource for teachers and students full stop.
speaker-0: put those notes ⁓ in in the show notes. But thank you so much for this conversation today, Shane. It's been an absolute pleasure.
speaker-1: Thank you both Jill and James. It's been an an absolute pleasure chatting with you. Thank you.
speaker-2: Hello, Jill. In previous episodes, we've heard from experts pushing us forward on evidence-based instruction and AI in education. Today, we're doing something a little bit different. We're talking to someone who's been building a company on those principles for nearly two decades and asking them how that has looked in practice.
speaker-0: So our guest today is none other than Shane Smith, co founder of Education Perfect. Hi Shane.
speaker-1: Hi there, Jill and James. It's a real pleasure to be here. ⁓ and congratulations on this really successful podcast series. There have been some really interesting ideas flowing through and looking forward to today's chat.
speaker-0: Well, we have been looking forward to this series for quite a while because for those of you who don't know, Shane co-founded EP with his brother Craig in 2007. And he did this remarkably while studying medicine at the University of Auckland. And from the very beginning, Shane and Craig built EP around principles of cognitive science. And that was way, way, way back before learning science became the movement that it is today. And after stepping away for a number of years, I think it has been quite a few. Shane is now back to lead EP's AI initiatives, including the development of our AI-powered feedback tool, which is being used by literally hundreds of thousands of students every year. Shane, welcome again to Well Measured.
speaker-1: Thank you so much, Jill.
speaker-0: Okay, well Shane, you started building Education Perfect almost twenty years ago, if you can believe that, and as a medical student, as you've told us. So clearly you didn't come from teaching. ⁓ I'm just wondering what made you want to change your game and innovate in education?
speaker-1: It's really interesting, Jill. ⁓ I I guess I had two entry points into EP. The first one was absolutely personal interest. And then the second one was very focused on impact. ⁓ so when I was going through university, ⁓ Craig had made the very first version of what was language perfect back then. ⁓ and I was just fascinated by ⁓ how our brains worked, ⁓ how ⁓ how we learned, ⁓ what good learning looked like. ⁓ and so I got pulled into some of those. really meaty, interesting challenges then as as a as a personal interest piece. But then I actually ⁓ finished my studies and and graduated and worked for a year as a junior doctor. and at the end of that, Craig called me and said, Shane, we're gonna go from teaching languages and teaching vocab in particular, ⁓ through to teaching ⁓ English and science and hit history and humanities. ⁓ and and at that stage, I guess I had a choice. ⁓ you know, I knew that this was something that I was passionate about and found really interesting. ⁓ but I also knew that as a doctor I could have quite a significant impact on each individual patient's life. And so I sort of did a bit of a a a weighted comparison and said, you know, I can either have a large impact on a very small number of people's lives as a doctor, or I can have a much smaller impact potentially, but on a much larger scale. And, you know, I've been really stoked that we've been able to scale EP ⁓ now to many millions of students who've been able to benefit from from the work we put in. and I think that that trade off ⁓ worked out pretty well in all in all.
speaker-2: That actually segues nicely into my into my question. So one of things I find really interesting about that early period is that you weren't just reading the research and applying it in a vacuum. You're also obsessively and I hope that word obsessively is OK. ⁓ Really close to what teachers and students were actually experiencing with the product. How did you balance what the science said should work?
speaker-1: It is accurate.
speaker-2: with what you're seeing and hearing from classrooms as you are going out there and seeing the product in action.
speaker-1: Yeah. So I guess we had a few ⁓ inputs to this. ⁓ the very first version of Language Perfect was obviously built ⁓ by a learner for a learner. ⁓ and and so ⁓ you know, that meant that we did a lot of testing on ourselves. ⁓ we we you know, Craig was using it to actively learn. ⁓ but then beyond that, when when he ⁓ graduated and went out to university himself, ⁓ I know that he spent many, many hours ⁓ in his student dorms. ⁓ pulling people aside and saying, Hey, ⁓ you're learning French or German or whatever language it was that the student was learning at the time. Can you sit with me for half an hour? I'd love to see you ⁓ to use this and and and see how you interact with it. ⁓ and he at that stage had an internal rule where he would sit silently no matter what the person did, ⁓ and just observe. ⁓ and if they got stuck, then then then he would observe that and then go away and and and and try and improve things. ⁓ But we were also fortunate enough that ⁓ right from the get-go, we're we were able to ⁓ get involved with some really supportive ⁓ schools. ⁓ at the time, I'm pretty sure Rangatoto College was one of those very supportive schools. ⁓ and we had a a great ⁓ relationship with our own school, St. Kenigan College. and with that, we were able to spend time in classrooms. ⁓ we were able to talk to teachers, we were able to talk to students, ⁓ and really overlay that over this ⁓ almost theoretical ⁓ basis for things ⁓ and and and mesh these these ideas and these concepts together because ⁓ it's all very well ⁓ to implement an idea that in a study has proven in a certain context ⁓ but what you really need to ensure that your is that your implementation of that idea in your context actually moves the dial and actually delivers what the research suggests that it should. ⁓ and and that very close ⁓ contact with students and teachers I guess was our it was our initial layer of validation there. Obviously since then we've we've gone and got a lot more formal validation which which ⁓ backs up ⁓ that that plan that this really does the ⁓ move the dial as well, ⁓ which I'm sure we'll be talking about today.
speaker-0: Yes, well we we will because fast forward twenty years and from those humble beginnings of language is perfect in New Zealand, ⁓ we're now covering all subjects across the curriculum, across New Zealand, Australia, Canada, internationally, ⁓ in in a lot of schools across the world. So so that has really expanded. And at the same time, the science of learning has become a much bigger conversation as well, especially in the last five or ten years. ⁓ it's now shaping public policy, it's in the mainstream, everyone's talking about it. So it must be nice watching the world catch up to where you were two decades ago. ⁓ but what I was thinking about is how has the evidence ⁓ around cognition, around the learning science changed anything in your mind about what EP should be doing right now?
speaker-1: It's a it's it's a great question. And I think, you know, watching ⁓ learning science become far more front and center has been a really healthy and I'd say really necessary evolution ⁓ in the space. ⁓ certainly when we ⁓ started building Language Perfect, it was a bit of a wild west. ⁓ and ⁓ I I I think you know ed tech itself has gone through a few phases during this time. You know, I think initially ⁓ there was this initial phase of excitement to say, hey, we have these devices in classrooms. ⁓ we want things that engage our students, we want things that are exciting. and and, you know, off the back of that, there were that were there were a series of tools that were, you know, engaging and exciting. ⁓ not necessarily so heavily learning paced. ⁓ there was also, you know, during COVID, there was this massive boost and rollout of of these tools. And since then we have seen some degree of disillusionment from from teachers saying, you know, a series of these tools which we really thought were going to move the dial for us, we have implemented, we've done our best, ⁓ and haven't have the haven't had the impact ⁓ that we thought. ⁓ and no, I think our journey ⁓ in parallel to that at EP has been ⁓ to ensure that as we've gone through this process, We've continued going back to the evidence. ⁓ but then also we have gone out ⁓ and more and more strengthened the evidence base ⁓ and and I guess the proof point around the product ⁓ to ensure that when we do get to this point of disillusionment, we can say, okay, well, it is true that some tools out there don't seem to move the dial. ⁓ but you can see here that that actually EP really cares about this. We have from from the ground up, it's been part of our DNA. ⁓ so why don't you give this a go? and I think, you know, if anything, it it it's it's more imp imperative than ever that we learn from the strong events base. ⁓ and ⁓ I've been really stoked ⁓ with the development of ⁓ our impact team at EP, ⁓ which which you, James and Jill are heavily involved with, ⁓ to see that ⁓ you know, on an ongoing basis. ⁓ every new thing we build as well as our historical things do have that really strong underpinning. ⁓ I think that's just becoming ⁓ table stakes as as we go forward now. It and which is fantastic for schools.
speaker-2: So let's now get into what it actually looks like to translate learning science into a product that works at scale because that's where it can get really hard. There's a real gap between knowing what the research says and building something that delivers on it in a real classroom with real constraints on a daily basis. Where have you found the transition from theory to product the hardest and what have you learned about getting it right?
speaker-1: I I think that one of the challenges in the classroom is that there is sometimes a a disjunct ⁓ between the things that feel good ⁓ to the learner ⁓ and the things that really move the dial. ⁓ and so, you know, we have a constant point attention to ensure that our learners not only feel good about their learning, but are genuinely challenged and pushed to the point that we're actually improving recall and and and and and building their ⁓ building their skills and their and their thinking capabilities. ⁓ you know, probably that the the classic example here is that it's so easy to go on YouTube and to watch a video and then, you know, a day or two later have completely forgotten all of the details. ⁓ I know that there are a number of ⁓ other ed tech firms out there that have that have you know lent very heavily heavily on passive video consumption.Feels great for the students. ⁓ it's something that we have veered away from ⁓ to be a lot more focused on actually really challenging our students on an ongoing basis because we know that the evidence ri really does support that that that approach. ⁓ that obviously in the classroom raises ⁓ challenges because you you you need to be able to ⁓ remind teachers of what the evidence says ⁓ in a in a head-to-head you know ⁓ comparison, ⁓ if it's purely the learner giving feedback. particularly if it's over a short time period where they haven't yet seen the the the results ⁓ of the effort that they've put in. ⁓ you know, sometimes the the edutainment ⁓ version ⁓ can can feel better and and and and get rated better. ⁓ but we also do know that teachers ⁓ also have ⁓ you know a a fantastic educ education themselves in this evidence, which I think is growing as well. ⁓ Probably another example is is differentiation. ⁓ and and I think this is an this is another ⁓ tug of war that we have ⁓ worked through over the years. ⁓ you know, an ideal learning system would be a hundred percent adaptive to each student. but the reality is that these students sit in a classroom with a teacher. ⁓ and one of our principles from day one is that we are a tool. that teachers deploy in their classrooms and out of class to support their teaching and to support students learning. And that means that sometimes we have to dial back the level of adaptivity to ensure that they have a classroom that they can keep track of. They have a classroom that that that they are in control of and in in in the most positive sense that they can guide and shepherd. There are ⁓ you know, mechanisms in EP where teachers can differentiate as much as they would would like to and many do use those. ⁓ and it allows them to move up ⁓ I I I guess the spectrum to that next level without forcing a hundred percent adaptivity and and differentiation on them when they're perhaps not ready for that yet.
speaker-0: Such a good point because it reminds me of that tug of war you've just talked about between research and practice and having, you know, the two work really well together. And it made me think you you've often talked about ⁓ how important teach and teacher and student feedback is to the product. ⁓ and that tug same tug of war between what teachers and students say they want and what the res what the research is is is saying. ⁓ so I wondered if you must have had a lot of moments along the journey where ⁓ users or teachers and students were saying that they wanted something and it made you rethink what the evidence says. were there examples you can share about that.
speaker-1: Yeah, I mean probably the the best example of that was when we shifted from the vocab tool language perfect out to education perfect. ⁓ so EP ⁓ is a set of structured lessons, whereas ⁓ our ⁓ the language perfect was was very much a targeted vocabulary acquisition tool. ⁓ and when we came ⁓ up with the idea of of of rolling ⁓ some of the best ideas from language perfect out to other subjects, ⁓ we knew that we had a strong evidence base behind us. We knew that it worked, at least we knew that it worked in languages. ⁓ and initially the very first version of ⁓ of of EP that we built was very much a quiz tool. It had it had a lot of these same mechanics of of our ⁓ language perfect tool. and but we built it out for history and geography and science and English. And we took it out into the classroom and spent quite a bit of time with with teachers and students. At that stage, to to be if I'm ⁓ honest with myself, we thought ⁓ that we kind of knew it all. ⁓ and we had this vision for how, you know, we were going to transform education with this with this new tool that we were ⁓ bringing forward, which, you know, was science-based and and did work. ⁓ But when we got into the classrooms, ⁓ we found that really a history classroom and a geography classroom and a science classroom all have different needs. They're trying to teach different shapes of knowledge. ⁓ and that actually, you know, vocabulary with with its very sort of atomic ⁓ facts ⁓ was, if anything, the exception, the outlier ⁓ rather than the rule. And and so we had to adapt. And I think rather than us saying the evidence was wrong, I think perhaps we misread the evidence. And and and when you look a little bit a little bit more broadly, ⁓ you know, there is a whole lot of ⁓ of evidence that then supports this transition that we made. ⁓ Bloom's Taxonomy probably is is the best example is saying actually we were catering incredibly well to one of the levels of Bloom's taxonomy, ⁓ but teachers were trying to teach the whole taxonomy, right? ⁓ and so to support them, to support their classrooms, to support the different shapes of knowledge and skills and to push students, we needed to build a platform that genuinely was able to to map ⁓ these these different shapes. And so EP's lessons ended up looking quite different to what what Language Perfect did. ⁓ we ended up building in very careful scaffolding. ⁓ we ended up ensuring that Anything we were testing students on. ⁓ We built in information up front so that they were actually able to get to the quiz having engaged with the information before they before they got there. And we also ⁓ made a more explicit ⁓ separation between between the learning and and and the quizzing side of things and our assessments, ⁓ and then closed that loop between that larger outer loop between assessments in the ⁓ the information we can gain from those ⁓ and tying those back into which lessons each student need needed to be focused on. So we ended up with a set of series of nested loops within loops ⁓ with with with EP. But you know that was very much based on ⁓ teachers and students saying, look, we see what you're building, ⁓ and it's interesting But it doesn't solve the problem that we need to have solved in our classroom at the moment. ⁓ and and and that forced us to ad adapt and and shift there.
speaker-2: Now Shane, we've talked a bit about evidence. We've talked about evidence in terms of product design and evidence sort of being gathered to make sure the product's working. And I to call out a bit of attention here because I think it's a fairly important tension to recognize. There can be a credibility question ⁓ that any tech company like UP faces when they claim to be evidence based because there's also the commercial reality of the incentive that they have to say that. Schools have been burned by products that dress up engagement metrics as learning outcomes and we've referred to that a little bit already. How do you think about that tension and what do you think EP still needs to do to keep earning that trust?
speaker-1: Jazz, I think this is a great question and and and it's a great call out. I I think when I reflect on this, ⁓ that there for are roughly three stages ⁓ that each ed tech firm goes through when they are grappling with evidence. ⁓ the first is really, you know, the marketing team cherry picking ideas from the research that sort of vaguely matches up ⁓ to concepts or mechanisms that you have in your product. obviously that this is the least valuable for students because ⁓ there's you know there there's not necessarily alignment there. ⁓ the second one, ⁓ which we do see quite a bit of is building in surface level implementations to kind of tick a box and to say we do Bloom's taxonomy because look, we have we have mapped it out here, therefore it's done. ⁓ and but I think that there is a third layer and and and to some extent. I would hope that all ed tech firms sort of move through these phases and get to this third point, the third layer, which is building an evidence base to demonstrate that not only have you understood ⁓ the ideas and the literature, ⁓ but you've crossed the chasm between a great idea and successfully implementing it in a way that moves the dial for students. ⁓ and this is an area that I think I'm incredibly proud to s to have seen EP mature in. ⁓ We have always had evidence as part of our DNA. ⁓ we probably have gone through these three phases ourselves to some extent. ⁓ I'd like to think much earlier than other ⁓ ed techs out there and and have got to this third phase a lot more quickly. but you know, there have definitely been moments in the past ⁓ where we where we've sort of looked around and said, ⁓ okay, well our our product obviously works. I wonder why. ⁓ And I think, you know, this is this is an area where ⁓ our impact team is doing a fantastic job ⁓ of ensuring that, you know, as we go forward, ⁓ every new feature that we that we build in has a very strong evidence space behind it. But not only that, that we're going out and after we build it, validating that the ideas that we know are good ideas actually work in our context and that we've implemented them in a way that actually moves the dial.
speaker-0: Yeah, look, I I think credibility and evidence are a perfect segue into this big topic of AI, ⁓ because we see a lot of players coming to that table trying to quickly spin up evidence to show that their AI tool is working. And I've heard you talk a lot about ⁓ making sure that it's AI for learning. ⁓ and ⁓ in recent years you've been heavily involved, and I mentioned in the introduction, coming back to be personally involved. in the use of AI in ⁓ to power the the formative feedback loop ⁓ and not just in a small way but at scale. ⁓ and it's led us to partner with Professor Rose Lucken and have that that impact study on it. So I thought it'd be really interesting to hear from your point of view what you are hoping ⁓ EP's AI feedback tool can ⁓ really create value for teachers and students and what's the problem it's trying to solve eventually.
speaker-1: Yeah, I mean, ⁓ certainly if I ⁓ think back gee, almost three years ago now, ⁓ when we saw these first ⁓ AI chatbots come out, ⁓ there, you know, obviously I'd been ⁓ in the ZedTech space for many years prior to that and had, I guess, an open set of challenges that I didn't yet know how to solve. ⁓ and ⁓ these AI tools came through and there was this light bulb moment that I think went out. off in many of our heads to say, hey, some of these unsolved challenges might actually be tractable now. One of the ones that ⁓ like really sprang to mind for us ⁓ was how ⁓ how to use AI to assist students as they engage with open-ended ⁓ responses and open-ended questioning. ⁓ so you know EP I I see EP as a as a series of nested feedback loops as as I might have mentioned. ⁓ and We've always had great feedback mechanisms for simple questions. so whether it's vocab, whether it's ⁓ multi-choice questions, for simple questions that computers could traditionally tractably gauge how well a student has done from. We were able to feed that into all of our repetition ⁓ algorithms and the space repetition, ⁓ and and achieve a fantastic recall with those. ⁓ however, As we move from the language perfect tool to EP, ⁓ and as we grappled with how this tool is going to work in ⁓ in a history classroom or an English classroom, we realized that these ⁓ these closed questions were actually really limiting. ⁓ and that a lesson that focused just on those ⁓ could never really open up ⁓ higher order thinking opportunities for students. ⁓ and so our EP lessons these days. have a whole bunch of open-ended textual responses in them. And we we did build in feedback mechanisms for these, but they they involved humans. So so we built, you know, initially a teacher feedback mechanism where teachers could give their students feedback on on their responses. We heard from teachers that this was a huge load on them ⁓ and also that they wanted to engage their students more on this process. And so we built in a peer review mechanism, which was a second attempt. At trying to trying to get feedback to the students about these sizable responses. We heard them from teachers to say theoretic is great, but we can't use it all the time. The overhead's too great for most of our lessons. It's fantastic from the occasional English essay or history essay where everyone's put a lot of time into the response and it's a great chance for reflection, but not all the time. ⁓ And so when when AI came through, ⁓ we saw that there was a fantastic opportunity ⁓ to really tighten that feedback loop that that we had there, the existing feedback loops that we had, ⁓ and to give students feedback in real time that was 100% personalized to their response and situation. ⁓ the other thing that we had seen prior to bringing AI into this was that when teachers were swamped, ⁓ which which we know that they are, ⁓ And when students knew that there was a chance that the work that they submitted may or may not get a review, ⁓ that there was a significant incentive for the students who wanted to take the path of least resistance ⁓ to put rubbish in, ⁓ to not really try. ⁓ and we really wanted also to solve ⁓ that that motivation piece ⁓ there ⁓ and and get all students up to a baseline level ⁓ so that actually they'd put in their best effort. ⁓ and had an opportunity to learn. ⁓ and so what we implemented was that now when students engage with ⁓ these open-ended r questions NEP, ⁓ we have an AI ⁓ tool in the background which analyzes the response. It's given as much context as we can get on on on on the background. You know, this is ⁓ year nine question, it's it's teaching a certain amount. This is how This is what a great answer looks like. This is what's expected of the student. And off the back of that, it gives a set of targeted feedback to the student within a few seconds. ⁓ And the student then has an opportunity to reflect on this feedback. And initially, when we brought this out, we stopped there. But we realized, well, hey, we've got an opportunity here to also address the motivation perspective of this as well. ⁓ and so we introduced something called the learning loop. And what the learning loop does is that answers that are either way off base because there's a significant misunderstanding, ⁓ or where the student hasn't put in the effort really required for this question, ⁓ the student is given that feedback, but then the system says, actually, we would really like you to give this another go. ⁓ and and there's a loop there where that where they need to reach a certain level ⁓ before they can proceed. ⁓ and what we found is that implementing that loop ⁓ had a significant impact on the style of responses that students ⁓ gave, the amount of effort that they put in, and off the back of that, ⁓ the quality of their responses and the amount of learning that they were able to do in the process.
speaker-2: That's great. Thank you, Shannon. Really, really good to get sort of the context and the problems that tool is solving as well as ⁓ hear about how it's working. And yeah, just to confirm the early signals from that AI feedback tool have been incredibly encouraging. So students are engaging with the feedback. Teachers are telling us it's very much filling that gap and that they couldn't fill themselves all the time in terms of giving the feedback. But we are still also very actively. developing that and tracking its performance and measuring its impact on learning. How do you think about that? That idea of shipping something that's showing real promise while we're still sort of building that evidence and iterating it.
speaker-1: Yeah, it's a it's a really great question. And I think it's particularly pertinent for AI at the moment because the technology itself is is evolving so rapidly ⁓ that it's really difficult for the evidence to keep up. Just literally the you know, the time that it takes to run the studies, ⁓ the models have shifted significantly from where they were when when you started. ⁓ I I think I've got a few ⁓ pieces to that answer. I mean, the first is that We had a sound set of principles to suggest that what we were looking at had a very high chance of working. ⁓ we knew ⁓ you know, from from the evidence that timely, personalized ⁓ feedback has a significant impact on students' ability to adapt and respond and to learn. ⁓ and so and and then the other thing is that we knew that, you know. This was an area of the product that we were quite dissatisfied with the status quo with and that we knew that there was quite a bit of room to to improve the the ESSM situation. So we were starting from what was an acceptable base point before, but actually a b a a baseline point that had lots of room for improvement. ⁓ and then applying some sensible principles around how we might improve that. The other thing to say is that we spent ⁓ over 12 months going from our first idea of this. ⁓ through to our official release. And and you know, the first that had a number of parts to it. The first is, you know, very consciously framing how the AI was going to ⁓ be integrated into the product. You know, there is ⁓ a temptation, ⁓ it's very easy to drop a chat bot anywhere ⁓ and say, right, I have AI on my product now. ⁓ but, you know, chatbots are open-ended. ⁓ and ⁓ there is some interesting ⁓ evidence that suggests that depending on who the student is, if you've got very, very motivated students, ⁓ they can get value out of a chat bot in isolation. ⁓ but there's also always a temptation to get the chatbot to do the heavy lifting for you. ⁓ at which stage ⁓ at which stage you've lost the learning opportunity. And so we were very conscious about how we integrated our AI feedback. It's not a chat bot. ⁓ it and it doesn't do the heavy lifting. It refuses to do the heavy lifting for for students. ⁓ but it does give them guidance around ⁓ where they can go to next. ⁓ the second piece was extensive dog fooding. So ⁓ we used ⁓ this tool for hundreds of hours ⁓ ourselves, ⁓ both in our own learning and and and and in simulated learning environments to see does this actually feel right? Does this that does it does this seem to work? ⁓ we also spent numerous hours, you know, doing interviews with both teachers and students, ⁓ and then went into classrooms with early versions of this ⁓ to get direct feedback from both teachers and students ⁓ about their experience of it up front. and so by the time we launched, we had every reason to believe ⁓ that this had a very high chance of working. And then upon launch, we immediately started tracking. ⁓ how people engaged with it and collecting ⁓ a first set of effica efficacy evidence ⁓ to validate that what we had built was actually out in the real world and working.
speaker-0: Thank you for sharing that because it it's so interesting reading the research about AI in the feedback loop. ⁓ I think everyone is excited about the possibilities of what it can do. ⁓ but we know that there can be a gap between, you know, what what what we're hoping for and what the reality is in practice. ⁓ that point around the feedback needing to be actionable is a big piece and that the AI needs to know exactly what, you know, needs to be close to the student to know exactly what comes next. And in concert with teachers. We know what an important role teachers play in in that feedback loop. And it made me think about you, James, because you're out there in classrooms with ⁓ hundreds of teachers all the time, especially through our impact study. So I just wondered if you're seeing firsthand what the feedback loop ⁓ you know, is is achieving in practice in classrooms.
speaker-2: I'm really confident and comfortable saying that the feedback loop is already doing something really, really meaningful. So students are getting responses to work that in a lot of cases would have gone unmarked and they're improving their answers. They're having another go. Teachers, they're building trust in it. They can see the feedback. They're increasingly comfortable with the quality. But the big open question is for us at the moment, sort of causation versus correlation almost, which is we're seeing improvement in the task in the moment. But we need to understand whether that's translating to deeper learning over time. That's where this really matters. We're also, and Shane, you refer to this, also when we spoke to Rose Lackin, we're also really watching for this risk of offloaded thinking and making sure that AI is sort of prompting students to think harder and not giving them an easy out. I'm interested though. So Shane, what's your ambition for where this goes next? not just as a feature, but in terms of the evidence that we build around it.
speaker-1: I I I think really I I think you hit the nail on the head saying that what we'd really like to see here is ⁓ is to track this from ⁓ implementation all the way through to long long term recall ⁓ and and prove at end to end ⁓ that having implemented this, ⁓ students have learnt more and and are better off in the process. ⁓ so far we've got some fantastic data to indicate that in the moment ⁓ they have responded more more effectively. ⁓ and I think ⁓ that it it's gonna be it's gonna be fascinating but also fantastic to follow this through and to see what that does their long-term recall ⁓ and so hopefully demonstrate that actually a more personalized adaptive set of feedback for them ⁓ does genuinely move the dial for them. ⁓ in terms of the feature itself, I I I think particularly as the technology continues to evolve. I'm really looking forward to being able to make this more and more personalized for each student, ⁓ and and being able to give ⁓ the AI as much context as possible, ⁓ so that when it gives feedback, it really is speaking to ⁓ this individual ⁓ and and really able to move them ⁓ through their journey ⁓ towards a deeper understanding, I guess.
speaker-0: I I'm really glad you called out long term memory because a previous guest, Nitty Stage Dava, ⁓ helped us understand that the definition of learning is a stable change in long term memory. And we know that the the interplay between short term memory ⁓ and long term re memory ⁓ and and a role that feedback has in that loop. So and AI can help in concert with the teacher and and it makes the impact study that that ⁓ I know you're leading, James, ⁓ so crucial for measuring that. ⁓ and it makes me think about the future and how we look forward to to where we're going with all of this. We've we've subtitled this episode, It's More Urgent Than Ever. And it does really feel like that with stakes are very high, schools are under a lot of pressure and it's coming at them from every direction. We've got workload issues for teachers, we've got budget constraints. We obviously have AI disrupting everything and and everyone ⁓ trying to work out how to make the most of that technology. We've also got this back to basics movement or or rather forward to fundamentals as we've called it ⁓ in in other conversations. But all of it is really asking for accountability, ⁓ which which is not a bad thing, but it's making me think about the approach that you've described, Shane. ⁓ evidence first, close to teachers, iterate honestly. does it matter more now? for you personally than when you started?
speaker-1: I I think it's you know, if I if I look back to where we started from and we think back to one of Cray's underlying principles, that underlying principle was all of our students have a limited amount of time in their lives. And so any time that they spend on our platform is precious. ⁓ and, you know, that applies to school environments too, you know, teachers and students of limited time in their day. And every time a teacher decides to ⁓ go to EP for ⁓ a a task to assist their teaching, there's a real opportunity cost for them choosing this rather than another style of teaching, whether it's ⁓ being up the front or going into small groups, for example. ⁓ and I think we have a duty of care ⁓ to ensure that they're spending this precious time in a way that's really gonna move the dial for their students. ⁓ and I think I think that is It has always been true, but it is very true today. and ⁓ you know, ⁓ it it is interesting to reflect on, you know, both schools ⁓ for both teachers and students, on on the value of an hour of of their day. and it's also then interesting to reflect on well, in in the scale of things, ⁓ when we're, you know. purchasing software or when we're when we're when we're thinking about what we're going to get our teachers and students to, do we really take into account just how valuable that time is? And do we take into account the opportunity cost of of applying a certain technique in a place as opposed to other options at that time?
speaker-2: Nice, it sort of pains me to say, but this is the last question for you Shane. So if another founder of an edtech firm came to you tomorrow and said that they genuinely want to put learning first, what would you tell them that they have to get right?
speaker-1: I think if there was one action that they could take that would move them in the best possible direction, it would be to spend as much time as possible with both teachers and students. ⁓ because ⁓ teachers have an amazing depth of knowledge themselves around both the evidence but also what works in practice. they ⁓ they tend to be very willing to share that. I know that we have always been warmly welcomed. ⁓ into ⁓ into classrooms ⁓ every everywhere that we've been and teachers have been jet very generous with their ideas and with their time with us. ⁓ and I you know of everything I think that puts you on the right pathway. ⁓ beyond that, obviously, you know, ⁓ go out and try and understand what the evidence actually says. ⁓ ensure that when you're implementing idea that once you've implemented you actually check that it works. ⁓ but I think the founding principle is that if you're spending a lot of time with teachers and students, ⁓ they're probably gonna tell you if it's not working. ⁓ and and that's gonna set you it set you back on track. So if I had one recommendation, it would be that.
speaker-0: Yep. It's ⁓ very hard to keep it simple i i in many ways, and there's many problems that we can solve, and you've certainly solved a lot of them over your time. ⁓ so thank you. Thank you for for ⁓ everyone who has used EP for you and and Craig starting this incredible company. where should listeners go to find out more about the journey and everything we've talked about today?
speaker-1: I think probably the the most interesting ⁓ place for them to go, we've just recently launched an impact page on our website, ⁓ which walks through a whole lot of ⁓ the work that we've recently been doing ⁓ in demonstrating this. So that's ⁓ w dot educationperfect.com slash impact. I think that's a fantastic starting point. ⁓ and it's a fantastic resource for teachers and students full stop.
speaker-0: put those notes ⁓ in in the show notes. But thank you so much for this conversation today, Shane. It's been an absolute pleasure.
speaker-1: Thank you both Jill and James. It's been an an absolute pleasure chatting with you. Thank you.
