ETS CEO: Could AI Make Standardized Testing More Important?의 전체 녹취록

Mm-hmm. Amit Sevak: that kind of thing All of the opportunities for personalization. So students with different learning approaches can still learn the material because AI can help iterate against that particular learning difference. The big risk we have right now are what? Number one, The tendency of AI to parrot back what it thinks you want to hear, that tendency leads to a lack of real meaningful exposure to different perspectives. The opportunity to really look at how we can think about AI's usage as it relates to critical thinking, really providing a push. Because the reality is so many of the skills we're going to need in an AI world are requiring. Deep critical thinking, let me give you an example. An AI system produces an output. My ability to evaluate that output is a basis of critical thinking. It's okay, wait a minute, is this what I want? Do I want something different? Do I buy this? Is it accurate? This is the power of evaluation. These kinds of skills are human skills coordinating with AI. And these are the kinds of things that we have to find new ways to. To define those kinds of skills and then using AI effectively to help the student better develop that skill set of evaluating, of analyzing the output of AI. So these are just some examples as you go deeper of why I'm recommending we still lean in, particularly for high school students, on teaching them how to work with AI because they need it for after high school, but being a little bit more cautious early on as students are still not yet comfortable in their own skin with many of these concepts. Hope King: I mean, I'm an adult that I'm not comfortable with a lot of the things that I see. where I'm going with this really is if you have personalization on one end as some of the benefits, but organizations like ETS where you are trying to test for standards, those two seem at odds. Amit Sevak: So let's let's break this down. Let's first talk about a standard. Let's imagine, for example, we want a standard of X for a eighth grader or a ninth grader. Let's say it is a standard that is approaching algebra for math, for example. Okay, that's a standard. We want a certain level of algebra-like skills for math. Great. To check across a population, how are we doing? We can use an assessment, a standardized test, to measure. How is this population, this group of high school students, this state, this country is doing? And we use that. ETS is honored to be the stewards of NAEP the National Assessment for Educational Progress, done every two years across 50 states on behalf of the U.S. government. We determine what is the level of literacy, of math competency for eighth graders. So that's an example of where we use an assessment to track across a population how are we doing relative to what our standard is. And it just so happens that most students are not proficient at eighth grade math in the United States today. The second solve that we have is all right, now that we know that broadly, how do we now go to the individual? And because we have students that are at different levels, that could be because of the education they had up to that point or other experiences that they have or their aptitude, we now can personalize the learning to help the student population get to a standard. And so in the case I just gave of eighth grade Proficiency, only half of the country is proficient. You might you have a population here that is at varying levels. Some might be at fifth grade, some at sixth grade, some at seventh grade. So that's where a personalized approach where you have mini assessments, what's often called a formative assessment or an interim assessment, that tracks progress that of the building blocks. And that over time can help the population move forward. So there are two kinds of ways you can measure. Once a year, for example, the summative. Type of assessment to check against a standard and more of these personalized or formative assessments. Think like a quiz, if you will, that has different levels depending on where students are at, a more adaptive quiz mechanism versus something that is a one-time across the population. So we get relative to a standard where folks are at. Different tools for different needs. Hope King: I think the other challenge with personalized learning is that you know it takes more time for teachers, you need more of them in some cases, or you need them to have more transparency into how the personalized learning process is progressing if they're like using AI on their own devices. Are you thinking about integrating what you're doing into the bigger platforms like Chat GPT and and Claude, Have you had conversations with the LLMs about this? Amit Sevak: Yes. we've had extensive conversations and we're working with many of them as we speak. We have found that the LLMs are looking at the problem differently. Some are seeking to have their particular platform to be the default platform for students, and therefore they're seeking ways to be able to drive an understanding of progress on the competencies around that LLM, specifically for evangelizing or moving forward their their platform. Usage. Others are thinking about this in the context of particular student needs, and we're starting to see some of the frontier model players starting to go deeper into education and really go to the level of teacher. And so you're seeing different players responding in different ways. Claude for Teachers just recently launched, for example, as a way to start to really get at how do we see teachers' usage of Claude as a way to help understand progression there. What all of the LLMs are are really I think telling us is that we have an incredible opportunity right now to really get engaged with them. To your point around the opportunity for us to embed our assessments into their platforms, we're also doing that. We're also looking at different ways that we can do that. We have to be very careful with that, also, because

remember, student data, individual data, is private, and so our interaction with different tools is going to be based. And our level of comfort with that. So right now we're largely leveraging their tools for the purpose of building, delivering, and scoring our assessments. We use multiple models to ensure that we're able to scale effectively. For example, item writing, which is writing questions, writing items. That's something we're using AI extensively, or scoring when we have to score across tens of millions of test takers in a year. We use AI for that now. Those are the examples of how we're using them for our processes. And but we are starting to experiment with different more creative. ways, always trying to protect and ensure that the integrity of the of the questions, the security of the questions, and also the safety, the security of the test taker is foremost. And that's where we're still on that curve. Hope King: when you're thinking about building, are you building for the students? Are you building for the universities? Are you building for the employers? Are you building for the parents? Like who do you have in mind when you're thinking about the Overall goal of these assessments. Amit Sevak: Yeah. That I love your question because we're thinking about all of those. Our journey started first with using AI to build capabilities for us as ETS. We thought of it initially as an automation vehicle. And we were leveraging machine learning and earlier versions before we got into LLMs of AI. And so that's how we were able to so effectively during the last four years manage this incredible transformation of ETS. We're now much more automated. We're now able to scale. We're able to provide all kinds of assessments on the fly. We have a lot more ability to productize and launch new assessments because we're using AI, if you will, in the factory, in how we build and how we deliver and how we score assessments. So the first use case, if you will, was all of the needs that we had internally. I mean, we have thousands of customers, comp companies, countries, school districts, and so we wanted to be able to build, deliver, and score for them more rapidly So, like many companies and many organizations, we started first to drive efficiency and scalability of the use of AI. Now, in our current focus, we're looking at specific external use cases. We're particularly looking at, I gave the example of the LLMs. There are many LLMs and there are also many organizations right now hope that are really trying to figure out AI literacy. And this is a hamster wheel of a problem, right? Because as you said a few minutes ago, change it's changing so quickly. And so as you're trying to certify someone's fluency in AI or AI literacy at this level versus this level versus this level across a population, or if you're trying to go deep in a particular domain, take healthcare, the use of AI by job function, right? Or finance, or if you're thinking about in a country,

we're currently working with a Central Asian country that's looking to drive AI literacy across the country. All of these Hope King: Mm-hmm. Amit Sevak: kinds of use cases are very, different. And so we're seeing that we'll be announcing shortly in one of our 50 states that the entire population of teachers are looking for a way to measure their progression of AI literacy from when they start the school year and are leveraging AI tools. If you think about it, think about the role of a teacher to just break down one example. A teacher is creating a lesson plan. You can use AI for that. They're coming up with assessments, quizzes, pop pop quizzes. They're coming up with ways to manage challenging student interactions. Hmm, how do I handle this? Almost like coaching. They're looking at ways to themselves get brushed up on new types of assignments. And they're using AI for assessments at the end. You've got all these kinds of questions come in and you want to validate something. You have AI give you a different score just to see or do a double check on that. So just in that example, like I mentioned there's so many different cases And so we own a offering called Praxis, which is a teacher certification, a teacher licensure assessment. And so we see many, many use cases, over 350,000 potential teachers take praxis to certify their ability to teach in a classroom, like a driver's license. Praxis is the way you get licensed to teach in 46 of the 50 states. And so we're now creating all kinds of tools that go beyond just certifying, you can teach math. to third graders, but can you use AI to teach math to third graders responsibly? So now we're moving into those territories. So it's role specific, industry specific, and country specific as we think about how do we apply AI externally as well. Hope King: Why keep standardized testing the way it is, Why not have each of the tests maybe adapt more to how the child has been learning? Amit Sevak: Great question. And i th th the the way I would start with the answer with is first recognizing that there are different uses for different types of assessments. So the question you're asking is the right question. Why are we measuring this individual at this time and for what purpose? What what's actually the purpose? So if it's for compliance or for measuring against a standard, the federal government is wanting to s check progress on math and reading across fifty states. And so that type of assessment allows you to measure against a benchmark. But if you're measuring for individuals, then you can use different types of tools, right? And so to your example, adaptive testing is awesome. When you're trying to drive for efficiency, you're trying to have computers rapidly respond with a score. When you're trying to use technology to build the assessments more quickly, so there's a lot of benefits of adaptive. We use adaptive testing all the time. So we have two kinds of assessments: those products that we own, like a TOEFL or a GRE or an ACT, and then those where we use it for clients. So we use it extensively for third parties. And to your point, I think the the key here is why? Why are we using the assessment and for what purpose? In some cases, a standardized, more traditional, multiple. Choice question is the right answer because we're trying to measure across a large population against a benchmark. Versus in other cases, you're trying to either drive for efficiency, you're trying to recognize learning differences, or you're just simply trying to drive a much more interactive experience, something more simulation-based, for example, for younger kids, can be much more powerful than an multiple choice question. So depending on the use case, you can go more traditional or you can go more innovative, more adaptive. Hope King: And and the you know, the spirit of the question is also just looking at the context of a lot of schools moving away from standardized testing requirements for becoming you know, enrolled in in a school or for even for assessing a a potential student's fit for for these schools. I know like these trends have been happening over the course of the past few years. You know, it's I it feels like and sounds like you're you're doing the job of adapting ETS to this new future. Amit Sevak: Yeah. Well, one piece of good news is that in terms of many of these assessments, there is a the beginning of a swing back. you're seeing more and more institutions recognize the value of a third party high quality assessment. more and more institutions are recognizing the limitations of the college essay, for example, that's been largely impacted by AI. And we all recognize that there's significant grade inflation. All kinds of studies have come out recently. reinforcing the traditional report card, the traditional GPA, has become quite inflated for a whole range of reasons. So when you think about if I'm a college admissions officer, how do I validate that this individual has some of these core competencies for readiness to college, particularly at a national level? And so you're seeing more and more institutions coming back. University of California Regents, the board, has recently announced that they're doing a review to determine if they should bring back. Testing for people applying into the California system. There was a petition of 800 math and science teachers across the California system that are petitioning to bring back math college admissions. And you're seeing that in institutions like MIT and Caltech and many others that are starting to bring it back. But to your question, do I see a shift happening? Absolutely, because the point here is that it's not about the test, right? It's about are we actually able to drive We bought the ACT not just to have another test in the portfolio of ETS. We bought it because we think there's an amazing opportunity right now to significantly modernize the whole portfolio of ACT assessments. I'm just fortunate to

have just spent literally the last three days hope here in Washington, D.C. with the ACT leadership. And we literally went through each of the products and said, okay, what can we do to reinvent this, to reimagine this, to make the products more engaging, to make them more career relevant. to drive better outcomes so that we're not just producing folks that took an assessment, but folks that are really demonstrating readiness in more and different ways. So I think I think this is actually a moment right now where I do see assessments as being front and center to measurement, right? As particularly when Hope King: Mm-hmm. Amit Sevak: there's so many other things that are confusing out there. does this person know this skill? Wouldn't you want this for somebody that's flying a pile flying an airplane, the pilot, to have been assessed to fly it properly? I'd want that person. I don't care what's on their GPA or how well they wrote an essay saying they can fly, right? I think even for hiring, there the there's the there's a re-looking at how can we use assessments to drive even hiring in new and fresh ways. And so I think it is changing to your point, but I think it's changing in places that's actually in favor of what we do, which is to advance the science of measurement. Hope King: Do you think this trend of reinstating the scores is an admission that it was a mistake to drop them in the first place? Amit Sevak: There were motivations for many institutions to take a pause on using traditional assessments. The pauses were varied, right? We had COVID. I mean, once in a century pandemic, we had all kinds of dislocation. They were trying to, in an empathetic way, provide access and opportunity. So for the moment, I understand why, but I think that folks are starting to realize that something we've had for decades independent, high quality. Rigorous, validated, and reliable ways to measure a person's skills at a given point in time as input to decision making is super powerful. And while the multiple choice question might change, this the tests over time are, and we're advocating for more interactive, more dynamic, more simulation-based assessments, we still think that these kinds of in particularly in a point in which there's AI can be used to fake anything. Hope King: Mm-hmm. Amit Sevak: how do we validate that person is Able to do that skill, right? Like the pilot example, or a surgeon or a nurse. you want that confidence as an employer, as a school, and even as the individual. Can I do this? Self-assessments, high-quality self-assessments are starting to pop up all over the place as a way for people to get comfortable, that they're that they're ready. Game on, let's go. Hope King: would you agree with the statement that your bet right now is that AI is actually pushing for a greater need of standardized testing? Amit Sevak: AI is pushing in certain use cases, yes, for a need for more assessment. Let me give a couple of examples. One is the college admissions process, right? As I mentioned, AI is disrupting the college essay, which was a really critical part of the college admissions process for decades. I did college essays, even my eldest son did college essays. And now the reliance on the traditional college essay has been impacted. So I think I do see that balance. And shifting there. But I think in other cases it's more of a nuanced answer because I think when you look at hiring assessments, the word you use was standardized, and I would say there's so many ways now that hiring my eldest son and my daughter both just recently completed interviews for their first jobs, and in that process, they went through an interactive AI tool that it captured answers to their questions. Or interacting with it with a bot. And those can't answers were then packaged into a score report that was sent to a hiring manager that was used as a part of the process. So there's an example of using AI to enable the process. So we're seeing assessments change significantly, but do assessments still add value to confirm what people know? Absolutely. Do I see that going away? Absolutely not. Hope King: I have to say, in in many ways, you are like the perfect archetype to lead this company. You went to Harvard for your MBA. You, I mean, you have cleared a lot of the hurdles where these test scores have been important traditionally. You that there's a bigger conversation around do college education, do college degrees even matter in your mind going forward, with some companies even saying college degrees don't don't matter. how big of an impact would that have Amit Sevak: Yeah, I no, I think that I'm really honored to be in this position. This is an organization that is a mission-driven organization. We are on this planet to advance the science of measurement to power human progress, to help students become college and career ready, to help companies really identify talent for their organizations, to help governments really raise the standard and boost life chances. I mean, we're so privileged to be in an organization that is waking up everyone. Every day trying to do good things in this world. And so for me, I come with that lens. I'm saying, all right, how do we help these schools solve what they're solving for? Bringing enrollments in that fit their culture, that fit their academic standards, how do we help companies do the same thing? And so we're constantly looking for ways because these organizations are changing what they need and what they're doing. We're trying to keep the student or the learner in the center of this process. And so every opportunity we can do that to do that, which means sometimes we have to break with the old products and find new and completely Hope King: Mm-hmm. Amit Sevak: different products for them to do that and so that's a that that's a big part I'm really excited hope that in the last couple of months we've launched three new centers a new center for AI responsible use of AI a new

center for advancing skills work ready skills and a new center for advancing innovation in policies around measurement and this is part of our commitment to really serve to really serve countries to serve people with innovative ways to measure what people know, what people can do. Hope King: Yeah, I think the, you know, like idea of the linear path though, which is why I brought up your own background. Like, y you know, you and I both grew up in that environment where every single part of our life was toward like the next goal and the next goal, and And now you're talking about Gen Z who are, unfortunately, if they're not able to find an entry-level job, they have to strike out strike out on their own. I'm hearing from other executives and workforce experts that linear path that you and I both walked for a long time, like that's gone away now. maybe this new matrix sort of career is popping up. that's really where the question is kind of coming from is like, Where does standardized testing fit into a world where everything does seem more amorphous? Amit Sevak: Yeah. Well, i i you're you're you're spot on that the world of work is where I'd start. The world of work has changed so radically and so fundamentally. What I really see is we're moving from milestone. Milestones that are passages in time — I finished high school, I got a diploma, I finished college, I got a degree, I completed this thing and I got this — to much more validation of skills in that moment. Because it's not just the sum of what did I learn, math, reading, did I complete biology at a high school level, I check the box with an AP biology. We've got to move beyond that. We've got to move beyond time-based and content-based and really recognize that this is changing. The closest proxy to this is what we saw the last decade in the AI in the IT field. So, if you look at a Fortune 1000 company in the IT department, the technology department, a software developer was on A was on a treadmill. They had to constantly learn new skills, they had to constantly learn new applications, new tools. And the CIO typically had an investment in training and certification. You've got to be certified for this language or certified to do this. That is, I believe, gonna increasingly be happening outside of IT and it's on this. constant change of the half-life of a skill right is is is shifting and therefore we've got to have validation of skills at different moments of time for different roles. and that's that's where I think ETS can play a huge role because we're really really good at at at measuring it decomposing down to the components what you know and and how you can apply that knowledge. Hope King: Are you seeing demand for more assessments around emotional proficiencies? You know, we know that the AI stuff is getting us all like skilled up and ways and areas of expertise that I wouldn't have been able to do on my own. But like everyone talks about the soft skills being the thing. Are companies

coming to you asking you for this? Amit Sevak: Well, here's a great stat. So when I took my job in twenty twenty two, I checked around the fifty states and we were looking to determine which states want to identify some of these kinds of durable skills or soft skills or emotional skills as really key priorities, going beyond math, reading, science, et cetera. And at that time in 2022, there were 10 states. So we said, all right, came on. So we had an event and we brought in representatives from those states and we said, okay, what are you trying to measure? Why are you trying to measure it? When in the elementary, middle school, high school journey do you want to measure? Those things. Today, Hope, states have something called a portrait of a graduate that includes these durable skills. And they want to really particularly focus Hope on the high school students because they see these durable skills as increasingly a life readiness. Set of skills. And what are they? Collaboration, communication, empathy, civics, often like civic reasoning, civic conversation. These are the kinds of skills that are now increasingly about relational intelligence, being able to relate to another person that is very different from just from just a decade ago when everybody was pounding the pavement saying we gotta strengthen math and reading. We still care about math and reading. These are still foundational skills for a functioning adult in America today. But these durable skills, particularly in a world of AI in which so much is getting automated, are starting to become even more important. So on your question, I would say I'm hearing it more from governments more than companies right now. They're the ones really saying, we really need to define this and measure this. So they've defined it. We now have six of those 40 plus states that are doing pilots with ETS in classrooms to measure these durable skills. And several of the states want to in the next two years make it a statewide requirement at the level of math and reading, which is just warms my heart that they're really recognizing the seriousness of us starting to measure these things. I mean, you you know, you got to measure what matters, otherwise how do you know if it's getting any better? So these states are really forward looking in that way. Hope King: So the answer is yes, but just not where I think they might be arising from that people Amit Sevak: That's right. Hope King: are people are thinking about this. That's great. In the remaining minute and and so that we have, I would love to hear what advice you have for teachers Amit Sevak: I would start with foundationals and AI. I really would, and let me explain why. Recent research shows that about 60% of all American teachers, we've got about two million teachers in the United States today. 60% of them are using AI during the educational, during the academic year, 60%. But less than 20% have had any formal training at all. So I think it's as much as I'd like to say, hey principals, hey superintendents, hey district leaders, we've got to have responsible training of AI for teachers. The reality is budgets are tight, there are other priorities, they're concerned about burdening teachers with more responsibilities. So my first advice would be for teachers to really seek out tools to apply in responsible ways AI. ETS is a party to an organization called Teach AI that is really driving this. On the Praxis website, you can find some information about AI literacy, but there's a lot of other great tools out there. ISTE has a great set of offerings. AI EDU. These are just a couple of websites that give great tools to help teachers responsibly use AI in the classroom. Second, as it relates to students and individuals that are anxious that are out there, the ETS Human Progress Report did find what we often call the Adaptability paradox where it shows that on the one hand, individuals are feeling incredibly anxious and a need to adapt, but the paradox is they don't quite know where they're going. And so there's this tension that's out there, this fear of being obsolete, this FOBO Our advice and my advice is to take things one step at a time. it's as old as time is this what is my purpose? Why am I here and what am I trying to solve? I'm I literally had that conversation with my daughter a couple of days ago. She's in her twenties and she's trying to figure it all out. She's trying to get ready for her next application for a job. Why am I interested in that? What is it about this interest that's for me? What are my motivations? Self-assessment is as important as anything at this point in time. And that's an ongoing conversation, whether in your 20s, 30s, 40s, or 50s. constantly having that conversation about what is my why, what's my purpose is the is the greatest gift that I received. in having friends and family members ask me that question often has got me to realize that's the gift I want to give to others to help people find their why. a new product offering that we want to have come out soon from ACT is going to be exactly about this. So more on that to come. Hope King: I like it. it sounds like we're getting into sort of personality and like individual, like you said, self-assessment. Is there anything that we didn't talk about today that you wanted to to talk about Amit Sevak: I'm just really grateful, hope, for this chance to chat with you. we're on an exciting journey at ETS. We really see that America does have a readiness problem, but we also believe it's solvable. We need better measurement, we need a higher level of engagement with policymakers and teachers and families, but we think that there's such tremendous opportunity with the use of technology responsibly to help students become more ready for what comes next. Hope King: Awesome. Well, thank you so much for spending time and hope to see you back real soon. Amit Sevak: Thanks so much, Hope, for having me.