Teaching Math with AI with Amy Langville, Chloe Lewis, Lew Ludwig, and Kathryn Pedings-Behling

WEBVTT
00:00:34.759 --> 00:00:41.560
I had students being like, I literally had no idea that that AI could be used in this way.
00:00:41.560 --> 00:00:48.119
Like I had no they they said I will never study the same way again, like for a test, period.
00:00:48.119 --> 00:00:51.719
Like I had no clue that I could do this.
00:00:52.679 --> 00:01:00.679
Welcome to Intentional Teaching, a podcast aimed at educators to help them develop foundational teaching skills and explore new ideas in teaching.
00:01:00.679 --> 00:01:02.519
I'm your host, Derek Bruff.
00:01:02.519 --> 00:01:07.640
A few years ago, it was pretty easy for math educators to ignore generative AI.
00:01:07.640 --> 00:01:13.319
The chatbots of 2022 and 2023 were notoriously bad at math.
00:01:13.319 --> 00:01:15.560
That, however, is no longer true.
00:01:15.560 --> 00:01:23.159
Today's Frontier AI models are very good at math to the point of proving mathematical conjectures that have been open for decades.
00:01:23.159 --> 00:01:31.000
Today on the show, I have a round table of math faculty sharing what AI aware mathematics teaching looks like here in 2026.
00:01:31.240 --> 00:01:43.480
Lew Ludwig is professor of mathematics at Denison University, and co-author with Todd Zakrajsek of the new book, The Science of Learning Meets AI, which I believe is out to the very week this episode airs.
00:01:43.480 --> 00:01:56.200
Lew is one of the organizers of the Teaching Math with Generative AI Google Group, which is how I found out about our second guest, Chloe Lewis, assistant professor of mathematics at the University of Wisconsin at Eau Claire.
00:01:56.200 --> 00:02:10.599
That Google group meets twice monthly on Zoom, and Chloe recently shared her Critique the Chatbot assignment, in which students evaluate ChatGPT's attempts to prove mathematical statements that students had previously disproved.
00:02:11.319 --> 00:02:24.039
And I'm glad to welcome back onto the podcast Amy Langville, professor of mathematics at the College of Charleston, and Kathryn Pedings-Behling, online education and e-learning coordinator, also at the College of Charleston.
00:02:24.039 --> 00:02:35.240
You may remember them from way back in episode 10 of the podcast in April of 2023 when they shared about their line of innovative mathematics textbooks called Deconstruct Calculus.
00:02:35.240 --> 00:02:45.400
These books are packed with creative activities that invite students into embodied learning, often requiring students to do something in the physical world that helps explain the mathematical concept.
00:02:45.719 --> 00:02:47.960
Yes, it's another four-guest episode.
00:02:47.960 --> 00:02:54.039
And with all these smart people, the conversation runs a little long, so I'm going to wrap up this introduction right now.
00:02:54.039 --> 00:02:59.479
Thank you all for being on Intentional Teaching Today.
00:02:59.479 --> 00:03:08.039
I'm excited to have some fellow uh mathematicians on the podcast and to talk about how AI is changing the work of math education today.
00:03:08.039 --> 00:03:08.919
Thanks for being here.
00:03:08.919 --> 00:03:10.359
Very excited to have you all here.
00:03:10.359 --> 00:03:13.560
Um I'm gonna start with my usual opening question.
00:03:13.560 --> 00:03:21.800
Um, and it is this uh can you tell us about a time when you realized that you wanted to be an educator?
00:03:21.800 --> 00:03:24.759
And uh Kathryn, maybe I'll start with you.
00:03:25.479 --> 00:03:25.879
Sure.
00:03:25.879 --> 00:03:28.120
Um, yeah, so I was the typical child.
00:03:28.120 --> 00:03:40.199
I always knew I wanted to be an educator, so I'll answer this by saying when I knew I wanted to be a math educator, and that's when I had class um with a high school math teacher whom I had for my junior and senior years named Dr.
00:03:40.199 --> 00:03:43.960
Nancy Long, who actually like completely changed my life.
00:03:43.960 --> 00:03:49.240
Um, and I knew at that point that I wanted to be just like her and be a math educator just like her.
00:03:49.479 --> 00:03:50.120
That's awesome.
00:03:50.120 --> 00:03:50.840
That's awesome.
00:03:50.840 --> 00:03:51.400
I love that.
00:03:51.400 --> 00:03:56.759
I had I had a couple of key high school teachers too that kind of got me in that direction.
00:03:56.759 --> 00:03:58.199
What about you, Amy?
00:03:59.080 --> 00:04:01.080
Uh mine was later than Kathryn.
00:04:01.080 --> 00:04:16.360
I was in graduate school trying to decide what to do with my uh degree in operations research, and I was a TA in the math department, and they forced me to teach my own class, and I didn't want to do it and protested, but they they made me and then I found out I loved it.
00:04:16.360 --> 00:04:22.519
In hindsight, it made me made sense because I had been all throughout high school and um college.
00:04:22.519 --> 00:04:26.680
I was an instructor of physical education type things like basketball and volleyball.
00:04:26.840 --> 00:04:29.960
So uh kind of a natural teacher, perhaps.
00:04:29.960 --> 00:04:30.919
Yeah.
00:04:30.919 --> 00:04:32.439
What about you, Lew?
00:04:33.319 --> 00:04:34.840
Um I would say so.
00:04:34.840 --> 00:04:40.360
I kind of in college, I kind of landed in what was called our CLC, the computer learning center.
00:04:40.360 --> 00:04:52.039
And uh I they had a very interesting thing set up where you would actually kind of coach or teach one-on-one different uh software packages, Lotus 123, Microsoft, or no, it wasn't even Microsoft back then.
00:04:52.039 --> 00:04:53.879
It was MultiMate, all these crazy things.
00:04:53.879 --> 00:04:57.320
But anyway, it was something that I kind of naturally took to and really enjoyed.
00:04:57.320 --> 00:05:03.000
And the fact that I look at my pedigree and both parents were teachers, and I have a grandparent that was a teacher, and I married it.
00:05:03.000 --> 00:05:04.600
It was just kind of like in the water there.
00:05:04.600 --> 00:05:06.519
So yeah, always been interested.
00:05:06.759 --> 00:05:11.639
Yes, I've had many folks on the podcast who point to a family pedigree of teaching.
00:05:11.639 --> 00:05:12.360
Yeah.
00:05:12.360 --> 00:05:15.159
Um, and Chloe, what about you?
00:05:15.799 --> 00:05:23.480
Um, I was an undergraduate at Northern Michigan University, which is in the upper peninsula of Michigan, pretty remote rural area.
00:05:23.480 --> 00:05:28.519
Um, and when I was there, actually, it was sort of undergoing a really big like economic shift.
00:05:28.519 --> 00:05:35.639
Like we were transitioning out of being like a coal mining area into, I mean, it was just like Obama-era energy sort of efficiency kind of thing.
00:05:35.639 --> 00:05:37.320
We're not going to be coal miners anymore, right?
00:05:37.320 --> 00:05:46.120
So there was um a lot of money to like re-education grants for um people who were like returning to school after a really long time.
00:05:46.120 --> 00:05:51.080
And I got the chance to work with the developmental mathematics course as an undergraduate while I was there.
00:05:51.080 --> 00:05:56.600
And I just really liked the like diversity of a college classroom in that setting.
00:05:56.600 --> 00:06:00.840
So that really made me want to be an educator and also to work at a public institution.
00:06:01.080 --> 00:06:01.799
Oh, that's great.
00:06:01.799 --> 00:06:02.759
That's great.
00:06:02.759 --> 00:06:07.399
That's maybe a little closer to my own, is I knew I wanted to be a math major.
00:06:07.399 --> 00:06:13.639
Um, but when I started actually kind of doing some tutoring as a college student, that's when I realized I really love this education bit.
00:06:13.639 --> 00:06:14.919
Well, thank you all.
00:06:14.919 --> 00:06:19.320
I think that gives us a little, a little sense, a little glimpse of the kind of who you are.
00:06:19.320 --> 00:06:23.159
Um, but let's jump into this AI discussion.
00:06:23.159 --> 00:06:29.159
Um and Lew, I'm gonna start with you mainly because I know that you're one of the organizers of this math and AI group.
00:06:29.159 --> 00:06:34.759
Um, and I know that you've been very active in kind of writing and speaking about AI over the last few years.
00:06:34.759 --> 00:06:43.960
What how would you describe AI's impact on college math teaching since Chat GPT was launched in 2022?
00:06:43.960 --> 00:06:46.519
Have you seen any kind of trajectory there?
00:06:46.919 --> 00:06:47.240
Yeah.
00:06:47.240 --> 00:06:50.200
So okay, so the day that we'll live in infamy, right?
00:06:50.200 --> 00:06:57.000
It was November 30th, 2022, as you pointed out, when ChatGPT 3.5, which is now retired when that landed.
00:06:57.000 --> 00:07:08.919
Um, I think by this early spring of 2023, kind of the game for mathematicians, especially if you were teaching a proofs course, was hey, let's find the error that AI made in its proof.
00:07:08.919 --> 00:07:10.279
And we'll laugh at the chat bot.
00:07:10.279 --> 00:07:13.399
And wasn't that kind of a funny and cute little thing to do, right?
00:07:13.399 --> 00:07:16.120
Um, that lasted about a semester.
00:07:16.120 --> 00:07:25.159
Now you can only really reliably get errors if you specifically prompt the AI to produce mistakes that students typically make, right?
00:07:25.159 --> 00:07:26.759
So the game has really shifted.
00:07:26.759 --> 00:07:34.120
Um, I think also trajectory-wise, more significantly, AI has just decimated graded homework, uh, you know, with take-home stuff.
00:07:34.120 --> 00:07:38.679
Uh as a result, myself, I've actually moved this semester to a standards-based grading.
00:07:38.679 --> 00:07:44.600
You know, students demonstrate mastery of specific skills rather than just kind of accumulate points, that little game that we have.
00:07:44.600 --> 00:07:51.559
Uh, the interesting thing is, this whole structure, if you've ever never tried this kind of standards-based grading, a lot of overhead.
00:07:51.559 --> 00:07:58.360
I couldn't have done all this overhead without the very technology that broke my old system, right?
00:07:58.360 --> 00:08:01.720
So the thing that broke my course actually has helped me to redesign it.
00:08:01.720 --> 00:08:09.799
So, you know, the trajectory for me has been instead of trying to work around AI, is actually to kind of uh you know more lean into it and try to use it to my advantage.
00:08:09.799 --> 00:08:12.759
And and for this semester, I really like where I've ended up with that.
00:08:13.159 --> 00:08:13.480
Okay.
00:08:13.480 --> 00:08:14.200
Okay.
00:08:14.200 --> 00:08:25.000
Um, Amy, Kathryn, I want to turn to you to next, in part because I think you both teach online math courses and online education has been particularly impacted by AI.
00:08:25.000 --> 00:08:32.679
Um, what's what's that trajectory been for you over the last few years uh as AI has has entered our world?
00:08:33.399 --> 00:08:40.679
Similar to Lew, like there was the slow, what is this tool gonna do, and then realizing it just keeps getting better and better and better.
00:08:40.679 --> 00:08:48.919
And so you can't uh ignore it, but on the same, on the same hand, it it's uh related to prior technology.
00:08:48.919 --> 00:08:54.360
So like Kathryn and I've had many discussions with colleagues about tech tool use prior to AI.
00:08:54.360 --> 00:09:06.840
And so we were already a little bit ahead of the curve leading towards the let's encourage tech tool use so that we can solve more interesting applied problems with students in an online space.
00:09:06.840 --> 00:09:16.680
And I guess maybe one of the first things we did, we used that typical you see a lot of folks do red, yellow, green on particular assignments as whether it's prohibited or allowed.
00:09:16.680 --> 00:09:26.680
And and Kathryn's got a little bit found a little bit more nuanced one where we have for every assignment it's clear to the student use is allowed or prohibited.
00:09:26.680 --> 00:09:42.440
Um, and then we also with our uh textbooks that we use, we created our own thing for deconstruct calculus uh for our students, and it has become a really useful physical component of the class.
00:09:42.440 --> 00:09:47.480
So much of an online class is on the screen, and to have this physical book has been very helpful.
00:09:47.480 --> 00:10:10.680
And in having that, I think, also eased the transition into AI in some ways because the book itself, we we already focused on process over product, so every every page had answers on it, and you know, and maybe I'll let Kathryn jump in here and some more about the use of the book, maybe and how it helps with AI.
00:10:12.040 --> 00:10:17.000
Yeah, so uh we Amy and I always love a new technology.
00:10:17.000 --> 00:10:27.240
Um we we love to jump in, we love to learn about things, and we we realized pretty early on that this was gonna be a new norm for the world.
00:10:27.240 --> 00:10:36.360
Um and so uh instead of putting our head in the sand, we really decided to figure out like, what does this mean for us?
00:10:36.360 --> 00:10:38.120
What does this mean for our classes?
00:10:38.120 --> 00:10:40.759
What does this mean for assessment of students?
00:10:40.759 --> 00:10:58.519
Uh, what does this mean for the expected knowledge that students have in a class and what um skills they need to be learning, and and how that can then apply to their own lives when they leave college.
00:10:58.519 --> 00:11:04.200
Um, so Amy and I in particular love to teach the business calculus class.
00:11:04.200 --> 00:11:07.720
These are the non-STEM major students.
00:11:07.720 --> 00:11:16.360
A lot of them are business majors, but we have quite a few psychology majors and um you know, we have some education majors, people like that in there.
00:11:16.360 --> 00:11:30.040
And so we really started thinking about how we can teach them to use the tool in a way that it's going to be helpful when they leave the classroom, right?
00:11:30.040 --> 00:11:36.200
And they may never do insert math topic here again, right?
00:11:36.200 --> 00:11:38.360
This is what they want to tell you, right?
00:11:38.360 --> 00:11:47.800
But we use math as an impetus for like how do we use these tools to help us gain knowledge when we need to learn new things.
00:11:48.040 --> 00:11:53.800
Um, and so our textbook, again, our students really enjoy this physicality of the textbook.
00:11:53.800 --> 00:12:05.639
Um, and something that we have always done that AI cannot replicate is we have quite a few three-dimensional things that students are required to do with their textbook.
00:12:05.639 --> 00:12:13.639
They're writing in their textbook, they're tearing out pages of their textbook and making paper airplanes and throwing them and graphing functions based on that, right?
00:12:13.639 --> 00:12:18.759
They're drawing on their hands to think about how three-dimensional maps work, right?
00:12:18.759 --> 00:12:24.920
So there are these sort of physical features that AI cannot replicate at this point.
00:12:24.920 --> 00:12:29.320
And so we're constantly thinking about that with our curriculum, right?
00:12:29.320 --> 00:12:30.440
What can we do?
00:12:30.440 --> 00:12:38.279
What can we do with our students that can't be replicated by a chatbot at this point in time?
00:12:38.279 --> 00:12:44.360
Um, and so that's just a little, a little glimpse into sort of how we're using it and thinking about it in our classes.
00:12:44.759 --> 00:13:03.639
Yeah, and one of the reasons I wanted to have the two of you back on the podcast, y'all were on a couple years ago to talk about um your very unique textbook, um, is that um I I hear from some faculty, uh mostly like English faculty who are who are moving the writing process back into the classroom.
00:13:03.639 --> 00:13:11.080
I had a I had a colleague on the podcast recently, Jim Seitz, who's do who's has all of his students, they do all their writing in class.
00:13:11.080 --> 00:13:14.759
Um and and there's lots of interesting things to say about that.
00:13:14.759 --> 00:13:18.040
But I know for online courses, there's not an in-class option.
00:13:18.040 --> 00:13:23.960
Um, but I love the kind of embodied elements of learning that you've built in um to that book.
00:13:23.960 --> 00:13:26.040
Um Chloe, what about you?
00:13:26.040 --> 00:13:29.320
What is the last three plus years uh in AI?
00:13:29.320 --> 00:13:31.399
How does that look like in your teaching?
00:13:31.879 --> 00:13:36.279
I don't usually consider myself like an early tech adopter.
00:13:36.279 --> 00:13:38.600
I'm still hanging on to like my iPhone 7 or something, right?
00:13:38.600 --> 00:13:47.560
But like I um sort of that fall when it first was released to the public, I was interested in it from a pedagogical standpoint.
00:13:47.560 --> 00:13:56.519
Um, and so actually, I think the first assignment I gave, I was teaching uh uh advanced linear algebra class as a graduate student at Michigan State at this point.
00:13:56.519 --> 00:14:03.080
And like one of the first kinds of exercises we had proved in that class was like, oh, prove that these two sets are equal.
00:14:03.080 --> 00:14:09.160
And so they had like written down a proof of these two sets are equal on their homework, and then I gave them two other sets, and I was like, prove these two sets are not equal.
00:14:09.160 --> 00:14:12.759
And they this was like homework one, they like submitted it.
00:14:12.759 --> 00:14:17.160
I gave them feedback, they got to resubmit it if they wanted, like that was just sort of the normal process.
00:14:17.399 --> 00:14:31.639
Um, in their final assignment, I I used to give a proof portfolio, which I don't assign anymore, um, because I think it's too easy to do with AI, but uh it was sort of a metacognitive reflection on like how do you think about proof?
00:14:31.639 --> 00:14:35.720
And then it was also some concrete, like include some proofs from this class.
00:14:35.720 --> 00:14:48.680
Um, and so as one of the sort of metacognitive reflective questions, I I asked ChatGPT to prove that these two sets are equal when they were not in fact equal, uh, and then asked the students to like assess the validity of that proof.
00:14:48.840 --> 00:14:53.560
Um, that question has actually been sort of my like benchmark for the last three years.
00:14:53.560 --> 00:15:01.080
Like uh still when falsely a lot of times now, if you command it to prove something falsely, it will say, This statement is not true.
00:15:01.080 --> 00:15:02.600
Are you sure you want me to prove that?
00:15:02.600 --> 00:15:10.360
Um this particular question of are these two sets equal, when I when I ask it to prove that the two sets are equal, it still returns a false proof every time.
00:15:10.360 --> 00:15:25.560
It it has even on um one of my research students had um joined like the temporary pro plan um when it was like available to college students, and even that model still like wrote me a false proof of that statement.
00:15:25.560 --> 00:15:38.440
So I guess that was the first thing I'd done with it, and then I've sort of continued that same um exercise in a lot of my classes, not just in proof-based mathematics classes.
00:15:38.440 --> 00:15:52.040
So I teach a lot of our uh math for the little arts type classes here, uh, where these are students who not just are not math majors, but they are not STEM majors usually, they are art majors and music majors and things like this.
00:15:52.040 --> 00:15:58.279
Um and so we cover, you know, voting theory and graph theory and topics like this.
00:15:58.279 --> 00:16:08.200
Um I have in the past like asked Chat GPT to like produce me something that is false like related to voting theory.
00:16:08.200 --> 00:16:10.440
Make me a voting theory table that does this.
00:16:10.440 --> 00:16:15.960
Um, and so we'll do an exercise in class where we talk about can this happen, and then we go through and think about it.
00:16:15.960 --> 00:16:18.360
And then I share a QR code with them.
00:16:18.360 --> 00:16:22.120
And I'm like, all right, here's what ChatGPT said when I asked it to prove that what do you think?
00:16:22.120 --> 00:16:36.519
Um, and one I think distinction that I've noticed when I give this type of exercise to my math for liberal art students, as opposed to when I give a similar type of exercise to my math students where I say, here's a proof that I made with AI, what do you think about it?
00:16:36.519 --> 00:16:42.840
Um, is that the math real art students are much more likely to doubt their own answer, right?
00:16:42.840 --> 00:16:50.759
Like we had just talked about it, we just come up to the conclusion before I even introduce anything about AI that was like, here's here's how this works, here's what we think.
00:16:50.759 --> 00:16:55.480
But once they see the generated solution, they're like, Well, the computer can't be wrong.
00:16:55.480 --> 00:16:56.840
I had a student say that to me one time.
00:16:56.840 --> 00:16:59.879
Well, it's from the computer, and the computer can't be wrong, so we must be wrong.
00:16:59.879 --> 00:17:00.840
What did we do wrong?
00:17:00.840 --> 00:17:07.000
Whereas my math major students are so much more likely to like read through a proof and be like, this makes no sense.
00:17:07.000 --> 00:17:08.119
Uh, this is stupid.
00:17:08.119 --> 00:17:09.240
Like, why did it do this?
00:17:09.240 --> 00:17:09.480
Right.
00:17:09.480 --> 00:17:18.759
And so I I think that's a really interesting distinction that I've noticed in the way that students respond to AI generated output.
00:17:19.480 --> 00:17:22.200
Well, do you have a conjecture as to why that would be?
00:17:22.200 --> 00:17:25.319
I I have some, but I'm curious what you've what you've thought about.
00:17:25.879 --> 00:17:27.160
I think it's twofold.
00:17:27.160 --> 00:17:35.400
I think one, the culture of mathematics, or at least the culture of mathematics that I try to like build in my mathematics classrooms.
00:17:35.400 --> 00:17:37.160
And again, these are more advanced math majors, right?
00:17:37.160 --> 00:17:39.480
They've had math under their belt for a couple of years.
00:17:39.480 --> 00:17:43.880
And um, is that like uh right?
00:17:43.880 --> 00:17:48.920
We we are a we are a classroom that like asks a lot of questions, right?
00:17:48.920 --> 00:17:53.240
So like I frequently like write something up on the board, I'm like, I don't know, do we think this is true?
00:17:53.240 --> 00:17:53.559
Right.
00:17:53.559 --> 00:17:56.039
And then we'll like have a discussion about whether or not we think that's true.
00:17:56.039 --> 00:17:59.400
So that's sort of like questioning like, does this work or does this not work?
00:17:59.400 --> 00:18:04.039
I think that is sort of inherent to the kinds of math proof classrooms that I built.
00:18:04.039 --> 00:18:07.160
It's also just a broader part of the mathematics discipline, right?
00:18:07.160 --> 00:18:13.400
As a professional mathematician, one of one function of your job is to, you know, you are refereeing a paper, right?
00:18:13.400 --> 00:18:16.279
And so your whole job is to say, is this proof valid or not?
00:18:16.279 --> 00:18:18.360
Like that's that's built into the job.
00:18:18.360 --> 00:18:33.960
Um so I think that's one part of it is it's just this explicit training in argument validation that we get in mathematics, that is maybe not the same kind of training one would get in a different like kind of degree program.
00:18:34.279 --> 00:18:43.880
Um, but I think the second theory I have is like the mathematical uh confidence in these students is very different.
00:18:43.880 --> 00:18:50.600
Usually the kinds of students I see in a math for liberal arts course are the kind who write to me at the beginning of the semester and say, I hate math, I've never been good at math, right?
00:18:50.600 --> 00:18:52.759
Like the the whole litany of everything.
00:18:52.759 --> 00:18:58.920
But um I think one that makes them more doubtful of their own mathematical ability, right?
00:18:58.920 --> 00:19:07.319
So even though we just came up with an argument for why this is true, that like just like is completely shot by the time they see something that a computer made.
00:19:07.559 --> 00:19:15.480
I also wonder if like there's some, I think in the public, there's some kind of base knowledge about like how do large language models work, right?
00:19:15.480 --> 00:19:19.480
Like there's some some sense that like, oh, these are statistical prediction models.
00:19:19.480 --> 00:19:24.200
I think that's like a relatively well understood phenomenon by most of the population.
00:19:24.200 --> 00:19:29.000
I think that what does that mean to be like a statistically predictive model?
00:19:29.000 --> 00:19:34.360
I think that has a significantly different meaning to a student who is like an upper level math major, right?
00:19:34.360 --> 00:19:41.400
They understand statistics and probability at a different kind of level than a student who is, you know, a self-proclaimed, oh, I don't like math kind of person.
00:19:41.400 --> 00:19:47.319
Like I think that underlying understanding of how these models work is also a factor.
00:19:47.720 --> 00:19:48.839
Yeah, yeah.
00:19:48.839 --> 00:19:53.079
Um I want to throw this open to to the rest of you.
00:19:53.079 --> 00:20:03.559
Do you see similar um similar situations where like some students are kind of over trusting the AI or or more skeptical?
00:20:03.559 --> 00:20:06.519
What does that look like in your in your courses, Lew?
00:20:07.799 --> 00:20:10.519
Um so I'm doing the standards-based grade thing, right?
00:20:10.519 --> 00:20:13.880
And uh I do a lot of uh actually true-false with this.
00:20:13.880 --> 00:20:21.559
It's it's kind of an interesting system, but there's a STEM question that has like maybe here's a linear system, and then I ask them a series of questions about that specific thing.
00:20:21.559 --> 00:20:24.839
And anyway, we were talking about uh differential equations, right?
00:20:24.839 --> 00:20:29.559
Linear differential equations and uh whether things would have the Wronskian or whatever.
00:20:29.559 --> 00:20:37.720
I had a kid roll in, this is a sophomore level class, and I you know, my true-false question, he goes, but isn't it in this context, isn't this true due to Abel's theorem?
00:20:37.720 --> 00:20:48.200
And I looked at him because my AI also pointed out in this very weird thing at the 600-level course, right, that you might invoke Abel's theorem, but this wouldn't fit in an undergraduate thing.
00:20:48.200 --> 00:20:50.200
So I impressed him on him, like, where'd you get that?
00:20:50.200 --> 00:20:52.200
He kind of like, you know, got it pulled back.
00:20:52.200 --> 00:20:57.799
So, you know, it it's it's interesting how the students, I think, sometimes blindly just accept what's going on.
00:20:57.799 --> 00:21:05.160
This was a computer science student taking a linear algebra differential equations course, but uh, you know, they were just uh, you know, bound to determine again.
00:21:05.160 --> 00:21:10.920
There's kind of that mindset, I want to get this right, I need this problem to get my meet my expectations and the whole nine yards.
00:21:10.920 --> 00:21:17.880
But it's just like, you know, he thought as long as I just throw out the word Abel's theorem, that that will be good enough, and that will kind of give me the passing grade that I need.
00:21:18.200 --> 00:21:28.759
So Chloe, it sounds like part of your goal there is to help your students understand AI's strengths and limitations a little bit.
00:21:28.759 --> 00:21:30.360
Is that is that fair to say?
00:21:31.079 --> 00:21:43.799
I very much see it as like part of my job in in like working through these exercises in class is to uh sow doubt in the like complete and utter truth that is produced by a large language model, right?
00:21:43.799 --> 00:21:51.559
Like that I think that's really important, especially for the students, like I said, who are taking this math for liberal arts class and are maybe not as comfortable with mathematics.
00:21:52.279 --> 00:21:52.600
Okay.
00:21:52.600 --> 00:21:58.519
Um, because I think you're also trying to give them a chance to practice their their proof skills, right?
00:21:58.519 --> 00:22:00.839
It's it's kind of serving both goals, right?
00:22:00.839 --> 00:22:06.839
A kind of math learning objective, but then also uh kind of an AI literacy, AI know-how piece.
00:22:06.839 --> 00:22:13.799
Um, Amy and Kathryn, do you do you also see AI literacy as kind of part of what you're trying to teach now in your courses?
00:22:15.640 --> 00:22:24.200
So I I will say something that I think um I think it's not 100% related just to AI literacy.
00:22:24.200 --> 00:22:33.640
So what Chloe was talking about is a phenomenon I have seen with my non STEM major students, the whole time I've taught post-secondary.
00:22:33.640 --> 00:22:36.279
So I used to teach secondary, I used to teach high school level.
00:22:36.279 --> 00:22:49.079
The whole time I've taught post-secondary, it takes me almost until the second half of the semester for any non-STEM major student to correct me if I have made a mistake in class.
00:22:49.079 --> 00:22:55.400
And most of the time it's just because they tr they blindly trust me, right?
00:22:55.400 --> 00:22:58.279
They're like, well, I'm I'm not a math person.
00:22:58.279 --> 00:23:03.799
So like if she's saying this is positive five instead of negative five, then that's gotta be what it is, right?
00:23:03.799 --> 00:23:07.160
Even though inside they're like, wait a, wait a minute, is that right?
00:23:07.160 --> 00:23:08.360
I don't know if that's right.
00:23:08.360 --> 00:23:18.920
I spend almost half my class, the first half of the semester of my um, I also teach a contemporary math class, which is like math for the liberal arts, um, business calculus class.
00:23:18.920 --> 00:23:23.000
I spend almost the first half saying, you have you belong here, right?
00:23:23.000 --> 00:23:24.519
You have a place here.
00:23:24.519 --> 00:23:27.880
Your voice matters, what you think matters.
00:23:27.880 --> 00:23:32.039
You can think about mathematics, like you have that ability.
00:23:32.039 --> 00:23:51.079
And so I think I don't know that I don't personally know that this is any more blind trusting of a large language model as it was a uh blind trusting of Kathryn Pedings-Behling standing in front of them, right, as their math teacher, right?
00:23:51.079 --> 00:23:54.600
So I feel like I'm seeing the same stuff.
00:23:54.600 --> 00:24:08.519
It's the to me, I I I don't really see a difference between um them questioning something that's coming out of our, you know, AI agent versus them questioning something that's coming out of my mouth, right?
00:24:08.519 --> 00:24:13.240
It doesn't happen either way for those non-STEM major students.
00:24:13.240 --> 00:24:14.039
I want to stress.
00:24:14.360 --> 00:24:15.400
They doubt themselves.
00:24:15.400 --> 00:24:16.759
They doubt themselves.
00:24:16.759 --> 00:24:22.039
And so if if someone appears to have some expertise or authority, then they're just gonna go with that.
00:24:22.039 --> 00:24:22.440
Yeah.
00:24:22.440 --> 00:24:26.120
Um, and that could be their instructor or a confident sounding chat bot, either one.
00:24:26.279 --> 00:24:26.680
Yes.
00:24:27.079 --> 00:24:27.559
Yeah.
00:24:27.559 --> 00:24:28.440
Yeah.
00:24:28.440 --> 00:24:31.240
Um, Chloe, I'm gonna circle back to you again.
00:24:31.240 --> 00:24:33.400
Uh, your your example is really interesting.
00:24:33.400 --> 00:24:40.200
Um uh so you show your students an output that ChatGPT has generated.
00:24:40.200 --> 00:24:45.799
You do not have the students go and kind of take that prompt and take it to Chat GPT themselves.
00:24:45.799 --> 00:24:56.120
And so I'm wondering how you think about um your students using AI and and like your role in encouraging them or discouraging them from using AI.
00:24:56.120 --> 00:25:02.759
I'm guessing that's an intentional choice on your part to kind of do it for them instead of have them use the tool.
00:25:03.319 --> 00:25:16.120
Uh yeah, I don't want anybody to be required to register with any particular uh AI, like to get an account on any particular AI website in order to participate in the learning activities that we're doing in class.
00:25:16.120 --> 00:25:22.920
So that has been the main motivation for generating uh all of the things myself and then sharing it with them.
00:25:22.920 --> 00:25:26.759
I mean, ChatGPT makes it really easy to like share a link out to a chat and things like that.
00:25:26.759 --> 00:25:35.000
We do, I mean, like all institutions or most institutions, I guess I should say now, we do have like uh institutional access to Microsoft Copilot.
00:25:35.000 --> 00:25:46.920
Um but again, I I've not had this conversation with the student, but I've had this conversation with several other just people who have ethical and environmental objections to the use of AI.
00:25:46.920 --> 00:25:47.319
Right.
00:25:47.319 --> 00:25:55.079
So I I don't feel like I want to put anybody in a particular position where they would have to make such a choice.
00:25:55.079 --> 00:25:57.240
Now, are most of them probably already using it?
00:25:57.240 --> 00:25:58.680
Do they already have their own Chat GPT account?
00:25:58.680 --> 00:26:00.360
Yeah, like I I I know that, right?
00:26:00.360 --> 00:26:10.440
But I I do not want anybody to be forced to use something that they do not want to if uh like in order to like participate in the activities we are doing in class.
00:26:10.759 --> 00:26:11.000
Sure.
00:26:11.000 --> 00:26:13.720
And it sounds like for your goals, they don't need to either.
00:26:13.720 --> 00:26:14.600
So Right.
00:26:14.600 --> 00:26:14.839
Yeah.
00:26:15.000 --> 00:26:15.480
Exactly.
00:26:15.480 --> 00:26:26.680
Um I actually I wanted to like come back around to something that Kathryn said and bring in uh a particular uh reference to the literature um that I think is supporting what her point is.
00:26:26.680 --> 00:26:33.319
Uh there is some work of Harel and Sowder in the 1990s about um student proof schemes.
00:26:33.319 --> 00:26:46.120
So they sort of diagram like uh the things that like when someone is sitting there reading or watching someone prove something, like uh how do you how do you validate that proof?
00:26:46.120 --> 00:26:51.160
Like, what is your sort of thinking process in order to determine do I think this is a valid proof or not?
00:26:51.160 --> 00:26:57.880
Um and they studied the appearance of different types of proof schemes in different contexts and different classes of students.
00:26:57.880 --> 00:27:05.000
Um, and one particular proof scheme that they found uh showed up in a lot of students was an authoritative proof scheme, right?
00:27:05.000 --> 00:27:11.559
Which is exactly what Kathryn was just talking about, the idea that like this thing has to be true because my teacher said it, because it is printed in a textbook.
00:27:11.559 --> 00:27:12.759
I think that is another, right?
00:27:12.759 --> 00:27:15.160
It is like authority broadly construed.
00:27:15.160 --> 00:27:32.519
And so uh Harel and Sowder sort of argue that these proof schemes kind of sit in a hierarchy of like, you know, uh what is I don't know, what is like the not good kind of mathematician do versus like what do you what is the robustly thinking mathematician do?
00:27:32.519 --> 00:27:36.759
And sure authoritative proof schemes sit relatively low on that totem pole.
00:27:37.000 --> 00:27:40.600
Um but I think more novice notion of proof, yeah.
00:27:40.600 --> 00:27:41.160
Correct.
00:27:41.319 --> 00:27:49.400
Yeah, but I just I wanted to add that uh to support her point that I I that this is one of the things that that we continue to observe from Harel and Sowder's work.
00:27:49.720 --> 00:27:50.440
Yeah, yeah.
00:27:50.440 --> 00:27:51.880
What about you, Lew?
00:27:51.880 --> 00:27:54.680
Do you well how do you feel about your students using AI?
00:27:54.680 --> 00:27:55.720
Do you encourage that?
00:27:55.720 --> 00:27:57.319
Do you discourage that?
00:27:59.720 --> 00:28:04.680
I assume that as soon as they walk out the door, they have access to AI, right?
00:28:04.680 --> 00:28:07.240
I mean, there's just there's just really no way of getting around it.
00:28:07.240 --> 00:28:12.759
I know uh Leon Furze is the other uh folks that we're referring to, the you know, the red, green light and all that kind of stuff.
00:28:12.759 --> 00:28:18.360
Uh Leon, he he's first developed that and he's kind of modified, he's got some like bubblegum-colored things now.
00:28:18.360 --> 00:28:23.720
But but he's uh thinking about that more kind of as an assessment way of how you might assess students.
00:28:23.720 --> 00:28:29.000
But again, he kind of believes also that once you step out, it's just naive to think that they wouldn't have access to that.
00:28:29.079 --> 00:28:40.440
So that's why I've really kind of directed things in a way to think with my students about how am I going to get to think about what they actually know and understand and can kind of demonstrate for me as opposed to just producing this product.
00:28:40.440 --> 00:28:44.360
Because it used to be you produce this product, that was your homework, that was your thinking.
00:28:44.360 --> 00:28:46.759
AI has kind of thrown all that out the window.
00:28:46.759 --> 00:28:52.279
So that's why I really moved towards this uh this standards-based grading to try to kind of eliminate that.
00:28:52.279 --> 00:28:54.200
Now, they know full on.
00:28:54.200 --> 00:28:56.440
I mean, I'm very heavily involved with AI.
00:28:56.440 --> 00:28:57.240
I talk about a lot.
00:28:57.240 --> 00:29:05.160
A lot of my things at the bottom will have a little AI disclosure about how I use it to modify or you know, make formatting changes or whatever the case may be.
00:29:05.160 --> 00:29:09.400
I talk to them about creating prompts for practice questions and stuff like that.
00:29:09.640 --> 00:29:14.200
But we also talk about, you know, at the end of the day, they need to do the hard part of learning.
00:29:14.200 --> 00:29:17.079
They need to do the you know, the friction that's involved.
00:29:17.079 --> 00:29:19.480
Because otherwise, I mean, I have a lot of athletes, right?
00:29:19.480 --> 00:29:23.319
You know, the old analogy of just, you know, let's just have the forklift lift the uh heavy weights for us.
00:29:23.319 --> 00:29:25.640
They know that there's not going to be any benefit to that.
00:29:25.640 --> 00:29:32.759
Uh the nice thing is, I think the uh the standard space grading, because they don't have to get it right the first time, gives them a little bit of a safety net.
00:29:32.759 --> 00:29:37.160
And they're for once in my I mean, I've been teaching for a long time, man, 30 years plus.
00:29:37.559 --> 00:29:38.360
You said Lotus Notes.
00:29:38.360 --> 00:29:40.759
So that that aged you a little.
00:29:41.079 --> 00:29:41.240
Right.
00:29:41.240 --> 00:29:41.559
Yeah.
00:29:41.559 --> 00:29:42.440
So yeah, exactly.
00:29:42.440 --> 00:29:46.200
So they but they're actually learning from their mistakes, right?
00:29:46.200 --> 00:29:51.240
So they they have to get like eight out of 10 or better, and they get a seven, and they don't just say, oh, it's a seven out of ten.
00:29:51.240 --> 00:29:55.240
They look at that 30% and they focus on it for next time to do better, right?
00:29:55.240 --> 00:29:58.759
Uh I never would have done that without AI messing everything up.
00:29:58.759 --> 00:30:00.120
So it's kind of interesting.
00:30:00.120 --> 00:30:11.640
I'm I'm using AI to redevelop my whole my class, but I'm still kind of, I guess, teaching in an old school, not really analog, but you know, kind of in an old school way as far as uh getting students to learn and think.
00:30:12.279 --> 00:30:21.319
So I'm gonna go back to Amy and Kathryn, um, because I I I know you do have your students use a particular AI tool that you two have developed.
00:30:21.319 --> 00:30:23.160
Um, can you tell us a little bit about that?
00:30:23.559 --> 00:30:34.279
Kathryn took a class, I think it might have even been loose class on how to uh make your own custom GPT, basically.
00:30:34.279 --> 00:30:46.120
And so we have um ChatGPT with uh custom instructions, and we fed it the knowledge base because we already had so many hundreds of pages of our deconstruct calculus book.
00:30:46.120 --> 00:30:53.319
It was easy to just feed that as a PDF into the knowledge base, say only use this and give it some instructions.
00:30:53.319 --> 00:31:00.920
And so we basically have a 24-hour tutor for our students that is customized to our book, to our language, to our notation.
00:31:00.920 --> 00:31:12.120
And um, then Kathryn could talk about some of the student uh reactions to that, and we've started to collect data on custom versus general GPT, interacting with these as a tutor.
00:31:12.120 --> 00:31:15.240
Uh, we've gotten some really great feedback.
00:31:16.279 --> 00:31:17.079
Kathryn?
00:31:17.559 --> 00:31:24.360
We at the College of Charleston, so we do not provide a license for open AI, ChatGPT.
00:31:24.360 --> 00:31:27.079
They have a license for Microsoft Copilot.
00:31:27.079 --> 00:31:29.960
Um, I'll leave my thoughts on that to myself.
00:31:29.960 --> 00:31:40.440
And um but like Lew said, the assumption is they basically leave our classroom and they're using some sort of AI.
00:31:40.440 --> 00:31:59.240
And um I don't want to get into this whole ethical thing, but this is what if I ever have students talk to me about it, my whole thing is that uh the college provides Microsoft Office 365, which sits in a data center that we require our students to use, right?
00:31:59.240 --> 00:32:01.559
And we don't have a problem with that.
00:32:01.559 --> 00:32:08.279
There's nobody talking about ethical implications of our students all having a Microsoft Office 365 account.
00:32:08.279 --> 00:32:14.360
But then there's all this conversation about ethical implications of them using Chat GPT.
00:32:14.360 --> 00:32:24.200
And so that anyway, my whole point is I can have a conversation all day long about um reducing our cloud usage as a whole.
00:32:24.200 --> 00:32:35.240
I can have a conversation about that all day, but I have a lot of trouble pointing the finger at AI, which is why we decided to dive in on this hole.
00:32:35.480 --> 00:32:49.000
So um I started doing a lot of thinking after this MAA workshop about AI agents that have a reduced um knowledge base, right?
00:32:49.000 --> 00:32:53.480
So we don't let our AI agent access the web for answers.
00:32:53.480 --> 00:32:55.240
Okay, so it has the knowledge base.
00:32:55.240 --> 00:32:59.799
We have our book, we have our syllabus uploaded there, assessments, et cetera.
00:32:59.799 --> 00:33:17.400
Um is that enough for our students to be able to ask it any questions that it has versus them just going to chatgbt.com, open, you know, not an agent, right?
00:33:17.400 --> 00:33:19.960
Um, and asking it questions, right?
00:33:19.960 --> 00:33:22.440
So how do they interact with those two things?
00:33:22.440 --> 00:33:29.720
So I pulled a couple of tools together, um, and we have some survey and some open-ended questions.
00:33:29.720 --> 00:33:37.799
Um, and so at the midterm in the final exam, we invite them uh to study with the AIs, right?
00:33:37.799 --> 00:33:43.079
So the the AI agent that's specific to their class and then general chat GPT.
00:33:43.079 --> 00:33:44.600
So general chat GPT.
00:33:44.600 --> 00:33:46.440
And we give them a few, sorry, go ahead.
00:33:46.440 --> 00:33:46.839
Please go ahead.
00:33:47.079 --> 00:33:48.440
Yeah, let me jump in just to clarify.
00:33:48.440 --> 00:33:54.680
So I think I'm hearing two attributes of your agent that are potentially relevant to what you're about to say.
00:33:54.680 --> 00:34:00.600
One is that it is uh essentially trained, it has access to your course materials through your textbook.
00:34:00.600 --> 00:34:10.039
And I think it's also been prompted to behave more like what Anna Mills calls an ethical tutor, someone who's not just giving away all the answers to students, right?
00:34:10.039 --> 00:34:13.719
Which off-the-shelf chat GPT does not have that quality.
00:34:14.119 --> 00:34:14.599
For sure.
00:34:14.599 --> 00:34:15.400
A hundred percent.
00:34:15.400 --> 00:34:16.599
Yes, absolutely.
00:34:16.599 --> 00:34:23.400
Yes, we um we had played around, Amy and I had both played around with Khanmigo some before we did this, right?
00:34:23.400 --> 00:34:27.400
And this idea of the Socratic type um model.
00:34:27.400 --> 00:34:34.920
And I did do some playing around with bullying our agent AI and figuring out how long it would take it before it would give them the answer and all this kind of stuff.
00:34:34.920 --> 00:34:39.239
And I I tried to help train it in ways to make that last longer, right?
00:34:39.239 --> 00:34:42.359
To make it so you had to bully it longer before it gave you the answer.
00:34:42.359 --> 00:34:45.319
So we give them some prompts to start off with, right?
00:34:45.319 --> 00:34:52.599
Pick a topic that's gonna be on the midterm and ask the AI agent to make you flashcards for it, whatever, right?
00:34:52.599 --> 00:34:56.119
Ask it to make you a multiple choice quiz, 10 questions.
00:34:56.119 --> 00:35:01.319
Ask it about one of those questions on the quiz that you didn't understand and ask it to go through, right?
00:35:01.319 --> 00:35:03.559
So we give them some prompts.
00:35:03.559 --> 00:35:16.119
And the first time we did this, which was fall midterm, this past fall 2025, I had students being like, I literally had no idea that that AI could be used in this way.
00:35:16.119 --> 00:35:22.679
Like I had no, they they said, I will never study the same way again, like for a test, period.
00:35:22.679 --> 00:35:26.279
Like I had no clue that I could do this.
00:35:26.599 --> 00:35:30.119
Um which I think anyway, yeah, go ahead, Amy.
00:35:30.119 --> 00:35:33.480
I just think that's particularly important in our online class, right?
00:35:33.480 --> 00:35:36.439
And face-to-face or students have access to the math lab.
00:35:36.439 --> 00:35:45.719
These are distance ed students, and to be able to have this 24-hour access to a tutor that speaks the same language and and they understand it's a conversation.
00:35:45.719 --> 00:35:48.679
It's not just a way to go grab an answer and cheat, right?
00:35:48.679 --> 00:35:54.439
It's to have a conversation and to help me prepare for this midterm and final exam.
00:35:54.439 --> 00:35:58.039
So they were using it because of the coaching that we've provided.
00:35:58.039 --> 00:36:02.519
We, you know, there's it's hard to check on some of this, but they were using it in a constructive way, right?
00:36:02.519 --> 00:36:04.679
So we coached them on how to use it.
00:36:05.079 --> 00:36:05.719
Yeah.
00:36:05.719 --> 00:36:11.319
Um, yeah, I don't know that I had uh any more to say about that, but we were collecting data again this semester.
00:36:11.319 --> 00:36:19.319
We're excited this summer to analyze that um to figure out sort of uh what they liked about each one.
00:36:19.319 --> 00:36:23.959
They have to answer things that they liked or didn't like about each one that they've studied with.
00:36:23.959 --> 00:36:31.959
Um and I I mean, I personally think it's should it's not really a surprise, but I personally think agent tech AI is where we're going, right?
00:36:31.959 --> 00:36:35.559
Like everybody's got an AI chatbot somewhere.
00:36:35.719 --> 00:36:47.239
And I think I think the neat thing about that is for me to witness, you know, Kathryn went to Lew's class, but then to see um how easy it is to make one for every single class that you have.
00:36:47.239 --> 00:36:51.319
Like it's it Lew, Lew had this set up, and it's like, oh, you make the custom instructions.
00:36:51.319 --> 00:36:56.359
We're luckily had our own you know textbook material, but not every movie has that.
00:36:56.359 --> 00:36:59.399
But there's it there's certain facets of your class that you can make.
00:36:59.399 --> 00:37:03.480
And Kathryn did some really cool ones for an independent study student.
00:37:03.480 --> 00:37:20.919
It was a student class of once, just one student in a class that you know it was able to sort of mimic and virtual classmates and have um conversations as if you were it was a um math ed student that was going to be teaching and mimic conversations that you might have with high school parents.
00:37:20.919 --> 00:37:27.079
And so you can do these things rather easily with not much startup, not much know-how about coding.
00:37:27.079 --> 00:37:30.199
Like that's I mean, we was talking about Lotus 123 .
00:37:30.199 --> 00:37:36.839
Imagine if you had to do something like that 30 years ago now as an instructor, it's so easy to make these tools.
00:37:37.239 --> 00:37:37.879
Yeah.
00:37:37.879 --> 00:37:48.919
Chloe, I'm curious what you make of this idea of providing students with a kind of AI-powered tutoring resource.
00:37:48.919 --> 00:37:56.679
What what what did you just hear that stood out in your mind as as interesting or useful or maybe concerning?
00:37:57.559 --> 00:38:06.199
Um I haven't like uh set up any sort of particular tools for my students use.
00:38:06.199 --> 00:38:09.799
Like I said, I mostly sort of bring them something already generated.
00:38:09.799 --> 00:38:18.519
Um but I do have a lot of conversations with my students, especially I'm teaching abstract algebra this semester, um, with actually like a returning cohort of students that I'd already taught in geometry.
00:38:18.519 --> 00:38:20.279
So it's like small class, nice size.
00:38:20.279 --> 00:38:21.879
I know these students pretty well.
00:38:21.879 --> 00:38:29.799
Um and uh all of my proofs classes are very like you try something once, I give you a lot of feedback on it, and then you try it again.
00:38:29.799 --> 00:38:35.319
Uh not quite a standards-based grading, but like a specs grading kind of situation, I would say more.
00:38:35.319 --> 00:38:40.919
Um and in a lot of those conversations, I guess my I'm just always sort of curious.
00:38:40.919 --> 00:38:42.599
I'm like, Do we use AI?
00:38:42.599 --> 00:38:51.319
And my students themselves are actually quite negative um on how well it does explaining mathematical ideas.
00:38:51.319 --> 00:38:57.239
Again, these are abstract algebra students, so I might get a different response if I asked my calculus student something about this, right?
00:38:57.239 --> 00:39:04.919
Um, but I I was talking to the student recently, uh, they're like working on some exam revisions right now.
00:39:04.919 --> 00:39:11.639
And uh like before I'd even give them their test back, she was like, she's like, I couldn't figure this one out on the test, and I got here, here, and here.
00:39:11.639 --> 00:39:18.839
And she's like, and I went and asked, you know, ChatGPT or Claude which everyone, she was like, the answer it gave me was like really bad.
00:39:18.839 --> 00:39:21.879
I was like, she's like, it didn't, it didn't make sense.
00:39:21.879 --> 00:39:27.319
And we sort of looked at it together and like flicked it into Claude, and it was like it wasn't all that great.
00:39:27.319 --> 00:39:32.119
Um I I think there's like more promise in this sort of self-contained ecosystem.
00:39:32.119 --> 00:39:42.679
Uh I I have a colleague at uh UW Stevens Point, uh, Grant Kopitzke, who uh also teaches a lot of distance learning, hybrid sort of things.
00:39:42.679 --> 00:39:55.879
Um, and he like one of his assignments for his calculus class is like uh they like have to have a conversation with his particular uh calculus tutor, and like they have to like send the log of that conversation to him.
00:39:55.879 --> 00:39:58.599
This is one of his assignments, and he really enjoys that.
00:39:58.599 --> 00:40:03.159
Um so I I think those use cases are intriguing.
00:40:03.319 --> 00:40:20.679
Um I also do like encounter quite often, I think, especially in my math for liberal arts class, I have a lot of English majors, not all of them, but I have a lot of English majors who take those sort of who feel very negatively about the future of generative AI as a whole, right?
00:40:20.679 --> 00:40:39.239
And so uh when I like because I have them uh one one assignment in that class is we read uh Cathy O'Neil's Weapons of Math Destruction, uh, and then they do a journal assignment in response to that, uh, which is a really nice book, and it's got a great interchapter that is really accessible to a lot of people.
00:40:39.239 --> 00:40:44.919
Um, but that is I sort of ask them to reflect on the role of, you know, did they see a weapon of math destruction in their own life?
00:40:44.919 --> 00:40:51.959
And I get a lot of papers from English students that are like, what what is my future gonna be if if all of this exists?
00:40:51.959 --> 00:40:58.919
And it feels uh insulting to or just incredibly negative, like sort of perspective on that.
00:40:58.919 --> 00:41:06.839
So I like I do think there are students out there who who don't it like aren't using AI tools in this way.
00:41:06.839 --> 00:41:12.119
Um whether that's like good for their employment prospects or whatnot later on, I'm not so sure.
00:41:12.119 --> 00:41:17.079
But I but I do think this is an attitude that does exist among the population of our students.
00:41:19.319 --> 00:41:24.759
Yeah, I just want to, as a uh innocent bystander, I just want to point out there's uh two different things going on.
00:41:24.759 --> 00:41:29.639
So our group down in the Carolinas, they're using ChatGPT, but what's it called?
00:41:29.639 --> 00:41:31.319
The corporate account, what's it called?
00:41:31.319 --> 00:41:32.599
Um what is it?
00:41:32.919 --> 00:41:34.039
Are you talking about teams?
00:41:34.279 --> 00:41:35.000
Teams, yeah.
00:41:35.000 --> 00:41:37.639
So what they're using, they're using it's a pro model.
00:41:37.639 --> 00:41:39.239
So this is a deep thinking model.
00:41:39.239 --> 00:41:40.919
This thing crushes stuff.
00:41:40.919 --> 00:41:46.279
Gazam and I, when we ran the workshop, even last semester, she's like, yes, it will do all of undergraduate mathematics.
00:41:46.279 --> 00:41:49.239
So you have to be careful when you say AI can't do things.
00:41:49.239 --> 00:41:55.399
Free model, I always, when I give my talks, I usually put up like a three-speed Schwinn bicycle from the 1960s.
00:41:55.399 --> 00:41:56.599
That's the free model.
00:41:56.599 --> 00:42:01.079
The paid-for model is usually Doc Brown's DeLorean from Back to the Future.
00:42:01.079 --> 00:42:03.559
Very fast, we'll take you, but dangerous, right?
00:42:03.559 --> 00:42:13.559
But but again, the thing is these these thinking models, um Abel's theorem, again, there's hardly anything that the thinking models that the paid for $20 models cannot do.
00:42:13.559 --> 00:42:16.679
So you're hearing slightly two different conversations going on.
00:42:16.679 --> 00:42:23.159
Sometimes it's making stuff up, sometimes just keep in mind that we're talking about totally two different models of what's happening.
00:42:23.159 --> 00:42:24.759
So just kind of point that out.
00:42:25.319 --> 00:42:30.039
Hey, this is Future Derek jumping in with a recommended reading on Lew's Point.
00:42:30.039 --> 00:42:36.359
Just last week, historian Mark Humphries wrote a post called The Agents Are Waking Up that made a similar point.
00:42:36.359 --> 00:42:44.119
If you're basing your estimation of generative AI's capabilities on free chatbots, you're missing out on a lot of the power of these tools.
00:42:44.119 --> 00:42:49.639
It's a good read that gives a sense of where AI is going in the near future, and it's a little scary.
00:42:49.639 --> 00:42:51.639
See the show notes for a link.
00:42:52.599 --> 00:42:53.079
Amy?
00:42:53.319 --> 00:42:54.519
I'd like to jump in on that too.
00:42:54.519 --> 00:43:05.559
Um, Lew, I agree that even further, uh besides just the free models, there are those explore custom GPTs that users contribute or companies contribute.
00:43:05.559 --> 00:43:07.719
So like the Wolfram one, I think, is amazing.
00:43:07.719 --> 00:43:12.439
The Wolfram GPT and the MATLAB one now, like, you know, the things that they can do.
00:43:12.439 --> 00:43:18.039
And so I just want to come back to Lew's comment before how it just decimated he had to totally revamp his class.
00:43:18.039 --> 00:43:19.159
I'm at that position.
00:43:19.159 --> 00:43:23.239
I haven't taught a class, a graduate class that I haven't done in a couple years.
00:43:23.239 --> 00:43:32.679
And now with these tools, especially with the MATLAB plus GPT, I feel like I need to totally redo my graduate class and start totally over.
00:43:32.679 --> 00:43:34.759
And but Lew's right, there's opportunity, right?
00:43:34.759 --> 00:43:36.039
That's that's something new.
00:43:36.039 --> 00:43:37.079
I've been teaching a long time.
00:43:37.079 --> 00:43:38.359
This is now fun and interesting.
00:43:38.359 --> 00:43:39.239
What am I gonna do?
00:43:39.239 --> 00:43:41.399
How and I come back to some of Chloe's things.
00:43:41.399 --> 00:43:43.399
What is it we're trying to get students to do?
00:43:43.399 --> 00:43:47.719
And I keep hearing over and over in hers is like verification and validation.
00:43:47.719 --> 00:43:49.319
And that was prior to AI, right?
00:43:49.319 --> 00:43:57.799
Because we talked about there are other tech tools, or there is the authoritative expert that, and so those skills become all the more heightened.
00:43:57.799 --> 00:44:02.359
Like we really have to get students good at doing those, verifying and validating.
00:44:02.359 --> 00:44:03.959
Because the tools aren't going anywhere.
00:44:03.959 --> 00:44:08.919
I do think they're just going to become more and more ingrained, and it's hard to not use them.
00:44:08.919 --> 00:44:14.199
Like you try you can't even do a regular search without a Google search anymore without getting AI response.
00:44:15.079 --> 00:44:19.559
A couple of years ago on the podcast I had on Brett Becker, who's a computer science educator.
00:44:19.559 --> 00:44:24.679
Um, and the the AI tools were pretty good at coding before they got pretty good at math.
00:44:24.679 --> 00:44:27.239
Um, and so CS has had to deal with this for a while.
00:44:27.239 --> 00:44:29.480
And and he argued that hey, it may be that.
00:44:29.480 --> 00:44:38.199
Our first year computer science curriculum is more about um verifying that programs do what we want them to do than actually coding them ourselves.
00:44:38.199 --> 00:44:41.879
And and he said, Look, this is all this is a useful skill already, right?
00:44:41.879 --> 00:44:47.000
Like if you're a if you're a software engineer, you've got to read other people's code and make sure it does what you want it to do.
00:44:47.000 --> 00:44:54.119
And so we just may get to that kind of uh earlier in the process um uh of CS education.
00:44:54.759 --> 00:44:58.679
Um Yeah, and I want to say, if I could, can I just piggyback on that really quickly?
00:44:58.679 --> 00:45:07.000
The College of Charleston has a newer major called Computing in the Arts, where they're focused on like the front-end design of things.
00:45:07.000 --> 00:45:16.839
And these students used to stumble quite a bit over having to design the back end before they could do all the front-end art stuff that they were doing.
00:45:16.839 --> 00:45:27.000
And I had a student in my business class who she and I were talking, and she said, AI has absolutely changed the way I'm able to do what I want to do with my major.
00:45:27.000 --> 00:45:33.079
She's like, I'm no longer up all night trying to figure out how to code the back end of these things.
00:45:33.079 --> 00:45:36.119
And I get to focus on the artistic piece, right?
00:45:36.119 --> 00:45:38.119
On the front end of all of this.
00:45:38.119 --> 00:45:41.559
That's what I want to be doing with my major, right?
00:45:41.559 --> 00:45:46.359
And I just think that's a great example of um of what it can do, right?
00:45:46.359 --> 00:45:53.879
What it can, what cognitive demand it can free up for our students who are in these artistic um type majors.
00:45:54.039 --> 00:45:56.199
So Lew, I want to go back to you.
00:45:56.199 --> 00:46:08.039
Um, you mentioned using, speaking of getting a little AI-powered assist in your professional work, you mentioned using AI to help you implement standards-based grading.
00:46:08.039 --> 00:46:15.559
Um, I know you talk a lot about ways we've we focus mostly on student use of AI, and I think that's a great, a great place to be.
00:46:15.559 --> 00:46:20.199
But but I know you talk a lot about how instructors can use AI in their work.
00:46:20.199 --> 00:46:30.279
Are there are there a few kind of strategies or ideas that you find particularly useful to share with math educators who are maybe interested in using some AI in their their course prep?
00:46:31.079 --> 00:46:31.799
Oh my God.
00:46:31.799 --> 00:46:33.319
So the course prep.
00:46:33.319 --> 00:46:36.519
So, you know, I'm looking at Amy when she's going to do that class.
00:46:36.519 --> 00:46:43.639
She's going to have an AI graduate student, right, Amy, over your shoulder, like talking back and forth with this and pushing back and forth.
00:46:43.639 --> 00:46:50.679
The thing that our math colleagues have to realize is that they've done a lot of hard work over the years and they become experts in their field, right?
00:46:50.679 --> 00:46:53.799
They're experts about their content, their discipline.
00:46:53.799 --> 00:46:57.079
They know their students very well and they know the context as well.
00:46:57.079 --> 00:47:01.480
They can then take that and leverage that while using AI, right?
00:47:01.639 --> 00:47:03.639
So I want to redesign my course.
00:47:03.639 --> 00:47:06.279
So, okay, um, I'm teaching Calc 2, right?
00:47:06.279 --> 00:47:13.000
And um okay, I know that students are gonna mess up some of the sequence tests and they're gonna, you know, what are they gonna do?
00:47:13.000 --> 00:47:15.159
Uh they're gonna or sorry, the convergence test, right?
00:47:15.159 --> 00:47:19.719
So they're gonna like um they're gonna confuse two different tests going on.
00:47:19.719 --> 00:47:29.399
So I can ask AI, say, hey, give me a series where students will be tempted, maybe you like to use the ratio test, but the comparison test would actually be the better move, right?
00:47:29.399 --> 00:47:33.639
So as an expert, you know your students will struggle with that and you can ask that question.
00:47:33.639 --> 00:47:39.959
Then you can follow up with and also give me two follow-up uh questions that will actually build distinction for my students.
00:47:39.959 --> 00:47:42.279
So now you're kind of channeling the AI.
00:47:42.279 --> 00:47:43.959
I'm not having my students use it.
00:47:43.959 --> 00:47:45.159
I mean that you could, right?
00:47:45.159 --> 00:47:54.199
But I'm having them, you know, AI is giving me a better question that I can ask for my students because I'm tapping into that expertise that I've seen them struggle with for over the years, right?
00:47:54.359 --> 00:47:57.879
Um my son is teaching AP stats as a 25-year-old.
00:47:57.879 --> 00:47:59.639
It's like, holy cow, how's he doing that?
00:47:59.639 --> 00:48:03.239
Because he leans into the AI and he says, hey, explain this to me.
00:48:03.239 --> 00:48:06.199
You know, I know they're gonna, there's linear regression, they're not gonna understand this.
00:48:06.199 --> 00:48:08.599
Give me three different ways to explain that, right?
00:48:08.599 --> 00:48:14.199
And then he can go through and filter out which one works best for him and then help to explain that to the to his students.
00:48:14.199 --> 00:48:20.359
So I think, you know, math colleagues, we have to understand that we are a bit in the driver's seat with AI.
00:48:20.359 --> 00:48:26.439
If we ask it to do something, if it gives us the wrong answer, I mean, if a student gave you a wrong answer, you wouldn't say, Oh, stupid student.
00:48:26.439 --> 00:48:29.079
You say, Oh, let me ask that in a slightly different way.
00:48:29.079 --> 00:48:34.679
So use your power as a mathematician, use your power as an educator to kind of rechannel that energy.
00:48:34.679 --> 00:48:37.399
And when you're working with the AI, treat it like a student.
00:48:37.399 --> 00:48:39.079
If it didn't give you the right answer, oh, you know what?
00:48:39.079 --> 00:48:40.519
Let me ask that in a slightly different way.
00:48:40.519 --> 00:48:42.599
Let me kind of move that in a different direction.
00:48:42.599 --> 00:48:48.279
Um, and man, you can just, as an expert, you can just make this AI stuff sing for you, right?
00:48:48.279 --> 00:48:49.799
It does some really, really strong stuff.
00:48:49.959 --> 00:48:54.199
It like I said, it's it's re it's revamped my whole course, or I should say, let me rephrase that.
00:48:54.199 --> 00:48:58.919
I've revamped my whole course with it helping me, you know, kind of provide the structure of what's going on.
00:48:58.919 --> 00:49:01.719
But it's just been a blast doing that.
00:49:01.719 --> 00:49:08.439
Uh now again, I I, you know, I like I said, I don't know if it's chickened out as the right word, but I I have pulled things back in the classroom.
00:49:08.439 --> 00:49:12.599
I'm not doing as much um deliberate homework, but I have these notes, right?
00:49:12.599 --> 00:49:14.759
I create videos and I do a little stuff with things.
00:49:14.759 --> 00:49:19.239
I have 61 students, my video notes, I'm getting like 80 hits.
00:49:19.239 --> 00:49:19.480
Okay.
00:49:19.480 --> 00:49:28.439
So either I have some psychotic student who's watching it 50 times, which I don't think is the case, but they're actually learning, they're looking at the notes because they know they will be tested on this material.
00:49:28.439 --> 00:49:33.719
And again, getting back to that standards-based grading, they're trying to focus on the stuff that they missed.
00:49:33.719 --> 00:49:40.759
So if they didn't get it the first time, that's why I'm getting more hits on a view than you know, just the minimal of 61 or something like that.
00:49:40.759 --> 00:49:42.199
It's really amazing.
00:49:42.199 --> 00:49:43.959
And I'm not even, I'm not giving any grades for this.
00:49:43.959 --> 00:49:44.919
I'm not assigning it, right?
00:49:44.919 --> 00:49:46.439
They're actually doing the work for once.
00:49:46.439 --> 00:49:47.559
I'm I'm stunned.
00:49:47.719 --> 00:49:54.279
So I um I keep referencing to old podcast episodes, but it's one reason I do this is to see these connections.
00:49:54.279 --> 00:50:01.480
I had uh Sharona Krinsky and Robert Bosley on a couple of episodes ago who who run the grading conference and the Center for Grading Reform.
00:50:01.480 --> 00:50:08.679
Um, and they said uh alternative grading practices just keep coming up, right?
00:50:08.679 --> 00:50:15.799
We didn't come here today to talk about alternative grading practices, but like like it's one very sensible response to the advent of AI.
00:50:15.799 --> 00:50:20.439
And so I hear it a lot in the guests that I have on my podcast, even if that's not the ostensible topic.
00:50:20.439 --> 00:50:23.239
And so a little more proof of their their theory today.
00:50:23.399 --> 00:50:29.399
Um, we are almost out of time, but I want to do um, let's do a round robin with one last question.
00:50:29.399 --> 00:50:36.839
What do you wish more of your math colleagues knew about generative AI's impact on teaching and learning?
00:50:37.319 --> 00:50:38.119
I will start.
00:50:38.119 --> 00:50:48.359
Um I uh part of what I've been working on has been uh a research project with two of my undergraduate students, Ellah Olson and Madison Schwartz.
00:50:48.359 --> 00:50:50.679
They're both uh mathematics education majors.
00:50:50.679 --> 00:51:03.480
Um and we have been um testing how uh students respond differently to proofs when we tell them this has been generated by AAI versus when there's no mention of the author.
00:51:03.480 --> 00:51:05.639
Like, are you more or less likely to believe that?
00:51:05.639 --> 00:51:14.439
Um, so we've been working on this research project for the last couple of years, and um, as part of that, we just ended up prompting the AI to write and prove a lot of different mathematical theorems.
00:51:14.439 --> 00:51:20.359
So um in doing so, we just were like, well, we kind of know a lot about what AI does with math.
00:51:20.359 --> 00:51:24.839
And so we've we're talking at our um department meeting, the three of us.
00:51:24.839 --> 00:51:32.919
Um, and I think the revelation of like you know, like it can do your entire take-home exam, like just full stop.
00:51:32.919 --> 00:51:35.399
I I do think that was surprising to some people.
00:51:35.399 --> 00:51:44.759
And so um for people who have like maybe really never tried those things, like I think they'd be shocked by like I said, it can it can do it all.
00:51:44.759 --> 00:51:47.319
Well, it could give you an answer for everything, right?
00:51:47.719 --> 00:51:57.639
So, and especially if you if you last checked in in 2023 or 2024, when when the tools were were often really bad at math, um, that has changed.
00:51:57.639 --> 00:51:58.039
Yeah.
00:51:58.039 --> 00:51:59.079
Kathryn?
00:52:00.439 --> 00:52:07.480
Um, so I would like to stop having people ask me the question, how do I keep my students from cheating with AI?
00:52:07.480 --> 00:52:10.359
That is the constant question people are asking.
00:52:10.359 --> 00:52:11.799
How do I keep my students from cheating?
00:52:11.799 --> 00:52:13.239
How do I keep them from cheating?
00:52:13.239 --> 00:52:26.119
I wish we had so many more people in our department who thought like Lew thought, where it's like, this is an opportunity for me to completely revamp the way that I'm teaching mathematics, right?
00:52:26.119 --> 00:52:28.279
And I have that opportunity, right?
00:52:28.279 --> 00:52:31.559
Um so that's I don't I don't know.
00:52:31.559 --> 00:52:32.679
That's that's what I'll say.
00:52:32.679 --> 00:52:45.000
And I don't know if Amy's gonna bring up from Solve Khan's book, but we we like, we have really linked on to this idea of um now that AI is here, you can expect more of your students.
00:52:45.000 --> 00:52:51.000
You can ask them to do more harder things because they have these tools.
00:52:51.000 --> 00:52:54.119
And we're really taking advantage of that in our classes.
00:52:54.439 --> 00:52:55.959
Amy, what would you add to that?
00:52:56.279 --> 00:53:00.599
Um I've heard some people say, well, AI only knows what's already known.
00:53:00.599 --> 00:53:07.079
And so what the human can, but I I don't know that that's necessarily I think it has elements of incremental creativity.
00:53:07.079 --> 00:53:12.679
I don't think it'll ever do paradigm shifting creativity, but but so then that leaves like what is left for the human.
00:53:12.679 --> 00:53:22.359
And uh if you have this encyclopedic resource, and Kathryn was saying, yeah, we can now ask grander assignments, they can expect more because you have leveraged this amazing assistant.
00:53:22.359 --> 00:53:26.519
Um but I also come back to okay, well then we need to foster creativity.
00:53:26.519 --> 00:53:27.559
And what exactly?
00:53:27.559 --> 00:53:29.639
That's such a vague word, and what does that really mean?
00:53:29.639 --> 00:53:31.000
And what does that mean in mathematics?
00:53:31.000 --> 00:53:37.000
And so like these broader discussions about what other things could we focus on?
00:53:37.000 --> 00:53:46.759
Um and I come back to Conrad Wolfram's four steps of computer-based math, and at one point he was like, oh, in step three, we're focusing on the computation step, and that's the thing that computers are good at.
00:53:46.759 --> 00:53:53.480
Well, now it's like computers are actually getting better at even the first step, and sometimes the second, and certainly the fourth.
00:53:53.480 --> 00:53:56.759
So so what is, you know, again, what is left then?
00:53:56.759 --> 00:53:59.480
What what we need to rethink what we're doing.
00:53:59.799 --> 00:54:01.559
I don't know if that's exciting or scary.
00:54:01.559 --> 00:54:04.039
Um of both.
00:54:04.039 --> 00:54:08.599
Lew, what do you wish more of our math colleagues knew about AI and teaching and learning?
00:54:08.839 --> 00:54:13.639
Uh I'm gonna piggyback a little bit on what Kat or what Amy said about, you know, kind of getting involved with it.
00:54:13.639 --> 00:54:16.759
Um, I I had a thing earlier today, and I was kind of thinking about this.
00:54:16.759 --> 00:54:23.719
Uh for those of us who've had young children, right, and you try to get them to uh eat something new like broccoli, right?
00:54:23.719 --> 00:54:29.399
You can't, you know, I'm not gonna have my kid decide whether they're gonna eat broccoli by listening to some other kid.
00:54:29.399 --> 00:54:31.000
They have to try it themselves, right?
00:54:31.000 --> 00:54:34.279
But moreover, it's one and done is not the deal, right?
00:54:34.279 --> 00:54:42.279
In order to try something new and kind of get used to the flavors and all that kind of stuff, unless you have that weird gene for broccoli, uh, you know, it takes like five or six times.
00:54:42.279 --> 00:54:53.000
So, you know, I think for for the math colleagues who have tried broccoli a long time ago, or maybe just listen to somebody else complain about broccoli, yeah, it's probably time to pony up and try the broccoli again.
00:54:53.000 --> 00:55:02.439
Uh again, I would encourage, you know, if you do have the means to do it, uh, to poke around with one of the paid models or the thinking models because it will really surprise you.
00:55:02.599 --> 00:55:08.599
But again, kind of spinning it from, oh, what are our students, you know, with Kathryn saying about our students cheating about how could I leverage this?
00:55:08.599 --> 00:55:13.079
How could I make myself into, you know, there we've had a lot of issues with with education.
00:55:13.079 --> 00:55:17.319
It wasn't like, you know, things were running along fine and all of a sudden AI hit and then things were bad, right?
00:55:17.319 --> 00:55:18.919
There was a lot of stuff going on.
00:55:18.919 --> 00:55:21.480
And I think AI just has kind of amplified that.
00:55:21.480 --> 00:55:39.399
But I do think we can kind of use, we can kind of channel this negative thing that it does do on the on for ourselves and kind of actually turn it into something positive to help ourselves and help our students better with their learning if we you know use it appropriately and and kind of you know do all the stuff that we already knew was good.
00:55:39.399 --> 00:55:44.759
We now actually have to pay attention to those, uh, you know, how students learn and cognitive science and all that stuff.
00:55:45.000 --> 00:55:45.480
Yeah.
00:55:45.480 --> 00:55:48.119
Well, um, that's a good place to leave it.
00:55:48.119 --> 00:55:52.039
I know there's lots more in this topic we could explore together, but um our time is nigh.
00:55:52.039 --> 00:55:54.039
Thank you so much for being here and being part of this panel.
00:55:54.039 --> 00:55:55.239
I really appreciate it.
00:55:55.239 --> 00:55:57.480
Um yeah, thanks so much.
00:55:57.799 --> 00:55:58.679
Thank you.
00:55:59.159 --> 00:55:59.799
Thank you all.
00:55:59.799 --> 00:56:00.519
Nice to meet you guys.
00:56:00.839 --> 00:56:01.239
Thank you.
00:56:03.959 --> 00:56:13.319
Thank you to Amy Langville, Chloe Lewis, Lew Ludwig, and Kathryn Pedings Behling for coming on the show and sharing their current AI-aware teaching practices.
00:56:13.319 --> 00:56:30.439
In the show notes, you'll find links to Lew's new book, The Science of Learning Meets AI, and to Amy and Kathryn's Deconstruct Calculus Project, and to Chloe's website, and to all the books, articles, and podcast episodes we mentioned during our conversation.
00:56:30.439 --> 00:56:36.599
And if you're a math educator, I highly recommend joining the Teaching Mathematics with Generative AI Google Group.
00:56:36.599 --> 00:56:41.159
That's where some of the best discussion of AI in math education is happening right now.
00:56:41.159 --> 00:56:43.879
See the show notes for a link to that group too.
00:56:44.279 --> 00:56:49.719
Intentional Teaching is sponsored by UPCEA, the Online and Professional Education Association.
00:56:49.719 --> 00:56:58.279
In the show notes, you'll find a link to the UPCEA website where you can find out about their research, networking opportunities, and professional development offerings.
00:56:58.279 --> 00:57:04.119
This episode of Intentional Teaching was produced and edited by me, Derek Bruff.
00:57:04.119 --> 00:57:12.279
See the show notes for links to my website and socials and to the Intentional Teaching newsletter, which goes out most weeks on Thursday or Friday.
00:57:12.279 --> 00:57:18.359
If you found this or any episode of Intentional Teaching useful, would you consider sharing it with a colleague?
00:57:18.359 --> 00:57:19.799
That would mean a lot.
00:57:19.799 --> 00:57:22.279
As always, thanks for listening.