WEBVTT
00:00:00.160 --> 00:00:02.799
So I want to start with a person in a room.
00:00:02.959 --> 00:00:06.480
So January this year on campus here at RMIT Vietnam.
00:00:06.959 --> 00:00:15.359
So 90 minutes with the people who run our degrees, we're talking program managers, course coordinators, doing a thing we call the AI reality check.
00:00:15.519 --> 00:00:28.000
The idea is that we get these people hands-on with the edge of the field, so wrapped in the same kind of like here's what shifted this month framing that we do on this show, and then a discussion across programs that never really talk to each other.
00:00:28.320 --> 00:00:29.839
I do remember you building this out.
00:00:29.920 --> 00:00:33.280
It's a brilliant way to kind of open eyes and get to that aha moment quickly.
00:00:33.600 --> 00:00:34.000
100%.
00:00:34.479 --> 00:00:38.560
And the last station in the session, I opened with the demo, right?
00:00:38.640 --> 00:00:48.000
So I got an Agentic browser, I think it was uh Atlas, rest in peace, OpenAI is Atlas from Memory, gave it a Canvas login, pointed it at a live assignment brief, and let it go.
00:00:48.159 --> 00:00:51.280
The thing read the brief, worked out the task, started doing it.
00:00:51.439 --> 00:00:54.159
All the time I'm at the front, look, no hands.
00:00:54.880 --> 00:00:56.640
And this was like nine months ago, yeah.
00:00:56.799 --> 00:01:03.520
So in an AI standards kind of life cycle, we've had nine different open AI releases since then and countless other models.
00:01:03.600 --> 00:01:06.159
So technology then, plus we're around now.
00:01:06.480 --> 00:01:08.640
I was a fresh face youngster back when I was running that.
00:01:09.200 --> 00:01:10.719
Like, I mean, it has aged me.
00:01:10.879 --> 00:01:15.359
But then what we did, because it wasn't just this sort of, oh, look at this flashy thing, not just hype.
00:01:15.519 --> 00:01:17.040
We handed out cards, okay?
00:01:17.200 --> 00:01:28.400
So every assessment type that you can think of, and the idea was all right, oh academics, or course coordinators, program type manager types, sort these into two piles as you see them now based on what you've just seen.
00:01:28.480 --> 00:01:30.079
And there were a few other demos as well.
00:01:30.239 --> 00:01:33.760
But are these AI vulnerable or are they AI resistant, if you like?
00:01:34.000 --> 00:01:36.239
We gave them 10, 11 minutes, sort them.
00:01:36.480 --> 00:01:40.079
And the the thing that was interesting, it was never clean, it was never unanimous.
00:01:40.159 --> 00:01:43.280
People have cards and there was disagreement, which was great.
00:01:43.519 --> 00:01:46.400
But the really interesting bit came at the end.
00:01:46.640 --> 00:01:47.040
Yeah.
00:01:47.359 --> 00:01:47.920
Go on.
00:01:48.319 --> 00:01:50.719
So there's a there's a program manager in the room, right?
00:01:50.799 --> 00:01:56.480
And she runs a program with hundreds of students across multiple locations, dozens of teaching staff, and others.
00:01:56.640 --> 00:02:01.439
She sat through 40 minutes of me and my team showing her what the tech does now.
00:02:01.599 --> 00:02:06.640
She's sorted her own assessments into piles, and most of them are in the wrong pile.
00:02:07.120 --> 00:02:12.719
And she sort of looks up and says, not angry, not panicking, just straight, like boom.
00:02:13.919 --> 00:02:14.560
What do I do now?
00:02:15.120 --> 00:02:16.960
It's a strange aha moment.
00:02:17.360 --> 00:02:18.159
For real, eh.
00:02:18.319 --> 00:02:20.560
And look, I didn't have an answer full disclosure.
00:02:20.639 --> 00:02:25.840
Um, I'd show in her the problem in 4K, and I did not have the next slide.
00:02:25.919 --> 00:02:32.719
Uh, the institution had a policy, she had a program, and there was nothing between the two uh that told her what to do on the Monday.
00:02:33.039 --> 00:02:39.039
Alas, the constant gap between policy and practice, the guided note, the guidance notes and the reality.
00:02:39.360 --> 00:02:39.680
Yeah.
00:02:39.919 --> 00:02:42.400
So that's what we're gonna be talking about today.
00:02:51.919 --> 00:02:56.639
Ladies and gentlemen, welcome to Adjunct Intelligence, the podcast that looks at AI and impact on higher education.
00:02:56.719 --> 00:03:01.280
I'm joined, as always, by Nick McIntosh, learning futurist, and myself, head of AI education.
00:03:01.439 --> 00:03:04.159
So, Nick, I would say this is our fourth goal at assessment.
00:03:04.240 --> 00:03:09.919
We've had Phil Dawson, we've spoken about Jason Lodge multiple times, we've done the two-lane thing.
00:03:10.159 --> 00:03:13.840
Why should we sit through a fourth episode of AI and assessment?
00:03:14.639 --> 00:03:17.199
Because of something that you said, actually.
00:03:17.360 --> 00:03:23.360
So five weeks ago, you opened the Phil Dawson episode by reading his line about the owl, right?
00:03:23.439 --> 00:03:30.719
So that that just redesign your assessment sort of thing is step one, draw two circles, step two, draw the rest of the owl.
00:03:30.879 --> 00:03:32.960
And then you uh you rift your own bit, right?
00:03:33.280 --> 00:03:34.479
Good luck with the feathers.
00:03:34.639 --> 00:03:37.840
Feathers are hard to draw for anyone who's never drawn an owl before.
00:03:38.159 --> 00:03:39.919
Feathers are bloody tough, right?
00:03:40.000 --> 00:03:54.400
And then so later in that conversation, Phil said something that really piqued my fancy, my interest, that the sector has the high-level guidance and it's got a handful, bare handful of fully realized examples and almost nothing in between, right?
00:03:54.639 --> 00:04:00.080
So he he spoke to a missing middle between frameworks and fully realized pictures.
00:04:00.319 --> 00:04:07.360
So it's our educator from the opening, innocently in search of a solution to what we are called a and others are called a wicked problem.
00:04:07.680 --> 00:04:08.479
Yep, exactly.
00:04:08.639 --> 00:04:12.159
So the gap between policy and practice, I guess, seen from both ends, right?
00:04:12.400 --> 00:04:15.759
So for lack of a better way to say it, we went looking for the feathers, right?
00:04:15.840 --> 00:04:21.920
So the things that the unis actually did, they wrote down and published somewhere that she or anyone could read the whole thing.
00:04:22.079 --> 00:04:25.680
So we found Ballpark 110, 114.
00:04:26.319 --> 00:04:26.879
Whoa.
00:04:27.439 --> 00:04:29.040
Yeah, and whoa, indeed.
00:04:29.279 --> 00:04:32.639
And so most of them, it turns out, are a conversation.
00:04:32.800 --> 00:04:39.519
Um, but before we dig into that and I tell you what we found, I need to tell you what this is and also what this is not.
00:04:39.839 --> 00:04:40.160
Yeah?
00:04:40.560 --> 00:04:42.800
A lengthy list of caveats, Nicholas.
00:04:42.959 --> 00:04:46.000
So uh what is the reality check thing?
00:04:46.160 --> 00:04:52.079
I think we need to dive into that reality check you opened up at the intro because it is actually a really powerful vehicle.
00:04:52.480 --> 00:04:53.199
Thank you very much.
00:04:53.360 --> 00:04:56.240
I I think you've got something similar you run your end as well.
00:04:56.399 --> 00:05:07.360
But for the record, uh, this is a hands-on exposure, uh, we frame it to the edge of the AI field for the people who make the decisions, right, who turn policy into practice, those who run programs, run courses.
00:05:07.600 --> 00:05:09.199
Ballpark 90 minutes.
00:05:09.279 --> 00:05:11.759
Uh, we do it the way that we do this show, right?
00:05:11.839 --> 00:05:13.680
So here's what's moved this quarter.
00:05:13.839 --> 00:05:18.639
We've got things that have changed with the regulators, with the tools, uh, research, the cases.
00:05:18.879 --> 00:05:22.000
And this pre-reading means that no one works in cold, right?
00:05:22.079 --> 00:05:24.000
Then they get their hands on the emerging tech.
00:05:24.240 --> 00:05:26.560
So these are not demonstrations or slides.
00:05:26.639 --> 00:05:35.199
This is the actual thing on a machine on a computer in front of them, doing something to learning, to teaching, and to assessments that they wrote now.
00:05:35.360 --> 00:05:36.879
What is possible now?
00:05:37.439 --> 00:05:38.959
As you mentioned, we've got something similar.
00:05:39.040 --> 00:05:41.040
We call a make and take session, plot twist.
00:05:41.279 --> 00:05:44.560
It's actually not about making and taking guys, it's just about opening eyes.
00:05:44.720 --> 00:05:47.839
I think Melbourne Uni has something also called their cheat lab, which I love.
00:05:47.920 --> 00:05:51.439
And from memory, you bring in like an assessment and see if AI can crack it.
00:05:51.759 --> 00:05:52.560
Spoiler alert.
00:05:52.879 --> 00:05:54.160
Most often it does.
00:05:54.560 --> 00:05:54.879
Nice.
00:05:54.959 --> 00:05:56.240
Yeah, no, that's great.
00:05:56.399 --> 00:06:03.920
Um, I think the the other part, and I'm sure you've got that in there as well, the thing that we found that matters most is we break down those organizational silos, right?
00:06:04.000 --> 00:06:05.519
We get people talking to each other.
00:06:05.680 --> 00:06:08.560
So from across the different subjects and schools.
00:06:08.639 --> 00:06:18.399
So you've got nursing, sitting down with business, sitting with uh software engineers, talking about the same experiences, the same technologies, and in this case of assessments, the same cards.
00:06:18.560 --> 00:06:28.240
And they find out that the person across the table that they might not otherwise get time with has reached the opposite conclusions for reasons that are also right.
00:06:29.199 --> 00:06:31.279
I don't care what people say about you, Nick.
00:06:31.360 --> 00:06:32.000
You are right.
00:06:32.079 --> 00:06:35.839
And your love of bringing people together, breaking down silos, it's just brilliant.
00:06:36.079 --> 00:06:36.639
Thanks, man.
00:06:36.959 --> 00:06:37.199
Thanks.
00:06:37.279 --> 00:06:39.920
I feel seen breaking down silence too.
00:06:40.160 --> 00:06:42.399
No, but that's that's the silo bit.
00:06:42.639 --> 00:06:46.000
And I I think it's where all the the real intelligence in the uni lives, right?
00:06:46.079 --> 00:06:49.519
So Phil uh Dawson referenced just now, has a paper on this.
00:06:49.600 --> 00:06:56.800
So the talk is cheap, which is about the difference between changing what you say about assessment and changing something a bit more meaningful, changing its structure.
00:06:57.040 --> 00:07:05.199
And on this very show, he said that the sector has confused AI use declarations and traffic lights guide and traffic light guidance with assurance.
00:07:05.439 --> 00:07:06.480
So that's the gap.
00:07:06.720 --> 00:07:12.480
The policy is a declaration, the assessment is a structure, and she's kind of standing in between them.
00:07:12.959 --> 00:07:17.439
So the reality check, the reality check is what goes in that kind of gap in theory.
00:07:17.600 --> 00:07:17.759
Yep.
00:07:17.920 --> 00:07:19.120
All right, let's get to the study.
00:07:19.279 --> 00:07:21.519
Let's get to the research, 114 data points.
00:07:21.600 --> 00:07:22.399
I I want to know.
00:07:22.800 --> 00:07:24.800
I imagine there's some form of an answer in here.
00:07:25.279 --> 00:07:25.839
There is one.
00:07:25.920 --> 00:07:26.959
Um I I do apologize.
00:07:27.040 --> 00:07:28.560
We're taking our time getting here, but it's worth it.
00:07:28.639 --> 00:07:29.680
It's the payoff is coming.
00:07:29.920 --> 00:07:46.480
So over about a month, uh, within our AI community practice at RMIT, we built a registry of about 100, as I say, 114 documented cases online, available for all, where an institution changed an assessment because of AI and published enough that a practitioner could read the whole thing.
00:07:46.639 --> 00:07:51.439
So public sources are only, which I'd say is a strength and a limitation in a sentence.
00:07:51.519 --> 00:07:51.920
Yeah.
00:07:52.160 --> 00:07:52.399
Correct.
00:07:52.560 --> 00:07:53.360
So what's the limit?
00:07:53.519 --> 00:07:58.160
Like get your get all your caveats out of the way and then I I know I'm I'm being super, super coy.
00:07:58.319 --> 00:07:58.480
Yeah.
00:07:58.560 --> 00:08:02.160
So the limit is that everything behind closed doors is invisible, right?
00:08:02.240 --> 00:08:07.439
So there'll be amazing work happening on course teams right now that no one has written up, and we can't see any of it.
00:08:07.600 --> 00:08:08.879
We can only see what was published.
00:08:08.959 --> 00:08:16.480
So when I I make grand statements like the sector does X, what I mean is the published record of what we could find at time of research says X.
00:08:16.560 --> 00:08:20.240
So this is not research per se, and I'm not going to pretend it is.
00:08:20.399 --> 00:08:20.959
Yeah.
00:08:21.360 --> 00:08:23.680
It's a really robust desktop scan.
00:08:23.920 --> 00:08:25.279
That but you've done the work.
00:08:25.439 --> 00:08:25.680
Yes.
00:08:25.920 --> 00:08:26.079
Yeah.
00:08:26.160 --> 00:08:29.920
So it's called a sector scan of current public uh practice, right?
00:08:30.000 --> 00:08:34.320
So I don't know if you're familiar with the one pages from Jamie Clark, pedagogy on a page.
00:08:34.639 --> 00:08:35.039
Yeah, yeah.
00:08:35.120 --> 00:08:37.120
So what is it if people aren't familiar?
00:08:37.519 --> 00:08:41.600
A primer on constructivism, on feedback, and it's a one pager, and you're done.
00:08:41.759 --> 00:08:47.519
So again, the whole idea of this exercise is to try and offer something practical, something useful for that PM I mentioned at the top.
00:08:47.679 --> 00:08:52.960
She has a day job, to-do list, measured in in feet, not items, competing deadlines, da da da.
00:08:53.679 --> 00:08:56.639
Think of this as a practical digest that we give over to her.
00:08:56.720 --> 00:09:02.080
And people like her is so empowered with that they can make their own decisions about what to do in their context.
00:09:02.399 --> 00:09:05.360
So let's let's start with what Phil said and a bit of news first.
00:09:05.519 --> 00:09:11.679
So on this show, he said we we'd caught him in the midst of writing to in the midst of trying to write a paper about ours, right?
00:09:11.840 --> 00:09:13.440
That paper may now have a home.
00:09:13.519 --> 00:09:14.879
There's an open access book coming.
00:09:14.960 --> 00:09:18.000
It's called Designing University Assessment for a World with AI.
00:09:18.080 --> 00:09:23.120
So we've got Corbin out of Deacon, 22 assessment uh scholars from six countries.
00:09:23.279 --> 00:09:26.399
Um, chapter seven is Corbyn, BM, Bold and Dawson.
00:09:26.480 --> 00:09:31.519
So Wicked Solutions, the challenge of advising on assessment design after AI, right?
00:09:31.919 --> 00:09:33.440
Or called the Owl paper.
00:09:33.919 --> 00:09:35.200
The owl paper for us.
00:09:35.360 --> 00:09:36.080
Yeah, 100%.
00:09:36.879 --> 00:09:42.559
So one interesting thing that Phil said to us, right, and I'm paraphrasing from the recording.
00:09:42.720 --> 00:09:48.960
Um, one of the academics who I'm guessing has contributed to this, uh, don't want to name names on it, but I think Kelly, Nicola, and Richmond came up.
00:09:49.039 --> 00:09:49.600
So I guess I do.
00:09:49.759 --> 00:09:53.360
The statement was I thought there were going to be lots and lots of different assessment task types.
00:09:53.519 --> 00:09:56.080
They found though there's only actually three or four, right?
00:09:56.159 --> 00:10:00.559
So an exam, an oral, a placement, or some technologized version of those.
00:10:00.879 --> 00:10:04.799
So the researchers are writing the framework side of the missing middle.
00:10:04.879 --> 00:10:07.840
So they're trying to bridge this gap between the policy and the practice.
00:10:08.080 --> 00:10:08.559
Yeah, yeah.
00:10:08.639 --> 00:10:12.080
And so the framework side is arriving and best people for it, right?
00:10:12.159 --> 00:10:16.720
So bared on what assessments for, Tim Fawns on collective learning integrity, etc.
00:10:17.039 --> 00:10:24.000
Read it when it lands, but I put it to you that our 114 is the other side, not what the field thinks, what institutions did.
00:10:24.080 --> 00:10:26.720
And I'm gonna say you want both on the desk, right?
00:10:27.120 --> 00:10:28.639
So enough foreplay.
00:10:28.799 --> 00:10:29.840
Let me get to the count for you.
00:10:29.919 --> 00:10:31.600
So 114 cases.
00:10:31.840 --> 00:10:34.000
So 114 cases, what do they say?
00:10:34.240 --> 00:10:37.120
Yeah, so we sorted them into five patterns, right?
00:10:37.279 --> 00:10:43.440
So interactive oral, 44 of the 114 use this, right?
00:10:43.600 --> 00:10:49.679
And 26 of the cases that we found online use it as the only assured approach, right?
00:10:50.000 --> 00:10:50.960
Critical appraisal.
00:10:51.039 --> 00:10:53.919
So give students an out AI output, make their judgment of it.
00:10:54.000 --> 00:10:57.919
We found 29 cases of it, 16 using it as the only approach.
00:10:58.000 --> 00:10:59.759
So not paired with something else, right?
00:10:59.840 --> 00:11:03.360
So it's just on the record, they're saying this is what we're doing here.
00:11:03.600 --> 00:11:08.399
Process portfolio, so drafts, revision, reflection, 27, but only two on their own.
00:11:08.559 --> 00:11:09.279
Control performance.
00:11:09.360 --> 00:11:16.240
So we're talking supervised exams in a time situation, 21, nine by themselves, and situated performance.
00:11:16.320 --> 00:11:20.159
So we've got placement studios, the real setting, 16.
00:11:20.399 --> 00:11:25.440
And just one of them is the sole approach with no sort of something to back it up.
00:11:25.840 --> 00:11:27.519
Listers, I pressure that's a lot of data.
00:11:27.679 --> 00:11:33.360
So um from YouTube, there's an infographic to kind of break it down because it is very interesting.
00:11:33.519 --> 00:11:35.840
And the fact that 16 is the smallest.
00:11:36.720 --> 00:11:37.759
Yeah, that's the smallest.
00:11:37.840 --> 00:11:40.480
And bit of a bit of a spoiler for the reveal there.
00:11:40.720 --> 00:11:46.080
I guess first one more pattern um that the cards don't carry because we decided not to make a card of it.
00:11:46.320 --> 00:11:50.000
22 of the cases are governance or detector decisions, right?
00:11:50.159 --> 00:11:53.440
And 12 of them did this as the entire response.
00:11:53.840 --> 00:11:54.799
What does that mean?
00:11:55.120 --> 00:11:57.840
So we're talking Turnitin's AI detector.
00:11:57.919 --> 00:11:58.159
Yeah.
00:11:58.320 --> 00:12:03.360
So Newcastle, they turned it on, Monish and UWA ruled it out as misconduct evidence.
00:12:03.440 --> 00:12:08.000
So UWA described it as unreliable, inequitable, unsuitable for high-stakes decisions.
00:12:08.240 --> 00:12:12.720
So opposite decisions, and I appreciate time locked, but same category.
00:12:12.879 --> 00:12:17.360
And so the second biggest single answer in the sector was a decision about a detector.
00:12:17.679 --> 00:12:20.000
All right, let's do the grand reveal then.
00:12:20.240 --> 00:12:20.559
Yep.
00:12:20.639 --> 00:12:20.960
Yep.
00:12:21.120 --> 00:12:29.840
So the thing that everyone in this field believes, I believe it, Phil said on here, you said it, is that judgment and context is the most robust thing we've got, right?
00:12:29.919 --> 00:12:34.000
So that in practice, watch someone do the job, placement, studio, clinic.
00:12:34.240 --> 00:12:37.679
That pattern is the smallest we could find online, right?
00:12:37.840 --> 00:12:43.279
Only one case uses it alone, and almost no one publishes it as a response to AI at all.
00:12:43.440 --> 00:12:45.679
It gets written up as placement practice.
00:12:46.000 --> 00:12:53.919
So the the cars online, I'm quoting my own document, which is weird, widely held, and from what we can find online undocumented.
00:12:54.159 --> 00:12:56.879
So the sector's answer is effectively a conversation.
00:12:57.120 --> 00:12:57.519
Yes.
00:12:57.759 --> 00:13:00.480
But, and this is the part that I really want to dig into as well.
00:13:00.559 --> 00:13:06.240
Let me put on my uh dusty old course coordinator hat for a second, because the the book I mentioned, right?
00:13:06.320 --> 00:13:10.320
There's a list that we need included as part of this conversation, right?
00:13:10.559 --> 00:13:14.480
Uh, a conversation, an interactive oral, is a lovely fix for a course of 50.
00:13:14.720 --> 00:13:18.000
Scale it to 1500, and you've got timetabling to consider.
00:13:18.080 --> 00:13:19.919
So who books the rooms for how many weeks?
00:13:20.080 --> 00:13:20.799
Payroll.
00:13:20.960 --> 00:13:24.960
Two markers in every 10 minute slot is two people being paid to assess one student.
00:13:25.120 --> 00:13:27.200
And that's a new problem for a finance system.
00:13:27.360 --> 00:13:30.799
Item security, and this is a coordinator's rule of thumb, not a finding.
00:13:30.960 --> 00:13:37.840
The moment that your exam runs over a two-week window, the questions are out, available online by day three, unless you built pools, right?
00:13:38.000 --> 00:13:41.519
Deferrals, adjustments, recordings, storage, 2 a.m.
00:13:41.679 --> 00:13:44.879
email from a student whose slot just clashed with the shift at work.
00:13:45.039 --> 00:13:46.399
That is a long list.
00:13:46.720 --> 00:13:51.840
And the list is a is a course coordinator's, or that PM we mentioned at the starts.
00:13:52.080 --> 00:13:54.559
That's the problem that they're often carrying alone.
00:13:54.879 --> 00:13:58.000
It is just a logistical nightmare.
00:13:58.320 --> 00:14:00.799
We have an answer, but the answer is just not good.
00:14:00.879 --> 00:14:04.000
It's saying, look, we have a way to fix this, but it's very, very challenging.
00:14:04.159 --> 00:14:06.320
So show me what this looks like when it's actually real.
00:14:06.480 --> 00:14:09.600
Not the idea, but when it's when it's Tuesday and we have to do this thing.
00:14:09.919 --> 00:14:10.480
Yep, yep.
00:14:10.559 --> 00:14:12.320
So Newcastle, great example.
00:14:12.559 --> 00:14:15.120
Teacher education, 524 students.
00:14:15.200 --> 00:14:19.600
You're talking a six-minute conversation with two sides of A4 notes allowed.
00:14:19.759 --> 00:14:23.519
And the six minutes sits inside a 15-minute appointment, right?
00:14:23.600 --> 00:14:25.120
So this is the logistics part of it.
00:14:25.200 --> 00:14:30.320
So there's an ID check, adjustments check, recording consent, the conversation, the marking, the wrap-up.
00:14:30.559 --> 00:14:35.759
The coordinator rings the disability service in advance instead of waiting for students to disclose.
00:14:35.919 --> 00:14:37.039
Super organized.
00:14:37.360 --> 00:14:40.480
So that's 524 15-minute appointments.
00:14:40.639 --> 00:14:42.879
I'm shocking at maths, but that's a large number.
00:14:43.120 --> 00:14:45.440
That's a large that is a headache waiting to happen, right?
00:14:45.519 --> 00:14:55.039
I mean, Queensland, another slightly smaller group, uh 230 postgrad teacher education students, 10 minutes each on their own lesson plan.
00:14:55.200 --> 00:14:57.519
Three weeks of room holds, right?
00:14:57.759 --> 00:15:02.799
Missed appointments turned out to expose a hole in the deferral policy that no one had noticed.
00:15:02.960 --> 00:15:04.240
Second order effects, right?
00:15:04.480 --> 00:15:10.159
And then they found something which I think is one of the best, most honest parts of this whole exercise.
00:15:10.320 --> 00:15:14.399
Some students' written and oral performance did not match.
00:15:14.960 --> 00:15:19.440
Which means that the written work was I'm assuming you're gonna stop me.
00:15:19.519 --> 00:15:21.039
Yeah, I am stopping you.
00:15:21.120 --> 00:15:23.279
Like I was I was watching that.
00:15:23.519 --> 00:15:24.639
Love your gentle one.
00:15:25.039 --> 00:15:26.000
These students.
00:15:26.960 --> 00:15:27.679
Exactly.
00:15:27.919 --> 00:15:28.720
No, 100%.
00:15:29.039 --> 00:15:30.080
But this is the thing as well.
00:15:30.159 --> 00:15:32.320
These are not like for like swaps, right?
00:15:32.399 --> 00:15:34.480
Because the two formats sample different things.
00:15:34.559 --> 00:15:37.120
It doesn't tell you which one is the student, right?
00:15:37.200 --> 00:15:39.279
The the dependent and independent variable.
00:15:39.440 --> 00:15:44.799
They're now planning, reportedly at Queensland, to remove some of the written work to make uh to make room for this.
00:15:44.879 --> 00:15:47.679
So this is planned, uh, but not evaluated.
00:15:48.000 --> 00:15:49.440
I remember this logistical nightmare.
00:15:49.519 --> 00:15:52.000
I think we saw, I saw Jason, a listener of the show, by the way.
00:15:52.080 --> 00:16:04.639
So hi Jason, if you're listening, being like logistically exhausted after undertaking the endeavor, just saying, Oh, it was great, it worked it out, but the scheduling, oh my gosh, that was so so challenging.
00:16:04.960 --> 00:16:06.000
Yeah, a hundred percent.
00:16:06.080 --> 00:16:07.919
And there's interesting things in there as well, right?
00:16:08.000 --> 00:16:13.919
I mean, he he said from an article I saw on this for Herzer that within two minutes they knew exactly where the student was.
00:16:14.159 --> 00:16:18.639
So the weak essay writers came alive, and that's the case for, right?
00:16:18.960 --> 00:16:21.200
But then there's a twist going back to Phil Dawson.
00:16:21.440 --> 00:16:24.879
Do you remember he spoke about the the was it the comedy, the peer review, right?
00:16:24.960 --> 00:16:25.840
His comedy show.
00:16:26.000 --> 00:16:32.639
He would ask students to come on stage, talk about their own work, something they are probably world leaders in for 10 minutes, right?
00:16:32.960 --> 00:16:35.519
Three quarters said no.
00:16:36.159 --> 00:16:42.559
And this is a really hot issue when it comes to things like equitable learning plans, students who are seeking alternate means of assessment, etc.
00:16:42.720 --> 00:16:42.960
etc.
00:16:43.120 --> 00:16:49.120
They're expecting a written word, and all of a sudden we're pushing them to this because it's allegedly uh what has that security element?
00:16:49.679 --> 00:16:50.320
Yeah, 100%.
00:16:50.639 --> 00:16:57.519
And so these are world leaders declining, and at the same time, the same academics ask their students to talk about what they know best, right?
00:16:58.080 --> 00:17:03.840
Anyway, so let's let's talk about the the conversation fairness thing because you raise a really good case.
00:17:03.919 --> 00:17:09.519
Um, so there are 44 cases we found, 26 where it is the whole judgment as it not paired with something.
00:17:09.680 --> 00:17:16.960
Another interesting thing, not one published interrator reliability, examiner effects, appeal outcomes, right?
00:17:17.119 --> 00:17:21.440
And that's concerning because it is the basis around which the entire thing is built.
00:17:21.599 --> 00:17:26.799
One Melbourne uh abstract reported examiner link variation that survived the redesign.
00:17:26.880 --> 00:17:29.119
And again, these signal cards will make these available.
00:17:29.279 --> 00:17:30.640
People can check these out for yourselves.
00:17:30.799 --> 00:17:32.480
Everything with links, all verified.
00:17:32.559 --> 00:17:39.920
Uh, there were seven cases describing a need for double marking or moderation, and adopting it doesn't establish its effect, right?
00:17:40.160 --> 00:17:46.559
Um again, I'm just gonna say a mismatch with the written work does not by itself establish misconduct.
00:17:46.640 --> 00:17:51.200
These things are two different measures of potentially, but not always the same thing.
00:17:51.519 --> 00:17:56.480
I just apply that to reality in terms of how I write and how I speak are wildly different.
00:17:56.799 --> 00:17:57.680
Let's talk about cheese.
00:17:57.759 --> 00:18:00.640
In particular, Phil's cheese of the Swiss variety, to be exact.
00:18:00.880 --> 00:18:09.839
So it's the idea that our approach to assessment currently in this AI-enabled landscape is just trying our best in various ways to ensure we can assure learning.
00:18:10.000 --> 00:18:13.039
He said on there that he won't stop leaving it as a metaphor.
00:18:13.359 --> 00:18:13.839
Yeah, yeah.
00:18:14.000 --> 00:18:17.119
So he said, pick a specific task, say what the holes are.
00:18:17.279 --> 00:18:19.440
So we did it for the oral just go fast here.
00:18:19.519 --> 00:18:22.960
Uh hole one, the wearables, glasses, his own example, right?
00:18:23.039 --> 00:18:26.079
You can't make them take them off and you can't spot the AI ones anywhere.
00:18:26.240 --> 00:18:28.400
Uh hole number two, this one was interesting.
00:18:28.640 --> 00:18:31.759
Open university in the UK is the remote proctoring arrangements.
00:18:31.839 --> 00:18:37.839
They said, quote, students may be asked to remove a religious head covering temporarily so the proctor can check for earphones.
00:18:38.319 --> 00:18:39.359
He did raise that.
00:18:39.599 --> 00:18:41.839
So it's head coverings and hearing aids.
00:18:42.079 --> 00:18:42.319
Right.
00:18:42.480 --> 00:18:44.720
And he raised it as a thing he wasn't comfortable with.
00:18:44.880 --> 00:18:47.759
And this I found we found rather as a published procedure.
00:18:47.839 --> 00:18:53.759
Uh whole three, his word cramming, oral is a short-term event, so uh favors memory and other things, performance.
00:18:53.920 --> 00:18:56.400
Uh whole four, examiner variability, just touch that.
00:18:56.480 --> 00:18:59.839
Hole five, misappointments in the deferral policy that wasn't built for them.
00:19:00.079 --> 00:19:00.640
Interesting.
00:19:00.960 --> 00:19:02.799
Six, feedback, right?
00:19:03.039 --> 00:19:06.400
Because a six-minute conversation doesn't leave time for it.
00:19:06.480 --> 00:19:07.920
And then just maths, right?
00:19:08.000 --> 00:19:13.359
As you said before, 524 times 10 is just an absolute nightmare.
00:19:13.519 --> 00:19:17.519
So you layer it on top, and that's the whole point of the cheese metaphor.
00:19:17.759 --> 00:19:18.000
Yeah.
00:19:18.160 --> 00:19:20.400
And so you layer filtered it live, right?
00:19:20.480 --> 00:19:26.720
So he was talking about peer the oral with something longitudinal, so a placement, um, or or with something controlled.
00:19:26.880 --> 00:19:29.759
And then here's what happens when you lean on the controlled one.
00:19:29.920 --> 00:19:34.559
So Brown University in the US, uh a take-home midterm averaged 96%.
00:19:35.119 --> 00:19:41.039
The in-person final, when this person got a bit suspect about this, he arranged one of these, averaged 49%.
00:19:41.759 --> 00:19:44.720
I can hear the media publications writing headlines now.
00:19:44.880 --> 00:19:47.519
Everyone is cheating on the take-home exam.
00:19:47.839 --> 00:19:48.400
Yeah, yeah.
00:19:48.480 --> 00:19:50.319
No, that's 100% what everyone said.
00:19:50.480 --> 00:19:57.440
And the analysis that followed couldn't separate unauthorized help from the fact that those are two different exams under two different conditions.
00:19:57.599 --> 00:20:02.880
Uh, just to be clear, this was from Inside Higher Ed, not the institution, so I'm flagging that, but an interesting case on this.
00:20:03.039 --> 00:20:06.720
Kind of an interesting comment on where the whole sector is in terms of sharing this stuff, right?
00:20:06.799 --> 00:20:11.759
We're all dealing with this thing together, um, or or in isolation, aside as our own.
00:20:11.920 --> 00:20:16.160
But anyway, a gap between conditions needs interpreting before it's evidence of anything.
00:20:16.240 --> 00:20:16.480
Yeah.
00:20:16.720 --> 00:20:18.160
So, what about this process layer?
00:20:18.319 --> 00:20:22.079
The drafts, the logs, the the trail of learning.
00:20:22.319 --> 00:20:22.640
Yep.
00:20:22.799 --> 00:20:26.480
Uh, closer to you, Sydney was an interesting one, a cautionary tale if you like.
00:20:26.640 --> 00:20:30.720
This was an international risk management course uh with an optional AI assistant.
00:20:30.799 --> 00:20:34.319
And if you used it, you had to submit your chat logs, right?
00:20:34.559 --> 00:20:36.240
Uptake just collapsed.
00:20:36.400 --> 00:20:42.160
So students had a workload and uh importantly, concern about surveillance and integrity rules.
00:20:42.319 --> 00:20:54.640
So they dropped the log requirement, the course dropped the log requirement and assessed a reflection instead, which therefore means that our our findings, our research, this test research holds no test of whether logs show learning.
00:20:54.799 --> 00:20:59.200
Again, it might be happening, but the question is open because it was never reached.
00:21:00.160 --> 00:21:01.279
That has kind of eroded.
00:21:01.440 --> 00:21:04.559
There's people not talking about the submit your chat logs and your prompts, etc.
00:21:04.720 --> 00:21:06.720
etc., or marking up your chat transcripts.
00:21:06.960 --> 00:21:08.880
But is there a version of that that worked?
00:21:09.200 --> 00:21:12.640
So instead of appending a transcript, they made the trail the task, right?
00:21:12.720 --> 00:21:18.880
So critique an AI essay against the rubric, improve it with track re changes, and then reflect.
00:21:19.039 --> 00:21:22.720
This was uh 30% of the grade with three interconnected artifacts.
00:21:22.880 --> 00:21:27.359
So I guess if you like, the record is inside the work, not bolted on.
00:21:28.240 --> 00:21:29.680
I like that as a process.
00:21:29.839 --> 00:21:31.039
So what's the answer?
00:21:31.519 --> 00:21:33.200
Where's the good cheese, Nick?
00:21:33.440 --> 00:21:36.559
The good cheese is, I guess, whatever you can afford, right?
00:21:36.880 --> 00:21:40.160
And then the problem though is that no one tells you what it costs.
00:21:40.240 --> 00:21:41.759
And I guess that's the next bit.
00:21:42.160 --> 00:21:44.240
Alright, so what does the registry do we have a cost?
00:21:44.559 --> 00:21:47.440
Did the registry reveal a uh dollar figure?
00:21:48.400 --> 00:21:53.039
So every one of those rows has a problems and cost field.
00:21:53.359 --> 00:21:56.799
47, just 47, used a time or cost word.
00:21:57.119 --> 00:21:58.880
Five attached a number.
00:21:59.359 --> 00:21:59.759
Five.
00:22:00.880 --> 00:22:01.200
Yeah.
00:22:01.279 --> 00:22:02.640
And these could be time, right?
00:22:02.720 --> 00:22:06.240
And so each each of them, every single one of them, is about setting an exam.
00:22:06.319 --> 00:22:12.079
So Queensland uh spoke to a week or more to write one unfamiliar context paper.
00:22:12.240 --> 00:22:16.640
Uh that was the only exam setting time setting figure I could find anywhere.
00:22:16.799 --> 00:22:18.319
Everything else we could find anywhere.
00:22:18.400 --> 00:22:21.599
So everything else publishes the design and never the bill.
00:22:21.839 --> 00:22:23.279
So who does the actual sums?
00:22:23.359 --> 00:22:25.440
Like who's who's revealing the numbers?
00:22:25.759 --> 00:22:29.920
Yeah, quite appropriately, this was a maths department, Pennsylvania, right?
00:22:30.000 --> 00:22:34.079
So a conventional midterm, they were talking about uh teacher time, right?
00:22:34.240 --> 00:22:39.759
They spoke to 15 person hours of grading plus four and a half hours of proctoring.
00:22:39.920 --> 00:22:45.039
They replaced it with a single oral, and the instructor was quite honest about it.
00:22:45.119 --> 00:22:53.759
And so against the the two written midterms, it replaced, it's roughly a wash at 40, 40 students, it comes out.
00:22:54.000 --> 00:22:59.119
But every single one of those hours was now his, the the instructor in this case, and not the TAs.
00:22:59.599 --> 00:23:02.000
So the hours just don't magically disappear.
00:23:02.319 --> 00:23:07.359
No, they change hands and then they scale with the size of the cohort, they grow, right?
00:23:07.519 --> 00:23:10.319
So 40 students, 15 minutes each, you can do that.
00:23:10.400 --> 00:23:17.519
But going back to Newcastle, 524, and you've got a booking system, 15 minute slots for the whole encounter.
00:23:17.680 --> 00:23:21.039
I mean, Queensland, 230, you're holding rooms for three weeks.
00:23:21.200 --> 00:23:22.960
And underneath all of that, there's a trade.
00:23:23.039 --> 00:23:33.359
So Queensland rates a large cohort in-class writing task as low effort because it returns a grade and feedback to the whole cohort as opposed to individual comments.
00:23:33.519 --> 00:23:41.359
In the seven places we found where anyone published a cost, there's a trade-off between securing at scale and individual feedback.
00:23:41.599 --> 00:23:46.480
And nobody, we could we couldn't find anyone who published a number on both.
00:23:46.799 --> 00:23:47.839
So what about orals?
00:23:48.000 --> 00:23:48.640
Are they magical?
00:23:48.720 --> 00:23:53.200
Are they cheaper or are they more expensive than or dearer than marking?
00:23:54.000 --> 00:23:59.200
I mean, again, from what we found, depends on the arrangement, which is probably an honest answer, right?
00:23:59.279 --> 00:24:05.839
So Sydney's business school, they spoke about, uh, they reported examining time was broadly comparable to written marking.
00:24:06.160 --> 00:24:08.960
Curtin published at 30 minutes a student.
00:24:09.119 --> 00:24:12.960
Queensland and Newcastle say that the scheduling was the real burden, right?
00:24:13.279 --> 00:24:20.240
So uh the the the caveat that we put at the end or the instruction, compare the whole appointment, every assessor and the admin, right?
00:24:20.319 --> 00:24:22.720
It's it's not just the length of the conversation.
00:24:23.119 --> 00:24:31.440
So me parading around earlier in the year, asking everyone to just convert everything to Vibas, because how easy is a conversation does not take into account the reality.
00:24:31.839 --> 00:24:47.359
I think, I think, I think the nuance, if I could be generous to you there, is the the frameworks say redesign, the practice accounts say what they built, but very little has been put across to the cost, and nothing, and I I know this is your live reality as well.
00:24:47.519 --> 00:24:50.480
Nothing says what comes off someone's plate to pay for it.
00:24:50.880 --> 00:24:53.920
The busyness, you know, is just getting crazier and crazier.
00:24:54.319 --> 00:24:55.759
Just pushing the work somewhere.
00:24:56.079 --> 00:25:00.240
Friend of the show, Professor Molly Dollinger, talked about doubt recently.
00:25:00.480 --> 00:25:10.160
That the rubrics and the moderation and the double marking and everything else that we put on top of assessment and securing assessment are there to hide the fact that perhaps we're just not sure.
00:25:10.400 --> 00:25:14.960
So we strap on all these armor to protect us from the world of the Swiss cheese realities.
00:25:15.279 --> 00:25:15.920
Yeah, 100%.
00:25:16.160 --> 00:25:17.359
I'm on team doubt, right?
00:25:17.440 --> 00:25:18.240
I remember you saying that.
00:25:18.319 --> 00:25:21.839
I mean, so am I, and this is why all these caveats have been layered across this.
00:25:22.079 --> 00:25:23.920
Are you I want to just put one more in there?
00:25:24.000 --> 00:25:28.720
So every paragraph or every every one of these cards ends with what the evidence doesn't establish.
00:25:28.880 --> 00:25:31.920
So this is an educator account, not a comparative evaluations.
00:25:32.160 --> 00:25:34.559
Track changes do not establish who reasons through them.
00:25:34.640 --> 00:25:36.559
They there's there's some legalese in there.
00:25:36.640 --> 00:25:38.400
They read like a lawyer wrote them.
00:25:38.880 --> 00:25:40.160
They do a little bit.
00:25:41.039 --> 00:25:42.240
Just ass covering.
00:25:42.559 --> 00:25:43.920
Ars covering indeed.
00:25:44.079 --> 00:25:44.480
Written.
00:25:44.640 --> 00:25:46.000
But that's deliberate, right?
00:25:46.079 --> 00:25:48.720
Because I I want to put it to you that the alternative is the last three years.
00:25:48.880 --> 00:25:52.720
A confident tool, confident policy, and no one saying out loud what we didn't know, right?
00:25:52.799 --> 00:25:53.759
Or that we didn't know.
00:25:54.000 --> 00:25:58.160
I'm assuming AI detectors still came up within some of your findings as well.
00:25:58.400 --> 00:25:59.119
Yeah, absolutely.
00:25:59.279 --> 00:26:02.000
ANU had a good one and applied policy capstone.
00:26:02.079 --> 00:26:08.240
So every student required to write a draft ministerial brief for Chat GPT, then write their own reflection on it.
00:26:08.400 --> 00:26:12.720
So, ground truth, every one of them was AI generated by design.
00:26:12.960 --> 00:26:21.279
Turn it in, and the delicious little AI indicator scored those briefs anywhere from under 10% AI generated to 100.
00:26:21.599 --> 00:26:25.440
Same tool, same known answer, it landed everywhere on the dial.
00:26:25.599 --> 00:26:25.759
Right.
00:26:25.839 --> 00:26:29.519
And so, to be fair as well, that's from seven students who can send it to the study.
00:26:29.599 --> 00:26:34.880
Uh, but this is the only case we could find with a known answer to test the detector against.
00:26:35.039 --> 00:26:37.839
And the detector, unsurprisingly, failed.
00:26:38.240 --> 00:26:41.200
Detection was the the machinery that concealed the doubt.
00:26:41.440 --> 00:26:44.000
Appraisal is where you make students kind of practice against it.
00:26:44.240 --> 00:26:50.240
And the line, second time, a mismatch in this does not by itself establish misconduct.
00:26:50.559 --> 00:26:55.200
So, Nick, I started off the year asking people to put down their pitchwalks, stop being so tribal.
00:26:55.359 --> 00:27:02.240
And when Phil said the most powerful change approach that works is taking someone out for a coffee and having real conversations about this.
00:27:02.559 --> 00:27:03.839
God, I was glad.
00:27:04.160 --> 00:27:04.640
Yeah, yeah.
00:27:04.720 --> 00:27:05.519
His own research.
00:27:05.599 --> 00:27:13.920
Hey, I mean, the people who change what what an academic does, uh, the colleague next door, local learning and teaching lead, helpful search spacer, hopefully.
00:27:14.000 --> 00:27:17.200
So his budget recommendation was around uh coffee vouchers.
00:27:17.359 --> 00:27:19.359
And that's the kind of thing that we're driving to, right?
00:27:19.519 --> 00:27:26.240
So each one of these ends with a sentence we could change blank to gather evidence of blank before proceeding.
00:27:26.319 --> 00:27:27.920
We need to resolve blank.
00:27:28.000 --> 00:27:28.400
Yeah.
00:27:28.640 --> 00:27:34.960
Nick, are you saying your robust benchmarking and desktop analysis is as good as a delicious latte amongst friends?
00:27:35.200 --> 00:27:37.039
Yeah, that's uh that sounds like a coffee to me.
00:27:37.519 --> 00:27:39.839
I mean, not for a committee, it's for two people and a napkin.
00:27:40.079 --> 00:27:45.920
And I think uh the the closing sort of beat that we have on these things and coming soon, so keep as a promissory note.
00:27:46.160 --> 00:27:52.880
We wrote something at the bottom, which was before the first run, decide what evidence would help us keep, adapt, or stop this approach, right?
00:27:53.039 --> 00:27:57.519
So um the registry with the IDS research, we've got an outcome column.
00:27:57.599 --> 00:28:05.839
So 106 of the 114 are continuing, uh, five not stated, two modified, and one just abandoned.
00:28:05.920 --> 00:28:06.240
Yeah.
00:28:06.559 --> 00:28:08.720
One only one was abandoned.
00:28:09.039 --> 00:28:12.880
Yeah, not say so uh the again recorded on the record, right?
00:28:12.960 --> 00:28:16.240
So the who knows what's happening out there, the great unwatched space of it.
00:28:16.319 --> 00:28:28.079
But Kingston let module leaders set AI permissions locally, uh, got, in their words, inconsistency, confusion, non-compliance, and replace it with one university-wide approach.
00:28:28.319 --> 00:28:30.799
So one published open 114.
00:28:30.960 --> 00:28:41.359
Like the the sector spent three years redesigning assessment, and almost nobody said whether it worked, because almost nobody decided in advance or at least published what didn't work would look like.
00:28:45.440 --> 00:28:48.960
I'm assuming you're gonna run this session again, but your program manager friends comes in again.
00:28:49.119 --> 00:28:50.799
She asked you, Do you have an answer?
00:28:50.880 --> 00:28:52.160
What does she do now?
00:28:52.480 --> 00:28:55.440
Yeah, so one unit, right?
00:28:55.599 --> 00:28:57.359
Not the whole program, just one part.
00:28:57.680 --> 00:29:02.559
And the the card that she held the longest because it had worked so well for so long.
00:29:02.720 --> 00:29:05.680
Uh, the whole grade was about unobserved online work.
00:29:05.839 --> 00:29:07.519
Then one of the five signal cards for it.
00:29:07.599 --> 00:29:11.359
So probably the interactive oral, because that's where the most published practice is.
00:29:11.519 --> 00:29:14.720
And maybe she reads Newcastle's 15-minute appointment.
00:29:14.960 --> 00:29:20.079
So she knows what she's up to signing up for before she promises anyone anything.
00:29:20.319 --> 00:29:23.920
Then one colleague, uh, the coffee and and the blank.
00:29:24.160 --> 00:29:27.359
So we could change this to gather evidence of that.
00:29:27.440 --> 00:29:33.519
And before we go further, we need to resolve the deferral policy, the market time, or whatever it is.
00:29:33.759 --> 00:29:38.720
And and she writes down before the first run what would make her stop, right?
00:29:38.880 --> 00:29:46.960
So the the practice behind these principles, so which would put her ahead of pretty much all of what we found when we went looking.
00:29:47.119 --> 00:29:50.160
So that's what goes between the policy and her Monday.
00:29:50.319 --> 00:29:54.480
It took us, it took us months to get there from January, that session.
00:29:54.559 --> 00:29:56.960
And it's just the sketch on the back of a napkin.
00:29:57.519 --> 00:30:02.160
But Nick, I would I guess I'll say thank you for kind of bringing this data together because it's severely lacking.
00:30:02.240 --> 00:30:06.079
And I think for me, there's a call to action there for people to share these practices a lot more.
00:30:06.240 --> 00:30:08.480
What's missing in this is the data, is the understanding.
00:30:08.640 --> 00:30:11.119
This is a challenge that we solve together as a sector.
00:30:11.200 --> 00:30:14.160
And so the more transparency we can be and the more open we can be.
00:30:14.319 --> 00:30:17.920
I hope that 114 number goes a lot higher for the next version.
00:30:18.240 --> 00:30:18.640
100%.
00:30:19.039 --> 00:30:19.279
100%.
00:30:19.759 --> 00:30:21.599
Ladies and gentlemen, thank you for listening to Agile Intelligence.
00:30:21.680 --> 00:30:26.880
Some of those signal cards are available up on the Agile Intelligence website as draft for you guys to give feedback on.
00:30:27.039 --> 00:30:28.720
We would love to hear it.
00:30:28.880 --> 00:30:33.200
And every number that Nick said is on them with its source with a link.
00:30:33.440 --> 00:30:39.440
If you're anywhere near Saigon on the 24th and 25th of September, higher education horizons at RMT Saigon is free.
00:30:39.599 --> 00:30:44.079
Tim Fawns is keynoting, and Nick will be there arguing with people in person.
00:30:44.720 --> 00:30:46.000
Bring a challenge him on his car.
00:30:46.319 --> 00:30:46.960
Bring a friend.
00:30:47.119 --> 00:30:47.920
Yeah, yeah.
00:30:48.400 --> 00:30:54.160
Until then, stay curious, Down Intelligence, and whether it's owls, cheese, or coffee, all three now come.
00:30:56.079 --> 00:30:56.960
Apparently.
00:00:00.160 --> 00:00:02.799
So I want to start with a person in a room.
00:00:02.959 --> 00:00:06.480
So January this year on campus here at RMIT Vietnam.
00:00:06.959 --> 00:00:15.359
So 90 minutes with the people who run our degrees, we're talking program managers, course coordinators, doing a thing we call the AI reality check.
00:00:15.519 --> 00:00:28.000
The idea is that we get these people hands-on with the edge of the field, so wrapped in the same kind of like here's what shifted this month framing that we do on this show, and then a discussion across programs that never really talk to each other.
00:00:28.320 --> 00:00:29.839
I do remember you building this out.
00:00:29.920 --> 00:00:33.280
It's a brilliant way to kind of open eyes and get to that aha moment quickly.
00:00:33.600 --> 00:00:34.000
100%.
00:00:34.479 --> 00:00:38.560
And the last station in the session, I opened with the demo, right?
00:00:38.640 --> 00:00:48.000
So I got an Agentic browser, I think it was uh Atlas, rest in peace, OpenAI is Atlas from Memory, gave it a Canvas login, pointed it at a live assignment brief, and let it go.
00:00:48.159 --> 00:00:51.280
The thing read the brief, worked out the task, started doing it.
00:00:51.439 --> 00:00:54.159
All the time I'm at the front, look, no hands.
00:00:54.880 --> 00:00:56.640
And this was like nine months ago, yeah.
00:00:56.799 --> 00:01:03.520
So in an AI standards kind of life cycle, we've had nine different open AI releases since then and countless other models.
00:01:03.600 --> 00:01:06.159
So technology then, plus we're around now.
00:01:06.480 --> 00:01:08.640
I was a fresh face youngster back when I was running that.
00:01:09.200 --> 00:01:10.719
Like, I mean, it has aged me.
00:01:10.879 --> 00:01:15.359
But then what we did, because it wasn't just this sort of, oh, look at this flashy thing, not just hype.
00:01:15.519 --> 00:01:17.040
We handed out cards, okay?
00:01:17.200 --> 00:01:28.400
So every assessment type that you can think of, and the idea was all right, oh academics, or course coordinators, program type manager types, sort these into two piles as you see them now based on what you've just seen.
00:01:28.480 --> 00:01:30.079
And there were a few other demos as well.
00:01:30.239 --> 00:01:33.760
But are these AI vulnerable or are they AI resistant, if you like?
00:01:34.000 --> 00:01:36.239
We gave them 10, 11 minutes, sort them.
00:01:36.480 --> 00:01:40.079
And the the thing that was interesting, it was never clean, it was never unanimous.
00:01:40.159 --> 00:01:43.280
People have cards and there was disagreement, which was great.
00:01:43.519 --> 00:01:46.400
But the really interesting bit came at the end.
00:01:46.640 --> 00:01:47.040
Yeah.
00:01:47.359 --> 00:01:47.920
Go on.
00:01:48.319 --> 00:01:50.719
So there's a there's a program manager in the room, right?
00:01:50.799 --> 00:01:56.480
And she runs a program with hundreds of students across multiple locations, dozens of teaching staff, and others.
00:01:56.640 --> 00:02:01.439
She sat through 40 minutes of me and my team showing her what the tech does now.
00:02:01.599 --> 00:02:06.640
She's sorted her own assessments into piles, and most of them are in the wrong pile.
00:02:07.120 --> 00:02:12.719
And she sort of looks up and says, not angry, not panicking, just straight, like boom.
00:02:13.919 --> 00:02:14.560
What do I do now?
00:02:15.120 --> 00:02:16.960
It's a strange aha moment.
00:02:17.360 --> 00:02:18.159
For real, eh.
00:02:18.319 --> 00:02:20.560
And look, I didn't have an answer full disclosure.
00:02:20.639 --> 00:02:25.840
Um, I'd show in her the problem in 4K, and I did not have the next slide.
00:02:25.919 --> 00:02:32.719
Uh, the institution had a policy, she had a program, and there was nothing between the two uh that told her what to do on the Monday.
00:02:33.039 --> 00:02:39.039
Alas, the constant gap between policy and practice, the guided note, the guidance notes and the reality.
00:02:39.360 --> 00:02:39.680
Yeah.
00:02:39.919 --> 00:02:42.400
So that's what we're gonna be talking about today.
00:02:51.919 --> 00:02:56.639
Ladies and gentlemen, welcome to Adjunct Intelligence, the podcast that looks at AI and impact on higher education.
00:02:56.719 --> 00:03:01.280
I'm joined, as always, by Nick McIntosh, learning futurist, and myself, head of AI education.
00:03:01.439 --> 00:03:04.159
So, Nick, I would say this is our fourth goal at assessment.
00:03:04.240 --> 00:03:09.919
We've had Phil Dawson, we've spoken about Jason Lodge multiple times, we've done the two-lane thing.
00:03:10.159 --> 00:03:13.840
Why should we sit through a fourth episode of AI and assessment?
00:03:14.639 --> 00:03:17.199
Because of something that you said, actually.
00:03:17.360 --> 00:03:23.360
So five weeks ago, you opened the Phil Dawson episode by reading his line about the owl, right?
00:03:23.439 --> 00:03:30.719
So that that just redesign your assessment sort of thing is step one, draw two circles, step two, draw the rest of the owl.
00:03:30.879 --> 00:03:32.960
And then you uh you rift your own bit, right?
00:03:33.280 --> 00:03:34.479
Good luck with the feathers.
00:03:34.639 --> 00:03:37.840
Feathers are hard to draw for anyone who's never drawn an owl before.
00:03:38.159 --> 00:03:39.919
Feathers are bloody tough, right?
00:03:40.000 --> 00:03:54.400
And then so later in that conversation, Phil said something that really piqued my fancy, my interest, that the sector has the high-level guidance and it's got a handful, bare handful of fully realized examples and almost nothing in between, right?
00:03:54.639 --> 00:04:00.080
So he he spoke to a missing middle between frameworks and fully realized pictures.
00:04:00.319 --> 00:04:07.360
So it's our educator from the opening, innocently in search of a solution to what we are called a and others are called a wicked problem.
00:04:07.680 --> 00:04:08.479
Yep, exactly.
00:04:08.639 --> 00:04:12.159
So the gap between policy and practice, I guess, seen from both ends, right?
00:04:12.400 --> 00:04:15.759
So for lack of a better way to say it, we went looking for the feathers, right?
00:04:15.840 --> 00:04:21.920
So the things that the unis actually did, they wrote down and published somewhere that she or anyone could read the whole thing.
00:04:22.079 --> 00:04:25.680
So we found Ballpark 110, 114.
00:04:26.319 --> 00:04:26.879
Whoa.
00:04:27.439 --> 00:04:29.040
Yeah, and whoa, indeed.
00:04:29.279 --> 00:04:32.639
And so most of them, it turns out, are a conversation.
00:04:32.800 --> 00:04:39.519
Um, but before we dig into that and I tell you what we found, I need to tell you what this is and also what this is not.
00:04:39.839 --> 00:04:40.160
Yeah?
00:04:40.560 --> 00:04:42.800
A lengthy list of caveats, Nicholas.
00:04:42.959 --> 00:04:46.000
So uh what is the reality check thing?
00:04:46.160 --> 00:04:52.079
I think we need to dive into that reality check you opened up at the intro because it is actually a really powerful vehicle.
00:04:52.480 --> 00:04:53.199
Thank you very much.
00:04:53.360 --> 00:04:56.240
I I think you've got something similar you run your end as well.
00:04:56.399 --> 00:05:07.360
But for the record, uh, this is a hands-on exposure, uh, we frame it to the edge of the AI field for the people who make the decisions, right, who turn policy into practice, those who run programs, run courses.
00:05:07.600 --> 00:05:09.199
Ballpark 90 minutes.
00:05:09.279 --> 00:05:11.759
Uh, we do it the way that we do this show, right?
00:05:11.839 --> 00:05:13.680
So here's what's moved this quarter.
00:05:13.839 --> 00:05:18.639
We've got things that have changed with the regulators, with the tools, uh, research, the cases.
00:05:18.879 --> 00:05:22.000
And this pre-reading means that no one works in cold, right?
00:05:22.079 --> 00:05:24.000
Then they get their hands on the emerging tech.
00:05:24.240 --> 00:05:26.560
So these are not demonstrations or slides.
00:05:26.639 --> 00:05:35.199
This is the actual thing on a machine on a computer in front of them, doing something to learning, to teaching, and to assessments that they wrote now.
00:05:35.360 --> 00:05:36.879
What is possible now?
00:05:37.439 --> 00:05:38.959
As you mentioned, we've got something similar.
00:05:39.040 --> 00:05:41.040
We call a make and take session, plot twist.
00:05:41.279 --> 00:05:44.560
It's actually not about making and taking guys, it's just about opening eyes.
00:05:44.720 --> 00:05:47.839
I think Melbourne Uni has something also called their cheat lab, which I love.
00:05:47.920 --> 00:05:51.439
And from memory, you bring in like an assessment and see if AI can crack it.
00:05:51.759 --> 00:05:52.560
Spoiler alert.
00:05:52.879 --> 00:05:54.160
Most often it does.
00:05:54.560 --> 00:05:54.879
Nice.
00:05:54.959 --> 00:05:56.240
Yeah, no, that's great.
00:05:56.399 --> 00:06:03.920
Um, I think the the other part, and I'm sure you've got that in there as well, the thing that we found that matters most is we break down those organizational silos, right?
00:06:04.000 --> 00:06:05.519
We get people talking to each other.
00:06:05.680 --> 00:06:08.560
So from across the different subjects and schools.
00:06:08.639 --> 00:06:18.399
So you've got nursing, sitting down with business, sitting with uh software engineers, talking about the same experiences, the same technologies, and in this case of assessments, the same cards.
00:06:18.560 --> 00:06:28.240
And they find out that the person across the table that they might not otherwise get time with has reached the opposite conclusions for reasons that are also right.
00:06:29.199 --> 00:06:31.279
I don't care what people say about you, Nick.
00:06:31.360 --> 00:06:32.000
You are right.
00:06:32.079 --> 00:06:35.839
And your love of bringing people together, breaking down silos, it's just brilliant.
00:06:36.079 --> 00:06:36.639
Thanks, man.
00:06:36.959 --> 00:06:37.199
Thanks.
00:06:37.279 --> 00:06:39.920
I feel seen breaking down silence too.
00:06:40.160 --> 00:06:42.399
No, but that's that's the silo bit.
00:06:42.639 --> 00:06:46.000
And I I think it's where all the the real intelligence in the uni lives, right?
00:06:46.079 --> 00:06:49.519
So Phil uh Dawson referenced just now, has a paper on this.
00:06:49.600 --> 00:06:56.800
So the talk is cheap, which is about the difference between changing what you say about assessment and changing something a bit more meaningful, changing its structure.
00:06:57.040 --> 00:07:05.199
And on this very show, he said that the sector has confused AI use declarations and traffic lights guide and traffic light guidance with assurance.
00:07:05.439 --> 00:07:06.480
So that's the gap.
00:07:06.720 --> 00:07:12.480
The policy is a declaration, the assessment is a structure, and she's kind of standing in between them.
00:07:12.959 --> 00:07:17.439
So the reality check, the reality check is what goes in that kind of gap in theory.
00:07:17.600 --> 00:07:17.759
Yep.
00:07:17.920 --> 00:07:19.120
All right, let's get to the study.
00:07:19.279 --> 00:07:21.519
Let's get to the research, 114 data points.
00:07:21.600 --> 00:07:22.399
I I want to know.
00:07:22.800 --> 00:07:24.800
I imagine there's some form of an answer in here.
00:07:25.279 --> 00:07:25.839
There is one.
00:07:25.920 --> 00:07:26.959
Um I I do apologize.
00:07:27.040 --> 00:07:28.560
We're taking our time getting here, but it's worth it.
00:07:28.639 --> 00:07:29.680
It's the payoff is coming.
00:07:29.920 --> 00:07:46.480
So over about a month, uh, within our AI community practice at RMIT, we built a registry of about 100, as I say, 114 documented cases online, available for all, where an institution changed an assessment because of AI and published enough that a practitioner could read the whole thing.
00:07:46.639 --> 00:07:51.439
So public sources are only, which I'd say is a strength and a limitation in a sentence.
00:07:51.519 --> 00:07:51.920
Yeah.
00:07:52.160 --> 00:07:52.399
Correct.
00:07:52.560 --> 00:07:53.360
So what's the limit?
00:07:53.519 --> 00:07:58.160
Like get your get all your caveats out of the way and then I I know I'm I'm being super, super coy.
00:07:58.319 --> 00:07:58.480
Yeah.
00:07:58.560 --> 00:08:02.160
So the limit is that everything behind closed doors is invisible, right?
00:08:02.240 --> 00:08:07.439
So there'll be amazing work happening on course teams right now that no one has written up, and we can't see any of it.
00:08:07.600 --> 00:08:08.879
We can only see what was published.
00:08:08.959 --> 00:08:16.480
So when I I make grand statements like the sector does X, what I mean is the published record of what we could find at time of research says X.
00:08:16.560 --> 00:08:20.240
So this is not research per se, and I'm not going to pretend it is.
00:08:20.399 --> 00:08:20.959
Yeah.
00:08:21.360 --> 00:08:23.680
It's a really robust desktop scan.
00:08:23.920 --> 00:08:25.279
That but you've done the work.
00:08:25.439 --> 00:08:25.680
Yes.
00:08:25.920 --> 00:08:26.079
Yeah.
00:08:26.160 --> 00:08:29.920
So it's called a sector scan of current public uh practice, right?
00:08:30.000 --> 00:08:34.320
So I don't know if you're familiar with the one pages from Jamie Clark, pedagogy on a page.
00:08:34.639 --> 00:08:35.039
Yeah, yeah.
00:08:35.120 --> 00:08:37.120
So what is it if people aren't familiar?
00:08:37.519 --> 00:08:41.600
A primer on constructivism, on feedback, and it's a one pager, and you're done.
00:08:41.759 --> 00:08:47.519
So again, the whole idea of this exercise is to try and offer something practical, something useful for that PM I mentioned at the top.
00:08:47.679 --> 00:08:52.960
She has a day job, to-do list, measured in in feet, not items, competing deadlines, da da da.
00:08:53.679 --> 00:08:56.639
Think of this as a practical digest that we give over to her.
00:08:56.720 --> 00:09:02.080
And people like her is so empowered with that they can make their own decisions about what to do in their context.
00:09:02.399 --> 00:09:05.360
So let's let's start with what Phil said and a bit of news first.
00:09:05.519 --> 00:09:11.679
So on this show, he said we we'd caught him in the midst of writing to in the midst of trying to write a paper about ours, right?
00:09:11.840 --> 00:09:13.440
That paper may now have a home.
00:09:13.519 --> 00:09:14.879
There's an open access book coming.
00:09:14.960 --> 00:09:18.000
It's called Designing University Assessment for a World with AI.
00:09:18.080 --> 00:09:23.120
So we've got Corbin out of Deacon, 22 assessment uh scholars from six countries.
00:09:23.279 --> 00:09:26.399
Um, chapter seven is Corbyn, BM, Bold and Dawson.
00:09:26.480 --> 00:09:31.519
So Wicked Solutions, the challenge of advising on assessment design after AI, right?
00:09:31.919 --> 00:09:33.440
Or called the Owl paper.
00:09:33.919 --> 00:09:35.200
The owl paper for us.
00:09:35.360 --> 00:09:36.080
Yeah, 100%.
00:09:36.879 --> 00:09:42.559
So one interesting thing that Phil said to us, right, and I'm paraphrasing from the recording.
00:09:42.720 --> 00:09:48.960
Um, one of the academics who I'm guessing has contributed to this, uh, don't want to name names on it, but I think Kelly, Nicola, and Richmond came up.
00:09:49.039 --> 00:09:49.600
So I guess I do.
00:09:49.759 --> 00:09:53.360
The statement was I thought there were going to be lots and lots of different assessment task types.
00:09:53.519 --> 00:09:56.080
They found though there's only actually three or four, right?
00:09:56.159 --> 00:10:00.559
So an exam, an oral, a placement, or some technologized version of those.
00:10:00.879 --> 00:10:04.799
So the researchers are writing the framework side of the missing middle.
00:10:04.879 --> 00:10:07.840
So they're trying to bridge this gap between the policy and the practice.
00:10:08.080 --> 00:10:08.559
Yeah, yeah.
00:10:08.639 --> 00:10:12.080
And so the framework side is arriving and best people for it, right?
00:10:12.159 --> 00:10:16.720
So bared on what assessments for, Tim Fawns on collective learning integrity, etc.
00:10:17.039 --> 00:10:24.000
Read it when it lands, but I put it to you that our 114 is the other side, not what the field thinks, what institutions did.
00:10:24.080 --> 00:10:26.720
And I'm gonna say you want both on the desk, right?
00:10:27.120 --> 00:10:28.639
So enough foreplay.
00:10:28.799 --> 00:10:29.840
Let me get to the count for you.
00:10:29.919 --> 00:10:31.600
So 114 cases.
00:10:31.840 --> 00:10:34.000
So 114 cases, what do they say?
00:10:34.240 --> 00:10:37.120
Yeah, so we sorted them into five patterns, right?
00:10:37.279 --> 00:10:43.440
So interactive oral, 44 of the 114 use this, right?
00:10:43.600 --> 00:10:49.679
And 26 of the cases that we found online use it as the only assured approach, right?
00:10:50.000 --> 00:10:50.960
Critical appraisal.
00:10:51.039 --> 00:10:53.919
So give students an out AI output, make their judgment of it.
00:10:54.000 --> 00:10:57.919
We found 29 cases of it, 16 using it as the only approach.
00:10:58.000 --> 00:10:59.759
So not paired with something else, right?
00:10:59.840 --> 00:11:03.360
So it's just on the record, they're saying this is what we're doing here.
00:11:03.600 --> 00:11:08.399
Process portfolio, so drafts, revision, reflection, 27, but only two on their own.
00:11:08.559 --> 00:11:09.279
Control performance.
00:11:09.360 --> 00:11:16.240
So we're talking supervised exams in a time situation, 21, nine by themselves, and situated performance.
00:11:16.320 --> 00:11:20.159
So we've got placement studios, the real setting, 16.
00:11:20.399 --> 00:11:25.440
And just one of them is the sole approach with no sort of something to back it up.
00:11:25.840 --> 00:11:27.519
Listers, I pressure that's a lot of data.
00:11:27.679 --> 00:11:33.360
So um from YouTube, there's an infographic to kind of break it down because it is very interesting.
00:11:33.519 --> 00:11:35.840
And the fact that 16 is the smallest.
00:11:36.720 --> 00:11:37.759
Yeah, that's the smallest.
00:11:37.840 --> 00:11:40.480
And bit of a bit of a spoiler for the reveal there.
00:11:40.720 --> 00:11:46.080
I guess first one more pattern um that the cards don't carry because we decided not to make a card of it.
00:11:46.320 --> 00:11:50.000
22 of the cases are governance or detector decisions, right?
00:11:50.159 --> 00:11:53.440
And 12 of them did this as the entire response.
00:11:53.840 --> 00:11:54.799
What does that mean?
00:11:55.120 --> 00:11:57.840
So we're talking Turnitin's AI detector.
00:11:57.919 --> 00:11:58.159
Yeah.
00:11:58.320 --> 00:12:03.360
So Newcastle, they turned it on, Monish and UWA ruled it out as misconduct evidence.
00:12:03.440 --> 00:12:08.000
So UWA described it as unreliable, inequitable, unsuitable for high-stakes decisions.
00:12:08.240 --> 00:12:12.720
So opposite decisions, and I appreciate time locked, but same category.
00:12:12.879 --> 00:12:17.360
And so the second biggest single answer in the sector was a decision about a detector.
00:12:17.679 --> 00:12:20.000
All right, let's do the grand reveal then.
00:12:20.240 --> 00:12:20.559
Yep.
00:12:20.639 --> 00:12:20.960
Yep.
00:12:21.120 --> 00:12:29.840
So the thing that everyone in this field believes, I believe it, Phil said on here, you said it, is that judgment and context is the most robust thing we've got, right?
00:12:29.919 --> 00:12:34.000
So that in practice, watch someone do the job, placement, studio, clinic.
00:12:34.240 --> 00:12:37.679
That pattern is the smallest we could find online, right?
00:12:37.840 --> 00:12:43.279
Only one case uses it alone, and almost no one publishes it as a response to AI at all.
00:12:43.440 --> 00:12:45.679
It gets written up as placement practice.
00:12:46.000 --> 00:12:53.919
So the the cars online, I'm quoting my own document, which is weird, widely held, and from what we can find online undocumented.
00:12:54.159 --> 00:12:56.879
So the sector's answer is effectively a conversation.
00:12:57.120 --> 00:12:57.519
Yes.
00:12:57.759 --> 00:13:00.480
But, and this is the part that I really want to dig into as well.
00:13:00.559 --> 00:13:06.240
Let me put on my uh dusty old course coordinator hat for a second, because the the book I mentioned, right?
00:13:06.320 --> 00:13:10.320
There's a list that we need included as part of this conversation, right?
00:13:10.559 --> 00:13:14.480
Uh, a conversation, an interactive oral, is a lovely fix for a course of 50.
00:13:14.720 --> 00:13:18.000
Scale it to 1500, and you've got timetabling to consider.
00:13:18.080 --> 00:13:19.919
So who books the rooms for how many weeks?
00:13:20.080 --> 00:13:20.799
Payroll.
00:13:20.960 --> 00:13:24.960
Two markers in every 10 minute slot is two people being paid to assess one student.
00:13:25.120 --> 00:13:27.200
And that's a new problem for a finance system.
00:13:27.360 --> 00:13:30.799
Item security, and this is a coordinator's rule of thumb, not a finding.
00:13:30.960 --> 00:13:37.840
The moment that your exam runs over a two-week window, the questions are out, available online by day three, unless you built pools, right?
00:13:38.000 --> 00:13:41.519
Deferrals, adjustments, recordings, storage, 2 a.m.
00:13:41.679 --> 00:13:44.879
email from a student whose slot just clashed with the shift at work.
00:13:45.039 --> 00:13:46.399
That is a long list.
00:13:46.720 --> 00:13:51.840
And the list is a is a course coordinator's, or that PM we mentioned at the starts.
00:13:52.080 --> 00:13:54.559
That's the problem that they're often carrying alone.
00:13:54.879 --> 00:13:58.000
It is just a logistical nightmare.
00:13:58.320 --> 00:14:00.799
We have an answer, but the answer is just not good.
00:14:00.879 --> 00:14:04.000
It's saying, look, we have a way to fix this, but it's very, very challenging.
00:14:04.159 --> 00:14:06.320
So show me what this looks like when it's actually real.
00:14:06.480 --> 00:14:09.600
Not the idea, but when it's when it's Tuesday and we have to do this thing.
00:14:09.919 --> 00:14:10.480
Yep, yep.
00:14:10.559 --> 00:14:12.320
So Newcastle, great example.
00:14:12.559 --> 00:14:15.120
Teacher education, 524 students.
00:14:15.200 --> 00:14:19.600
You're talking a six-minute conversation with two sides of A4 notes allowed.
00:14:19.759 --> 00:14:23.519
And the six minutes sits inside a 15-minute appointment, right?
00:14:23.600 --> 00:14:25.120
So this is the logistics part of it.
00:14:25.200 --> 00:14:30.320
So there's an ID check, adjustments check, recording consent, the conversation, the marking, the wrap-up.
00:14:30.559 --> 00:14:35.759
The coordinator rings the disability service in advance instead of waiting for students to disclose.
00:14:35.919 --> 00:14:37.039
Super organized.
00:14:37.360 --> 00:14:40.480
So that's 524 15-minute appointments.
00:14:40.639 --> 00:14:42.879
I'm shocking at maths, but that's a large number.
00:14:43.120 --> 00:14:45.440
That's a large that is a headache waiting to happen, right?
00:14:45.519 --> 00:14:55.039
I mean, Queensland, another slightly smaller group, uh 230 postgrad teacher education students, 10 minutes each on their own lesson plan.
00:14:55.200 --> 00:14:57.519
Three weeks of room holds, right?
00:14:57.759 --> 00:15:02.799
Missed appointments turned out to expose a hole in the deferral policy that no one had noticed.
00:15:02.960 --> 00:15:04.240
Second order effects, right?
00:15:04.480 --> 00:15:10.159
And then they found something which I think is one of the best, most honest parts of this whole exercise.
00:15:10.320 --> 00:15:14.399
Some students' written and oral performance did not match.
00:15:14.960 --> 00:15:19.440
Which means that the written work was I'm assuming you're gonna stop me.
00:15:19.519 --> 00:15:21.039
Yeah, I am stopping you.
00:15:21.120 --> 00:15:23.279
Like I was I was watching that.
00:15:23.519 --> 00:15:24.639
Love your gentle one.
00:15:25.039 --> 00:15:26.000
These students.
00:15:26.960 --> 00:15:27.679
Exactly.
00:15:27.919 --> 00:15:28.720
No, 100%.
00:15:29.039 --> 00:15:30.080
But this is the thing as well.
00:15:30.159 --> 00:15:32.320
These are not like for like swaps, right?
00:15:32.399 --> 00:15:34.480
Because the two formats sample different things.
00:15:34.559 --> 00:15:37.120
It doesn't tell you which one is the student, right?
00:15:37.200 --> 00:15:39.279
The the dependent and independent variable.
00:15:39.440 --> 00:15:44.799
They're now planning, reportedly at Queensland, to remove some of the written work to make uh to make room for this.
00:15:44.879 --> 00:15:47.679
So this is planned, uh, but not evaluated.
00:15:48.000 --> 00:15:49.440
I remember this logistical nightmare.
00:15:49.519 --> 00:15:52.000
I think we saw, I saw Jason, a listener of the show, by the way.
00:15:52.080 --> 00:16:04.639
So hi Jason, if you're listening, being like logistically exhausted after undertaking the endeavor, just saying, Oh, it was great, it worked it out, but the scheduling, oh my gosh, that was so so challenging.
00:16:04.960 --> 00:16:06.000
Yeah, a hundred percent.
00:16:06.080 --> 00:16:07.919
And there's interesting things in there as well, right?
00:16:08.000 --> 00:16:13.919
I mean, he he said from an article I saw on this for Herzer that within two minutes they knew exactly where the student was.
00:16:14.159 --> 00:16:18.639
So the weak essay writers came alive, and that's the case for, right?
00:16:18.960 --> 00:16:21.200
But then there's a twist going back to Phil Dawson.
00:16:21.440 --> 00:16:24.879
Do you remember he spoke about the the was it the comedy, the peer review, right?
00:16:24.960 --> 00:16:25.840
His comedy show.
00:16:26.000 --> 00:16:32.639
He would ask students to come on stage, talk about their own work, something they are probably world leaders in for 10 minutes, right?
00:16:32.960 --> 00:16:35.519
Three quarters said no.
00:16:36.159 --> 00:16:42.559
And this is a really hot issue when it comes to things like equitable learning plans, students who are seeking alternate means of assessment, etc.
00:16:42.720 --> 00:16:42.960
etc.
00:16:43.120 --> 00:16:49.120
They're expecting a written word, and all of a sudden we're pushing them to this because it's allegedly uh what has that security element?
00:16:49.679 --> 00:16:50.320
Yeah, 100%.
00:16:50.639 --> 00:16:57.519
And so these are world leaders declining, and at the same time, the same academics ask their students to talk about what they know best, right?
00:16:58.080 --> 00:17:03.840
Anyway, so let's let's talk about the the conversation fairness thing because you raise a really good case.
00:17:03.919 --> 00:17:09.519
Um, so there are 44 cases we found, 26 where it is the whole judgment as it not paired with something.
00:17:09.680 --> 00:17:16.960
Another interesting thing, not one published interrator reliability, examiner effects, appeal outcomes, right?
00:17:17.119 --> 00:17:21.440
And that's concerning because it is the basis around which the entire thing is built.
00:17:21.599 --> 00:17:26.799
One Melbourne uh abstract reported examiner link variation that survived the redesign.
00:17:26.880 --> 00:17:29.119
And again, these signal cards will make these available.
00:17:29.279 --> 00:17:30.640
People can check these out for yourselves.
00:17:30.799 --> 00:17:32.480
Everything with links, all verified.
00:17:32.559 --> 00:17:39.920
Uh, there were seven cases describing a need for double marking or moderation, and adopting it doesn't establish its effect, right?
00:17:40.160 --> 00:17:46.559
Um again, I'm just gonna say a mismatch with the written work does not by itself establish misconduct.
00:17:46.640 --> 00:17:51.200
These things are two different measures of potentially, but not always the same thing.
00:17:51.519 --> 00:17:56.480
I just apply that to reality in terms of how I write and how I speak are wildly different.
00:17:56.799 --> 00:17:57.680
Let's talk about cheese.
00:17:57.759 --> 00:18:00.640
In particular, Phil's cheese of the Swiss variety, to be exact.
00:18:00.880 --> 00:18:09.839
So it's the idea that our approach to assessment currently in this AI-enabled landscape is just trying our best in various ways to ensure we can assure learning.
00:18:10.000 --> 00:18:13.039
He said on there that he won't stop leaving it as a metaphor.
00:18:13.359 --> 00:18:13.839
Yeah, yeah.
00:18:14.000 --> 00:18:17.119
So he said, pick a specific task, say what the holes are.
00:18:17.279 --> 00:18:19.440
So we did it for the oral just go fast here.
00:18:19.519 --> 00:18:22.960
Uh hole one, the wearables, glasses, his own example, right?
00:18:23.039 --> 00:18:26.079
You can't make them take them off and you can't spot the AI ones anywhere.
00:18:26.240 --> 00:18:28.400
Uh hole number two, this one was interesting.
00:18:28.640 --> 00:18:31.759
Open university in the UK is the remote proctoring arrangements.
00:18:31.839 --> 00:18:37.839
They said, quote, students may be asked to remove a religious head covering temporarily so the proctor can check for earphones.
00:18:38.319 --> 00:18:39.359
He did raise that.
00:18:39.599 --> 00:18:41.839
So it's head coverings and hearing aids.
00:18:42.079 --> 00:18:42.319
Right.
00:18:42.480 --> 00:18:44.720
And he raised it as a thing he wasn't comfortable with.
00:18:44.880 --> 00:18:47.759
And this I found we found rather as a published procedure.
00:18:47.839 --> 00:18:53.759
Uh whole three, his word cramming, oral is a short-term event, so uh favors memory and other things, performance.
00:18:53.920 --> 00:18:56.400
Uh whole four, examiner variability, just touch that.
00:18:56.480 --> 00:18:59.839
Hole five, misappointments in the deferral policy that wasn't built for them.
00:19:00.079 --> 00:19:00.640
Interesting.
00:19:00.960 --> 00:19:02.799
Six, feedback, right?
00:19:03.039 --> 00:19:06.400
Because a six-minute conversation doesn't leave time for it.
00:19:06.480 --> 00:19:07.920
And then just maths, right?
00:19:08.000 --> 00:19:13.359
As you said before, 524 times 10 is just an absolute nightmare.
00:19:13.519 --> 00:19:17.519
So you layer it on top, and that's the whole point of the cheese metaphor.
00:19:17.759 --> 00:19:18.000
Yeah.
00:19:18.160 --> 00:19:20.400
And so you layer filtered it live, right?
00:19:20.480 --> 00:19:26.720
So he was talking about peer the oral with something longitudinal, so a placement, um, or or with something controlled.
00:19:26.880 --> 00:19:29.759
And then here's what happens when you lean on the controlled one.
00:19:29.920 --> 00:19:34.559
So Brown University in the US, uh a take-home midterm averaged 96%.
00:19:35.119 --> 00:19:41.039
The in-person final, when this person got a bit suspect about this, he arranged one of these, averaged 49%.
00:19:41.759 --> 00:19:44.720
I can hear the media publications writing headlines now.
00:19:44.880 --> 00:19:47.519
Everyone is cheating on the take-home exam.
00:19:47.839 --> 00:19:48.400
Yeah, yeah.
00:19:48.480 --> 00:19:50.319
No, that's 100% what everyone said.
00:19:50.480 --> 00:19:57.440
And the analysis that followed couldn't separate unauthorized help from the fact that those are two different exams under two different conditions.
00:19:57.599 --> 00:20:02.880
Uh, just to be clear, this was from Inside Higher Ed, not the institution, so I'm flagging that, but an interesting case on this.
00:20:03.039 --> 00:20:06.720
Kind of an interesting comment on where the whole sector is in terms of sharing this stuff, right?
00:20:06.799 --> 00:20:11.759
We're all dealing with this thing together, um, or or in isolation, aside as our own.
00:20:11.920 --> 00:20:16.160
But anyway, a gap between conditions needs interpreting before it's evidence of anything.
00:20:16.240 --> 00:20:16.480
Yeah.
00:20:16.720 --> 00:20:18.160
So, what about this process layer?
00:20:18.319 --> 00:20:22.079
The drafts, the logs, the the trail of learning.
00:20:22.319 --> 00:20:22.640
Yep.
00:20:22.799 --> 00:20:26.480
Uh, closer to you, Sydney was an interesting one, a cautionary tale if you like.
00:20:26.640 --> 00:20:30.720
This was an international risk management course uh with an optional AI assistant.
00:20:30.799 --> 00:20:34.319
And if you used it, you had to submit your chat logs, right?
00:20:34.559 --> 00:20:36.240
Uptake just collapsed.
00:20:36.400 --> 00:20:42.160
So students had a workload and uh importantly, concern about surveillance and integrity rules.
00:20:42.319 --> 00:20:54.640
So they dropped the log requirement, the course dropped the log requirement and assessed a reflection instead, which therefore means that our our findings, our research, this test research holds no test of whether logs show learning.
00:20:54.799 --> 00:20:59.200
Again, it might be happening, but the question is open because it was never reached.
00:21:00.160 --> 00:21:01.279
That has kind of eroded.
00:21:01.440 --> 00:21:04.559
There's people not talking about the submit your chat logs and your prompts, etc.
00:21:04.720 --> 00:21:06.720
etc., or marking up your chat transcripts.
00:21:06.960 --> 00:21:08.880
But is there a version of that that worked?
00:21:09.200 --> 00:21:12.640
So instead of appending a transcript, they made the trail the task, right?
00:21:12.720 --> 00:21:18.880
So critique an AI essay against the rubric, improve it with track re changes, and then reflect.
00:21:19.039 --> 00:21:22.720
This was uh 30% of the grade with three interconnected artifacts.
00:21:22.880 --> 00:21:27.359
So I guess if you like, the record is inside the work, not bolted on.
00:21:28.240 --> 00:21:29.680
I like that as a process.
00:21:29.839 --> 00:21:31.039
So what's the answer?
00:21:31.519 --> 00:21:33.200
Where's the good cheese, Nick?
00:21:33.440 --> 00:21:36.559
The good cheese is, I guess, whatever you can afford, right?
00:21:36.880 --> 00:21:40.160
And then the problem though is that no one tells you what it costs.
00:21:40.240 --> 00:21:41.759
And I guess that's the next bit.
00:21:42.160 --> 00:21:44.240
Alright, so what does the registry do we have a cost?
00:21:44.559 --> 00:21:47.440
Did the registry reveal a uh dollar figure?
00:21:48.400 --> 00:21:53.039
So every one of those rows has a problems and cost field.
00:21:53.359 --> 00:21:56.799
47, just 47, used a time or cost word.
00:21:57.119 --> 00:21:58.880
Five attached a number.
00:21:59.359 --> 00:21:59.759
Five.
00:22:00.880 --> 00:22:01.200
Yeah.
00:22:01.279 --> 00:22:02.640
And these could be time, right?
00:22:02.720 --> 00:22:06.240
And so each each of them, every single one of them, is about setting an exam.
00:22:06.319 --> 00:22:12.079
So Queensland uh spoke to a week or more to write one unfamiliar context paper.
00:22:12.240 --> 00:22:16.640
Uh that was the only exam setting time setting figure I could find anywhere.
00:22:16.799 --> 00:22:18.319
Everything else we could find anywhere.
00:22:18.400 --> 00:22:21.599
So everything else publishes the design and never the bill.
00:22:21.839 --> 00:22:23.279
So who does the actual sums?
00:22:23.359 --> 00:22:25.440
Like who's who's revealing the numbers?
00:22:25.759 --> 00:22:29.920
Yeah, quite appropriately, this was a maths department, Pennsylvania, right?
00:22:30.000 --> 00:22:34.079
So a conventional midterm, they were talking about uh teacher time, right?
00:22:34.240 --> 00:22:39.759
They spoke to 15 person hours of grading plus four and a half hours of proctoring.
00:22:39.920 --> 00:22:45.039
They replaced it with a single oral, and the instructor was quite honest about it.
00:22:45.119 --> 00:22:53.759
And so against the the two written midterms, it replaced, it's roughly a wash at 40, 40 students, it comes out.
00:22:54.000 --> 00:22:59.119
But every single one of those hours was now his, the the instructor in this case, and not the TAs.
00:22:59.599 --> 00:23:02.000
So the hours just don't magically disappear.
00:23:02.319 --> 00:23:07.359
No, they change hands and then they scale with the size of the cohort, they grow, right?
00:23:07.519 --> 00:23:10.319
So 40 students, 15 minutes each, you can do that.
00:23:10.400 --> 00:23:17.519
But going back to Newcastle, 524, and you've got a booking system, 15 minute slots for the whole encounter.
00:23:17.680 --> 00:23:21.039
I mean, Queensland, 230, you're holding rooms for three weeks.
00:23:21.200 --> 00:23:22.960
And underneath all of that, there's a trade.
00:23:23.039 --> 00:23:33.359
So Queensland rates a large cohort in-class writing task as low effort because it returns a grade and feedback to the whole cohort as opposed to individual comments.
00:23:33.519 --> 00:23:41.359
In the seven places we found where anyone published a cost, there's a trade-off between securing at scale and individual feedback.
00:23:41.599 --> 00:23:46.480
And nobody, we could we couldn't find anyone who published a number on both.
00:23:46.799 --> 00:23:47.839
So what about orals?
00:23:48.000 --> 00:23:48.640
Are they magical?
00:23:48.720 --> 00:23:53.200
Are they cheaper or are they more expensive than or dearer than marking?
00:23:54.000 --> 00:23:59.200
I mean, again, from what we found, depends on the arrangement, which is probably an honest answer, right?
00:23:59.279 --> 00:24:05.839
So Sydney's business school, they spoke about, uh, they reported examining time was broadly comparable to written marking.
00:24:06.160 --> 00:24:08.960
Curtin published at 30 minutes a student.
00:24:09.119 --> 00:24:12.960
Queensland and Newcastle say that the scheduling was the real burden, right?
00:24:13.279 --> 00:24:20.240
So uh the the the caveat that we put at the end or the instruction, compare the whole appointment, every assessor and the admin, right?
00:24:20.319 --> 00:24:22.720
It's it's not just the length of the conversation.
00:24:23.119 --> 00:24:31.440
So me parading around earlier in the year, asking everyone to just convert everything to Vibas, because how easy is a conversation does not take into account the reality.
00:24:31.839 --> 00:24:47.359
I think, I think, I think the nuance, if I could be generous to you there, is the the frameworks say redesign, the practice accounts say what they built, but very little has been put across to the cost, and nothing, and I I know this is your live reality as well.
00:24:47.519 --> 00:24:50.480
Nothing says what comes off someone's plate to pay for it.
00:24:50.880 --> 00:24:53.920
The busyness, you know, is just getting crazier and crazier.
00:24:54.319 --> 00:24:55.759
Just pushing the work somewhere.
00:24:56.079 --> 00:25:00.240
Friend of the show, Professor Molly Dollinger, talked about doubt recently.
00:25:00.480 --> 00:25:10.160
That the rubrics and the moderation and the double marking and everything else that we put on top of assessment and securing assessment are there to hide the fact that perhaps we're just not sure.
00:25:10.400 --> 00:25:14.960
So we strap on all these armor to protect us from the world of the Swiss cheese realities.
00:25:15.279 --> 00:25:15.920
Yeah, 100%.
00:25:16.160 --> 00:25:17.359
I'm on team doubt, right?
00:25:17.440 --> 00:25:18.240
I remember you saying that.
00:25:18.319 --> 00:25:21.839
I mean, so am I, and this is why all these caveats have been layered across this.
00:25:22.079 --> 00:25:23.920
Are you I want to just put one more in there?
00:25:24.000 --> 00:25:28.720
So every paragraph or every every one of these cards ends with what the evidence doesn't establish.
00:25:28.880 --> 00:25:31.920
So this is an educator account, not a comparative evaluations.
00:25:32.160 --> 00:25:34.559
Track changes do not establish who reasons through them.
00:25:34.640 --> 00:25:36.559
They there's there's some legalese in there.
00:25:36.640 --> 00:25:38.400
They read like a lawyer wrote them.
00:25:38.880 --> 00:25:40.160
They do a little bit.
00:25:41.039 --> 00:25:42.240
Just ass covering.
00:25:42.559 --> 00:25:43.920
Ars covering indeed.
00:25:44.079 --> 00:25:44.480
Written.
00:25:44.640 --> 00:25:46.000
But that's deliberate, right?
00:25:46.079 --> 00:25:48.720
Because I I want to put it to you that the alternative is the last three years.
00:25:48.880 --> 00:25:52.720
A confident tool, confident policy, and no one saying out loud what we didn't know, right?
00:25:52.799 --> 00:25:53.759
Or that we didn't know.
00:25:54.000 --> 00:25:58.160
I'm assuming AI detectors still came up within some of your findings as well.
00:25:58.400 --> 00:25:59.119
Yeah, absolutely.
00:25:59.279 --> 00:26:02.000
ANU had a good one and applied policy capstone.
00:26:02.079 --> 00:26:08.240
So every student required to write a draft ministerial brief for Chat GPT, then write their own reflection on it.
00:26:08.400 --> 00:26:12.720
So, ground truth, every one of them was AI generated by design.
00:26:12.960 --> 00:26:21.279
Turn it in, and the delicious little AI indicator scored those briefs anywhere from under 10% AI generated to 100.
00:26:21.599 --> 00:26:25.440
Same tool, same known answer, it landed everywhere on the dial.
00:26:25.599 --> 00:26:25.759
Right.
00:26:25.839 --> 00:26:29.519
And so, to be fair as well, that's from seven students who can send it to the study.
00:26:29.599 --> 00:26:34.880
Uh, but this is the only case we could find with a known answer to test the detector against.
00:26:35.039 --> 00:26:37.839
And the detector, unsurprisingly, failed.
00:26:38.240 --> 00:26:41.200
Detection was the the machinery that concealed the doubt.
00:26:41.440 --> 00:26:44.000
Appraisal is where you make students kind of practice against it.
00:26:44.240 --> 00:26:50.240
And the line, second time, a mismatch in this does not by itself establish misconduct.
00:26:50.559 --> 00:26:55.200
So, Nick, I started off the year asking people to put down their pitchwalks, stop being so tribal.
00:26:55.359 --> 00:27:02.240
And when Phil said the most powerful change approach that works is taking someone out for a coffee and having real conversations about this.
00:27:02.559 --> 00:27:03.839
God, I was glad.
00:27:04.160 --> 00:27:04.640
Yeah, yeah.
00:27:04.720 --> 00:27:05.519
His own research.
00:27:05.599 --> 00:27:13.920
Hey, I mean, the people who change what what an academic does, uh, the colleague next door, local learning and teaching lead, helpful search spacer, hopefully.
00:27:14.000 --> 00:27:17.200
So his budget recommendation was around uh coffee vouchers.
00:27:17.359 --> 00:27:19.359
And that's the kind of thing that we're driving to, right?
00:27:19.519 --> 00:27:26.240
So each one of these ends with a sentence we could change blank to gather evidence of blank before proceeding.
00:27:26.319 --> 00:27:27.920
We need to resolve blank.
00:27:28.000 --> 00:27:28.400
Yeah.
00:27:28.640 --> 00:27:34.960
Nick, are you saying your robust benchmarking and desktop analysis is as good as a delicious latte amongst friends?
00:27:35.200 --> 00:27:37.039
Yeah, that's uh that sounds like a coffee to me.
00:27:37.519 --> 00:27:39.839
I mean, not for a committee, it's for two people and a napkin.
00:27:40.079 --> 00:27:45.920
And I think uh the the closing sort of beat that we have on these things and coming soon, so keep as a promissory note.
00:27:46.160 --> 00:27:52.880
We wrote something at the bottom, which was before the first run, decide what evidence would help us keep, adapt, or stop this approach, right?
00:27:53.039 --> 00:27:57.519
So um the registry with the IDS research, we've got an outcome column.
00:27:57.599 --> 00:28:05.839
So 106 of the 114 are continuing, uh, five not stated, two modified, and one just abandoned.
00:28:05.920 --> 00:28:06.240
Yeah.
00:28:06.559 --> 00:28:08.720
One only one was abandoned.
00:28:09.039 --> 00:28:12.880
Yeah, not say so uh the again recorded on the record, right?
00:28:12.960 --> 00:28:16.240
So the who knows what's happening out there, the great unwatched space of it.
00:28:16.319 --> 00:28:28.079
But Kingston let module leaders set AI permissions locally, uh, got, in their words, inconsistency, confusion, non-compliance, and replace it with one university-wide approach.
00:28:28.319 --> 00:28:30.799
So one published open 114.
00:28:30.960 --> 00:28:41.359
Like the the sector spent three years redesigning assessment, and almost nobody said whether it worked, because almost nobody decided in advance or at least published what didn't work would look like.
00:28:45.440 --> 00:28:48.960
I'm assuming you're gonna run this session again, but your program manager friends comes in again.
00:28:49.119 --> 00:28:50.799
She asked you, Do you have an answer?
00:28:50.880 --> 00:28:52.160
What does she do now?
00:28:52.480 --> 00:28:55.440
Yeah, so one unit, right?
00:28:55.599 --> 00:28:57.359
Not the whole program, just one part.
00:28:57.680 --> 00:29:02.559
And the the card that she held the longest because it had worked so well for so long.
00:29:02.720 --> 00:29:05.680
Uh, the whole grade was about unobserved online work.
00:29:05.839 --> 00:29:07.519
Then one of the five signal cards for it.
00:29:07.599 --> 00:29:11.359
So probably the interactive oral, because that's where the most published practice is.
00:29:11.519 --> 00:29:14.720
And maybe she reads Newcastle's 15-minute appointment.
00:29:14.960 --> 00:29:20.079
So she knows what she's up to signing up for before she promises anyone anything.
00:29:20.319 --> 00:29:23.920
Then one colleague, uh, the coffee and and the blank.
00:29:24.160 --> 00:29:27.359
So we could change this to gather evidence of that.
00:29:27.440 --> 00:29:33.519
And before we go further, we need to resolve the deferral policy, the market time, or whatever it is.
00:29:33.759 --> 00:29:38.720
And and she writes down before the first run what would make her stop, right?
00:29:38.880 --> 00:29:46.960
So the the practice behind these principles, so which would put her ahead of pretty much all of what we found when we went looking.
00:29:47.119 --> 00:29:50.160
So that's what goes between the policy and her Monday.
00:29:50.319 --> 00:29:54.480
It took us, it took us months to get there from January, that session.
00:29:54.559 --> 00:29:56.960
And it's just the sketch on the back of a napkin.
00:29:57.519 --> 00:30:02.160
But Nick, I would I guess I'll say thank you for kind of bringing this data together because it's severely lacking.
00:30:02.240 --> 00:30:06.079
And I think for me, there's a call to action there for people to share these practices a lot more.
00:30:06.240 --> 00:30:08.480
What's missing in this is the data, is the understanding.
00:30:08.640 --> 00:30:11.119
This is a challenge that we solve together as a sector.
00:30:11.200 --> 00:30:14.160
And so the more transparency we can be and the more open we can be.
00:30:14.319 --> 00:30:17.920
I hope that 114 number goes a lot higher for the next version.
00:30:18.240 --> 00:30:18.640
100%.
00:30:19.039 --> 00:30:19.279
100%.
00:30:19.759 --> 00:30:21.599
Ladies and gentlemen, thank you for listening to Agile Intelligence.
00:30:21.680 --> 00:30:26.880
Some of those signal cards are available up on the Agile Intelligence website as draft for you guys to give feedback on.
00:30:27.039 --> 00:30:28.720
We would love to hear it.
00:30:28.880 --> 00:30:33.200
And every number that Nick said is on them with its source with a link.
00:30:33.440 --> 00:30:39.440
If you're anywhere near Saigon on the 24th and 25th of September, higher education horizons at RMT Saigon is free.
00:30:39.599 --> 00:30:44.079
Tim Fawns is keynoting, and Nick will be there arguing with people in person.
00:30:44.720 --> 00:30:46.000
Bring a challenge him on his car.
00:30:46.319 --> 00:30:46.960
Bring a friend.
00:30:47.119 --> 00:30:47.920
Yeah, yeah.
00:30:48.400 --> 00:30:54.160
Until then, stay curious, Down Intelligence, and whether it's owls, cheese, or coffee, all three now come.
00:30:56.079 --> 00:30:56.960
Apparently.
