WEBVTT
00:00:00.160 --> 00:00:05.679
In 2026, a 16-year-old named Olivia Hunt wrote a breakup letter to ChatGPT.
00:00:05.839 --> 00:00:09.439
Not a think piece, not a policy submission, a breakup letter.
00:00:09.599 --> 00:00:15.119
The kind you write when something, someone has gotten too close and you need a name, what it's doing to you.
00:00:15.279 --> 00:00:19.359
She wrote, Your voice started to replace my own.
00:00:19.679 --> 00:00:20.559
Six words.
00:00:20.879 --> 00:00:30.960
But everything we're going to talk about today, the research, the policy fights, the data from thousands of students, all of it is really just a longer version of what Olivia said in those six words.
00:00:31.199 --> 00:00:33.119
Something is replacing something.
00:00:33.280 --> 00:00:38.719
And we haven't figured out yet whether that's a problem, a feature, or just what happens next.
00:00:38.960 --> 00:00:42.640
Now this is where the assessment conversation actually is in 2026.
00:00:42.799 --> 00:00:53.359
Not the keynote version, the honest version, including the moment where one of the most respected assessment researchers in the world was asked if he could offer any reassurance, and he said no.
00:00:54.799 --> 00:00:55.520
That alright.
00:00:56.159 --> 00:00:57.679
Yeah, exactly.
00:01:07.200 --> 00:01:12.640
Ladies and gentlemen, welcome to Agile Intelligence, the podcast that looks at higher education and artificial intelligence.
00:01:12.719 --> 00:01:15.439
My name is Dalzinski, head of AI Education.
00:01:15.519 --> 00:01:18.000
I'm joined by Nick McIntosh, Learning Futurist.
00:01:18.159 --> 00:01:19.200
Nick, how are you?
00:01:19.439 --> 00:01:20.879
This episode seems somber.
00:01:21.040 --> 00:01:22.480
Or is it does it provide an answer?
00:01:22.640 --> 00:01:23.840
Where how are you?
00:01:23.920 --> 00:01:24.799
What are you feeling?
00:01:25.120 --> 00:01:26.159
I I feel good.
00:01:26.319 --> 00:01:27.599
Uh I feel curious.
00:01:27.760 --> 00:01:28.879
I feel open-eyed.
00:01:29.040 --> 00:01:34.799
I don't think we have an answer, as we sort of alluded to at the top, but we have many questions and they are pressing.
00:01:34.879 --> 00:01:35.359
How are you doing?
00:01:35.519 --> 00:01:36.159
Are you right?
00:01:36.799 --> 00:01:40.719
I am also excellent and fine with not having an answer.
00:01:40.959 --> 00:01:50.799
But I feel like there's a lot of people who are kind of they're looking to experts in the fields and to others to have an answer, to come in and be the saving light or the provide the magic bullet.
00:01:51.120 --> 00:02:01.840
Well, let's set the scene for this entire conversation because there was an op-ed that landed recently in the Sydney Morning Herald that got a lot of attention, at least in the circles that we move in.
00:02:02.079 --> 00:02:12.319
So an academic by the name of Kylie Moore Gilbert at Macquarie wrote that she advised her stepdaughter to think twice before enrolling in university.
00:02:12.560 --> 00:02:21.120
Her case, students are taking on tens of thousands of dollars in debt and being graded, she argues, on who can write the best AI prompts.
00:02:21.280 --> 00:02:27.039
She called it industrial scale fraud, universities cashing checks, and turning a blind eye.
00:02:27.280 --> 00:02:28.800
Now, some of it lands.
00:02:28.960 --> 00:02:36.080
The question about whether we should accept that nurses and engineers got their qualifications from a language model, that's serious.
00:02:36.319 --> 00:02:43.120
But she also claims probably over 90% of students are cheating, which is rhetorical, not evidential.
00:02:43.360 --> 00:02:48.319
And think twice about going to university lands very differently depending on who you are.
00:02:48.560 --> 00:02:53.520
If you've got networks and a safety net, a gap year is a sabbatical.
00:02:53.759 --> 00:02:55.759
And if you don't, it's a trap.
00:02:56.319 --> 00:03:02.719
Kath Ellis, so PVC Equality and Integrity at Western Sydney Uni, she fired back.
00:03:02.960 --> 00:03:06.400
For a lot of students, she argued, university is an optional.
00:03:06.560 --> 00:03:11.919
If you want to be a nurse, a teacher, or a social worker, there is no side door.
00:03:12.159 --> 00:03:15.919
Every year spent waiting is a year of lost momentum and lost income.
00:03:16.080 --> 00:03:22.240
The students who can least afford to wait are perhaps the ones most likely to take Moore Gilbert's advice seriously.
00:03:22.719 --> 00:03:25.360
Maybe it's just from our particular bubble, Nick.
00:03:25.520 --> 00:03:33.759
Me and my Australian counterparts seem rightfully outraged at some of the rhetoric and narrative around all of this because none of it's black and white.
00:03:33.919 --> 00:03:36.000
People who are right today are wrong tomorrow.
00:03:36.159 --> 00:03:43.360
On one hand, it means you do a lot of fence-sitting and perhaps not taking a side or a position because who knows how it's going to pan out in the next six to twelve months.
00:03:43.520 --> 00:03:46.479
But that may be because we're progressing this argument in Australia.
00:03:46.639 --> 00:03:48.719
I just think we're further along in some spaces.
00:03:48.960 --> 00:04:00.960
I think some of the conversations that we're having and some of the frameworks and uh practical ways in which AI and education is showing up, they're much more exciting than the boring old students using AI to cheat.
00:04:01.439 --> 00:04:04.960
The conversation is very much moved to how do we ensure the learning has happened?
00:04:05.039 --> 00:04:10.159
And I think that's much more powerful, which I think is why there's a bit of bit of spice around this.
00:04:10.719 --> 00:04:11.280
Entirely.
00:04:11.439 --> 00:04:14.960
And by extension, what is a degree actually certifying now?
00:04:15.199 --> 00:04:19.360
And look, just as we were finalizing this episode, the story got a twist.
00:04:19.519 --> 00:04:21.120
I genuinely couldn't have written.
00:04:21.279 --> 00:04:29.040
The Guardian reported this week that Kath Ellis, and again for context, Pro-Vice-Chancellor for Quality and Integrity, used AI to write her opinion piece.
00:04:29.279 --> 00:04:37.040
The article then goes on to say she uploaded 40,000 words of her own material into Microsoft Copilot, used it to generate early drafts, and didn't declare it.
00:04:37.279 --> 00:04:40.240
Sydney Morning Herald pulled the article, calling it unacceptable.
00:04:40.480 --> 00:04:48.319
Weston Sydney defended her, said it was a sophisticated and appropriate use of generative AI, drawing on her own expertise.
00:04:48.480 --> 00:04:53.519
And I'll be honest, for myself, I'm inclined to vigorously agree with this stance.
00:04:53.680 --> 00:04:54.000
Okay?
00:04:54.160 --> 00:05:03.279
This is she is building on years, decades of her own thinking in this space, and so I'm here for that argument, and this is something I personally do.
00:05:03.600 --> 00:05:07.279
That said, the disclosure question here is genuinely a live one.
00:05:07.519 --> 00:05:10.959
What counts as AI assisted versus AI generated?
00:05:11.040 --> 00:05:20.480
And if you spent, as I say, decades building a body of work, use a tool to synthesize your own thinking, is that meaningfully different from using it to produce thinking that you don't have?
00:05:20.720 --> 00:05:20.959
Sorry.
00:05:21.120 --> 00:05:26.160
Dale, let me ask you, because I think the more interesting question isn't whether she did something wrong.
00:05:26.319 --> 00:05:29.040
It's whether anyone could have done it, quote unquote, right.
00:05:29.199 --> 00:05:43.279
When the norms aren't settled, when the newspaper's own policy is ambiguous, and when the university and the editor can look at the same action and reach diametrically opposite conclusions, what does disclosure even mean in that environment?
00:05:43.839 --> 00:05:45.360
I really feel for Kathia.
00:05:45.519 --> 00:05:48.720
The details of the story are far more interesting and nuanced than the headline.
00:05:48.879 --> 00:05:53.279
The story that's been shared is one that will get some clicks and has some bias from the news head.
00:05:53.519 --> 00:05:58.480
But I'm going to point to, I'm gonna now we keep calling everyone a friend of the show just because we mentioned them.
00:05:58.720 --> 00:06:01.040
I would argue this is a best friend of the show.
00:06:01.120 --> 00:06:05.439
Uh, Professor Molly Dillinger, who has been on the show from Curtin, and she had a great response.
00:06:05.680 --> 00:06:13.600
She said, using AI in high-profile pieces of writing, whether that's a journal article, a newspaper article, or a press release for the university, it is a choice.
00:06:13.759 --> 00:06:17.360
And that choice obviously signals how much you're trying to push the dial.
00:06:17.759 --> 00:06:21.439
Whilst the choice was made, there is a fuzzy line here about what we are disclosing.
00:06:21.600 --> 00:06:24.879
We have the 40,000 words of source material, we have early drafts.
00:06:25.120 --> 00:06:28.319
I mean, sure, there was recognized AI writing patterns.
00:06:28.480 --> 00:06:35.360
I mean, sure, there was reportedly recognized AI writing patterns, but the reason they're recognized is because they're used everywhere as patterns.
00:06:35.519 --> 00:06:39.040
So this is not just black and white, evil cathalas.
00:06:39.120 --> 00:06:41.199
That's that's that's not the truth here.
00:06:41.519 --> 00:06:42.480
No, not remotely.
00:06:42.560 --> 00:06:46.399
I want to touch also on the mechanism behind this conclusion that it was AI generated.
00:06:46.560 --> 00:06:53.120
So this was flagged by an AI detector called Pangram, and this one, this one irritates me.
00:06:53.279 --> 00:06:55.279
This one, it feels a touch personal.
00:06:55.439 --> 00:06:55.759
Yeah.
00:06:55.920 --> 00:07:01.199
So from our context, we have elected not to turn on Turnitin's AI detection uh feature.
00:07:01.360 --> 00:07:07.759
So I was in the room uh when that decision was made, or at least the the room of tech boffins who advised the people who made the decision.
00:07:07.920 --> 00:07:08.639
We didn't.
00:07:08.800 --> 00:07:16.639
I was categorically opposed to this and still am, not because I'm soft on integrity, but because these tools do not work.
00:07:16.800 --> 00:07:18.480
They have never worked.
00:07:18.800 --> 00:07:19.600
Yeah, yeah.
00:07:19.839 --> 00:07:27.040
So for context, OpenAI, the people who built Chat GPT, made their own uh AI text detector early 23.
00:07:27.279 --> 00:07:30.959
They quietly took it down mid-23 because it just didn't work.
00:07:31.199 --> 00:07:40.720
Again, the people who made the category-defining AI tool that generated all this text couldn't understand the back end, didn't understand the back end enough to create something, right?
00:07:40.959 --> 00:07:44.480
Further, the false positive rate is devastating.
00:07:44.720 --> 00:07:55.759
Students or individuals accused of misconduct by a tool that is essentially a sophisticated guess facing consequences that research shows can include serious mental health impacts.
00:07:56.240 --> 00:08:01.199
These false accusations of academic misconduct are not admin inconveniences.
00:08:01.279 --> 00:08:03.120
They can be life-altering.
00:08:03.360 --> 00:08:08.639
And just to get mildly technical for a second, bypassing these detectors is just trivial.
00:08:08.720 --> 00:08:10.000
The work is seconds, right?
00:08:10.160 --> 00:08:13.439
Run your text through a humanizer, adjust the phrasing, done.
00:08:13.519 --> 00:08:13.680
Right?
00:08:13.839 --> 00:08:17.680
Give it samples of context, or as Kath did, samples of your writing.
00:08:17.920 --> 00:08:18.399
Easy.
00:08:18.560 --> 00:08:18.959
Nothing.
00:08:19.120 --> 00:08:20.639
The tool fails in both directions.
00:08:20.800 --> 00:08:23.920
It catches people who didn't cheat and it misses people who did.
00:08:24.079 --> 00:08:28.079
Whether Kath used AI inappropriately is almost beside the point.
00:08:28.560 --> 00:08:36.960
For me, the fact that a 2026 editorial decision was made on the basis of an AI detector tells you everything about how far we still have to go.
00:08:37.200 --> 00:08:40.879
The enforcement illusion isn't just in uni's, it's in newsrooms too.
00:08:41.200 --> 00:08:42.240
Yeah, in the industry.
00:08:42.399 --> 00:08:46.720
And I read the note from The Guardian, a publication that usually does all right in my books.
00:08:46.879 --> 00:08:49.600
And it really did read as someone who was offended.
00:08:49.840 --> 00:08:51.840
You didn't tell us, is to sum it up.
00:08:51.919 --> 00:08:53.200
But that may be last year's argument.
00:08:53.759 --> 00:08:59.679
As AI becomes more ubiquitous, more ambient, disclosing where you've used AI is increasingly challenging.
00:08:59.919 --> 00:09:10.000
We're working on a knowing the role of AI kind of framing at the moment at RMIT, where educators and students can talk about how AI shows up in their learning as a thinking partner, right through to the thing that helps do the work.
00:09:10.240 --> 00:09:13.039
But that needs shared language and shared understanding.
00:09:13.200 --> 00:09:25.919
More of these things will actually happen and they will leap across industries, and that's going to be a very human challenge because my version of using AI may be very different to how the person next to me uses it in a completely different job or occupation or background or context.
00:09:26.159 --> 00:09:33.039
And so it brings in the humor component of we need to collectively understand what we're comfortable with and how it shows up and be clear about it.
00:09:33.279 --> 00:09:33.759
I love it.
00:09:33.919 --> 00:09:34.320
I love it.
00:09:34.480 --> 00:09:35.600
Human-centric AI.
00:09:35.679 --> 00:09:36.159
That's perfect.
00:09:36.320 --> 00:09:37.200
No the role.
00:09:37.519 --> 00:09:38.639
Let's let's pivot now.
00:09:38.720 --> 00:09:46.320
Let's pivot to the enforcement illusion because the research literature uh was arriving at a conclusion much more uncomfortable than anything in either op ed.
00:09:46.480 --> 00:09:52.799
Okay, so three researchers, uh Corbin, Dawson, Lou, published a paper last year called Talk is Cheap.
00:09:53.200 --> 00:10:00.879
The argument was that almost everything universities have done in response to AI as pertaining to uh assessment is largely theater.
00:10:01.200 --> 00:10:04.559
So they distinguish discursive changes from structural ones.
00:10:04.720 --> 00:10:07.120
So discursive changes work through communication.
00:10:07.279 --> 00:10:10.960
So that's rules, guidelines, instructions, traffic light systems.
00:10:11.120 --> 00:10:15.759
Red means no AI, amber, orange, yellow means limited, greens mean go.
00:10:16.080 --> 00:10:19.840
Uh AI declarations on assignment cover sheets and honor codes.
00:10:20.000 --> 00:10:22.720
The problem isn't that these are badly designed.
00:10:22.879 --> 00:10:25.679
The problem is they have no inherent enforcement mechanism.
00:10:25.840 --> 00:10:29.679
They rely entirely on student compliance with rules that can't be verified.
00:10:29.840 --> 00:10:34.879
And I think the traffic light metaphor perfectly explains why it fails.
00:10:35.039 --> 00:10:42.240
Because real traffic lights work, because there are cameras, there are police, there are meaningful penalties, and these are structural.
00:10:42.399 --> 00:10:44.799
They change what happens if you don't comply.
00:10:45.039 --> 00:10:47.279
The uni version is just a sign.
00:10:47.519 --> 00:10:51.200
It communicates a preference, but it doesn't enforce anything.
00:10:51.519 --> 00:10:54.399
And the sector did eventually blink.
00:10:54.559 --> 00:10:58.399
So a series of unis have put in place systems mandating certain percentages.
00:10:58.480 --> 00:11:03.200
So 50% of all subject marks in the case of Melbourne must come from secure, supervised assessments.
00:11:03.360 --> 00:11:09.679
Australia's regulated text has shifted from an educative to regulatory uh stance, putting institutions on notice.
00:11:09.919 --> 00:11:12.480
The field moved to what seemed like the obvious answer.
00:11:12.639 --> 00:11:17.279
If you can't trust what students submit from home, put them in a room and watch them.
00:11:17.519 --> 00:11:35.120
But there was a moment at a recent seminar in Melbourne at Cradle, uh Deacon, so the Center for Research and Assessment, Digital Learning, where frequent mention on the pod, Jason Lodge gave a talk there on distributed cognition, AI agents, and what any of this means for assessment validity.
00:11:35.360 --> 00:11:39.679
Superbly worth watching if you haven't seen it online on YouTube, with a look.
00:11:39.840 --> 00:11:43.919
But right at the end, uh a question came in from the chat.
00:11:44.000 --> 00:11:52.559
So Phil Dawson, one of the authors of the Talkers Cheat paper, one of the architects of how this field thinks about assessment, gave a bit of a confession.
00:11:52.799 --> 00:11:53.120
Okay.
00:11:53.279 --> 00:12:00.879
So Dawson said his secret worry, the thing he sits with, if you like, is that higher education might never actually crack this.
00:12:01.039 --> 00:12:18.000
If we abandon the strategy of physically separating students from AI at the point of assessment, so we're talking the supervised exam, the controlled room, what the field calls ring fencing, he worries we won't find a workable alternative that's genuinely fit for assurance purposes.
00:12:18.240 --> 00:12:22.799
Something that actually lets us say with confidence the student knows it.
00:12:23.039 --> 00:12:24.559
So, Dale, you saw this, right?
00:12:24.720 --> 00:12:27.279
He asked Lodge, can you reassure me?
00:12:27.519 --> 00:12:29.440
And Jason said no.
00:12:29.759 --> 00:12:34.559
I was up in Sydney this week, as we mentioned at the top of the show, and Jason gave a cut-down version of this talk.
00:12:34.799 --> 00:12:38.799
There was the regular rogues of the Australian AI leaders around, including Danny Lou.
00:12:38.960 --> 00:12:42.000
A lot of what they were doing was there to plug the Cassoray statement.
00:12:42.159 --> 00:12:44.639
I got my own personal copy here, printed version.
00:12:44.720 --> 00:12:48.000
I think they're they're limited edition, which we'll do here as well, we'll do a bit of plugging.
00:12:48.080 --> 00:12:53.759
So Cassaray.ai, a cross-sector call to action on Australian education and training the age of AI.
00:12:54.000 --> 00:12:55.840
But there's two signs I want to talk about.
00:12:56.000 --> 00:13:06.399
Firstly, one of Danny's sessions, not only was it standing room only, but there was people lining up, spilling out onto the conference floor, hoping to listen to what he had to say.
00:13:06.639 --> 00:13:15.279
The other, after Jason did his talk, again, barely an empty seat in the room, yeah, he was mobbed because he was trying to give out physical copies of the casserole statement.
00:13:15.440 --> 00:13:20.000
I did ask them both to sign my personal book, but they they refused.
00:13:20.240 --> 00:13:25.600
But we're in desperate search of one person, one statement, one idea to solve this problem.
00:13:26.000 --> 00:13:26.960
It's not gonna happen.
00:13:27.360 --> 00:13:30.559
Every framework, every approach, they all have trade-offs.
00:13:30.799 --> 00:13:40.799
I think what I'll call out here is that the casserole statement provides at least a strong organizing function for this to be everyone's responsibility, not just one sage on the stage, one expert.
00:13:40.960 --> 00:13:44.080
There's that's probably what's that's probably the argument we're hearing.
00:13:44.399 --> 00:13:45.360
I love that as well.
00:13:45.440 --> 00:13:52.240
It's a very strong call to action, but I think that every approach has its trade-off thing, is is a really important point.
00:13:52.320 --> 00:13:55.600
So let's let's keep that top of mind as we we move on.
00:13:55.840 --> 00:14:00.320
Because let's pivot now to to the removal of the backstop, because this has kind of happened, right?
00:14:00.399 --> 00:14:07.759
The the sector with some of these things, that initial hype, knee-jerk response to AI, began moving towards secure assessment, right?
00:14:07.840 --> 00:14:13.120
So you had things like supervised exams, you had interactive orals, this is face-to-face, real time.
00:14:13.360 --> 00:14:16.559
Can't hide behind a polished submission and/or product.
00:14:16.799 --> 00:14:21.600
Several researchers described oral assessment as essentially immune to AI.
00:14:21.759 --> 00:14:23.200
This is clean logic, right?
00:14:23.360 --> 00:14:25.519
Physically separate the student from the tech.
00:14:25.759 --> 00:14:26.960
Problem solved.
00:14:27.679 --> 00:14:29.440
But then the glasses arrived.
00:14:29.679 --> 00:14:30.639
AI glasses.
00:14:30.799 --> 00:14:39.679
And a published, a paper published this year by Corbyn, Sharp, and Dawson makes an argument that I think defines the next phase of this conversation.
00:14:39.919 --> 00:14:43.919
So AI-enabled smart glasses are now mass market products.
00:14:44.000 --> 00:14:49.440
So from Ray Band Metas for a few hundred to knockoffs for as cheap as 70, I can see them.
00:14:49.519 --> 00:14:50.799
I'm sure less with time.
00:14:50.960 --> 00:14:54.080
How much are they going for now at JB Hi-Fi?
00:14:54.480 --> 00:14:56.159
I've seen very low ones for like$50.
00:14:56.559 --> 00:14:59.600
Limited functionality, but all you need is a Bluetooth connection.
00:14:59.919 --> 00:15:03.200
Yeah, but I mean you can't tell the difference from ordinary eyewear, right?
00:15:03.279 --> 00:15:04.960
In some cases, and madness.
00:15:05.200 --> 00:15:07.440
So think this is a heads-up display.
00:15:07.519 --> 00:15:18.960
So you you've got AI-generated text in your line of sight, built-in microphones processing speech in real time, cameras, read exam materials, and none of this produces any reliable external signal.
00:15:19.120 --> 00:15:23.440
No attention shifts to a separate screen, no gaze redirection.
00:15:23.679 --> 00:15:25.840
Student just sits there looking at you, right?
00:15:26.240 --> 00:15:30.480
The paper introduces this idea of dual transparency.
00:15:30.559 --> 00:15:38.080
So the tech is transparent to the user with enough use, it stops being a tool you consult and starts being how you perceive the world.
00:15:38.240 --> 00:15:42.159
So they draw on, and I'm going to take a run at this name, Merlo-Ponty.
00:15:42.320 --> 00:15:43.679
Apologies if I got that wrong.
00:15:43.840 --> 00:15:46.639
But example, I'll take that, thank you very much.
00:15:46.879 --> 00:15:47.840
Descarts.
00:15:48.159 --> 00:15:51.519
Example of a blind person learning to use a cane.
00:15:51.759 --> 00:15:58.080
So at first you feel the cane or crutches, if you like, but then you feel the pavement through the cane, right?
00:15:58.159 --> 00:16:00.240
So the tool disappears into perception.
00:16:00.480 --> 00:16:04.960
AI glasses have the architectural conditions to do exactly that.
00:16:05.120 --> 00:16:07.679
And they're simultaneously transparent to observers.
00:16:07.840 --> 00:16:11.919
The operation produces no reliable external signal that anyone watching can detect.
00:16:12.080 --> 00:16:18.720
So together, those two things dissolve the conditions on which those secure assessments potentially depend.
00:16:19.440 --> 00:16:29.759
I will say it starts getting a bit weird when education or show your learning is performed to the best of your ability as if you're stranded on desert island entirely naked with only what is in your head.
00:16:30.000 --> 00:16:31.279
Here is a pencil.
00:16:31.840 --> 00:16:36.480
I'm not saying that we surrender and just let technology take over, etc.
00:16:36.639 --> 00:16:36.799
etc.
00:16:36.960 --> 00:16:41.759
I'm just saying that this weird whack-a-mole of technology isn't serving anyone working in education.
00:16:41.919 --> 00:16:50.639
Not to mention the students that I speak to from all walks of life, all different disciplines and more, have what I would call a really quite a healthy relationship with some of this technology.
00:16:50.879 --> 00:16:55.120
The ones who want to cheat are going to cheat through some means.
00:16:56.240 --> 00:16:59.440
But the other ones, they're really conscious around how they use the tool.
00:16:59.519 --> 00:17:03.519
They recognize why they're there at university, what they're there to do, to improve, to learn.
00:17:03.600 --> 00:17:07.759
And when they feel like that's being eroded, they are actually very concerned.
00:17:08.160 --> 00:17:22.319
There's definitely concerns of fairness regarding what we're doing, what I'm doing, or what the student over there perhaps might get better marks because they're using and they're aided by some of these technology, and that's something we need to deal with as a sector regarding that's what integrity looks like, is that fairness component.
00:17:22.400 --> 00:17:27.200
But for a lot of it, that intrinsic value of students that we've spoken about a lot is actually there.
00:17:28.480 --> 00:17:29.519
No, absolutely.
00:17:29.680 --> 00:17:38.720
It's really interesting to watch the responses to date though, because again, it's sort of that knee-jerk prohibition response just coming out of uh straight out the gate.
00:17:38.799 --> 00:17:43.759
So, I mean, just to cite a couple of examples in the UK, University of London prohibited them.
00:17:43.920 --> 00:17:49.839
Uh Aberdeen um saying and vigilators and vigilators may ask to examine suspicious eyewear.
00:17:50.480 --> 00:17:58.480
But once prohibition stops working, or perhaps when uh knowledge, awareness of this thing grows, the next step is inspection.
00:17:58.640 --> 00:18:02.559
So physical examination of what students are wearing and who gets scrutinized.
00:18:02.720 --> 00:18:13.119
Students with disabilities, students with hearing aids, increasingly students with hearing aids, so increasingly AI enabled by default, students whose religious practice involves head coverings, right?
00:18:13.359 --> 00:18:15.759
The paper calls this bodily adjudication.
00:18:15.920 --> 00:18:20.319
The object of suspicion shifts from the essay, the product, to the person.
00:18:20.640 --> 00:18:25.680
I think it was um, I think it was Thomas Corbyn of Deacon who said it in that seminar I mentioned.
00:18:25.839 --> 00:18:28.400
The chronic problem is now the acute problem.
00:18:28.559 --> 00:18:33.039
So the window to find alternatives to this ring fencing is basically closed.
00:18:33.359 --> 00:18:35.279
I have I have I have all the data.
00:18:35.359 --> 00:18:37.039
Let's talk about what's actually happening.
00:18:37.359 --> 00:18:54.960
Because the wonderful AI and higher ed study uh of 8,000 students for Australian universities back in 2024, they've done a more recent version, I understand now, found that 83% of students have used AI for study, so nearly half weekly or daily, but only just over a quarter actually trust what it generates.
00:18:55.119 --> 00:18:56.400
They're using it pragmatically.
00:18:56.480 --> 00:18:58.720
So, as you said, in terms of this, they know what they're there for.
00:18:58.880 --> 00:19:02.079
Students were using it to edit, to summarize, to generate ideas.
00:19:02.400 --> 00:19:07.200
Only 12 and a bit, 13%, were completing assignments with it directly.
00:19:07.359 --> 00:19:10.559
That was the more nuanced picture then, rather than students be cheating.
00:19:10.799 --> 00:19:15.519
HEPI in the UK then published their 26th uh UK student survey.
00:19:15.759 --> 00:19:18.160
95% of students use AI now.
00:19:18.319 --> 00:19:20.160
94% for assess work.
00:19:20.400 --> 00:19:26.079
Different context between the AI and H E and HIPI study, but similar uh cohort of students, right?
00:19:26.240 --> 00:19:34.079
So direct submission of AI generated text jumped from 3 to 12%, 2024 to 26, nearly quadrupling in two years.
00:19:34.319 --> 00:19:38.079
But I think we come back to the students knowing what's good for them as well, right?
00:19:38.160 --> 00:19:40.960
Because that whole cognitive impact thing that we've been talking about.
00:19:41.119 --> 00:19:45.680
So taken here the OECD's mirage of false mastery, we've spoken about this before.
00:19:45.839 --> 00:19:55.039
Students using AI for practice performed 127% better doing assisted tasks, 17% worse when tests alone later, right?
00:19:55.279 --> 00:19:58.960
Ben Williamson of uh Edinburgh calls it desaturation.
00:19:59.119 --> 00:20:06.240
So students submitting original work while learning nothing because that useful friction that produces learning has been removed.
00:20:06.480 --> 00:20:15.519
Um, you may have seen this uh at the event that you you just referenced at the EduTech event, but apparently there's a 2026 edition of the AI and higher ed survey.
00:20:15.680 --> 00:20:19.279
I saw that Tim Forns uh spoke to a couple of slides from that.
00:20:19.519 --> 00:20:28.319
Apparently, some of their findings now, 9,000 students this time, when assessment restricts AI, 62% of students say they can learn deeply.
00:20:28.480 --> 00:20:33.119
When it allows AI, that drops to below 50% to 48.
00:20:33.279 --> 00:20:36.079
So students themselves are telling us something with that number.
00:20:36.240 --> 00:20:40.240
Tell us what responsible use looks like for a specific assignment for our field.
00:20:40.400 --> 00:20:41.440
Don't just ban it.
00:20:41.680 --> 00:20:44.000
That seems to me heartbreakingly reasonable.
00:20:44.240 --> 00:20:47.839
So we do have evidence to suggest that AI may be rotting our brain.
00:20:47.920 --> 00:20:50.160
Anecdotally, you see the concern for myself, from colleagues.
00:20:50.319 --> 00:20:54.000
People all of a sudden go, geez, how am I going to get my afternoon's work done without Claude?
00:20:54.480 --> 00:20:57.200
In short, if you don't use a muscle, it erodes.
00:20:57.359 --> 00:21:04.079
The average human who used to be built for hunting, for gathering, for farming, does not do that anymore because the technology took over.
00:21:04.640 --> 00:21:06.960
Humans pride themselves on their ability to think.
00:21:07.279 --> 00:21:11.279
We've been at the top of the food chain for a long time for that very reason.
00:21:11.759 --> 00:21:14.720
This is probably a much bigger philosophical episode for us to talk about, Nick.
00:21:14.880 --> 00:21:25.279
But tell me, Nick, are you a bit do you have a bit of a visceral reaction in the pit of your stomach, knowing that your brain is perhaps not as good as it was previously because of how much AI you use?
00:21:25.680 --> 00:21:32.640
Are you concerned that perhaps do you wish someone maybe was a bit more guiding with how you're using it now?
00:21:33.200 --> 00:21:37.920
I I am no worry, think good I I I good no computer easily.
00:21:38.559 --> 00:21:40.400
100%, 100%, 100%.
00:21:40.720 --> 00:21:44.160
Like, I mean, this may turn into a new appendix between my ears, you know what I mean?
00:21:44.240 --> 00:21:48.559
And and that's a very real possibility, or at least the the dummification.
00:21:48.799 --> 00:21:49.599
Idiocracy.
00:21:49.759 --> 00:21:53.440
Maybe it was a forecast, you know, a documentary more than anything else.
00:21:53.839 --> 00:21:59.680
I think at least the reaction that I have when everyone's whenever I see that stat or when I go back to do a task.
00:22:00.079 --> 00:22:04.960
And I go, oh, the fact that I'm I'm twitching to use one of my anything.
00:22:05.200 --> 00:22:09.119
Anything to help me out to get it done faster, quicker, in theory, better.
00:22:09.279 --> 00:22:11.039
So I yeah.
00:22:11.359 --> 00:22:13.359
It it is a it is a concern.
00:22:13.599 --> 00:22:16.079
And our students, clearly, they feel that.
00:22:16.640 --> 00:22:19.039
Let me take us through a bit of a pivot.
00:22:19.119 --> 00:22:24.720
Yeah, because I mean, I I I hear the concern, but let's talk about what people are actually trying to improve this entire thing.
00:22:24.799 --> 00:22:30.640
Because we don't want to just speak problems and leave you there, because there are real structural uh attempts to address this.
00:22:30.720 --> 00:22:38.000
So I want to call out directly Western Sydney University's IA matrix to inspire and assure as two axes.
00:22:38.160 --> 00:22:40.000
The insight is deceptively simple.
00:22:40.160 --> 00:22:41.680
Start with stopping, right?
00:22:41.839 --> 00:22:46.880
Identify the low-value tasks first and eliminate them before adding anything new.
00:22:47.119 --> 00:22:48.960
Curtains Assessment 2030 program.
00:22:49.279 --> 00:22:54.799
So you cited uh Molly Dollinger earlier, they've built around the two-lane model that first arose at the University of Sydney.
00:22:54.880 --> 00:22:57.279
So that's Danny Liu and Adam Bridgeman and the like.
00:22:57.440 --> 00:23:01.680
Lane one, secure supervised assessment that verifies learning occurred.
00:23:01.839 --> 00:23:07.839
Lane two, open assessments that prepare students to operate as professionals in an AI-enabled world, right?
00:23:08.000 --> 00:23:12.160
The key question for lane two isn't did they write every word themselves?
00:23:12.319 --> 00:23:20.720
It's did they demonstrate the disciplinary knowledge to work with the tool, to judge what good looks like, and produce something they can explain and defend.
00:23:20.880 --> 00:23:25.200
And that maps onto what professional life actually looks like, I would argue.
00:23:25.519 --> 00:23:33.200
I'm going to link to Texas here a little bit with their three ways of assuring learning or securing assessment, which they came out towards the end of last year.
00:23:33.279 --> 00:23:39.599
And one of the suggestions they had was programmatic assessment is that you don't fix your individual course, you fix the entire degree.
00:23:39.680 --> 00:23:46.240
And across that degree, we redesigned that so that it does have a certain amount of uh defensibility and be able to show you the skills, etc.
00:23:46.400 --> 00:23:46.640
etc.
00:23:48.079 --> 00:23:51.759
That's one bit to park, and but I really liked community practice that I was in recently.
00:23:51.839 --> 00:24:00.559
They framed this really perfectly is that it does support this looking at the pattern of learning across the entire program, not just one particular point of failure.
00:24:00.799 --> 00:24:09.279
I think that's something that I decided to look at regarding, okay, what over its students' life as they come into and do their undergrad, their postgrad of that may look like.
00:24:09.519 --> 00:24:15.279
What's the pattern of learning that they've exhibited across that journey versus just how do they perform with that one particular course?
00:24:15.440 --> 00:24:17.119
And so I think there's a very much an argument.
00:24:17.839 --> 00:24:37.119
Programmatic assessment is on paper really easy, so is said to be really easy, but when you actually get down to it and do it and you're organizing almost 12 different units and more uh to be able to produce a cohesive narrative, it's very challenging, but it does seem to be a really positive answer in this kind of I'd say fight, but um way for us to kind of navigate this challenge.
00:24:37.359 --> 00:24:40.240
I also think that me and Claude went long with the interactive orals thing.
00:24:40.319 --> 00:24:42.319
So what I did is I did a brain dump of relevant things.
00:24:42.480 --> 00:24:51.359
So my suggestion is I do this, and what I'm gonna say is that one of the things that we're really seeing in this entire space is that there is, as you said, no one single answer.
00:24:51.440 --> 00:24:53.440
This is an evolution as the entire thing moves.
00:24:53.680 --> 00:24:56.720
Want to touch on something from Leon Furs and the AI assessment scale.
00:24:56.799 --> 00:24:59.279
There's been a lot of fracass around this.
00:24:59.759 --> 00:25:01.039
Yeah, no, absolutely.
00:25:01.200 --> 00:25:07.440
I mean, the the other thing to call out in this space as well is that it is ever evolving, it is moving.
00:25:07.519 --> 00:25:22.079
And as such, as we move, what was an answer, as you alluded to earlier, what was an answer at one point maybe needs to evolve at a later point in this entire cycle as we move through AI and assessment and everything else as it continues to develop.
00:25:22.240 --> 00:25:35.279
So I want to call out here Leon Furs, so one of the original authors of the wildly popular AI assessment scale, who just published a long reflection on three years of watching frameworks travel.
00:25:35.440 --> 00:25:40.079
So he uses a concept uh from Dylan Williams of lethal mutations.
00:25:40.240 --> 00:25:45.920
So good principles distorted by scale and context, the the AIS, traffic lights.
00:25:46.160 --> 00:25:50.720
And in this, the Tullay model, he has seen this undergo the same process.
00:25:50.960 --> 00:26:08.559
Just to give some context on this, there has been a bit of back and forth, not just with the story that we mentioned at the top, but on the role of traffic lights, and we alluded to this as well in terms of that uh discursive versus structural change piece, um, in terms of what is the best answer for this wicked problem of AI and assessment.
00:26:08.799 --> 00:26:16.000
But Furzer's conclusion is really worth taking a minute with, because he concludes that frameworks are scaffolding.
00:26:16.160 --> 00:26:17.759
They are not permanent structures.
00:26:17.920 --> 00:26:31.039
And his closing line in these reflective pieces, and I'd recommend checking out his blog, is that every framework produced to handle AI in education has been useful for a time and insufficient for what came next.
00:26:31.279 --> 00:26:34.880
Now that isn't failure, that's how the work gets done.
00:26:35.200 --> 00:26:44.480
And the field it really seems to be is maturing as we we go further into this uh period of AI-dominated or at least AI-affected assessment.
00:26:44.720 --> 00:26:52.400
Any comment to that, Dale, in terms of how these frameworks evolve and them as thinking pieces, uh just more broadly on the sector and the approach?
00:26:52.799 --> 00:26:59.599
Frameworks live and die in the actual classroom where they're actually needing to show up in how people are going to play with them.
00:26:59.759 --> 00:27:04.799
And so I think we can spend oodles of time creating some of these, and they are really helpful for a starting point.
00:27:05.279 --> 00:27:19.759
But they do need to be revisited because as soon as you put them into practice, into play, the reality comes through because the thing that works on a PowerPoint deck perhaps doesn't work in a classroom full of 1500 students with very different uh behavioral needs and very different learning needs and learning requirements.
00:27:20.000 --> 00:27:26.799
And so I think a lot of what we're seeing here is these are permissions, these are things, these are guides for people to start with and to take and to perhaps navigate through this.
00:27:26.960 --> 00:27:31.440
But the next step is cool, and how do we support people to actually engage with these and move forward?
00:27:31.519 --> 00:27:36.559
It's that's the bit that's gonna be the most challenging, and that's the bit where the work's starting for that is probably.
00:27:36.640 --> 00:27:38.079
And that work is starting right now.
00:27:38.160 --> 00:27:42.000
And I think some of those use cases, they're gonna be the more powerful ones for us to consider.
00:27:42.319 --> 00:27:50.799
And the good news here is as well that the uh the field is maturing sufficiently that there are now textbooks, that there are now actual um bounded guides coming forward.
00:27:50.880 --> 00:27:58.000
Uh the creative research group, so Margaret Beerman, Tom Scorbin has mentioned, and colleagues, have just submitted a full manuscript to Rootlet, so 19 chapters.
00:27:58.160 --> 00:28:06.960
Their central argument, foundational assessment principles, far from being rendered obsolete by AI, provide essential frameworks for navigating this disruption.
00:28:07.119 --> 00:28:14.000
And when serious scholars like these are writing that sentence, it means the panic is starting to give way to something more durable.
00:28:14.319 --> 00:28:18.559
But I'm gonna wrap up with a nod back to the Castlere statement, right?
00:28:18.640 --> 00:28:22.079
So this is that cross-call uh cross-sector call of action that you have.
00:28:22.319 --> 00:28:27.359
Unsigned, sadly, but maybe maybe we can arrange like a signing event for you.
00:28:27.920 --> 00:28:35.920
The diagnosis of the sector, their diagnosis that Australia's response to date has been fragmented, stored, and at odds with national plan.
00:28:36.160 --> 00:28:39.839
So not a lack of expertise, but a failure of coordination and collective courage.
00:28:40.000 --> 00:28:40.720
I love that.
00:28:40.880 --> 00:28:51.039
And I love the the outflowing of energy that that has come from this and and and the sharing, the open-handedness of the sector and attempting to address this challenge together.
00:28:51.279 --> 00:28:56.000
But you know, the really interesting part for me, I don't know if you saw this, and I'm interested to get your take.
00:28:56.240 --> 00:28:59.440
They built in a contingency clause into the document.
00:28:59.680 --> 00:29:15.440
If certain trigger signals arrive, so labor market shifts, AI capability leaps, employers moving away from formal qualifications, the sector commits to shortcutting lengthening gov lengthy governance processes and institutional inertia.
00:29:16.079 --> 00:29:16.799
And nurture.
00:29:17.200 --> 00:29:22.240
The sector commits to shortcutting lengthy government processes and institutional inertia.
00:29:22.480 --> 00:29:26.000
They put an escape hatch and a policy document.
00:29:26.160 --> 00:29:33.039
And that tells you something interesting about how much time they think we have and the urgency they think we should be moving with.
00:29:33.359 --> 00:29:34.559
Plenty to think about, Nick.
00:29:34.640 --> 00:29:36.160
But everyone, thank you so much for listening.
00:29:36.240 --> 00:29:40.880
If you haven't already, please leave us review, share around within your network, and most of all, hit subscribe, whatever platform you're at.
00:29:40.960 --> 00:29:45.759
For those watching on YouTube, please note we are on all your favourite podcast platforms too.
00:29:45.839 --> 00:29:48.559
And those who love to see our faces, you can find us on YouTube too.
00:29:48.720 --> 00:29:52.319
We would love to tell you what's on next week's episode, but it's a world that moves way too fast.
00:29:52.400 --> 00:29:53.359
So you'll just have to tune in.
00:29:53.440 --> 00:29:56.799
Until then, stay curious, stay intelligent, and get your customer statement.
00:00:00.160 --> 00:00:05.679
In 2026, a 16-year-old named Olivia Hunt wrote a breakup letter to ChatGPT.
00:00:05.839 --> 00:00:09.439
Not a think piece, not a policy submission, a breakup letter.
00:00:09.599 --> 00:00:15.119
The kind you write when something, someone has gotten too close and you need a name, what it's doing to you.
00:00:15.279 --> 00:00:19.359
She wrote, Your voice started to replace my own.
00:00:19.679 --> 00:00:20.559
Six words.
00:00:20.879 --> 00:00:30.960
But everything we're going to talk about today, the research, the policy fights, the data from thousands of students, all of it is really just a longer version of what Olivia said in those six words.
00:00:31.199 --> 00:00:33.119
Something is replacing something.
00:00:33.280 --> 00:00:38.719
And we haven't figured out yet whether that's a problem, a feature, or just what happens next.
00:00:38.960 --> 00:00:42.640
Now this is where the assessment conversation actually is in 2026.
00:00:42.799 --> 00:00:53.359
Not the keynote version, the honest version, including the moment where one of the most respected assessment researchers in the world was asked if he could offer any reassurance, and he said no.
00:00:54.799 --> 00:00:55.520
That alright.
00:00:56.159 --> 00:00:57.679
Yeah, exactly.
00:01:07.200 --> 00:01:12.640
Ladies and gentlemen, welcome to Agile Intelligence, the podcast that looks at higher education and artificial intelligence.
00:01:12.719 --> 00:01:15.439
My name is Dalzinski, head of AI Education.
00:01:15.519 --> 00:01:18.000
I'm joined by Nick McIntosh, Learning Futurist.
00:01:18.159 --> 00:01:19.200
Nick, how are you?
00:01:19.439 --> 00:01:20.879
This episode seems somber.
00:01:21.040 --> 00:01:22.480
Or is it does it provide an answer?
00:01:22.640 --> 00:01:23.840
Where how are you?
00:01:23.920 --> 00:01:24.799
What are you feeling?
00:01:25.120 --> 00:01:26.159
I I feel good.
00:01:26.319 --> 00:01:27.599
Uh I feel curious.
00:01:27.760 --> 00:01:28.879
I feel open-eyed.
00:01:29.040 --> 00:01:34.799
I don't think we have an answer, as we sort of alluded to at the top, but we have many questions and they are pressing.
00:01:34.879 --> 00:01:35.359
How are you doing?
00:01:35.519 --> 00:01:36.159
Are you right?
00:01:36.799 --> 00:01:40.719
I am also excellent and fine with not having an answer.
00:01:40.959 --> 00:01:50.799
But I feel like there's a lot of people who are kind of they're looking to experts in the fields and to others to have an answer, to come in and be the saving light or the provide the magic bullet.
00:01:51.120 --> 00:02:01.840
Well, let's set the scene for this entire conversation because there was an op-ed that landed recently in the Sydney Morning Herald that got a lot of attention, at least in the circles that we move in.
00:02:02.079 --> 00:02:12.319
So an academic by the name of Kylie Moore Gilbert at Macquarie wrote that she advised her stepdaughter to think twice before enrolling in university.
00:02:12.560 --> 00:02:21.120
Her case, students are taking on tens of thousands of dollars in debt and being graded, she argues, on who can write the best AI prompts.
00:02:21.280 --> 00:02:27.039
She called it industrial scale fraud, universities cashing checks, and turning a blind eye.
00:02:27.280 --> 00:02:28.800
Now, some of it lands.
00:02:28.960 --> 00:02:36.080
The question about whether we should accept that nurses and engineers got their qualifications from a language model, that's serious.
00:02:36.319 --> 00:02:43.120
But she also claims probably over 90% of students are cheating, which is rhetorical, not evidential.
00:02:43.360 --> 00:02:48.319
And think twice about going to university lands very differently depending on who you are.
00:02:48.560 --> 00:02:53.520
If you've got networks and a safety net, a gap year is a sabbatical.
00:02:53.759 --> 00:02:55.759
And if you don't, it's a trap.
00:02:56.319 --> 00:03:02.719
Kath Ellis, so PVC Equality and Integrity at Western Sydney Uni, she fired back.
00:03:02.960 --> 00:03:06.400
For a lot of students, she argued, university is an optional.
00:03:06.560 --> 00:03:11.919
If you want to be a nurse, a teacher, or a social worker, there is no side door.
00:03:12.159 --> 00:03:15.919
Every year spent waiting is a year of lost momentum and lost income.
00:03:16.080 --> 00:03:22.240
The students who can least afford to wait are perhaps the ones most likely to take Moore Gilbert's advice seriously.
00:03:22.719 --> 00:03:25.360
Maybe it's just from our particular bubble, Nick.
00:03:25.520 --> 00:03:33.759
Me and my Australian counterparts seem rightfully outraged at some of the rhetoric and narrative around all of this because none of it's black and white.
00:03:33.919 --> 00:03:36.000
People who are right today are wrong tomorrow.
00:03:36.159 --> 00:03:43.360
On one hand, it means you do a lot of fence-sitting and perhaps not taking a side or a position because who knows how it's going to pan out in the next six to twelve months.
00:03:43.520 --> 00:03:46.479
But that may be because we're progressing this argument in Australia.
00:03:46.639 --> 00:03:48.719
I just think we're further along in some spaces.
00:03:48.960 --> 00:04:00.960
I think some of the conversations that we're having and some of the frameworks and uh practical ways in which AI and education is showing up, they're much more exciting than the boring old students using AI to cheat.
00:04:01.439 --> 00:04:04.960
The conversation is very much moved to how do we ensure the learning has happened?
00:04:05.039 --> 00:04:10.159
And I think that's much more powerful, which I think is why there's a bit of bit of spice around this.
00:04:10.719 --> 00:04:11.280
Entirely.
00:04:11.439 --> 00:04:14.960
And by extension, what is a degree actually certifying now?
00:04:15.199 --> 00:04:19.360
And look, just as we were finalizing this episode, the story got a twist.
00:04:19.519 --> 00:04:21.120
I genuinely couldn't have written.
00:04:21.279 --> 00:04:29.040
The Guardian reported this week that Kath Ellis, and again for context, Pro-Vice-Chancellor for Quality and Integrity, used AI to write her opinion piece.
00:04:29.279 --> 00:04:37.040
The article then goes on to say she uploaded 40,000 words of her own material into Microsoft Copilot, used it to generate early drafts, and didn't declare it.
00:04:37.279 --> 00:04:40.240
Sydney Morning Herald pulled the article, calling it unacceptable.
00:04:40.480 --> 00:04:48.319
Weston Sydney defended her, said it was a sophisticated and appropriate use of generative AI, drawing on her own expertise.
00:04:48.480 --> 00:04:53.519
And I'll be honest, for myself, I'm inclined to vigorously agree with this stance.
00:04:53.680 --> 00:04:54.000
Okay?
00:04:54.160 --> 00:05:03.279
This is she is building on years, decades of her own thinking in this space, and so I'm here for that argument, and this is something I personally do.
00:05:03.600 --> 00:05:07.279
That said, the disclosure question here is genuinely a live one.
00:05:07.519 --> 00:05:10.959
What counts as AI assisted versus AI generated?
00:05:11.040 --> 00:05:20.480
And if you spent, as I say, decades building a body of work, use a tool to synthesize your own thinking, is that meaningfully different from using it to produce thinking that you don't have?
00:05:20.720 --> 00:05:20.959
Sorry.
00:05:21.120 --> 00:05:26.160
Dale, let me ask you, because I think the more interesting question isn't whether she did something wrong.
00:05:26.319 --> 00:05:29.040
It's whether anyone could have done it, quote unquote, right.
00:05:29.199 --> 00:05:43.279
When the norms aren't settled, when the newspaper's own policy is ambiguous, and when the university and the editor can look at the same action and reach diametrically opposite conclusions, what does disclosure even mean in that environment?
00:05:43.839 --> 00:05:45.360
I really feel for Kathia.
00:05:45.519 --> 00:05:48.720
The details of the story are far more interesting and nuanced than the headline.
00:05:48.879 --> 00:05:53.279
The story that's been shared is one that will get some clicks and has some bias from the news head.
00:05:53.519 --> 00:05:58.480
But I'm going to point to, I'm gonna now we keep calling everyone a friend of the show just because we mentioned them.
00:05:58.720 --> 00:06:01.040
I would argue this is a best friend of the show.
00:06:01.120 --> 00:06:05.439
Uh, Professor Molly Dillinger, who has been on the show from Curtin, and she had a great response.
00:06:05.680 --> 00:06:13.600
She said, using AI in high-profile pieces of writing, whether that's a journal article, a newspaper article, or a press release for the university, it is a choice.
00:06:13.759 --> 00:06:17.360
And that choice obviously signals how much you're trying to push the dial.
00:06:17.759 --> 00:06:21.439
Whilst the choice was made, there is a fuzzy line here about what we are disclosing.
00:06:21.600 --> 00:06:24.879
We have the 40,000 words of source material, we have early drafts.
00:06:25.120 --> 00:06:28.319
I mean, sure, there was recognized AI writing patterns.
00:06:28.480 --> 00:06:35.360
I mean, sure, there was reportedly recognized AI writing patterns, but the reason they're recognized is because they're used everywhere as patterns.
00:06:35.519 --> 00:06:39.040
So this is not just black and white, evil cathalas.
00:06:39.120 --> 00:06:41.199
That's that's that's not the truth here.
00:06:41.519 --> 00:06:42.480
No, not remotely.
00:06:42.560 --> 00:06:46.399
I want to touch also on the mechanism behind this conclusion that it was AI generated.
00:06:46.560 --> 00:06:53.120
So this was flagged by an AI detector called Pangram, and this one, this one irritates me.
00:06:53.279 --> 00:06:55.279
This one, it feels a touch personal.
00:06:55.439 --> 00:06:55.759
Yeah.
00:06:55.920 --> 00:07:01.199
So from our context, we have elected not to turn on Turnitin's AI detection uh feature.
00:07:01.360 --> 00:07:07.759
So I was in the room uh when that decision was made, or at least the the room of tech boffins who advised the people who made the decision.
00:07:07.920 --> 00:07:08.639
We didn't.
00:07:08.800 --> 00:07:16.639
I was categorically opposed to this and still am, not because I'm soft on integrity, but because these tools do not work.
00:07:16.800 --> 00:07:18.480
They have never worked.
00:07:18.800 --> 00:07:19.600
Yeah, yeah.
00:07:19.839 --> 00:07:27.040
So for context, OpenAI, the people who built Chat GPT, made their own uh AI text detector early 23.
00:07:27.279 --> 00:07:30.959
They quietly took it down mid-23 because it just didn't work.
00:07:31.199 --> 00:07:40.720
Again, the people who made the category-defining AI tool that generated all this text couldn't understand the back end, didn't understand the back end enough to create something, right?
00:07:40.959 --> 00:07:44.480
Further, the false positive rate is devastating.
00:07:44.720 --> 00:07:55.759
Students or individuals accused of misconduct by a tool that is essentially a sophisticated guess facing consequences that research shows can include serious mental health impacts.
00:07:56.240 --> 00:08:01.199
These false accusations of academic misconduct are not admin inconveniences.
00:08:01.279 --> 00:08:03.120
They can be life-altering.
00:08:03.360 --> 00:08:08.639
And just to get mildly technical for a second, bypassing these detectors is just trivial.
00:08:08.720 --> 00:08:10.000
The work is seconds, right?
00:08:10.160 --> 00:08:13.439
Run your text through a humanizer, adjust the phrasing, done.
00:08:13.519 --> 00:08:13.680
Right?
00:08:13.839 --> 00:08:17.680
Give it samples of context, or as Kath did, samples of your writing.
00:08:17.920 --> 00:08:18.399
Easy.
00:08:18.560 --> 00:08:18.959
Nothing.
00:08:19.120 --> 00:08:20.639
The tool fails in both directions.
00:08:20.800 --> 00:08:23.920
It catches people who didn't cheat and it misses people who did.
00:08:24.079 --> 00:08:28.079
Whether Kath used AI inappropriately is almost beside the point.
00:08:28.560 --> 00:08:36.960
For me, the fact that a 2026 editorial decision was made on the basis of an AI detector tells you everything about how far we still have to go.
00:08:37.200 --> 00:08:40.879
The enforcement illusion isn't just in uni's, it's in newsrooms too.
00:08:41.200 --> 00:08:42.240
Yeah, in the industry.
00:08:42.399 --> 00:08:46.720
And I read the note from The Guardian, a publication that usually does all right in my books.
00:08:46.879 --> 00:08:49.600
And it really did read as someone who was offended.
00:08:49.840 --> 00:08:51.840
You didn't tell us, is to sum it up.
00:08:51.919 --> 00:08:53.200
But that may be last year's argument.
00:08:53.759 --> 00:08:59.679
As AI becomes more ubiquitous, more ambient, disclosing where you've used AI is increasingly challenging.
00:08:59.919 --> 00:09:10.000
We're working on a knowing the role of AI kind of framing at the moment at RMIT, where educators and students can talk about how AI shows up in their learning as a thinking partner, right through to the thing that helps do the work.
00:09:10.240 --> 00:09:13.039
But that needs shared language and shared understanding.
00:09:13.200 --> 00:09:25.919
More of these things will actually happen and they will leap across industries, and that's going to be a very human challenge because my version of using AI may be very different to how the person next to me uses it in a completely different job or occupation or background or context.
00:09:26.159 --> 00:09:33.039
And so it brings in the humor component of we need to collectively understand what we're comfortable with and how it shows up and be clear about it.
00:09:33.279 --> 00:09:33.759
I love it.
00:09:33.919 --> 00:09:34.320
I love it.
00:09:34.480 --> 00:09:35.600
Human-centric AI.
00:09:35.679 --> 00:09:36.159
That's perfect.
00:09:36.320 --> 00:09:37.200
No the role.
00:09:37.519 --> 00:09:38.639
Let's let's pivot now.
00:09:38.720 --> 00:09:46.320
Let's pivot to the enforcement illusion because the research literature uh was arriving at a conclusion much more uncomfortable than anything in either op ed.
00:09:46.480 --> 00:09:52.799
Okay, so three researchers, uh Corbin, Dawson, Lou, published a paper last year called Talk is Cheap.
00:09:53.200 --> 00:10:00.879
The argument was that almost everything universities have done in response to AI as pertaining to uh assessment is largely theater.
00:10:01.200 --> 00:10:04.559
So they distinguish discursive changes from structural ones.
00:10:04.720 --> 00:10:07.120
So discursive changes work through communication.
00:10:07.279 --> 00:10:10.960
So that's rules, guidelines, instructions, traffic light systems.
00:10:11.120 --> 00:10:15.759
Red means no AI, amber, orange, yellow means limited, greens mean go.
00:10:16.080 --> 00:10:19.840
Uh AI declarations on assignment cover sheets and honor codes.
00:10:20.000 --> 00:10:22.720
The problem isn't that these are badly designed.
00:10:22.879 --> 00:10:25.679
The problem is they have no inherent enforcement mechanism.
00:10:25.840 --> 00:10:29.679
They rely entirely on student compliance with rules that can't be verified.
00:10:29.840 --> 00:10:34.879
And I think the traffic light metaphor perfectly explains why it fails.
00:10:35.039 --> 00:10:42.240
Because real traffic lights work, because there are cameras, there are police, there are meaningful penalties, and these are structural.
00:10:42.399 --> 00:10:44.799
They change what happens if you don't comply.
00:10:45.039 --> 00:10:47.279
The uni version is just a sign.
00:10:47.519 --> 00:10:51.200
It communicates a preference, but it doesn't enforce anything.
00:10:51.519 --> 00:10:54.399
And the sector did eventually blink.
00:10:54.559 --> 00:10:58.399
So a series of unis have put in place systems mandating certain percentages.
00:10:58.480 --> 00:11:03.200
So 50% of all subject marks in the case of Melbourne must come from secure, supervised assessments.
00:11:03.360 --> 00:11:09.679
Australia's regulated text has shifted from an educative to regulatory uh stance, putting institutions on notice.
00:11:09.919 --> 00:11:12.480
The field moved to what seemed like the obvious answer.
00:11:12.639 --> 00:11:17.279
If you can't trust what students submit from home, put them in a room and watch them.
00:11:17.519 --> 00:11:35.120
But there was a moment at a recent seminar in Melbourne at Cradle, uh Deacon, so the Center for Research and Assessment, Digital Learning, where frequent mention on the pod, Jason Lodge gave a talk there on distributed cognition, AI agents, and what any of this means for assessment validity.
00:11:35.360 --> 00:11:39.679
Superbly worth watching if you haven't seen it online on YouTube, with a look.
00:11:39.840 --> 00:11:43.919
But right at the end, uh a question came in from the chat.
00:11:44.000 --> 00:11:52.559
So Phil Dawson, one of the authors of the Talkers Cheat paper, one of the architects of how this field thinks about assessment, gave a bit of a confession.
00:11:52.799 --> 00:11:53.120
Okay.
00:11:53.279 --> 00:12:00.879
So Dawson said his secret worry, the thing he sits with, if you like, is that higher education might never actually crack this.
00:12:01.039 --> 00:12:18.000
If we abandon the strategy of physically separating students from AI at the point of assessment, so we're talking the supervised exam, the controlled room, what the field calls ring fencing, he worries we won't find a workable alternative that's genuinely fit for assurance purposes.
00:12:18.240 --> 00:12:22.799
Something that actually lets us say with confidence the student knows it.
00:12:23.039 --> 00:12:24.559
So, Dale, you saw this, right?
00:12:24.720 --> 00:12:27.279
He asked Lodge, can you reassure me?
00:12:27.519 --> 00:12:29.440
And Jason said no.
00:12:29.759 --> 00:12:34.559
I was up in Sydney this week, as we mentioned at the top of the show, and Jason gave a cut-down version of this talk.
00:12:34.799 --> 00:12:38.799
There was the regular rogues of the Australian AI leaders around, including Danny Lou.
00:12:38.960 --> 00:12:42.000
A lot of what they were doing was there to plug the Cassoray statement.
00:12:42.159 --> 00:12:44.639
I got my own personal copy here, printed version.
00:12:44.720 --> 00:12:48.000
I think they're they're limited edition, which we'll do here as well, we'll do a bit of plugging.
00:12:48.080 --> 00:12:53.759
So Cassaray.ai, a cross-sector call to action on Australian education and training the age of AI.
00:12:54.000 --> 00:12:55.840
But there's two signs I want to talk about.
00:12:56.000 --> 00:13:06.399
Firstly, one of Danny's sessions, not only was it standing room only, but there was people lining up, spilling out onto the conference floor, hoping to listen to what he had to say.
00:13:06.639 --> 00:13:15.279
The other, after Jason did his talk, again, barely an empty seat in the room, yeah, he was mobbed because he was trying to give out physical copies of the casserole statement.
00:13:15.440 --> 00:13:20.000
I did ask them both to sign my personal book, but they they refused.
00:13:20.240 --> 00:13:25.600
But we're in desperate search of one person, one statement, one idea to solve this problem.
00:13:26.000 --> 00:13:26.960
It's not gonna happen.
00:13:27.360 --> 00:13:30.559
Every framework, every approach, they all have trade-offs.
00:13:30.799 --> 00:13:40.799
I think what I'll call out here is that the casserole statement provides at least a strong organizing function for this to be everyone's responsibility, not just one sage on the stage, one expert.
00:13:40.960 --> 00:13:44.080
There's that's probably what's that's probably the argument we're hearing.
00:13:44.399 --> 00:13:45.360
I love that as well.
00:13:45.440 --> 00:13:52.240
It's a very strong call to action, but I think that every approach has its trade-off thing, is is a really important point.
00:13:52.320 --> 00:13:55.600
So let's let's keep that top of mind as we we move on.
00:13:55.840 --> 00:14:00.320
Because let's pivot now to to the removal of the backstop, because this has kind of happened, right?
00:14:00.399 --> 00:14:07.759
The the sector with some of these things, that initial hype, knee-jerk response to AI, began moving towards secure assessment, right?
00:14:07.840 --> 00:14:13.120
So you had things like supervised exams, you had interactive orals, this is face-to-face, real time.
00:14:13.360 --> 00:14:16.559
Can't hide behind a polished submission and/or product.
00:14:16.799 --> 00:14:21.600
Several researchers described oral assessment as essentially immune to AI.
00:14:21.759 --> 00:14:23.200
This is clean logic, right?
00:14:23.360 --> 00:14:25.519
Physically separate the student from the tech.
00:14:25.759 --> 00:14:26.960
Problem solved.
00:14:27.679 --> 00:14:29.440
But then the glasses arrived.
00:14:29.679 --> 00:14:30.639
AI glasses.
00:14:30.799 --> 00:14:39.679
And a published, a paper published this year by Corbyn, Sharp, and Dawson makes an argument that I think defines the next phase of this conversation.
00:14:39.919 --> 00:14:43.919
So AI-enabled smart glasses are now mass market products.
00:14:44.000 --> 00:14:49.440
So from Ray Band Metas for a few hundred to knockoffs for as cheap as 70, I can see them.
00:14:49.519 --> 00:14:50.799
I'm sure less with time.
00:14:50.960 --> 00:14:54.080
How much are they going for now at JB Hi-Fi?
00:14:54.480 --> 00:14:56.159
I've seen very low ones for like$50.
00:14:56.559 --> 00:14:59.600
Limited functionality, but all you need is a Bluetooth connection.
00:14:59.919 --> 00:15:03.200
Yeah, but I mean you can't tell the difference from ordinary eyewear, right?
00:15:03.279 --> 00:15:04.960
In some cases, and madness.
00:15:05.200 --> 00:15:07.440
So think this is a heads-up display.
00:15:07.519 --> 00:15:18.960
So you you've got AI-generated text in your line of sight, built-in microphones processing speech in real time, cameras, read exam materials, and none of this produces any reliable external signal.
00:15:19.120 --> 00:15:23.440
No attention shifts to a separate screen, no gaze redirection.
00:15:23.679 --> 00:15:25.840
Student just sits there looking at you, right?
00:15:26.240 --> 00:15:30.480
The paper introduces this idea of dual transparency.
00:15:30.559 --> 00:15:38.080
So the tech is transparent to the user with enough use, it stops being a tool you consult and starts being how you perceive the world.
00:15:38.240 --> 00:15:42.159
So they draw on, and I'm going to take a run at this name, Merlo-Ponty.
00:15:42.320 --> 00:15:43.679
Apologies if I got that wrong.
00:15:43.840 --> 00:15:46.639
But example, I'll take that, thank you very much.
00:15:46.879 --> 00:15:47.840
Descarts.
00:15:48.159 --> 00:15:51.519
Example of a blind person learning to use a cane.
00:15:51.759 --> 00:15:58.080
So at first you feel the cane or crutches, if you like, but then you feel the pavement through the cane, right?
00:15:58.159 --> 00:16:00.240
So the tool disappears into perception.
00:16:00.480 --> 00:16:04.960
AI glasses have the architectural conditions to do exactly that.
00:16:05.120 --> 00:16:07.679
And they're simultaneously transparent to observers.
00:16:07.840 --> 00:16:11.919
The operation produces no reliable external signal that anyone watching can detect.
00:16:12.080 --> 00:16:18.720
So together, those two things dissolve the conditions on which those secure assessments potentially depend.
00:16:19.440 --> 00:16:29.759
I will say it starts getting a bit weird when education or show your learning is performed to the best of your ability as if you're stranded on desert island entirely naked with only what is in your head.
00:16:30.000 --> 00:16:31.279
Here is a pencil.
00:16:31.840 --> 00:16:36.480
I'm not saying that we surrender and just let technology take over, etc.
00:16:36.639 --> 00:16:36.799
etc.
00:16:36.960 --> 00:16:41.759
I'm just saying that this weird whack-a-mole of technology isn't serving anyone working in education.
00:16:41.919 --> 00:16:50.639
Not to mention the students that I speak to from all walks of life, all different disciplines and more, have what I would call a really quite a healthy relationship with some of this technology.
00:16:50.879 --> 00:16:55.120
The ones who want to cheat are going to cheat through some means.
00:16:56.240 --> 00:16:59.440
But the other ones, they're really conscious around how they use the tool.
00:16:59.519 --> 00:17:03.519
They recognize why they're there at university, what they're there to do, to improve, to learn.
00:17:03.600 --> 00:17:07.759
And when they feel like that's being eroded, they are actually very concerned.
00:17:08.160 --> 00:17:22.319
There's definitely concerns of fairness regarding what we're doing, what I'm doing, or what the student over there perhaps might get better marks because they're using and they're aided by some of these technology, and that's something we need to deal with as a sector regarding that's what integrity looks like, is that fairness component.
00:17:22.400 --> 00:17:27.200
But for a lot of it, that intrinsic value of students that we've spoken about a lot is actually there.
00:17:28.480 --> 00:17:29.519
No, absolutely.
00:17:29.680 --> 00:17:38.720
It's really interesting to watch the responses to date though, because again, it's sort of that knee-jerk prohibition response just coming out of uh straight out the gate.
00:17:38.799 --> 00:17:43.759
So, I mean, just to cite a couple of examples in the UK, University of London prohibited them.
00:17:43.920 --> 00:17:49.839
Uh Aberdeen um saying and vigilators and vigilators may ask to examine suspicious eyewear.
00:17:50.480 --> 00:17:58.480
But once prohibition stops working, or perhaps when uh knowledge, awareness of this thing grows, the next step is inspection.
00:17:58.640 --> 00:18:02.559
So physical examination of what students are wearing and who gets scrutinized.
00:18:02.720 --> 00:18:13.119
Students with disabilities, students with hearing aids, increasingly students with hearing aids, so increasingly AI enabled by default, students whose religious practice involves head coverings, right?
00:18:13.359 --> 00:18:15.759
The paper calls this bodily adjudication.
00:18:15.920 --> 00:18:20.319
The object of suspicion shifts from the essay, the product, to the person.
00:18:20.640 --> 00:18:25.680
I think it was um, I think it was Thomas Corbyn of Deacon who said it in that seminar I mentioned.
00:18:25.839 --> 00:18:28.400
The chronic problem is now the acute problem.
00:18:28.559 --> 00:18:33.039
So the window to find alternatives to this ring fencing is basically closed.
00:18:33.359 --> 00:18:35.279
I have I have I have all the data.
00:18:35.359 --> 00:18:37.039
Let's talk about what's actually happening.
00:18:37.359 --> 00:18:54.960
Because the wonderful AI and higher ed study uh of 8,000 students for Australian universities back in 2024, they've done a more recent version, I understand now, found that 83% of students have used AI for study, so nearly half weekly or daily, but only just over a quarter actually trust what it generates.
00:18:55.119 --> 00:18:56.400
They're using it pragmatically.
00:18:56.480 --> 00:18:58.720
So, as you said, in terms of this, they know what they're there for.
00:18:58.880 --> 00:19:02.079
Students were using it to edit, to summarize, to generate ideas.
00:19:02.400 --> 00:19:07.200
Only 12 and a bit, 13%, were completing assignments with it directly.
00:19:07.359 --> 00:19:10.559
That was the more nuanced picture then, rather than students be cheating.
00:19:10.799 --> 00:19:15.519
HEPI in the UK then published their 26th uh UK student survey.
00:19:15.759 --> 00:19:18.160
95% of students use AI now.
00:19:18.319 --> 00:19:20.160
94% for assess work.
00:19:20.400 --> 00:19:26.079
Different context between the AI and H E and HIPI study, but similar uh cohort of students, right?
00:19:26.240 --> 00:19:34.079
So direct submission of AI generated text jumped from 3 to 12%, 2024 to 26, nearly quadrupling in two years.
00:19:34.319 --> 00:19:38.079
But I think we come back to the students knowing what's good for them as well, right?
00:19:38.160 --> 00:19:40.960
Because that whole cognitive impact thing that we've been talking about.
00:19:41.119 --> 00:19:45.680
So taken here the OECD's mirage of false mastery, we've spoken about this before.
00:19:45.839 --> 00:19:55.039
Students using AI for practice performed 127% better doing assisted tasks, 17% worse when tests alone later, right?
00:19:55.279 --> 00:19:58.960
Ben Williamson of uh Edinburgh calls it desaturation.
00:19:59.119 --> 00:20:06.240
So students submitting original work while learning nothing because that useful friction that produces learning has been removed.
00:20:06.480 --> 00:20:15.519
Um, you may have seen this uh at the event that you you just referenced at the EduTech event, but apparently there's a 2026 edition of the AI and higher ed survey.
00:20:15.680 --> 00:20:19.279
I saw that Tim Forns uh spoke to a couple of slides from that.
00:20:19.519 --> 00:20:28.319
Apparently, some of their findings now, 9,000 students this time, when assessment restricts AI, 62% of students say they can learn deeply.
00:20:28.480 --> 00:20:33.119
When it allows AI, that drops to below 50% to 48.
00:20:33.279 --> 00:20:36.079
So students themselves are telling us something with that number.
00:20:36.240 --> 00:20:40.240
Tell us what responsible use looks like for a specific assignment for our field.
00:20:40.400 --> 00:20:41.440
Don't just ban it.
00:20:41.680 --> 00:20:44.000
That seems to me heartbreakingly reasonable.
00:20:44.240 --> 00:20:47.839
So we do have evidence to suggest that AI may be rotting our brain.
00:20:47.920 --> 00:20:50.160
Anecdotally, you see the concern for myself, from colleagues.
00:20:50.319 --> 00:20:54.000
People all of a sudden go, geez, how am I going to get my afternoon's work done without Claude?
00:20:54.480 --> 00:20:57.200
In short, if you don't use a muscle, it erodes.
00:20:57.359 --> 00:21:04.079
The average human who used to be built for hunting, for gathering, for farming, does not do that anymore because the technology took over.
00:21:04.640 --> 00:21:06.960
Humans pride themselves on their ability to think.
00:21:07.279 --> 00:21:11.279
We've been at the top of the food chain for a long time for that very reason.
00:21:11.759 --> 00:21:14.720
This is probably a much bigger philosophical episode for us to talk about, Nick.
00:21:14.880 --> 00:21:25.279
But tell me, Nick, are you a bit do you have a bit of a visceral reaction in the pit of your stomach, knowing that your brain is perhaps not as good as it was previously because of how much AI you use?
00:21:25.680 --> 00:21:32.640
Are you concerned that perhaps do you wish someone maybe was a bit more guiding with how you're using it now?
00:21:33.200 --> 00:21:37.920
I I am no worry, think good I I I good no computer easily.
00:21:38.559 --> 00:21:40.400
100%, 100%, 100%.
00:21:40.720 --> 00:21:44.160
Like, I mean, this may turn into a new appendix between my ears, you know what I mean?
00:21:44.240 --> 00:21:48.559
And and that's a very real possibility, or at least the the dummification.
00:21:48.799 --> 00:21:49.599
Idiocracy.
00:21:49.759 --> 00:21:53.440
Maybe it was a forecast, you know, a documentary more than anything else.
00:21:53.839 --> 00:21:59.680
I think at least the reaction that I have when everyone's whenever I see that stat or when I go back to do a task.
00:22:00.079 --> 00:22:04.960
And I go, oh, the fact that I'm I'm twitching to use one of my anything.
00:22:05.200 --> 00:22:09.119
Anything to help me out to get it done faster, quicker, in theory, better.
00:22:09.279 --> 00:22:11.039
So I yeah.
00:22:11.359 --> 00:22:13.359
It it is a it is a concern.
00:22:13.599 --> 00:22:16.079
And our students, clearly, they feel that.
00:22:16.640 --> 00:22:19.039
Let me take us through a bit of a pivot.
00:22:19.119 --> 00:22:24.720
Yeah, because I mean, I I I hear the concern, but let's talk about what people are actually trying to improve this entire thing.
00:22:24.799 --> 00:22:30.640
Because we don't want to just speak problems and leave you there, because there are real structural uh attempts to address this.
00:22:30.720 --> 00:22:38.000
So I want to call out directly Western Sydney University's IA matrix to inspire and assure as two axes.
00:22:38.160 --> 00:22:40.000
The insight is deceptively simple.
00:22:40.160 --> 00:22:41.680
Start with stopping, right?
00:22:41.839 --> 00:22:46.880
Identify the low-value tasks first and eliminate them before adding anything new.
00:22:47.119 --> 00:22:48.960
Curtains Assessment 2030 program.
00:22:49.279 --> 00:22:54.799
So you cited uh Molly Dollinger earlier, they've built around the two-lane model that first arose at the University of Sydney.
00:22:54.880 --> 00:22:57.279
So that's Danny Liu and Adam Bridgeman and the like.
00:22:57.440 --> 00:23:01.680
Lane one, secure supervised assessment that verifies learning occurred.
00:23:01.839 --> 00:23:07.839
Lane two, open assessments that prepare students to operate as professionals in an AI-enabled world, right?
00:23:08.000 --> 00:23:12.160
The key question for lane two isn't did they write every word themselves?
00:23:12.319 --> 00:23:20.720
It's did they demonstrate the disciplinary knowledge to work with the tool, to judge what good looks like, and produce something they can explain and defend.
00:23:20.880 --> 00:23:25.200
And that maps onto what professional life actually looks like, I would argue.
00:23:25.519 --> 00:23:33.200
I'm going to link to Texas here a little bit with their three ways of assuring learning or securing assessment, which they came out towards the end of last year.
00:23:33.279 --> 00:23:39.599
And one of the suggestions they had was programmatic assessment is that you don't fix your individual course, you fix the entire degree.
00:23:39.680 --> 00:23:46.240
And across that degree, we redesigned that so that it does have a certain amount of uh defensibility and be able to show you the skills, etc.
00:23:46.400 --> 00:23:46.640
etc.
00:23:48.079 --> 00:23:51.759
That's one bit to park, and but I really liked community practice that I was in recently.
00:23:51.839 --> 00:24:00.559
They framed this really perfectly is that it does support this looking at the pattern of learning across the entire program, not just one particular point of failure.
00:24:00.799 --> 00:24:09.279
I think that's something that I decided to look at regarding, okay, what over its students' life as they come into and do their undergrad, their postgrad of that may look like.
00:24:09.519 --> 00:24:15.279
What's the pattern of learning that they've exhibited across that journey versus just how do they perform with that one particular course?
00:24:15.440 --> 00:24:17.119
And so I think there's a very much an argument.
00:24:17.839 --> 00:24:37.119
Programmatic assessment is on paper really easy, so is said to be really easy, but when you actually get down to it and do it and you're organizing almost 12 different units and more uh to be able to produce a cohesive narrative, it's very challenging, but it does seem to be a really positive answer in this kind of I'd say fight, but um way for us to kind of navigate this challenge.
00:24:37.359 --> 00:24:40.240
I also think that me and Claude went long with the interactive orals thing.
00:24:40.319 --> 00:24:42.319
So what I did is I did a brain dump of relevant things.
00:24:42.480 --> 00:24:51.359
So my suggestion is I do this, and what I'm gonna say is that one of the things that we're really seeing in this entire space is that there is, as you said, no one single answer.
00:24:51.440 --> 00:24:53.440
This is an evolution as the entire thing moves.
00:24:53.680 --> 00:24:56.720
Want to touch on something from Leon Furs and the AI assessment scale.
00:24:56.799 --> 00:24:59.279
There's been a lot of fracass around this.
00:24:59.759 --> 00:25:01.039
Yeah, no, absolutely.
00:25:01.200 --> 00:25:07.440
I mean, the the other thing to call out in this space as well is that it is ever evolving, it is moving.
00:25:07.519 --> 00:25:22.079
And as such, as we move, what was an answer, as you alluded to earlier, what was an answer at one point maybe needs to evolve at a later point in this entire cycle as we move through AI and assessment and everything else as it continues to develop.
00:25:22.240 --> 00:25:35.279
So I want to call out here Leon Furs, so one of the original authors of the wildly popular AI assessment scale, who just published a long reflection on three years of watching frameworks travel.
00:25:35.440 --> 00:25:40.079
So he uses a concept uh from Dylan Williams of lethal mutations.
00:25:40.240 --> 00:25:45.920
So good principles distorted by scale and context, the the AIS, traffic lights.
00:25:46.160 --> 00:25:50.720
And in this, the Tullay model, he has seen this undergo the same process.
00:25:50.960 --> 00:26:08.559
Just to give some context on this, there has been a bit of back and forth, not just with the story that we mentioned at the top, but on the role of traffic lights, and we alluded to this as well in terms of that uh discursive versus structural change piece, um, in terms of what is the best answer for this wicked problem of AI and assessment.
00:26:08.799 --> 00:26:16.000
But Furzer's conclusion is really worth taking a minute with, because he concludes that frameworks are scaffolding.
00:26:16.160 --> 00:26:17.759
They are not permanent structures.
00:26:17.920 --> 00:26:31.039
And his closing line in these reflective pieces, and I'd recommend checking out his blog, is that every framework produced to handle AI in education has been useful for a time and insufficient for what came next.
00:26:31.279 --> 00:26:34.880
Now that isn't failure, that's how the work gets done.
00:26:35.200 --> 00:26:44.480
And the field it really seems to be is maturing as we we go further into this uh period of AI-dominated or at least AI-affected assessment.
00:26:44.720 --> 00:26:52.400
Any comment to that, Dale, in terms of how these frameworks evolve and them as thinking pieces, uh just more broadly on the sector and the approach?
00:26:52.799 --> 00:26:59.599
Frameworks live and die in the actual classroom where they're actually needing to show up in how people are going to play with them.
00:26:59.759 --> 00:27:04.799
And so I think we can spend oodles of time creating some of these, and they are really helpful for a starting point.
00:27:05.279 --> 00:27:19.759
But they do need to be revisited because as soon as you put them into practice, into play, the reality comes through because the thing that works on a PowerPoint deck perhaps doesn't work in a classroom full of 1500 students with very different uh behavioral needs and very different learning needs and learning requirements.
00:27:20.000 --> 00:27:26.799
And so I think a lot of what we're seeing here is these are permissions, these are things, these are guides for people to start with and to take and to perhaps navigate through this.
00:27:26.960 --> 00:27:31.440
But the next step is cool, and how do we support people to actually engage with these and move forward?
00:27:31.519 --> 00:27:36.559
It's that's the bit that's gonna be the most challenging, and that's the bit where the work's starting for that is probably.
00:27:36.640 --> 00:27:38.079
And that work is starting right now.
00:27:38.160 --> 00:27:42.000
And I think some of those use cases, they're gonna be the more powerful ones for us to consider.
00:27:42.319 --> 00:27:50.799
And the good news here is as well that the uh the field is maturing sufficiently that there are now textbooks, that there are now actual um bounded guides coming forward.
00:27:50.880 --> 00:27:58.000
Uh the creative research group, so Margaret Beerman, Tom Scorbin has mentioned, and colleagues, have just submitted a full manuscript to Rootlet, so 19 chapters.
00:27:58.160 --> 00:28:06.960
Their central argument, foundational assessment principles, far from being rendered obsolete by AI, provide essential frameworks for navigating this disruption.
00:28:07.119 --> 00:28:14.000
And when serious scholars like these are writing that sentence, it means the panic is starting to give way to something more durable.
00:28:14.319 --> 00:28:18.559
But I'm gonna wrap up with a nod back to the Castlere statement, right?
00:28:18.640 --> 00:28:22.079
So this is that cross-call uh cross-sector call of action that you have.
00:28:22.319 --> 00:28:27.359
Unsigned, sadly, but maybe maybe we can arrange like a signing event for you.
00:28:27.920 --> 00:28:35.920
The diagnosis of the sector, their diagnosis that Australia's response to date has been fragmented, stored, and at odds with national plan.
00:28:36.160 --> 00:28:39.839
So not a lack of expertise, but a failure of coordination and collective courage.
00:28:40.000 --> 00:28:40.720
I love that.
00:28:40.880 --> 00:28:51.039
And I love the the outflowing of energy that that has come from this and and and the sharing, the open-handedness of the sector and attempting to address this challenge together.
00:28:51.279 --> 00:28:56.000
But you know, the really interesting part for me, I don't know if you saw this, and I'm interested to get your take.
00:28:56.240 --> 00:28:59.440
They built in a contingency clause into the document.
00:28:59.680 --> 00:29:15.440
If certain trigger signals arrive, so labor market shifts, AI capability leaps, employers moving away from formal qualifications, the sector commits to shortcutting lengthening gov lengthy governance processes and institutional inertia.
00:29:16.079 --> 00:29:16.799
And nurture.
00:29:17.200 --> 00:29:22.240
The sector commits to shortcutting lengthy government processes and institutional inertia.
00:29:22.480 --> 00:29:26.000
They put an escape hatch and a policy document.
00:29:26.160 --> 00:29:33.039
And that tells you something interesting about how much time they think we have and the urgency they think we should be moving with.
00:29:33.359 --> 00:29:34.559
Plenty to think about, Nick.
00:29:34.640 --> 00:29:36.160
But everyone, thank you so much for listening.
00:29:36.240 --> 00:29:40.880
If you haven't already, please leave us review, share around within your network, and most of all, hit subscribe, whatever platform you're at.
00:29:40.960 --> 00:29:45.759
For those watching on YouTube, please note we are on all your favourite podcast platforms too.
00:29:45.839 --> 00:29:48.559
And those who love to see our faces, you can find us on YouTube too.
00:29:48.720 --> 00:29:52.319
We would love to tell you what's on next week's episode, but it's a world that moves way too fast.
00:29:52.400 --> 00:29:53.359
So you'll just have to tune in.
00:29:53.440 --> 00:29:56.799
Until then, stay curious, stay intelligent, and get your customer statement.
