Justin: Hey everybody, welcome to Recombinant Teaching. I'm Justin Schaefer, your host, and on this show we're gonna do some QA about STEM education. So if you've been listening over the summer, I took to kind of took a break from the QA format to do the course design series, which I hope you like that. If you haven't listened to it yet, go back and check it out. There's 10 episodes on everything you need to know about designing effective and engaging STEM courses, from starting your objectives all the way to assessments to activities, talk about online, face-to-face, lab, everything. In between. It's a good listen, I hope, if you want to check that out. But now we're back here in the fall. It's fall 26, back in the semester. I'm in week four of my semester, teaching intra-bio right now, having a blast, and I'm going back to the QA format. So I'm excited to welcome our first guest of this fall, Noah Courtney, and I'll have him introduce himself. So go ahead, Noah.
Noah A Courtney: Hey Justin, thanks for having me. my name's Noah Courtney. I'm a second-year PhD student and NSF Graduate Research Fellow at the University of Rhode Island. I am a biology education researcher and I work in Dr. Lisa Lemary's Biology Education Research Lab. my work primarily focuses on leveraging quantitative methods and statistical methods to investigate ways to support community college STEM student persistence. Outside of my research, though, I focus really heavily on mentorship. Over the past few years, I've worked with multiple students in group settings, in one-on-one settings, and I've worked specifically to mentor students in writing applications for the National Science Foundation Graduate Research Fellowship Program.
Justin: Those are awesome things, Noah. You got great goals there with the mentorship and with the research and st C C student persistence and biology. all great things. I'm glad you're doing all that great work. And those of you listening, before we start recording, me and Noah were talking about how he's known as the stats guy in his department. So if anyone out there's looking for a statistician collaborator, you know who to hit up now.
Noah A Courtney: I it's grow growing in my own stats knowledge has been a fun part of graduate school and I look forward to continuing collaborating with people doing that kind of work.
Justin: Awesome, awesome. Great. So yeah, so I hope you got a question, we can try to unpack together. So what what you got?
Noah A Courtney: Yeah, so I I've been an avid listener over the past few months, and I heard you mention that you have worked with students in class on writing GRFP proposals. And
Justin: Yeah.
Noah A Courtney: in my work mentoring students primarily one-on-one in writing these proposals, I noticed that it takes a lot of time and and that time is absolutely worth it. But I'm really wondering what are what are some ways that I can scaffold research writing s skills to teach to students and and how can I start to work to scale the mentorship model that I've been developing.
Justin: sweet. Well, I'm glad you're interested in that. And yeah, and you you mentioned that you're doing that with your students and helping them write these GRFPs. So I was gonna actually say, hey, you know, I've done that in a class, but you know, hey, you asked it, so cool. so yeah, a lot of things, a lot of thoughts on this. So you're kind of scaffolding a whether it's research, writing, or just a kind of a bigger project in general, right? So and I I think this falls under the umbrella of what we call authentic assessment. not my term has been around for a while, but I I like to think about And this happened to me a couple years ago, where I felt like, are these multiple choice tests, are these kind of cookbook labs really representative of what are we want our students to be able to do in order to share their knowledge, right? Are they truly assessing what we want them to be able to do in the real world? So that's where this idea of authentic assessment I stumbled across. and I'll give you a link in the show notes for more detail and not for everyone listening. But you know, are these are assessments that are more realistic? These are ways that you can actually show have students do things that In the real world. You know, now, you know, we're doing, we're talking about biology right now, so that could be lab work, right? So, you know, pipe edding, PCR, running gels, you know, analyzing data, those are real lab skills, things you can assess. You know, if we were, you know, doing in a car mechanic school, we'd say, Hey, can you change that tire? Right? That's a truly authentic assessment, Rich, which by the way, no, I I I'm guessing you can change a tire. All
Noah A Courtney: I I can change the tire, yes.
Justin: right, good, sir. That's two of us in this room. Can you drive a stick?
Noah A Courtney: I can. I exclusively drive stick shift vehicles.
Justin: Me too. Awesome. I got a crappy Honda fit. I hope that you have a cooler car with a stick. But I love that you can drive a manual as well. Lost skills, everyone. You gotta keep those authentic skills going. but
Noah A Courtney: Absolutely.
Justin: absolutely good, good. But right, so whether it's something like that or when it comes to the writing, I think you can include authentic assessments. And then in the for those of you who don't know, the National Science Foundation offers this program called the Graduate Research Fellowship Program, GRFP. And this is a program where graduate students can apply or even senior undergraduates can apply. I believe nowadays it's you have two tries for this award and you you write a personal essay but then you also write a research proposal. So it's two pages long, including references, right? It's it's very rigid in format on purpose. You have to have a certain you know I think it's 12 point new times new moment font, you know, certain margins, you know, it's very, very regimented. But this is a real thing that students can apply for and you can earn that award which is amazing, life changing. It's usually much more in terms of a stipend than what you normally get at graduate school. It's prestigious. It can carry you forward in your career, things like that. So back in the day when I was in college in grad school, you know, gosh, 20 years ago now. I'm getting old, Noah. I I I back then we were allowed three tries at this award. And I I tried all three try times and I did fortunately earn it on my third try. So I had some experience as a student with this award. And then thinking about this authentic assessment, I said, okay, can we actually add this to the classroom? Can we have students do this at scale, as you mentioned, and not just one on one? Because it they do take a lot of time, they take a lot of energy to write, and then how do you get feedback on that? So so what I decided to do, and I did this in an introduction to biomedical engineering course, but you could do this in any course, doesn't have to be, 'cause it th NSF GRP GRFP is for any any research discipline really, right? So you could do this in physics, chemistry, biology, geology, computer science. It doesn't really matter. I I believe they even fund education research, right? Or
Noah A Courtney: I'm I'll pause here for a second.
Justin: Yeah.
Noah A Courtney: I'm not sure if you want to keep this in or not. That's up to you. I'm fine with this I'm happy to discuss it.
Justin: Okay. Okay.
Noah A Courtney: but the solicitation came out three weeks ago and they have cut everything related to it. I'm funded on a STEM education GRFP and and they cut a hundred percent of STEM education from it this year. Yeah. I I
Justin: really? Dude.
Noah A Courtney: I wrote a post about it on LinkedIn, you should check it out. It's yeah. It's
Justin: man.
Noah A Courtney: not a great time for STEM education like bioed research or deeper in general.
Justin: I was I was not aware of that. man,
Noah A Courtney: Yeah.
Justin: that's a bummer. I I I know it's sometimes more restrictive when it comes to some of the social sciences and research, but I didn't realize they actually this year cut out STEM education research.
Noah A Courtney: They Yeah, they nixed it completely.
Justin: dude. yeah, let let's let's keep this in and keep it going. And we could put a link to that LinkedIn post in the in the notes too, if that's okay with you.
Noah A Courtney: Yeah, absolutely.
Justin: Yeah, no, I can I can I'm looking no you guys can't see it listening, but his his face looks concerned. I is I could tell this has impacted you personally now.
Noah A Courtney: My my lab mate and I both both have STEM education graduate research fellowships and and there are so many awesome young bioed researchers and and and students in Deber and undergraduate researchers participating that that it really is a a shame that that they are are cutting off a a source of of possible funding for some of these students who want to pursue graduate school.
Justin: Damn, yeah, that that really sucks. Yeah, they're the the choices some of these agencies have made with limiting funding in recent years has not been good. yeah, my my postdoc program was one of the programs that were cut by the NIH. last year so I was in these programs called IRACTA IRA C D A and they funded postdoc training programs for people who wanted to go into teaching careers in STEM and specifically in biomedical sciences and I wouldn't be here without my program Spire which was at Carolina and that was cut last year. Supposedly some of that funding's come back at least to let them finish their grants and the current trainees but I don't know about the longer term future of those programs. So yeah it's unfortunate on a lot of levels. So yeah no but that's yeah good sidebar for every One listen, I think, if you didn't know about those programs and some of those agency changes that happened, bummer bummer bummer across the board. But If you have a students or courses that are want to be in, I guess, the fund the areas that are still funded, you know, right, unfortunately by these programs, such as biomedical sciences, such as, you know, whether it's chemistry or physics or engineering research, they can still apply for these for real. But again, even if you don't want to do this for real, you can do this as an assessment in your course. And that's what we're trying to talk about now with this scaling this up. So I did this in a class with 60 students, and they all had to write that two-page proposal. we didn't do the personal. site part which did two-page proposal and so we did this over the entire semester and I had lots of deliverables along the way so I think that's the first piece of advice I'd say for anyone trying to do something like this. Whether it's with 10 students, 60 or 150, whatever, just break it up into smaller chunks. Because you can't just say hey write this two-page research proposal. That's a big ask of of anybody, whether you're a you know introductory student, senior student, graduate student, that's something you might not have done before. So break it up. So what did I do? the first assignment was what's your research question? One sentence. Was your requirement for that project or that assignment, right? They had to submit a one-sentence research question. the next one was a one-sentence hypothesis that tests their question, right? So really tiny baby steps. With both of those, I they did peer feedback on Canvas, anonymous peer feedback on their question and on their on their hypothesis. I gave them feedback as well. Okay, then the next step was what is your what are little research review, a little lit review, right? So what are some papers about the project that related to your topic and your question that you're reading, you know, so a couple of sent you know, a couple of paragraphs there. Then they had to do a little experimental design plan. And then by the time we got to that point, they were kind of had a f a first round of a draft together. So so again, it was lots of deliverab deliverables along the way, a lot of peer feedback online along the way, peer feedback for me. But here here's the the kicker for me, which I wrote I wrote an article about this on LinkedIn too, and I'll put it in the notes, but I did this in biomedical engineering and I had such a wide variety of topics, right? I had I had gold nanoparticles for cancer treatment, I had prosthetic limbs, I had genetic engineering of plants for a new food source. I don't know anything about all these topics, right? So how am I gonna give feedback on it to help them write it better is the question. I can give them feedback on kind of the structure of the of the proposal and kind of the logical flow, right? Of you know, you go from you know very broad and more specific to your question, to your methods, specific aims, stuff like that. But so I I turned to AI and this was spring of twenty-four, I think. I can't remember, twenty three, yeah, spring twenty twenty-four, I think. so it was, you know, GPT four maybe at the time. Or even three five still. And I just told them, hey, I'm gonna use AI to help give you feedback. And I did. So I took the proposals, no names on them or anything, and I just copied and pasted the text into to LLM. And I said, Hey, can you give me feedback on this? And you know, within seconds, you get 10 bullet points of feedback. And so I copied that feedback and put it into the comment box on Canvas, and I said, Here's your feedback. By the way, this is not from me. I told it's very clear, right? This from AI, and that's what I did. And so I was really nervous about this. Felt really good. guilty about it. You know, I was raised Catholic, so I'm guilty about everything, but including this one. And but I went to class one day and I after I did this I said, hey, for your first draft, I use AF feedback. What do you all think? You read your feedback. And I I did a clicker question and 95% of the students said it was both useful and accurate. So I thought, okay, let's let's keep going there. So there's a second draft. I did it again. And this time I used Gemini just for a different flair, you know, see how that would work. but in the end, the the best way to do feedback and scale for this, Noah, that I can say is right, you kind of use this triangle. You have you know instructor feedback, you have the AI feedback, but then you have the peer feedback. And it's not just the peer feedback of the comment box on Canvas, it's sitting down in the classroom, printing. staring eye to eye. Okay. And so with sixty students, what we did was I during a certain we did this twice throughout the semester. I I burned two whole days of class for this, which was worth it. They brought in three copies of their proposal and that each time and they would sit down for about fifteen minutes with a peer. They would trade first, five minutes, read through it, make comments, and then talk through it. Then we'd rotate. So they got three rounds of peer review live in three different in two different class sessions. So six total rounds. At the end of the semester I asked the students what what's which type of feedback was best. it was like 65% or so said that that peer peer feedback face to face. And I asked them why, and they said, well, because we actually have to talk through it. So so you can do that kind of feedback to scale this up, right? You can use the AI, you can use the peer feedback. That way you're you yourself are not spending hours on each individual proposal, which I'm assuming you're doing with your mentees right now. You're spending lots of time with them, which is amazing that you can do that. But on a classroom scale, you can't really do that, right?
Noah A Courtney: Yeah, it's pretty that's pretty challenging. I am very curious how you see students' research thinking develop. They're they're through doing literature review, how are they thinking and growing in their thinking about framing a research question and arguing that this is a really important research question.
Justin: That's a great Question to ask, and I don't know if I have a great answer to that though, because I personally have a hard time teaching writing and writing development, especially that critical thinking, like you're talking about. I don't know if I was ever taught it well myself. I know when I was in grad school, my mentor always told me how hard it is to teach writing of proposals and things, and maybe I've just carried that ethos along. but I think the the the what I what I try to do with them at least is you know ask them you know it does is this a logical argument and not necessarily is it valuable or not but you know would would people care about it there's actually an acronym that I found in the biomedical liter literature called FINER that helps people kind of guide the development of their research questions. so it's let me see if I can run off the top of my head. I'll link to it in the notes too but F is for feasible I is for I think I'm putting a blank on that one.
Noah A Courtney: Ha ha.
Justin: N is for novel. E is for ethical, R is for relevant. I know that I'm I'm forgetting the I. But basically it's a way to help you kind of guide your idea, you know. You know, if it hits these criteria, it's it's a pretty solid idea, you know. Is it kind of the kind of and yeah, maybe the I will come to me, but yeah, that's kind of the idea there, right? So you can use you kind of test the you know, you're helping students, giving them kind of a rubric like that to say, you know, does it check some? Of these boxes so they can do that their own. And then when we review each other's proposals or when I review theirs, I'm keeping this in mind to say, okay, how is that coming along? But as they develop over time, that's the hard part too, right? So you get all this feedback, whether it's from the peer, the AI, or you as the instructor, how do you help them incorporate that feedback? How do you help them make that proposal better? you know, those are all hard things to do, especially at scale, because you don't have a lot of time to spend on each individual one. But even if you did, like I said, I don't know all the the technical details of these. So if you it's kind of hard for me to say like well is this is this actually a logical experiment or not? So I don't know. So what what do you and you're helping your students, I'm assuming, maybe on that on the bio ed side, so maybe this is a area of expertise for you, but ev or if it's not, you know, how are you helping them develop those skill sets?
Noah A Courtney: I've worked with students in who are writing proposals in the bioed research field, last year at least. And I've also worked with with a few students, some awesome students interested in in plant genetics and interested in in questions about ecological questions. So it it has been hard and I've found that a strong collaboration with not only between myself and and the mentee. but also with their graduate PI. I've been fortunate where where those kinds of meetings have gotten to happen. So we do bring a subject matter expert into the room who knows that literature base. So that has supported that portion of the writing process. But again, like you said, walking into a classroom with 25 students might mean 25 very unique interests. So so that's really good advice on on scaling that up.
Justin: Yeah.
Noah A Courtney: but it's definitely a challenge to work on it outside of like it's outside of my field. So so supporting
Justin: Yes.
Noah A Courtney: those students has been a a challenge.
Justin: Yeah, and I think, you know, if you view it too in a classroom setting of it, y yes, it's an authentic real thing they could do, but it's still as an exercise. Like you're this is not something they most likely would submit. although I do have a story, one of my former students here at Mines, she's now a PhD student in at Vanderbilt University in a molecular biosciences program. She was in this class with me, wrote the kind of dummy sample proposal as a undergrad student, and then she applied for the real thing then as a senior. didn't get it, but her last second and last chance as a graduate student, she told me, you know, like, hey, I got that first round of practice with you in the classroom, so I kind of got a feeling for what this is like. Then I did it for real on my own, and then I got to grad school and had more support and I got it. So, you know, like I I'm not saying I'm not trying to take credit for her her success here, but just the fact that you're trying it out as a student, you're giving a crack at this type of real writing assignment that could prove dividends down the line. Whether you write the real thing or you end up writing other proposals or something. Else. Another thing I thought about trying, I haven't pulled the trigger on this one, but is having in another, I was going to do this anatomy and physiology, but have students write the basically a cover page, the one page cover page for an RL1, an NIH proposal, where they have to come up with two specific aims, they have to make a figure that summarizes what their entire project is, right? So another kind of form of that authentic assessment, which would be super, I think, even more challenging than the NSFGRFP, but it could be another fun idea for an authentic assessment.
Noah A Courtney: Absolutely. My my intro genetics class in community college did did something similar where
Justin: cool.
Noah A Courtney: the instructor had us practice science reading and science writing and and and this really authentic assessment you're talking about in reading and choosing a researcher, reading three of their papers throughout the course of a semester and at the end of the semester write that professor a letter asking them for a job. So that kind of authentic assessment is something that really helped push me into science. So I think that that these opportunities are really good, especially f to get students reading and writing early.
Justin: that's sweet. I like I love that. I love that project that your professor had you do. and I bet that gave you some you know, experience again you could carry forward with you too. And then you know last thing I'll say on this too is that, you know, I mentioned AI earlier for feedback on, you know, what I did. I just kind of copied and pasted and put feedback in. But nowadays with especially Clyde, I mean, man, if you can help train your students on how to use that effectively to help them with their writing development and their research logic and or anything, right? You know. you know that you can give them prompts to tell you know have a Claude act like a you know seasoned professor in this area and they're trying to help you develop your proposal and you're going to give it your proposal and you want feedback you know from that point of view and man it can give incredible feedback and ask it asks great follow-up questions and will really help you develop your writing. So it's you know not not having it do it for you of course but having it act like a peer especially if it's in a you know a research area that you you don't might not have that subject matter expert like you had there to help you with in your project. So I think the more we can rely on AI tools like again, Claude's my favorite, I think is the best of the bunch right now, that can be really helpful for helping anyone, including myself, you know, develop my own writing skills.
Noah A Courtney: Absolutely. And students students are using those tools. they're using those tools every day. And and showing them the best ways to use that as an effective tool rather than asking them to write for them is definitely
Justin: Yes.
Noah A Courtney: a a point of emphasis that I I've been trying to make. It sounds like you have as well.
Justin: yeah. Yep. Yeah, that's one of my three I have kind of three big pillars of AI and education. Mindset, karma, and value. And the the mindset one is yeah, kind of modeling how to use these tools effectively and not ignoring it of the little devil on your shoulder saying, I'll do it for you. Let me do it for you, right? So how do how do you model that? And so yeah, if you can use it, have students learn how to use it in that way, that's gonna only help them become more creative and better writers and and better communicators. but again, you gotta ignore the temptation to have it do it for you for sure. So
Noah A Courtney: Absolutely.
Justin: yeah, and we could have a whole other conversation. Just on that too. But all right. Well, thank you, Noah, for being here. This was really, really fun. chat about all sorts of stuff.
Noah A Courtney: Yeah, thanks for having me.
Justin: Yeah, and thanks to all of you for listening, everybody. So that's it for this episode of Recombinant Teaching. If you want to learn more about Noah and his work, we'll put some notes in the show notes as well as some of those links we mentioned. I'll try to remember what that I stands for and finer, but I'll put that link in there too. And yeah, please check out my website too at Recombinant Education for everything that I do. If you ever need me to help you out with your courses and your campus and your faculty and trainees, I'd love to work with you. So thanks for listening, everybody, and happy teaching.
Noah A Courtney: Hey Justin, thanks for having me. my name's Noah Courtney. I'm a second-year PhD student and NSF Graduate Research Fellow at the University of Rhode Island. I am a biology education researcher and I work in Dr. Lisa Lemary's Biology Education Research Lab. my work primarily focuses on leveraging quantitative methods and statistical methods to investigate ways to support community college STEM student persistence. Outside of my research, though, I focus really heavily on mentorship. Over the past few years, I've worked with multiple students in group settings, in one-on-one settings, and I've worked specifically to mentor students in writing applications for the National Science Foundation Graduate Research Fellowship Program.
Justin: Those are awesome things, Noah. You got great goals there with the mentorship and with the research and st C C student persistence and biology. all great things. I'm glad you're doing all that great work. And those of you listening, before we start recording, me and Noah were talking about how he's known as the stats guy in his department. So if anyone out there's looking for a statistician collaborator, you know who to hit up now.
Noah A Courtney: I it's grow growing in my own stats knowledge has been a fun part of graduate school and I look forward to continuing collaborating with people doing that kind of work.
Justin: Awesome, awesome. Great. So yeah, so I hope you got a question, we can try to unpack together. So what what you got?
Noah A Courtney: Yeah, so I I've been an avid listener over the past few months, and I heard you mention that you have worked with students in class on writing GRFP proposals. And
Justin: Yeah.
Noah A Courtney: in my work mentoring students primarily one-on-one in writing these proposals, I noticed that it takes a lot of time and and that time is absolutely worth it. But I'm really wondering what are what are some ways that I can scaffold research writing s skills to teach to students and and how can I start to work to scale the mentorship model that I've been developing.
Justin: sweet. Well, I'm glad you're interested in that. And yeah, and you you mentioned that you're doing that with your students and helping them write these GRFPs. So I was gonna actually say, hey, you know, I've done that in a class, but you know, hey, you asked it, so cool. so yeah, a lot of things, a lot of thoughts on this. So you're kind of scaffolding a whether it's research, writing, or just a kind of a bigger project in general, right? So and I I think this falls under the umbrella of what we call authentic assessment. not my term has been around for a while, but I I like to think about And this happened to me a couple years ago, where I felt like, are these multiple choice tests, are these kind of cookbook labs really representative of what are we want our students to be able to do in order to share their knowledge, right? Are they truly assessing what we want them to be able to do in the real world? So that's where this idea of authentic assessment I stumbled across. and I'll give you a link in the show notes for more detail and not for everyone listening. But you know, are these are assessments that are more realistic? These are ways that you can actually show have students do things that In the real world. You know, now, you know, we're doing, we're talking about biology right now, so that could be lab work, right? So, you know, pipe edding, PCR, running gels, you know, analyzing data, those are real lab skills, things you can assess. You know, if we were, you know, doing in a car mechanic school, we'd say, Hey, can you change that tire? Right? That's a truly authentic assessment, Rich, which by the way, no, I I I'm guessing you can change a tire. All
Noah A Courtney: I I can change the tire, yes.
Justin: right, good, sir. That's two of us in this room. Can you drive a stick?
Noah A Courtney: I can. I exclusively drive stick shift vehicles.
Justin: Me too. Awesome. I got a crappy Honda fit. I hope that you have a cooler car with a stick. But I love that you can drive a manual as well. Lost skills, everyone. You gotta keep those authentic skills going. but
Noah A Courtney: Absolutely.
Justin: absolutely good, good. But right, so whether it's something like that or when it comes to the writing, I think you can include authentic assessments. And then in the for those of you who don't know, the National Science Foundation offers this program called the Graduate Research Fellowship Program, GRFP. And this is a program where graduate students can apply or even senior undergraduates can apply. I believe nowadays it's you have two tries for this award and you you write a personal essay but then you also write a research proposal. So it's two pages long, including references, right? It's it's very rigid in format on purpose. You have to have a certain you know I think it's 12 point new times new moment font, you know, certain margins, you know, it's very, very regimented. But this is a real thing that students can apply for and you can earn that award which is amazing, life changing. It's usually much more in terms of a stipend than what you normally get at graduate school. It's prestigious. It can carry you forward in your career, things like that. So back in the day when I was in college in grad school, you know, gosh, 20 years ago now. I'm getting old, Noah. I I I back then we were allowed three tries at this award. And I I tried all three try times and I did fortunately earn it on my third try. So I had some experience as a student with this award. And then thinking about this authentic assessment, I said, okay, can we actually add this to the classroom? Can we have students do this at scale, as you mentioned, and not just one on one? Because it they do take a lot of time, they take a lot of energy to write, and then how do you get feedback on that? So so what I decided to do, and I did this in an introduction to biomedical engineering course, but you could do this in any course, doesn't have to be, 'cause it th NSF GRP GRFP is for any any research discipline really, right? So you could do this in physics, chemistry, biology, geology, computer science. It doesn't really matter. I I believe they even fund education research, right? Or
Noah A Courtney: I'm I'll pause here for a second.
Justin: Yeah.
Noah A Courtney: I'm not sure if you want to keep this in or not. That's up to you. I'm fine with this I'm happy to discuss it.
Justin: Okay. Okay.
Noah A Courtney: but the solicitation came out three weeks ago and they have cut everything related to it. I'm funded on a STEM education GRFP and and they cut a hundred percent of STEM education from it this year. Yeah. I I
Justin: really? Dude.
Noah A Courtney: I wrote a post about it on LinkedIn, you should check it out. It's yeah. It's
Justin: man.
Noah A Courtney: not a great time for STEM education like bioed research or deeper in general.
Justin: I was I was not aware of that. man,
Noah A Courtney: Yeah.
Justin: that's a bummer. I I I know it's sometimes more restrictive when it comes to some of the social sciences and research, but I didn't realize they actually this year cut out STEM education research.
Noah A Courtney: They Yeah, they nixed it completely.
Justin: dude. yeah, let let's let's keep this in and keep it going. And we could put a link to that LinkedIn post in the in the notes too, if that's okay with you.
Noah A Courtney: Yeah, absolutely.
Justin: Yeah, no, I can I can I'm looking no you guys can't see it listening, but his his face looks concerned. I is I could tell this has impacted you personally now.
Noah A Courtney: My my lab mate and I both both have STEM education graduate research fellowships and and there are so many awesome young bioed researchers and and and students in Deber and undergraduate researchers participating that that it really is a a shame that that they are are cutting off a a source of of possible funding for some of these students who want to pursue graduate school.
Justin: Damn, yeah, that that really sucks. Yeah, they're the the choices some of these agencies have made with limiting funding in recent years has not been good. yeah, my my postdoc program was one of the programs that were cut by the NIH. last year so I was in these programs called IRACTA IRA C D A and they funded postdoc training programs for people who wanted to go into teaching careers in STEM and specifically in biomedical sciences and I wouldn't be here without my program Spire which was at Carolina and that was cut last year. Supposedly some of that funding's come back at least to let them finish their grants and the current trainees but I don't know about the longer term future of those programs. So yeah it's unfortunate on a lot of levels. So yeah no but that's yeah good sidebar for every One listen, I think, if you didn't know about those programs and some of those agency changes that happened, bummer bummer bummer across the board. But If you have a students or courses that are want to be in, I guess, the fund the areas that are still funded, you know, right, unfortunately by these programs, such as biomedical sciences, such as, you know, whether it's chemistry or physics or engineering research, they can still apply for these for real. But again, even if you don't want to do this for real, you can do this as an assessment in your course. And that's what we're trying to talk about now with this scaling this up. So I did this in a class with 60 students, and they all had to write that two-page proposal. we didn't do the personal. site part which did two-page proposal and so we did this over the entire semester and I had lots of deliverables along the way so I think that's the first piece of advice I'd say for anyone trying to do something like this. Whether it's with 10 students, 60 or 150, whatever, just break it up into smaller chunks. Because you can't just say hey write this two-page research proposal. That's a big ask of of anybody, whether you're a you know introductory student, senior student, graduate student, that's something you might not have done before. So break it up. So what did I do? the first assignment was what's your research question? One sentence. Was your requirement for that project or that assignment, right? They had to submit a one-sentence research question. the next one was a one-sentence hypothesis that tests their question, right? So really tiny baby steps. With both of those, I they did peer feedback on Canvas, anonymous peer feedback on their question and on their on their hypothesis. I gave them feedback as well. Okay, then the next step was what is your what are little research review, a little lit review, right? So what are some papers about the project that related to your topic and your question that you're reading, you know, so a couple of sent you know, a couple of paragraphs there. Then they had to do a little experimental design plan. And then by the time we got to that point, they were kind of had a f a first round of a draft together. So so again, it was lots of deliverab deliverables along the way, a lot of peer feedback online along the way, peer feedback for me. But here here's the the kicker for me, which I wrote I wrote an article about this on LinkedIn too, and I'll put it in the notes, but I did this in biomedical engineering and I had such a wide variety of topics, right? I had I had gold nanoparticles for cancer treatment, I had prosthetic limbs, I had genetic engineering of plants for a new food source. I don't know anything about all these topics, right? So how am I gonna give feedback on it to help them write it better is the question. I can give them feedback on kind of the structure of the of the proposal and kind of the logical flow, right? Of you know, you go from you know very broad and more specific to your question, to your methods, specific aims, stuff like that. But so I I turned to AI and this was spring of twenty-four, I think. I can't remember, twenty three, yeah, spring twenty twenty-four, I think. so it was, you know, GPT four maybe at the time. Or even three five still. And I just told them, hey, I'm gonna use AI to help give you feedback. And I did. So I took the proposals, no names on them or anything, and I just copied and pasted the text into to LLM. And I said, Hey, can you give me feedback on this? And you know, within seconds, you get 10 bullet points of feedback. And so I copied that feedback and put it into the comment box on Canvas, and I said, Here's your feedback. By the way, this is not from me. I told it's very clear, right? This from AI, and that's what I did. And so I was really nervous about this. Felt really good. guilty about it. You know, I was raised Catholic, so I'm guilty about everything, but including this one. And but I went to class one day and I after I did this I said, hey, for your first draft, I use AF feedback. What do you all think? You read your feedback. And I I did a clicker question and 95% of the students said it was both useful and accurate. So I thought, okay, let's let's keep going there. So there's a second draft. I did it again. And this time I used Gemini just for a different flair, you know, see how that would work. but in the end, the the best way to do feedback and scale for this, Noah, that I can say is right, you kind of use this triangle. You have you know instructor feedback, you have the AI feedback, but then you have the peer feedback. And it's not just the peer feedback of the comment box on Canvas, it's sitting down in the classroom, printing. staring eye to eye. Okay. And so with sixty students, what we did was I during a certain we did this twice throughout the semester. I I burned two whole days of class for this, which was worth it. They brought in three copies of their proposal and that each time and they would sit down for about fifteen minutes with a peer. They would trade first, five minutes, read through it, make comments, and then talk through it. Then we'd rotate. So they got three rounds of peer review live in three different in two different class sessions. So six total rounds. At the end of the semester I asked the students what what's which type of feedback was best. it was like 65% or so said that that peer peer feedback face to face. And I asked them why, and they said, well, because we actually have to talk through it. So so you can do that kind of feedback to scale this up, right? You can use the AI, you can use the peer feedback. That way you're you yourself are not spending hours on each individual proposal, which I'm assuming you're doing with your mentees right now. You're spending lots of time with them, which is amazing that you can do that. But on a classroom scale, you can't really do that, right?
Noah A Courtney: Yeah, it's pretty that's pretty challenging. I am very curious how you see students' research thinking develop. They're they're through doing literature review, how are they thinking and growing in their thinking about framing a research question and arguing that this is a really important research question.
Justin: That's a great Question to ask, and I don't know if I have a great answer to that though, because I personally have a hard time teaching writing and writing development, especially that critical thinking, like you're talking about. I don't know if I was ever taught it well myself. I know when I was in grad school, my mentor always told me how hard it is to teach writing of proposals and things, and maybe I've just carried that ethos along. but I think the the the what I what I try to do with them at least is you know ask them you know it does is this a logical argument and not necessarily is it valuable or not but you know would would people care about it there's actually an acronym that I found in the biomedical liter literature called FINER that helps people kind of guide the development of their research questions. so it's let me see if I can run off the top of my head. I'll link to it in the notes too but F is for feasible I is for I think I'm putting a blank on that one.
Noah A Courtney: Ha ha.
Justin: N is for novel. E is for ethical, R is for relevant. I know that I'm I'm forgetting the I. But basically it's a way to help you kind of guide your idea, you know. You know, if it hits these criteria, it's it's a pretty solid idea, you know. Is it kind of the kind of and yeah, maybe the I will come to me, but yeah, that's kind of the idea there, right? So you can use you kind of test the you know, you're helping students, giving them kind of a rubric like that to say, you know, does it check some? Of these boxes so they can do that their own. And then when we review each other's proposals or when I review theirs, I'm keeping this in mind to say, okay, how is that coming along? But as they develop over time, that's the hard part too, right? So you get all this feedback, whether it's from the peer, the AI, or you as the instructor, how do you help them incorporate that feedback? How do you help them make that proposal better? you know, those are all hard things to do, especially at scale, because you don't have a lot of time to spend on each individual one. But even if you did, like I said, I don't know all the the technical details of these. So if you it's kind of hard for me to say like well is this is this actually a logical experiment or not? So I don't know. So what what do you and you're helping your students, I'm assuming, maybe on that on the bio ed side, so maybe this is a area of expertise for you, but ev or if it's not, you know, how are you helping them develop those skill sets?
Noah A Courtney: I've worked with students in who are writing proposals in the bioed research field, last year at least. And I've also worked with with a few students, some awesome students interested in in plant genetics and interested in in questions about ecological questions. So it it has been hard and I've found that a strong collaboration with not only between myself and and the mentee. but also with their graduate PI. I've been fortunate where where those kinds of meetings have gotten to happen. So we do bring a subject matter expert into the room who knows that literature base. So that has supported that portion of the writing process. But again, like you said, walking into a classroom with 25 students might mean 25 very unique interests. So so that's really good advice on on scaling that up.
Justin: Yeah.
Noah A Courtney: but it's definitely a challenge to work on it outside of like it's outside of my field. So so supporting
Justin: Yes.
Noah A Courtney: those students has been a a challenge.
Justin: Yeah, and I think, you know, if you view it too in a classroom setting of it, y yes, it's an authentic real thing they could do, but it's still as an exercise. Like you're this is not something they most likely would submit. although I do have a story, one of my former students here at Mines, she's now a PhD student in at Vanderbilt University in a molecular biosciences program. She was in this class with me, wrote the kind of dummy sample proposal as a undergrad student, and then she applied for the real thing then as a senior. didn't get it, but her last second and last chance as a graduate student, she told me, you know, like, hey, I got that first round of practice with you in the classroom, so I kind of got a feeling for what this is like. Then I did it for real on my own, and then I got to grad school and had more support and I got it. So, you know, like I I'm not saying I'm not trying to take credit for her her success here, but just the fact that you're trying it out as a student, you're giving a crack at this type of real writing assignment that could prove dividends down the line. Whether you write the real thing or you end up writing other proposals or something. Else. Another thing I thought about trying, I haven't pulled the trigger on this one, but is having in another, I was going to do this anatomy and physiology, but have students write the basically a cover page, the one page cover page for an RL1, an NIH proposal, where they have to come up with two specific aims, they have to make a figure that summarizes what their entire project is, right? So another kind of form of that authentic assessment, which would be super, I think, even more challenging than the NSFGRFP, but it could be another fun idea for an authentic assessment.
Noah A Courtney: Absolutely. My my intro genetics class in community college did did something similar where
Justin: cool.
Noah A Courtney: the instructor had us practice science reading and science writing and and and this really authentic assessment you're talking about in reading and choosing a researcher, reading three of their papers throughout the course of a semester and at the end of the semester write that professor a letter asking them for a job. So that kind of authentic assessment is something that really helped push me into science. So I think that that these opportunities are really good, especially f to get students reading and writing early.
Justin: that's sweet. I like I love that. I love that project that your professor had you do. and I bet that gave you some you know, experience again you could carry forward with you too. And then you know last thing I'll say on this too is that, you know, I mentioned AI earlier for feedback on, you know, what I did. I just kind of copied and pasted and put feedback in. But nowadays with especially Clyde, I mean, man, if you can help train your students on how to use that effectively to help them with their writing development and their research logic and or anything, right? You know. you know that you can give them prompts to tell you know have a Claude act like a you know seasoned professor in this area and they're trying to help you develop your proposal and you're going to give it your proposal and you want feedback you know from that point of view and man it can give incredible feedback and ask it asks great follow-up questions and will really help you develop your writing. So it's you know not not having it do it for you of course but having it act like a peer especially if it's in a you know a research area that you you don't might not have that subject matter expert like you had there to help you with in your project. So I think the more we can rely on AI tools like again, Claude's my favorite, I think is the best of the bunch right now, that can be really helpful for helping anyone, including myself, you know, develop my own writing skills.
Noah A Courtney: Absolutely. And students students are using those tools. they're using those tools every day. And and showing them the best ways to use that as an effective tool rather than asking them to write for them is definitely
Justin: Yes.
Noah A Courtney: a a point of emphasis that I I've been trying to make. It sounds like you have as well.
Justin: yeah. Yep. Yeah, that's one of my three I have kind of three big pillars of AI and education. Mindset, karma, and value. And the the mindset one is yeah, kind of modeling how to use these tools effectively and not ignoring it of the little devil on your shoulder saying, I'll do it for you. Let me do it for you, right? So how do how do you model that? And so yeah, if you can use it, have students learn how to use it in that way, that's gonna only help them become more creative and better writers and and better communicators. but again, you gotta ignore the temptation to have it do it for you for sure. So
Noah A Courtney: Absolutely.
Justin: yeah, and we could have a whole other conversation. Just on that too. But all right. Well, thank you, Noah, for being here. This was really, really fun. chat about all sorts of stuff.
Noah A Courtney: Yeah, thanks for having me.
Justin: Yeah, and thanks to all of you for listening, everybody. So that's it for this episode of Recombinant Teaching. If you want to learn more about Noah and his work, we'll put some notes in the show notes as well as some of those links we mentioned. I'll try to remember what that I stands for and finer, but I'll put that link in there too. And yeah, please check out my website too at Recombinant Education for everything that I do. If you ever need me to help you out with your courses and your campus and your faculty and trainees, I'd love to work with you. So thanks for listening, everybody, and happy teaching.
