O TYM ODCINKU
A conversation between Harsh Kumar and Bart Everson on creativity and artificial intelligence.
Harsh Kumar is a fourth-year PhD student in the Department of Computer Science at the University of Toronto. His research focuses on developing algorithms and systems for social good, particularly in cognition, mental health, and education. Recent experiments have focused on problems related to AI and education.
Links for this episode:
- Kumar, H., Vincentius, J., Jordan, E., & Anderson, A. (2024). Human Creativity in the Age of LLMs: Randomized Experiments on Divergent and Convergent Thinking (arXiv:2410.03703). arXiv. https://doi.org/10.48550/arXiv.2410.03703
- One of the popular TikToks on the above paper
- The math education paper mentioned during our talk
- Another paper on guiding students to use LLMs, which educators might find useful
Transcript:
Bart Everson
I'm Bart Everson, your host for this episode of Teaching, Learning, and Everything Else. I'm delighted to be with Harsh Kumar today. Just let me introduce him real quickly. He's a fourth-year PhD student in the Department of Computer Science at the University of Toronto, and his research focuses on developing algorithms and systems for social good, particularly in cognition, mental health, and education, and some of his current work caught my attention because it focused on an issue that's near and dear to my heart, which is creativity. So we saw this paper. "Human Creativity in the Age of LLMs," which, as you may know, that's artificial intelligence, Large Language Models. And I just wanted to ask you, Harsh — well, first of all, thanks for joining us. What prompted you to do research in this particular area, the impact of large language models on human creativity?
Harsh Kumar
Thanks, Bart, for the introduction. I think just to start us off, I think the inspiration for this came just from my own use of these models, like ChatGPT and like other large language models, tools. Like I was realizing that it was allowing me to do a lot of things very fast compared to before, but it felt like it was also changing the way I was doing these things on my own when I didn't have these tools later on. Like, let's say, if it's about brainstorming, like, I realized I would be able to come up with ideas very well with ChatGPT, but I felt like I wasn't able to do that as well as before with other humans, like, when I didn't have this experience with ChatGPT. And I felt like people around me were having similar experiences. And this is not new. It has been there with spell-checkers, with calculators and other tools that came out. But it just felt like with tools like large language models, they are much more general than anything that we have had before. And it felt like it was important to understand, like, what are the impacts they are having on the way we do these things in a world where these tools didn't exist? What we found was like, a lot of the work within computer science, human computer interaction, AI, was mostly focusing on making the model very good at creative tasks without accounting for, like, what effect it is leaving on the users who are using this. So that's where we started thinking about, like, can we design, in a very controlled setting, a way to measure these kinds of effects?
Bart Everson
Yeah, that's so interesting. And your research focused on the impact of these large language models on two key aspects of creativity, divergent and convergent thinking. Those are things I've heard of certainly, but you know, some of our listeners might not know exactly what that means. Can you explain the difference between these two concepts and why that's crucial for understanding creativity?
Harsh Kumar
Yeah, for sure. So, yeah. So I think one of the challenges with creativity is around, like, measurement of creativity for any task. The problem is that a lot of creativity happens not just actively, but also kind of passively in the background. But just based on, like, the many theories that exist for creativity, you can think of two main processes that are part of the creative process. The first one is the process of coming up with ideas. And in this stage, you don't really think about how good each of the ideas is, but just thinking about what are the different ways something can be done? So that accounts for the divergent thinking part of this creativity process. And on the other side of the spectrum could be once you have these of pool of ideas from the first round, how do you pick? How do you, like, rationalize which one is good? Which one or two ideas are worth trying out or implementing for? So that's where you converge into something smaller than what you started out with. So that's the convergent thinking aspect of it. And it seemed like these two, like focusing on these two separately can allow us to narrow down a bit more around understanding how creativity is affected, while using large language models, or in general.
Bart Everson
Yeah, and measuring that does seem like a challenge. You used two tests. I guess they're considered classic tests. The Alternative Uses Test and the Remote Associates Test. Could you tell us a little bit about what these tests are?
Harsh Kumar
Yeah, for sure. So for the divergent thinking task, we use this test called Alternate Users Test. So in this particular test, the task is that you are giving a given an everyday object, something like a shoe, let's say. And then you are asked to come up with as many creative uses of a shoe as you can within a certain time limit. And the prompt is that it doesn't have to be like, a very useful way of doing, like, using shoe, but it has to be — the reader should get like, this thing that, okay, well, this is very creative, even if it's not very useful, but this is a creative way, something like, maybe using it as a potted plant, like a holder for a potted plant, or something like that. Similarly, for the convergent thinking task, we use this Remote Associates Test. And there's this very popular New York Times Connections game, which I think a lot of people play, where you're basically given three words which have something in common, like some seemingly common thread that connects them, and you're asked to come up with a fourth word that can fit in with all these three words. And that touches more on the convergent thinking side of things.
Bart Everson
All right. Great. Thanks. So your findings, I guess, indicated that different types of — having AI large language models assist people had different effects on creativity as measured by those tests. Can you elaborate into how you designed that whole thing?
Harsh Kumar
Yea h, I think I can talk about the findings that I think were most exciting for me. So, we tried two different methods of AI interaction. The first one was like a default ChatGPT interaction that people would have where, like we wanted to see how it was being played out in the real world. And we tried to design another treatment, which was supposed to act like more of a coach, in the sense that we thought maybe the response can be framed in a way that even while someone is using it, they do well, but even after they stop using it, they have learned something from that initial process, which makes them do better, and later on, when they don't have this support. And this was inspired by a lot of — so we don't have a lot of research literature on this, but there's a lot of professors and social media influencers, like science communicators, who have been saying that, okay, you can use these tools for learning, different frameworks of thinking. And we wanted to see, can that help in this task. You can think that maybe it might be helpful, because it is not giving you the answer, but it is telling you how to do like, giving you a path to do it, and maybe it will teach you how to do it better later on. But for creativity, you can also think that it might lead to homogenization of ideas, because if everyone learns the same framework for a problem, then everyone will lead to the same solution, which can be problematic later on, even when they have stopped using large numbers models. So what we found was so in our experiment setting, we had this first exposure rounds where we gave people a chance to interact with the LLM that was assigned to them and do the task, both Remote Associates and Alternate Uses tests. And later on, we gave them a distractor task just to make them forget a little bit of what they had done, to simulate the real world gap between actual exposure and test. And finally, we had another set of test rounds where none of the participants had any LLM anymore, but we wanted to see what are the residual effects from the initial exposure rounds on the person's way of doing this task. And we got signals showing that people who never had any LLMs, like the control condition, who were given none of the two, they seem to do seemingly better in later test rounds, even if they didn't do as well in the exposure rounds than the other LLM conditions, which kind of aligned with what I and others have been feeling, that okay these tools on their default way might be hampering the way we think, if we don't use them very carefully.
Bart Everson
So interesting, and I'm just curious to know, I know it must be outside of the actual experiment. But do you have any speculation on what's happening, you know what — what might cause that?
Harsh Kumar
Right. Yeah, I think, great question. So this is something that, like I explored in one of my previous papers, which is called, Math Education in the Age of LLMs. It's like about, more about the learning context. And over there the manipulation that we did was around, when should you give LLMs to people? You can think about giving that to them after they had done the problem on their own, and then you give it to them to improve that problem, like improve their solution. Or you can give it to them right away and then have them solve that problem with the LLM. And we found like later on, when you test people, the first group of people who spent enough cognitive like, enough of their like, cognitive engagement with the task, and then only used it to improve their work, like after fully being immersed in the task, end up doing much better in the task than people who kind of cheat, in some sense, like students who might cheat with LLMs. And I feel like similar mechanisms could be at play here, given that when you are getting readymade solutions, you don't actually engage with the task on your own and develop those cognitive mechanisms that can help you arrive at the solution better, almost similar to, like, doing any kind of exercise, like, the more you do it, your muscle gets better. But if you stopped using those muscles, then you have difficulty, like flexing them out, or like making use of them later on. So something like that.
Bart Everson
It makes sense. It does make sense. On an intuitive level I think it probably makes sense to people, but it's great to hear you explaining that. You know, given your findings, and keeping in mind, you know our audience of faculty members, people who are teaching in higher education, what advice would you give to a faculty member who's thinking about using AI as part of their teaching or research? You know what might be some of the practical advice, or also the ethical considerations, things that they should keep in mind in light of your research?
Harsh Kumar
Yeah, I think great question, Bart. I think this is something that I've been thinking a lot lately, especially since this particular paper on, like, human creativity — so it got viral on TikTok after some people read it. And these are very normal people who are not from research, and they also related to it. And I've been getting like, bunch of messages every day, like, people are saying, OK, I do this kind of work. Tell me, how should I use this so that I don't destroy my own creativity. So I think the first thing is that this fear, I think, is unfounded. It's not — like, the message of the paper isn't that you have to be worried about using ChatGPT on a day to day basis. I personally feel like these are, like, the most magical piece of technology that we have had in a while. I think the main question is about being a bit more mindful and like conscious about the way you are using it, like simple tweaks like this timing of use, like instead of making it do the task for you, maybe spend some more time doing the task on your own, and then use this for feedback. Like these minor tweaks can help out. Like, in terms of making, ensuring that the impact that it has on the user is, like, positive. And just for the teachers themselves, I think they have a role in terms of communicating, I think, even to their students, how to use these tools, because there is, like, no proper, like, formal guidelines that are rigorously experimentally tested about what works and what doesn't work for students. So like, trying to, like, I think the way you're doing with the podcast, like, engage more with the scientific literature around what is the right way to use these tools, and then communicating it to students, I think that will just have, like, lasting benefits for the whole society, because people will learn, like, how to use these tools properly for their day-to-day stuff.
Bart Everson
That is so interesting. Your allusion to the research going viral, kind of going viral. I mean, that is the dream, I think that many of us have, that our research into whatever will be seen as relevant and be picked up, maybe, outside of the hallowed halls of academia. You know, that the people would actually look at and read something in the broader culture is kind of the dream, and that's fantastic to hear that. Because I did not know, you know, I just came came across this because a colleague shared the link with me. But I can see, I can see why this research would capture people's imagination, because it does speak to some deep-seated interests and so forth. Hopefully we can find a way to share in the show notes, some of that virality, or some of that discussion elsewhere, maybe a key TikTok video or something. That'd be fantastic. Well, but let's orient a little bit to the future. I'm sure you're already thinking about future directions for research, and you know, that's where all of our questions seem to just to lead to more questions. Do you have any thoughts on, you know, what the future research directions that you see emerging from your work?
Harsh Kumar
Right? Yeah. Great point. I think — with writing this first paper, we felt that I think it focused more on the problem without giving concrete solution, because I think setting up the problem itself took a lot of effort. So I think one good conversation that I've been having with other researchers is about not just thinking about this period of LLM use and trying to optimize for outcomes during LLM use, but also trying to think of these later stage where you take that LLM away and see what effects are, and trying to make sure that whatever you are designing, whatever tools you are designing, whatever models you are designing, have like a positive long term impact on the user. Like, account for that in the experimental setting. That is, I think, one good outcome that I think, I hope, would come out of this, like, in terms of the way others are conducting research. I think, for me personally, like, I've been thinking a lot about the solution to this, like, given that there could be like, long-term like issues, if like, it's not used carefully, can we do something on the model side that can improve like these effects, and one aspect that we have been focusing on is on human curiosity, which has been shown to link very heavily with creativity. Like, if you are, like, very curious about a topic, you will naturally come up with moreexciting ideas. You would have more energy to converge on something good. And the way we talk to these models, you can imagine talking to one of your colleagues after — there are some people after like, once you have a conversation with them, you just come out beaming with more ideas and excitement about a topic. And we're trying to think, can we think of these models that way, like the responses? And even if the same content has to be communicated to the person, can we figure out experimentally how that content should be shown or designed so that it leads to increased curiosity for the person reading it. And hopefully, maybe that might lead to better outcomes for creativity, which we can try to establish experimentally. But yeah, this link of curiosity and creativity leading to better things. I think that is something that I'm excited about.
Bart Everson
I'm learning so much in this conversation. I mean, it makes sense that those attributes are linked, but I never really thought about it before, or never heard about the research supporting that. So it's all very, very interesting to me, personally. Well, you've been very good responding to my various questions, and I thank you for that. But I'm wondering, is there anything that I've left out or that you were hoping to want to address or to communicate that that maybe I didn't ask about.
Harsh Kumar
No, I think you were, like, pretty spot on with your questions, Bart. But I think just in general, I think, just as a consumer of this podcast, I would love to hear other researchers, if you can get them to talk about this AI literacy, in the sense — I think there's this question about how we should use these tools, and there is, I think, some discourse on like popular media, like social media, channels and places, but I feel like not a lot of them are very rigorously, scientifically backed. And I feel like as researchers, we don't have that energy to maybe publicize it, or talk about it, make videos about it, or make documentaries about it, but I think these podcasts could be very powerful way to communicate those right things. So I hope you keep doing that with other researchers and talking about it with your audience.
Bart Everson
Fantastic. Well, I will be sure to share this podcast through the usual channels and also on my — I'm on Mastodon, so I will post it there and see what responses I might get. I'll go fishing.
Harsh Kumar
Thanks. Sounds great.
Bart Everson
Well, thank you so much, Harsh, for talking with me today. I appreciate all your perspectives, and of course, we will be sharing a direct link to the paper as well as other relevant links in the show notes. So we look forward to having more conversations in the future.
Harsh Kumar
Sounds good. Thanks for inviting me again, Bart.
