TRANSKRİPT 🔗
Dan: Welcome to AI and Design, where we explore how artificial intelligence is reshaping the world of design. I'm Dan Saffer
Nik: And I'm Nik Martelaro, and we're faculty at Carnegie Mellon's Human Computer Interaction Institute. Each week, we break down the latest AI developments, dive deep into topics that matter to designers, and talk with fascinating guests who are right at the intersection of these fields.
Dan: Whether you're a designer working with AI or an AI practitioner interested in design, we're glad you're here. This week, we're talking about
Nik: Anton Sten writes about where design decisions live and provides a cool way to track them as context for AI.
Dan: It's not just design work that's changing in the AI era, product management work is changing too. But will there even be a title such as Product Manager in a few years? Or product designer? Ron Golden thinks the job title product designer is already over.
Nik: and finally, there's lots of talk about junior designers. And so we talked with some recent graduates from the HCII to hear about what their experiences have been like recently.
Dan: But before we get to all that, today is launch day for my debut novel, The Restoration. If you're a fan of this podcast and are interested in a possible future with AI, this book is probably for you.
Nik: Dan, I'm excited, but what is the book about?
Dan: So the book is set 150 years in the future. Over the next 20-ish years, there is an event or really series of events known as the shattering that happens to America, where lots of things go really wrong all around AI and climate change. And there is a period where America basically dissolves. And a couple years after that, the British come and recolonize America. And they recolonize the original 13 colonies and reinstitute a second Pax Britannica where they basically rule America as they did in the eighteenth century. The big difference is that they have a very powerful surveillance AI known as the sovereign. And the sovereign oversees everything. there's lampposts known as the standards that watch people on the streets. There are these loyalty clocks that people have in their homes. And the entire society is set up. To be run by the sovereign and is overseen by this dictatorial AI and the crown, King George VIII. And this is what happens 150 years from today. When a group of people decide that they no longer want to be ruled by King George and this AI and start fostering a rebellion. the book is told from the point of view of ten different characters scattered across British North America. And they all are trying to convene in one way or another. Some of them are crown operatives, some of them are rebels and resistance leaders at a second continental congress that will take place on July 4th, 2176. And so the book really builds to that moment
Nik: Wow. So this definitely sounds like dystopian sci-fi a little bit. but I'm curious, many of the technologies that you're talking about don't actually seem so far off from the perspective of what is in some ways possible with AI and surveillance systems today. I'm curious how have your observations of things that are happening today sort of inspired some of the elements of the book?
Dan: Yeah, this really extrapolates some of the things that we're seeing today and taking them out into a more certainly dystopian kind of world. Now I will say there is also on the resistance side of things, they are also building their own AIs that is different than the sovereign that will help them to fight the sovereign. And that is known as Prometheus. And that is being built in order to provide a different worldview that maybe the AI can provide. And that is a more a more person-centric as opposed to society centric AI. And so when I was writing the book, I was thinking a lot about these different future visions about what AI could be. Is it going to be all held in control by these major corporations, or in my case, the government, or is it going to be a lot of different personalized AI that is optimized for individuals and the clash between the two of those is a major theme in the book. And all that is of course based in what is what's happening now, the debates that are happening now, and I mean, yes, this is not that far away, although the shattering certainly certainly sets things back a bit, but it this is also what happens when a system like this is allowed to run for a long period of time and that people just start to accept that this is the way things are. That being surveilled and having a lot of your life dictated to you by the AI is just how things are. And a lot of the seeds of this are in things that we see, from like cognitive surrender or some of the other research that we've talked about on this podcast, where if you just start handing over pieces of your life to AI, you just get used to that feeling of like, well, they're just gonna handle it. And I guess in a certain sense this is a cautionary tale about what happens when you do that too much, when too much control is handed over to the AI and not to people. but I don't want to make it seem like it's homework. It's actually kind of a fun spy thriller action, lots of different factions playing off one another. Game of Thrones was a big influence on me with all the different. point of view characters and the different factions all vying for control and I hope it's a good fun read for everyone. It's the first book of a trilogy and it's been really fun world building and character building exercise to see how these themes would play out in a world where there's not only this superpowered AI, but also carriages and women wearing corsets because the crown has reverted back to a neo-Victorian 19th century style of of manners and moors as also another means of control. And the combination of those things, like electric carriages and super powered corsets, and lamps that are also surveillance objects. was really fascinating to create this
Load full transcript
very kind of Dickensian world that also has this AI layer on top of it.
Nik: within design practice, it's actually really common for us to try to imagine these potential futures and think about which ones could be possible and which ones are probable, and then which ones are desirable. And I feel like actually I can see some of the influence on the way that we often think as designers on how you're maybe thinking about the book. And I actually think from from the flip side, right. as designers, oftentimes we get so much inspiration from from fiction and actually oftentimes from sort of science fiction as to where things could go both as cautionary and as perspective visions of the future. I'm actually really excited about this duality that you're showing between a central AI systems and maybe more distributed and personalized AI. And I think that I'm quite curious as a designer reading that, where my my thoughts might go and what that might inspire within me.
Dan: Yeah, I have jokingly said, well, this is a just a giant design fiction exercise where I'm trying to play out a number of these threads we often are looking a couple years out into the future, maybe for these things with design fiction. But I was like, well, what if I take it out a hundred and fifty years? What could happen then? there's a lot in there that I think listeners of this podcast will will really enjoy. So pick up the book. It's out now. we'll definitely put a link into the show notes. It is out in ebook. It is out in paperback. And the audiobook should be available within a week or two. So whatever format you want to read it in or listen to it in, it's ready for you.
Nik: Right, So let's talk a little bit about some of the news stories this week. our first story is from an author, Anton Sten, who runs their own design studio and talks a little bit about their process in capturing the decisions that they're making and how they can give those decision files as context to AIs. This is something we've talked about on the show before, if you remember back to the episode where we discussed design ledgers, so capturing design decisions. And Anton talks about this is giving context to AI is important. we have to give it documents. And then honestly, lots of us are trying to give more and more context to it. Things like our Slack messages, our Google document drives, our meeting transcriptions using tools like granola that's recording when we're on a video call. and those are useful to give, however, they're really unstructured, right? And so what's it happened? Is that the AI has to go through and search those things, and maybe it pulls up the right stuff, maybe it doesn't. And so Anton actually talks about a process that he's been developing where he's got two text-based files, two markdown files, running notes and decisions.md. The concept here, running notes, is chronological, contains observations, questions, just sort of a stream of consciousness on things that you might be trying as a designer, what might matter, how your thinking has changed. And it is it's messy. So it's just a chronological thing, but it does have a timestamp. It does have a date to it. And then decisions is a stricter file. So that one is about recording what decision you made, what actually is the thing you're moving forward with? The newest decisions are at the top of the file, so it's at the beginning of it, also timestamped. the decisions could be your own decisions, they could be team decisions. and then these are things that basically are really captured. So, again, this actually harkens back a lot to the design ledgers that we talked about. And really, the idea here is with these two files, this can really give your AI much. more structured context than the messy context that we're trying to give by connecting it to all these other apps and things. And I I thought this was a cool idea. I've been thinking more in my own process about how to capture decisions, how to basically build up memory because I do find that the memory file isn't very good. It doesn't always capture the things I want or it captures things that I did don't want and then I have to go edit it. Whereas actually me taking some time to go in and create a more structured file, especially with the time I really like the idea of saying, the most recent stuff is at the top. I think that this could be a strategy that works well for a lot of designers who are thinking about how to capture their decisions more.
Dan: I wish that I had read this article two months ago because I don't even know if I told you this, Nik, but we assigned our advanced students Who have to put together a design ledger for their last project and for their final big project coming up. I have actually read through about 20 design ledgers over the last couple weeks as I've been grading their work. And I found as a human being trying to figure out what was their running notes versus decisions so I can only imagine that it would be tricky for an AI to do it. And so I think splitting it into these kinds of running notes versus decisions and the timestamps turned out to be extremely crucial for figuring out what was happening when. I think it's just a great idea and in fact as we're starting to do design ledgers for their final project I think I'm gonna immediately ask them to do this to get to have a running notes log and a decisions log. Now I will say one thing that is still a problem with these things is a remembering to do them And well, B, it's it is just it's really just A. It's it's remembering to do them and remembering to update them as things get made. Because you can't always just have logs automatically happen. Maybe you can for things like running notes, but for the decisions, again, maybe, maybe you you'd be able to capture some of those automatically, but remembering to put in a decision or being like, okay, this is the direction we're going to go in. that is definitely the hurdle that some of
my students really bumped into that and because we were graded them on their design ledger actually went back and tried to fill in the design ledger after they'd made all the decisions and after they'd done stuff and I'm like, well that's not really how it should work. but I really like this idea of how do you structure context so it's not just a dump of things.
Nik: potentially we might see in design tooling or just designers developing their own add-ons to working with their agents where you can have something that checks in with you, where the agent is saying, Hey, this seems like a big thing. Can you write down a decision? It doesn't even have to do it for you. It could actually just bring up the moment in which to do it. And that could just be enough to say, Hey, remember to do this. I can also imagine, you know, people within their own prop Process, right? You just have a ritual like at the end of every work session, you make sure you have 10 minutes to do this. that's probably harder said than done, but you know, I have this amazing view of the calm, wise designer sitting there being like, okay, I've got 10 minutes before my next thing. I'm at a stopping point. I'm going to, do my notes and my decision log. and so I think that there's a mix there, right? One is can you develop the d discipline as a designer, or two, can we develop some interfaces and interactions or can, you know, the model companies build it in when they know you're doing design work and they this if this turns out to be really good, like it just comes up and says, Hey, let's let's actually do this, let's facilitate this and the agent works with you on this.
Dan: Right. Okay, our next article was about how product management work is changing in the AI era. And this is an article from Casey Winters, who's the CEO and co-founder of Super Me, which is an AI native professional network. And apparently Super Me interviewed a bunch of product managers ranging from people at startups all the way to public companies trying to figure out how their workflows have changed with the addition of AI. And interesting article, lots of insights in here, a couple ones that jumped out to me. So the first was that Everyone seems to be working harder and shipping more, but what was interesting was how few PMs felt like this dramatic increase in output had produced a similar increase in business results. And I think that is something that we have seen as one of the themes of this show. almost to the beginning of the show, I remember we were talking about like AI buzz and stuff like that where people were just getting burned out on AI because they were being so productive, but The impact of this has not radically changed anything.
Nik: Yeah, I think that this is something that designers everywhere are wrestling with, right? they're just doing more, with less support or staff, in some cases. and yeah, but is is the the outcome Of better products actually happening. I'll say though, I do wonder if
we're getting into a phase where, we might be getting at the top of we might have passed in this regard, maybe peak hype of AI design, and then we might be in a little bit of a trough of people getting a little disillusioned. I know myself, I've actually been thinking more about my own. experience working with AI and I'm I'm incredibly productive, but it doesn't feel great all the time. Not that work has to feel great all the time, but there's something about the process that isn't sitting as well with me and I'm trying to figure out ways in which to change my process. And I I'll say every once in a while though I do I do get something where I'm like, wow, I'm incredibly productive here and it's incredibly exciting. And actually this felt like an effective process that really did lead to a better outcome than I I would have ever been able to have done before. And so I think, i we're probably just getting to that point maybe in the like people are hitting the trough and then hopefully we're gonna start seeing some stuff come out of it.
Dan: You're talking about the Gartner trough of disillusionment? Right.
Nik: Yeah, the Gartner hype cycle. So we'll we'll put a link to that in the show if you haven't seen this. But this is a a common way that now we talk about hype cycles of products and services and new technologies. And typically what happens is everyone gets almost like way over excited. It gets overhyped. And then we realize, well, it's not as good as we we thought. It's not magically solving everything. We get disillusioned. But then eventually the technologies are almost always useful in some way, and it just takes time for. For us to adapt and figure things out. And I think that's actually one of the things here is that We are still we've been running the show for less than a year, and just the amount of change in what designers can do with AI has been so large. We haven't had time to keep to catch a breath and figure out what actually is the thing that's making us, I think, incredibly productive and leading to better product outcomes. It just I think that unfortunately that stuff still moves a little bit at human speed as opposed to technology speed.
Dan: And he does talk about this later in the article, which we'll get to, about how organizations have adapted to AI. And there's three ways, and we'll get to that in a second. But before that, I wanted to talk about prioritization and how the PMs that he talked to. felt like prioritization was now becoming more and more important almost like it was a year or two years ago that we need to relearn and rediscover prioritization skills because in the past engineering scarcity forced a prioritization of feature delivery. And AI, once it removes some of that scarcity it didn't replace the prioritization method. So everything could just move fast and move forward. prioritization Played a a really important role. Not only did it control the rate of change to customers so that customers just weren't
bombarded all the time with new features and the product just gets overburdened, but prioritization also forced the organization to reject bad ideas. That stuff that once you thought about it for a minute or for it where it sat on the shelf for a month, you could be like, maybe. this isn't really such such a great idea.
Nik: I've been thinking about that from the perspective of it's so fast to build out any idea. And actually this takes us back to the fastest gun in the West argument in the article there where
Dan: Yeah.
Nik: you can realize an idea and make it real within a day. But actually in some ways, the the fact that things take a long time, if they don't settle and sit and you're like In a in a week or two, you're still like, man, I really, I still really want to build this idea. I still think this is gonna really set our stuff apart. the fact that you can just build things in a day means any idea potentially can be shown to be bad, but also it could start to entrench you because you're like, well, well now we have something built. And now we g we're gonna continue working on this. Whereas if you take the time and let something percolate, and then you you see. If it sticks, you see if it's actually that important. And I've had this experience a ton with some of the work I've been doing and some of our startup stuff where we're like, actually, yeah, if we're not talking about it a week later. It doesn't matter anymore. we're that isn't the priority. clearly the priority is something else. And so we've actually been using that, but we've tried to be be more reflective about that, of what are we really still latching on to? And and so then we'll go, and maybe after we've said it to ourselves three times, man, we really ought to do this. Then we go, okay, third time's a charm. Let's actually go and put some effort into it.
Dan: It's like Beetlejuice, if you say the feature's name three times, it appears.
Nik: Yeah.
Dan: the next insight was all around a new skill that's emerging that I thought was an interesting nuance take to trust and judgment. Because it's like, yeah, of course, everything needs judgment now. that's been talked about quite a bit. But what kind of emerged was how to figure out what kinds of things to feed the AI to best augment your own human performance He was saying that there was a sense that by overfeeding the AI with context, it flattens out all the nuance. And the nuance is where a lot of really useful product insights lived. And that the more sophisticated that The PM's AI workflow was, the less the PM ended up blindly trusting the AI, which I thought felt about right to me. And so it was trying to figure out what kind of context to give the AI, the synthesis it's allowed to do, what actions it can take for you, and what what still requires a human being to either judge it or verify it before it goes out. So I thought this was a really interesting take on PM judgment about what the AI should know and what it should work on. it's almost like a meta judgment about how to use the AI. in the right way for your organization. So the last insight I thought was pretty interesting was that every organization starts out trying AI in roughly the same way. It's like, okay, we're gonna use AI. my god, it's magic. Let's build everything fast. And then like, crap. Now there's a lot of mistakes or a lot of problems. And then the organization splits into three ways that they deal with that. One is that they say, gosh, we shouldn't be using AI for a lot of this stuff. And they pull back and they say, okay, you could use AI in these very specific things. the second direction is the opposite. well, we just need to keep trying stuff to figure out if it's gonna work. And then the third way is really and they say the article that it really kind of comes out of that second, is that teams started to really care about what builds real impacts for users or business. and then they started to really focus on what those AI processes did that really allow them to have those kinds of impacts. So, what are the things that are actually making a difference? They care about productivity, but not if it doesn't channel into things like growth or profit, anything like that. If you build faster and you're still learning faster, awesome. But if you're building faster and you don't learn things, well you've just wasted a lot of time, money, and tokens. So I thought that was a that was a pretty interesting insight. But really lots to unpack in this article about the ways that AI is impacting not just designers but also PMs. Or are we even calling them designers anymore?
Nik: Yeah, that gets us into our next and last article for today, which is from Ron Golden, who's head of design in the money management and in-store wallet group at PayPal, with an article on product designer is no longer a job description. And in the article, Ron basically asserts that product designer isn't particularly descriptive anymore. It it mostly means like, yeah, you probably work on software, especially today. It probably means work on software, though product designer actually broadly could mean you work on products, and that you spend a lot of time thinking how a product works, how it looks, who it's for, and then figuring out how to actually create that and make it real for a customer. Instead though of thinking about this broad title of product designer, Ron breaks down what what are the attributes of a great product designer? And actually great product designers have four attributes that he's suggesting. And that in they have them in different levels. And actually they often have probably two areas that they're really strong in, and then they have some abilities in the others. That's his heuristic. And so the four areas are style, strategy, builder, and operator. So, really quickly, style is about creating products that have distinctive, unmistakable appearance and point of view. So, this is about creating something
that feels custom to your brand. It makes you feel something when you experience it, right? It really just takes the the visual. and the sort of aesthetic experience to a to another level. It doesn't, you're not making boring or average products. strategy is about being able to ask good questions and think really multidimensionally about business, about goals, constraints, metrics. So these are team members who are talking about trade-offs. They're figuring out what's really gonna move the needle, especially given all the things we've got to consider with our design. Builders are folks who have technical abilities that allow them to straddle the line between design and engineering. And actually, this is where probably we might see the role of design engineer kind of coming back. These are folks who are building prototypes, right? And they're getting concepts into real working systems as opposed to just just creating sort of mock-ups and wireframes and and drawings. And then lastly, This operator role is someone who is probably becoming close to a manager or a design operations lead. This is someone who's thinking about how the design work actually gets done. What's the process? What are the internal workflows? What are the internal tools that the team and they are using to actually be productive? The article has some nice descriptions of these and then actually has some nice discussions on when you might want more or less of each of these attributes, depending on where you're at, in your say your product your company's maturity.
Dan: we already do kind of see these different flavors in larger design teams and I wonder if they become more and more distinct or if they just all are blurred kind of like they are now. And I mean it it is definitely an interesting time when you do see people Out in the professional world who are now have the title of design builder or just builder. And some of our recent grads that we talked to in our interview, they they even mentioned that title as a thing. I I don't know if product designer is going away, but I know it's it's definitely changing.
Nik: And I think that's a great segue into actually hearing from some of our former students who are now junior designers out in the field. we were able to get on a call with four of them and talk a little bit about what it's like out there, being new in the professional world and navigating sort of this extremely turbulent time where things are changing so much because of the impact of AI
Dan: for those of you that are senior folks or managers or VPs and have forgotten what it's like to be a junior designer, this will be a really enlightening conversation with them. I know that I learned a lot about what their lives are like.
Nik: All right, so now we have a special segment here with a number of recent graduates from our program who actually are reached out to us and wanted to talk about what it's like out there being a recent grad and working in design and just dealing with all of the crazy changes that are happening. We're super excited to have four folks here. we're gonna actually have them introduce themselves one at a time. I'm gonna actually start. Lisa, do you wanna go first?
lisa: Yeah, hi everyone. My name is Lisa. I work at a fintech startup. and yeah, I'm a product designer.
Nava: Hi, my name is Nava. I work at an IT infrastructure company and I'm a user experience designer.
Sean: my name is Sean, currently a product manager at a health tech startup called Camir. but I work with a lot of designers in our team, so super excited to chat.
Qiyu: Hello, my name's Kiyu. Also work in tech and my job title is kinda interesting. they call me AI prototyper. So basically my day to day job is talking to Claude and then do AI products design.
Nik: Awesome. Well, thank you all for being on the show with us. I think we'll start with just a first question is what's it like out there as a new grad? I mean, I don't know if you want to talk a little bit about what it was like to get your position, what it was like to enter the the workforce in sort of these like heavily changing times. What's it like?
Dan: It's funny because we talk so much to design leaders who are like the crisis with junior staff right now. And so it's great to be able to actually talk to you. Good user research practice right there.
Nava: I would say it's pretty rough. If I were to describe it in one word, I would I would say this experience has been really confusing and conflicting. I think walking out of the program, it we graduated, Lisa Q and I graduated 2024 from MHCI. I think Sean was the BHCI. in 2024 as well. So that was right before I think everything started to snowball. I was in a very lucky position where I had an internship before the program and so I got a return offer and everything was very clear cut for me. But I know that wasn't the story for everyone. And I think because I was able to start my role right at the beginning of this entire new era of the age of AI I was able to see how my role specifically is changing. I didn't have to go through the experience of watching the job, the job hunt change. So I'm very grateful for that. But it was also very hard to see my friends go through that experience. And I'm sure we'll get a lot more into it in the rest of the episode. But AI design is definitely a lot different from what I think we experienced throughout the program. I think Getting our masters in HCI was, at least for me, my way of getting my head straight before walking into a career. And I definitely learned a lot of valuable lessons, but as the program is about ambiguity, we definitely dealt with a lot of ambiguity.
Sean: And just adding on to what Nala said, I guess I got to see it from a different site. So our company recently was hiring new grass, so we started our like new grad pipelines. We opened our application portal for about two days, and then we got 80,000 applicants. and
Dan: Wait, wait. Eighty thousand?
Sean: Yes,
because we have like 14 positions. Total. So we raised a new round and we I'm pretty sure we got 80,000 applicants. So we had to close the portal because there were just so many people that came in and thinking now it's not just undergrad that's searching for jobs. You get masters that also wants to change their positions, you get people who have experience, who wanna try out like the new grad to switch their fields or positions. So it became increasingly difficult as with NAMA said. the snowball starts rolling and everyone just waits to find a new job. So the bar gets higher and higher for everyone who wants to get into a new position, basically. And I think NABA already mentioned about MHCI. I graduated from BHCI, but something that I think from our side, like people start to look for hands on experience. Much, much more. Like back in twenty twenty three, I don't even think design system was talk that much. But then now like you have to know a design system, you have to know prototype. You have to use, let's say, cloud design Figma to hook up the MP MCP. Like that's one of our basic take-home assignments for you to do. And I realized like these hands-on experience really helps for you to develop that sense and I guess take a take a step back and understand what this industry is all about.
lisa: I guess for me building off what Sean is saying, it's like the bar is higher but also like I feel like all the roles start blending into each other. Like I feel now and I'm curious about Sean's experience as a PM but I'm like, we don't yet have a PM and so I feel like a lot of my work is actually like sculpting things out and writing almost like a PRD but then giving that to Claude and then once Claude makes it then I'm like designer role, like let's refine this and then one dad Once that's done then it's like they're like, Okay, well just push a PR at that point because you're you're basically there and it's like I'm kind of encroaching into like other roles. So it's like the bar within design feels a little bit higher, but then the bar for other roles that are technically not my role feel lowered because with AI you can kind of do a little bit of it. But I know for a fact that like the Whatever PRDs I'm attempting to write are probably not great and the PRs that I'm pushing out are probably not great. But it's like everyone's just like, just do it. If you can do it, just do it. so I feel like I have a lot of concerns around quality or accountability as the snowball gets bigger. But it is cool to just learn more and try new things. but yeah, interesting times.
Qiyu: Since we all mentioned job hunting, speaking of which, do we all notice that there are so many interesting job titles out there in the market? mine is AI prototyper. I've also seen there's a title called Builder, just given that how everybody's vibe coding right now. And then I also feel like the whole job hunting process itself is a prototype. Like people are still figuring out who to h hire and what quality they're looking for. So to me it's a prototype process itself as well.
Sean: I was about to say similar things because there's a lot more job titles out there. Like what Lisa said, the AI can just make things that looks about right. So people usually just go with it. And one thing I notice as like working with designers is previously when we go on on sites like talking with the customer face to face, like it's my job to hype up the customer, telling them like, look at this prototype. Like it like be be very excited when it released in two months. But now it became a little different because AI can do such a good job in prototyping. The customer usually get very hyped. They're like, when can I get this? Because it looks so similar to the current website and the current flow. So my job became like to cool them down, telling them that this is only a prototype. We actually need someone that's experienced to to work on this, give us a few more weeks or time and I felt like our role really shift as well and to what Q said, this idea of having more jobs, maybe it's just prototyping in a different sense. Like AI prototypers, I think we have a lot of field engineers, so forward deploy engineers that we get nowadays, etc. So back to Dan and Nick if there's anything you want want to talk.
Dan: Yeah, it's funny, I was thinking as as you were talking about this, that my own grad experience 20 years ago was in design and it was extremely narrow. It we never talked about data, we never talked about business, or rarely talked about the we certainly never talked about numbers and money and those kinds of things. That was that was somebody else's job to figure that part out. And now I feel like even in the almost four years that I've been at the HCII, how much I am teaching not just design, but also product management at the same time. I think it it is just because the nature of the roles have changed so much that there is this blending and our grads are going both into design and product management and blank in b builder, whatever whatever we're calling it today. And I know even that Nick has been teaching that blends even more engineering into it and that kind of developer thinking into it as well. about that. Nick?
Qiyu: It's also interesting that everybody's building something, but because of our different backgrounds, people build things with different focuses. Like I remember back in 2023, when I was still in CMU, Nick was already telling us all these different tools for vibe coding and how we could connect them together. And now everybody needs to build something using all these tools. But it's also interesting to see for engineers, they might focus more on okay, I want to build this tool to prove that feasibility-wise it works. But for designers, we might put more attention to the depth of different variations of interactions and to put more focus on how the things look and feel. So yeah, everybody's building something, but we build
things for different purposes.
Dan: At your organizations is there a standard tool or is everyone kind of using their own set of things now?
Nava: Yeah, we have the same tool pretty much across the board. There are some people that get special access to other tools, but because we're a very large company, there's a lot of like process that goes behind getting access. There's security restrictions. We're currently using VS Code with GitHub Copilot, which sounds like dinosaur compared to what Other tools people are using. I talk to Lisa a lot and she is using Claude all the time. I think she doesn't really use Figma very much. I won't speak for you. You can elaborate on that a little bit more, please. But yeah, we all have the same tool and the process is actively being defined. And I think what Lisa mentioned earlier as well about encroaching into other territories, everybody wanting to build something because we're all like now enabled. That's the beautiful thing about AI. It enables everybody to do so much more. But what if it's okay that we're all good at some things? And what if it's okay if we all sit in a room and collaborate and designate like delegate certain tasks to certain people? So it's yeah.
lisa: Yeah, I guess yeah, Nam and I do talk about this a lot. I feel like my company, it is a startup, so I feel like less like processed stuff. But I think the overall sentiment is just whatever tools you need, just try them all and whatever helps you do your job better and faster, just use it. so I know like I use a lot of cloud code. We've been using a lot of like paper, if you guys have heard of it. yeah, I I think it's just like trying everything. under the sun and then I could be using Cloud Code but then like my other coworker could be using like Codecs or whatever. Everyone has like their own process. I feel like that's not enforced as much. It's purely just try everything and work as fast as you can.
Nava: I think that connects to something that was very strongly taught in the program, which was it's not about the tool. I remember across the cohort, we were all like, Why aren't we learning how to use Figma? This should be a part of the curriculum. We need to know how to use this tool because it's the industry standard. and Looking back, I understand why every professor, including you you both, were telling us that it's not at all about the tool and we can't be attached to Figma because as this as this profession has developed over time, there have already been so many different tools people have had to adapt to and transition to. So this is just the next wave. And it's a little difficult now that there are so many different tools in this wave. It's not like Photoshop to sketch to Figma. I'm sure there were other competitor tools. There was a balsamic here and there. But yeah, now it seems like the inability to make a decision of what is the best thing and that's changing. And I think that just goes to show that the value we provide is not about design. It's about our thinking. And that's a very awakening thing at this stage in my career is that I don't just have to be really good at making wireframes, I have to be good throughout this entire process and it's about the ideas that I have and building the right thing the right way.
Dan: How has mentorship gone? One of the things that I've been very concerned about is it used to be you you'd kind of launch people and they would go work at agencies or big companies and They'd be a junior designer with other more senior designers. that would work with them and help mentor them. And now I worry that because everyone is very siloed, everyone has their own tools, that not that doesn't happen as much now. Wonder what your thoughts are on that.
Nava: Personally, I've had a lot of mentorship and I feel lucky to be in this position because I ha I'm I work on a large team of designers. We are a complete UX org. We have a research team, a design system team, and a product designer team of around 20. And we have two managers. And that just means that at any moment in time, half the team could be busy, but there is somebody else I could reach out to to like dig into their minds and understand what did they do when they were in this similar position that I don't know how to deal with. but I know that's not the case for everybody. It really depends. I would think, I would assume that it depends on the size of your team and how many people you have available to you. And I think oftentimes Mentorship now goes beyond the team and who you have at work. It's also about the industry at large and scrolling on LinkedIn and seeing what people have to say and reaching out and having random coffee chats here and there. You've got to get creative.
Qiyu: This is a interesting topic because I'm being also thinking about this question in terms of am I really learning the right thing and who I should approach to to learn the thing I want to learn. But two thoughts on my mind is First, there are a lot of things that I don't think can be taught by just talking to people. Like how do you wipe code? Those are the things I have to learn myself through trying things out. And also the same as design taste. Sometimes I want to learn like what is a good design by talking to my design mentors, like my managers. But still, people can tell us like those details. Okay, this one is off, interaction is off, color is off. But still, those are good things to be aware of, but it really to get that sense, I still need to learn through practice. That's one thing I've noticed. The second thing is I feel like mentorship nowadays is not just the manager and also I see type of mentorship. Like to me, because sometimes I also need to deal with code in my day-to-day job. So engineers are also my mentors, even being as a designer. So let's say I want to push a PR, but I don't really know what's the best practice in pushing PR in a repository. So now my mentor is no longer just the
designers. I also need to talk to more seniors engineers to see how they think about this, because these are all my unknowns. I remember Lisa also has some experience pushing PRs.
lisa: Yeah, I feel like it's actually it's been haunting me, to be honest. yeah, I I have no like engineering background at all. so I guess the the way for me to do it is just like beg an engineer to just get like my local repository set up and then talk to Cloud Code and hope like it looks good. but it's it's really hard, I think, for me to like figure out number one whether I want to be pushing PRs. because when the PR is out it's like I have no baseline sense of like is this decent code or is this like absolute like hallucination? and so yeah, I I don't know, I think a part of me is like, maybe I should learn a little bit of code so I can like roughly skim it and get a sense. but I think right now the process just like asking Claude over and over again like are you sure this is not gonna break anything? thing. Like can you triple check? Can you go through and like make the code clean, whatever clean means. Or like check current conventions and just like pretend you're an engineer at this company and like what would they say? So I it just feels to me maybe like I there's no way this is the most efficient way to use this to make a PR but that's kind of like the most I can do with the knowledge I have right now.
Nava: It sounds like your mentor
Qiyu: Go ahead, Nella.
Nava: is quad code. you you are reaching out to Quad Code to get this insight that you need, but you're also at odds with it at the same time.
lisa: Honestly, yeah. I think so. Yeah, and I feel like there's an awkward social element where it's like I feel almost irresponsible or guilty giving my engineers a PR that I haven't been able to like fully verify and then like force them to review it. like I feel like with designs I have the confidence like this is a good design or like this is not a good design. and so I can like take full accountability over that. But yeah, with stuff like that I'm just like I'm really sorry if it sucks and I guess you're welcome if it's great. But I don't know.
Qiyu: I was just about to say I feel like Cloud is my mentor as well. sometimes I don't know how everybody feels about this. I vibe code a lot, but after I close my laptop, I don't feel I've learned that much, just because a lot of decisions were made by Claude. And just like what Lisa said, I also asked Claude, like double check what is the root cause, and then fix that root cause. All this process was done by Claude Code, not by me. but to me, I do want to learn something. That's that's also my side project. I want to see if there's a way I could better leverage Claude as my mentor by creating a skill. So it tracks all my conversation with Claude and then kind of tracks the question I ask and then pinpoint the stuff that I could learn myself and generate a report by the end of the session. Let me see. I don't know if that's like a good way of learning, but that whole process is still a prototype as well, as we talked about.
Dan: Not a bad way of learning. I mean we've talked on this program about design ledgers and stuff, trying to keep track of things that you don't know or when you make a decision, like why you made that decision. So it seems like this could add-on to that. I've done a thing now where I've asked Claude to challenge me before it gives an answer sometimes where it says like well what do you think about this before it just tells me something and so I am forced to like write down okay I think this is the right way to go and then it's reveals Now it doesn't do that all the time 'cause I would probably wanna destroy it and stop using it. But every once in a while it jumps in and's like, What's your what's your take on this? And I and I find that to be helpful.
Nik: Actually, Dan, maybe to that point that you're making on how you and Q are trying to turn the AI agent into a partner that you want to work with to have it act the way you want to work. I'm curious from the rest of the folks here, right? What have you done that you've tried to sort of customize your experience and sort of make if you're working with an AI agent on stuff, make it actually do things in a way that you want them to do it or challenge you or what have you. Like what other what other things have you tried?
Nava: I think for the majority of my process, I'm actually not using AI to generate designs. I'm using it for the beginning part of research and just verifying if I'm headed in the right direction. I think also working in the IT space, I don't have a background in this type of stuff. So it's actually been a great tool to help teach me what's going on and help save my coworkers a lot of time from having to explain everything to me. So I think I just jump in there really quickly. I have yet to customize my experience through skills, but I'm constantly doing something similar to what Dan just said. I tell it not to affirm me at the end because I'm I go to it to ask, Am I on am I on the right track right now? And a lot of times I find I've I realized that it was just telling me that I was on the right track and I was actually working on stuff that, didn't match the expectations of whoever I was delivering for. And so I've started to tell it to not affirm me, tell me if I'm wrong, be honest with me. I don't need you to make me feel good about myself because this is work. This is not personal whatsoever. But I think I I would assume I'm a little bit further back in the AI journey, and there's a lot more I can learn from the rest of you here today as well, of like how I can actually customize that experience, especially in the beginning part of the process.
Dan: Do any you or your organizations have any kind of context documents that would help that are like, hey, this is like you were saying, like, I don't know this domain very well. is there anyone that's like put together like
a context.md file that is like, hey, here's everything we know about our user base or anything like that?
Sean: We do in our company, however, I think because we use so much context, it's very easy for like individuals AI to come to the same conclusion. So after that, our company started doing another so because I'm a product manager, our product team now has a rule of like you cannot write PRDs with AI at all. You can do some research with it. But what we realize is like because AI reads all the same context file, it usually derives into the same answer and then it's very hard for the human to go back to read it and then debug these because we like we all start with the same ideas. That's why we it's sort of a bummer so I guess I'm even further back behind the AI journey now. But we couldn't use AI to write PRD which I personally found helpful because that goes back to the other thing about like discoveries. I think now I need to be more fine with bad solutions at first. and then testing it out because AI gives so much of seemingly okay solutions, but maybe it doesn't have any of my thoughts in it. So I would say that's something I also learned. Asking the right questions. Be fine with bad solutions and iterate based on it.
Nava: That's really interesting for me to hear because I think in my organization, the design team is the only part of the process when we're considering how it starts from PM, hands off to designer, and ends up with engineers. We're the only part that is still remaining manual. Our PMs are 100% making. PRDs using AI and our designs are being used as a blueprint to be fed into AI, and that is what gets developed. And the manual intervention in those stages, from what I hear, is not that much. So it's really interesting that your company actually said you are not allowed to use AI. I think we've been in this era of like encouraging people to use AI as much as possible to find out where it really can be helpful. I think that hype has led to what we were talking about earlier with everyone encroaching in different roles. so it's been a very interesting thing to witness, especially as like the team that is trying to remain steadfast and do the right thing, which feels sometimes like going back to manual processes. But about whether our organizations have context for us to feed into whatever projects we're working on, we have something called an AI software defined life cycle, which is it sounds like a process, but it's really a repository of skills that are supposed to put on the different hats. So we have a UX designer hat, we have an architect hat, we have a PM hat, and they do provide some context, but I find that I have to feed the context more myself. So I get a meeting transcript. I tell it to make a markdown file summarizing that meeting transcript and I set I guess the start of my process is creating all of the context so that I can make sure I'm feeding it all of the right information. Because when I am trying to do something on the fly and just jump right in there, ask one prompt, That's when the response I get is not useful.
Dan: Yeah, and that's probably when it is the absolute
lisa: Us.
Dan: most generic too. When it's either it's either wrong or generic, where it's like, Well, this isn't very helpful. Lisa are we gonna say something?
lisa: Yeah, I guess I was thinking about kind of what Sean was touching on with like not being I guess AI always coming at the same conclusions and so we currently have kind of cloud connected to all our different like we use Fathom for meeting recordings and notes and so and it also like connects to like Slack and everything basically like your GitHub repo. So I feel like a really helpful tool is you can just ask it like for this client in the last like six months. months what have they been complaining about, what have been the requests and it'll just scrape through all of the calls, all the meeting notes, and it'll give you like a nice summary. However, I'm also like, I don't know how much I trust this summary because it's like I mean in six months there's just like so so much content and it's picking out maybe like three main points. And so part of me is always like, Wow this is really helpful but then I'm like, well maybe I should also dig in and see what all these recordings are actually saying. never like quote slack messages and stuff, but I feel like there's this kind of tension between like, wow, AI can do so much, but also like, ooh, do I trust it? Like it can do so much, but how reliable is that? I don't know. I feel like I deal with a lot of mistrust in the AI outputs. so it's interesting.
Qiyu: I'm having the same issue with that. Sometimes I don't really trust AI output. And I think part of the reason is because AI, in my opinion, does not really capture those human nuanced details. Like to Dan's point about the context.md, a lot of the context, in my opinion, lives in people's mind. And part of that is very hard to be described with words. Like, for example, the leadership just preferred this. for their own reason. And that might just be a human taste. That's really hard to describe with words. Even human cannot really describe it. We just like feel it. So that's one issue. And then the other issue is context really hard to update. There are a lot of things like just happen through real-world conversation and then no AI tool could capture those ad hoc-based conversations. So I think that could be the reason why sometimes I don't really trust AI output because the input quality is not that great.
Dan: Yeah, there's no way that AI can capture everything. a quick hallway conversation or just the tone that someone has about Yeah, maybe you shouldn't do that, or even though they're like, Yeah, you could try that, you know, and so it just reads the transcript is, Yeah, you could try that. And so it's like, well, the AI is like, Well, you just just try that then. but There's there is so much nuance and keeping a
context thing up to date is extremely hard. And and yeah, a a Sisophean task, right? Constantly pushing the boulder up the hill.
Nik: I will say, Dan, though, there's in the future we might be looking at things that are like if AI pins and they are recording every conversation. I'm imagining a future workplace where everywhere you go in the hallway is you everyone's wearing their pin and stuff. I I've I've seen concepts of this and I'm not gonna claim it's a good or a bad thing to do. I'll let people make their own judgments of that because there could be positives and negatives to all this, but but yeah. maybe as a last question, What would you all recommend to current students and current folks who are just entering the field right now? and maybe what would you recommend to us, Dan and I as educators, What should we be conveying to young designers and product developers and managers that you think now that you've had some experience out there in the real world?
Qiyu: I think mindset is very important. I know we've been talking about this AI in work, but to be honest, sometimes I vibe code because of FOMO, the fear of missing out. So I really wanna try using as much AI, like leveraging it in as many parts of my workflow as possible. Cause I don't wanna leave behind. But more importantly is the reason I do all these things is I want to learn how to live a happier life and also learn as much as I can. Learn the thing I want to learn. So something I would recommend people who, especially early in the career, is we know finding what you like is important, but I know that itself is a research process. And then know the real goal you want to solve. And then try to learn AI. Try to not learn it not because of FOMO, but because you find something that you want to create. That being said, use AI outside work. maybe use it to create some side projects. Find if there's any part of your life that you could you leverage AI. So for example, I create my own snack website and I share it with friends to have some fun in life. And then the more you do it, the more you can find, okay, AI could be helpful this way. And then that learning could be used back in work in some way.
Nava: I 100% agree that it's about the mindset. I think the advice that I would give to someone entering this space or early in their career or current students is that just try your best to instill some confidence in yourself. And that's easier said than done, but that's something that comes through experience. So just do what you can. Stay up to date with what's going on. I know that t feels like a lot of pressure because there's something going on every single day, but just practi learn by doing. And that also connects to the what I would say for Nick and Dan here and for the faculty at the HCII of CMU. I think the beautiful thing about the program is that we're constantly using our hands. And I think now there's just so many more ways to practice things. And I think the current framework of the classes, if it's anything like it was two years ago, is a great framework to keep students practicing and learning by doing. So keep that up. But adapted to the day and time that we are in now.
lisa: Yeah, I guess my thoughts on it would be like it's I feel like it's really easy to get lost in like the speed of AI and like because you can do so much so quickly, just like going down that hill. but I think maybe my advice would be like first principles are always first principles and they will always stand regardless of technology. And so I feel like as technology gets better and better and better, I feel like it's very easy to just tell Claude like, design does new features. and just let it produce something. But just because it did it fast doesn't mean it's like good design or a good experience and I feel like I have to remind myself a lot to like take a step back and like actually think about things that, we learned in school but maybe like people ten, twenty, thirty years ago were still learning in school because that's just like good human experience, good design. it's yeah, it regardless of the tech that you use to get there, it's it still stands. But yeah.
Sean: I don't have too much advice, but I think one thing I recently realized is there's a fine line between me understanding a subject versus AI telling me what I understood. So it this happened because there was one day that I used up all my cloud credit and had to think for the first time after like a few years and I realized I cannot think for myself. So thinking back, I think this is one thing that recently realized and want to do more, like making sure that I'm actually the one that understood it. Had metrics on that, but this is what I have.
Nik: Wow, that is excellent information. Actually, super helpful for us. And hopefully for our listeners who are out there, this is useful for all of you as you're navigating these interesting times. Thank you so much, everyone, for being on the show and for giving us your thoughts. we of course wish you well in all the things that you're doing. Super excited to see all the kinds of awesome work that you're gonna be putting out there yourself and with the orgs that you're part of.
Dan: And that's our show for this week. We'll be back next week with another episode of AI and Design. More AI, more design. We'll see you then.