Full Transcript for Google CC, Controlling Image Generators, and Planning with Agents

into the kids feels like it really an essential part if you're talking about managing a household or a Designing something for a family and leaving out kids feels like a very strange, very strange design choice. And obviously it's for privacy, security, cover your ass type, liability issues, but I think it really hobbles this concept. Nik: Yeah, it'll be interesting to see how well this plays out with families. I imagine that they're gonna be rolling it out and trying to get as much feedback as they can from people. Again, I could see where, you know, coordination among the adults would probably be still pretty useful and maybe there's a way if parents have s something set up on their kids' emails where everything forwards to them, which I have heard of people doing, that might kind of make it a non-issue. But yeah, I agree that that it does feel like a l limitation. And I do wonder it if though it's basically just because of the nature of where we're at right now in the AI landscape and age restrictions on the web. they might be doing this for reasons that aren't particularly because they want to do it this way. They might just be doing that again because of the the challenges that you're saying and they they haven't figured everything out yet. The thing that I think is really interesting here though is the idea of having an AI agent and designing for multiple users, multiple different users, arguably probably multiple different users with different AI literacy and different different AI Dan: Different use cases, yeah. Nik: yes, exactly. Because You know, right now I feel like so much of people's AI experience is very individual. And if anything, people are almost tuning their interactions to be hyper personalized. You know, they interact with their AI agents in a very specific way. and that's very different from how you might interact with yours and I might interact with mine. And so this is an interesting design problem. And I wonder if it's an early c use case where we might start to see Google designing for say teams in the future, right? Where I could actually see, you know, if I have a team that's working on a project and we're all together and we all have an AI agent that's helping as sort of another team member, how would I make that work? So this is kind of maybe an interesting lower cost use case. to try out as opposed to rolling this out to be like, and now we're embedded into Google Drive and Google workspace where stuff, you know, maybe has to work really well. Although that being said, managing your family is hard work and actually if things go wrong, they could go really wrong here. Dan: Right. And we we have seen a little bit of this in like Figma Design Agent, for example. On a one Figma file, multiple people can be talking to the same design agent. And I think the design agent treats every every every command, every prompt as the way it should be, there's no hierarchy of well, hey, you're not the designer, you probably can't make this change or anything like that. it's a straight up change. Just as though you're direct manipulating the canvas right right there. But yeah, it it'll be interesting with this, you know, hey, hey AI, tell grandpa that I'm at the store and really you're not. Or you know, like t tell my parents that I am I I was I was definitely there for my doctor's appointment, even though I was out smoking weed in the parking lot at the 7-Eleven. Nik: and this might not even also be just just the kids. Actually, we have some great work here from the HCII, that actually explored how elders were interacting and talking with AI agents and how they did and didn't want certain information shared with the rest of their family. I think one of the use cases I remember reading about was they used to say, my friend is out at the store getting me some grapes. Which was actually coded language for buy me some cigarettes. even though, Dan: Ha ha. Nik: you know, grandma wasn't really supposed to be smoking cigarettes anymore. And so there's actually all these things and in the case of say actually people who are taking care of elder family members, right, who are adults and have their own personal lives. Right, with children it's a little more like sometimes we can make an argument of like, well, you probably should know a bit more about your kids' life and you're a little more involved. But yeah, it's a different case. So yeah, I wonder how they're gonna manage this. you and we'll link to that research in the show notes that there's actually a lot of tricky situations where what information is shared among family members and which information is kept private. And how does an AI agent that's working on behalf of everyone, You know, how does it decide? Also, especially in situations where what if information could be very valuable, especially to someone who is a a head of household and, you know, if they did know something could potentially, help someone. Like let's say someone's talking about health issues that they're having, but they're not sharing that with their family, should should it share? Should it let someone Who's maybe a caretaker know about that, or should it keep it private? These are gonna be really challenging design problems for the team to work on. Dan: I mean basically this is this is social, right? This is designing for social. And any time that you have a social or community platform for things, which basically this is turning AI into, It creates a whole other set of dynamics that you have to be aware of and take into account and and try to negotiate and it becomes a lot harder. I mean I've I've worked on a lot of products in my time, but a lot of the stuff that I did at Twitter, just having to think about the social aspects of things definitely rank up there as some of the hardest stuff I've ever done. And applying it to AI where it has to think and reason and debate and discuss and make decisions for a group of people is a really tough challenge.

Nik: Yeah, and so I imagine from a design perspective, the the team at Google is going to probably have to put a lot of work into figuring out how it is moderating interactions and communications, say, between family members that happen via the AI, so through the AI. And then another thing that I'm not sure if if it's part of what they're gonna work on, but I I've heard ideas about is how does a a family, say, develop their own rules and develop their own guidelines for hey, this is how How you will, you know, maybe they will set out guidelines for, you know, we'll respect the privacy of certain information, but this information will always be shared. And so now, it it may not just be yep, we turn the AI AI on and it it works, but actually that we have to go through a a formal process of almost setting a a constitution for the AI within the group. Dan: But again, who sets that up? And do they then have all the power to do that? And does everyone have to agree to this constitution? Like, for example, you're gonna share with me your location and your calendar all the time. Well, I don't want to do that. there's just so many places it can break down. That being Nik: Well, Dan: said, it is actually interesting that they would pick home. I would think this would actually be a lot easier for a workplace that for like a team. Like this is our team AI. And because the bound the boundaries of a team and of a workplace, I think are maybe more rigid than a than i i it it's already almost a contract when you're at work. And so I would think that that would be an easier place to have some kind of team constitution about how we're all going to use this one AI. Nik: So next up, Darko Trepevsky wrote a nice article about how he has struggled with using image generators to lay out and create new brand content, and actually eventually worked out a nice system that provided him with more control over the things that should be constraints. So the things that really should always be a certain way, like a brand logo always in a certain place of a certain size. and then also pairing that with the ability to have the AI systems make a little bit more judgment on their own. You know, the the big challenge that he was having was he was offloading so many decisions to the AI where he really wanted to have more control. And it's a nice write-up. He talks about some of the technical implementations that he did with some of the image generators that he uses. I think for designers out there who are working on this, this is a nice kind of almost micro how to that might give you some inspiration. write up. Dan: Well, 'cause 'cause he's an engineer, so his solution to this is is pretty engineering heavy. But the core idea I thought was great was that like, hey, I am gonna generate a whole bunch of concepts, but I really want a couple things to be fixed in those concepts, and I want deep control over that, but the rest I really want the AI to go in and fill in. So I love that idea

of the like The like tug and play where it's like I I am in control of this piece and you are in control of that piece and then I will make the final call over the variations that you make of the stuff that I want the AI to make variations on. So I think that was really cool. Nik: Yeah, I agree. And, you know, the two kind of key points that I thought were sort of stop asking the AI to be your layout engine. I mean, in the case of the kind of work that he was producing where he was laying out images with text on it, right, that makes sense because that is where we probably would want to have more control. But then using AI where probabilistic variation is valuable was also a really key point and that's also where I think a lot of the the benefits of using AI are because you can generate all these options and then you can select among them, right? And that's really where I think you end up with a much more co-creative workflow. I think that now many of the workflows that I think a lot of us have maybe been experiencing where we sort of prompt and get something and prompt and get something. Sure, we might say that's co-creative, but it it doesn't feel co-creative in the same way because you are letting the AI do so much for you. It almost just feels like outsourcing, And then you kind of reviewing and coming back. Where I think that the the system that Darko describes, and the way that you describe it, Dan, really is much more of this co-creation. I own this part, you own this part, you're creating variation, and then from there we create something together. Dan: you can definitely see this applied to say a Figma file where you're like, Hey, I really like this top piece. Let's keep that top piece there. generate four or five variations of the bottom or Something as simple as, hey, keep the navigation the same in all of these. Don't don't even bother thinking about that. keep that steady. But the rest, feel free to play with. I think that there is something really good about that handoff. One of my teams last year the prioritization team really was looking at things like this, particularly when something they were looking at it from a tech standpoint where it was like, when something was produced, they would highlight a piece of it and be like, this this stays the same, but the rest, not so great. fix the rest. So it's not regenerating everything all the time. Because you're right, that does not feel like I'm doing the thing, it feels like the AI is doing the thing. And this way gives it a lot more structure, I think, that is really interesting. Nik: One of the other points that was made in the article too was this idea of designing for automated validation as opposed to always manually inspecting. And I thought that that is a really good point, especially when you want to leverage the probabilistic variation that an AI system can give you. Because if you are always getting something that's sort of new and different every time. then you kind of are

stuck having to always manually look at it and go like, no, that's not quite right. And you kind of are just working in this one at a time manner where you're sort of edging it towards hopefully what you want. Whereas if you have certain aspects of your in this case image generation that are automatedly checked. And I'll say again, as you pointed out, Dan, Darko is an engineer and so there's definitely parts of the system like drawing bounding boxes around things like the the brand logo to then have a system then And numerically check yes, it is the right aspect ratio, yes, it's the right size, it's in the right location. But if you can do that, I think this gives you the opportunity to generate much, much more. You can actually have more automation on the things you want to be probabilistic, knowing that, everything that you're getting out of this has the constraints you set, and you're not always checking those constraints, which is actually super tiring and really creatively draining. Actually, I was thinking about this myself where if I'm doing something right with a logo in it and it just keeps moving it or adjusting it, I'm just like, my gosh, this is you're killing me here. Where Whereas I don't get the the benefit and arguably hopefully the rich creative excitement of seeing new variations that are like, ooh, that's interesting. That's exciting to see, because I'm just focused on the you keep messing this part up. Dan: Mm-hmm. I wonder if this is also something that you could incorporate. I mean, almost like he does with the bounding box, like could you do that with with a canvas and be like, okay, don't touch these things. you could mark them as pinned basically, like don't mess with these no matter what you do. Yeah. I mean we we almost have this now it's almost at the other level where it's like fix this little thing, fix this little thing, where you're doing like tweaks. But I wonder if there is the the flip side of it, which is keep all these, tweak everything else. Nik: Yeah, actually the thing that I'm imagining now in something like a canvas tool, a Figma make, a Claude design, what have you, is the ability to lock elements and really quickly like, tweak and adjust, tweak and adjust, that's good, lock that. which is actually the the kind of thing that we do when we're working collaboratively with each other, right? We will actually Dan: Right. Nik: lock layers. and so I yeah, actually now that I think about it, I'm not sure I've really seen that. entirely it's possible it's there, but I don't know if I've entirely seen that on some of my AI design tools. Dan: Yeah. They might be building it even as we speak, so who knows? Nik: And I'll say it actually does lead a little bit into our next story because thinking about sort of how we really take advantage of sort of the probabilistic variation that these systems can provide us, well, how do you actually work with these things? Dan, you wanna talk to us about the last story for today?

Dan: Sure. This is an article called Planning with Agents by Maggie Appleton. And Maggie is a designer, engineer, and researcher at GitHub Next, which is the RD lab part of GitHub. And planning with agents is about why we need richer, thicker interfaces, and better boundary objects for planning with agents. That's a big mouthful. So let's break it down. So planning with agents in this article refers to that window when the human is working with the AI, is is awake in the loop, and the intent and judgment are being passed from the user to the agent before the agent runs off and goes and acts alone. And it is this small, scarce window, but it's super important, which is why it's worth designing. Now how this is done right now is you're there, you're working in a command line interface or your desktop app and you're kind of in planning mode and the system might be asking you some questions and Especially if it's a a lot of questions, eventually you're probably gonna be like, Yes, sure, that's that's the way to go. because the question's unclear or complex. And then the agent like gives you a long list of things that it's about to do and says, Do you want me to go do this? And you're like say yes or no. And More likely than not you're gonna say yes. Because at that point you're like, sure, just just go do it. Now, there's a couple problems that she calls out with this. So the plan, it can't be edited directly, especially in the command line interface. It is this long plan, and you you would have to then go in and be like, no, no, no, don't do this piece, don't do that piece. But you can't like target one specific piece of it very well. And it could be a super long Thing that most people aren't gonna read all the pieces of it. And it probably still contains some open questions, and there's no easy way of answering them. There is no kind of pace to it. You can't pause on a particular question and go deeper, or be like, huh, could you just research this little bit of it and prototype it and see what happens? And the agent treats everything as fixed. once you say yes, it's like go. And the the big kind of meta issue here that she calls out is that humans and agents just live in different worlds That are only partially legible to each other. She actually goes back to a really old study from the eighties with HCI researcher Lucy suchman about PhD researchers talking to a computer and the computer talking back to them and the metaphor is like looking through a keyhole at each other trying to figure out what the other thing is doing. And it's the same here that the humans and the agents, they live in worlds that they don't understand. And the artifacts that get passed between them, which are things like prompts and plans and markdown files, they're all built for the agent's way of seeing. And both parties get the same kind of markdown, and so the human beings have to do all the adaptation. And one of the things she critiques is that there's there's a school

of thought. in HCI world where it's like, well, the agent will just be able to look at all your stuff. And eventually the agent will will have sensors and it'll be in your home, like we talked about with with CC, or it'll be on your watch. It'll be everywhere and it'll know a lot more of your context. But she's like, well, no, you can't really bring the agent into the human world. You can't bring the real world into the agent. That'll probably never happen. And so what you need are better boundary objects. Nik: I thought this was a really cool article. I have to say I agree heavily with the idea of us needing better boundary objects to allow us to work more effectively with AI agents. actually friend of the show, Mike Kuniavsky and I, we used to work together and many years ago actually talked about in some presentation about how we should think about every step of the design process and all the different boundary objects and roles that an agent might take and that we might take and how we would actually work together. and we, you know, really we're thinking about what is the future of design communication between AI and people. And I think that Maggie here is really getting at that, especially in this case about the planning aspects of things, because I think it's one of the things that we're doing a lot right now. And I definitely feel a lot of the problems that she is raising. I mean, even in interfaces where, say in Claude with plan mode, at least it'll create like a markdown file, and now I can side screen it and look at the markdown in at least a nicely rendered version, but yeah, I d I don't really go in and edit it. I can add comments to it and say, you know, adjust this, adjust this, but it I'm not actually writing that document. I should say I could go in and edit it and say, hey, I made some manual updates. And I do that sometimes. Sometimes I just wanna tell it to change these things. But these often become so long, so detailed, you just cognitively surrender, and I really like Maggie's idea here of saying how do we make these plans more visual? How do we make them more interactive? At least at a minimum for my own individual work, I would love that. And then of course I agree very much so in the aspect of the social thing that really these need to be artifacts that a team is working around because most problems that we do are team based. Dan: Right, so she lays out a bunch of solutions here in the article that are probably we're talking through. So yeah, the first is can we move beyond text to other kinds of representations like charts or maps or timelines, architecture diagrams? Things that d aren't just a markdown file, but things that we can look at and compare and and make decisions on that are more that are more visual. That's I think a really interesting idea. the other one is kind of what you were just talking about, Nick, which is letting people manipulate the plan directly by like doing and testing, but making that easier than just Ugh, j than just editing a a massive four thousand word markdown file that I have to kind of sift through to look to make sure everything is as it should be because I'll do some cognitive surrender and and just let the AI figure that out. And then Yeah, like you were saying, like is there a way to make this more multiplayer? So if you are working with coworkers on something, can you can you point them to it? Can you share this? Can you have like discussions about things? So That's a that's that's one way of doing it. And then the other way is something that also some of my students looked at last year, which is how do you start to bring reality into planning? And right now the planning agents stop at observation, which is looking at files or tracing kind of dependencies, and then they predict which implementation will work and aim at that. But rather you could have Sub agents building several approaches in parallel, and then say, this one probably seems like the best way to do this, or here's two different ways to do this. This one will take this amount of time, this one will take this amount of time, and this is the kind of outcome that you would get, and this that kind of quick, like prediction of the future. is something that I think is currently underutilized and underdone, mostly because I'm theorizing that it's very expensive, that it's gonna burn a lot of tokens to do that. But maybe as we start to get more of these local models, stuff like this becomes much more commonplace. Nik: Yeah, I really like this idea of parallelizing things, trying to then have multiple, say, sub agents figure things out and then come back. I mean, that's something that I think is incredibly powerful about having these AI agents that can actually quote unquote reason about things. but it's something that, yeah, it is right now at least so expensive and you know, you'd burn through potentially your entire subscription, if you do this too many times. and so we're not really getting the benefit of it right now. Dan: yeah, this will eat through your free subscription or your twenty dollar subscription pretty fast by doing many, many possible future variations. Nik: I think it's important now for designers to start thinking about what that could look like because as these models get cheaper, or as you said, you know, we can start running them locally, there's ways for us to to do more parallelization. How do you design for that? Because also now you're thinking in five, ten directions and then having to manage it all, or your agent systems are having to manage it all, right? I think there's a an interesting design challenge about coordination and how you actually reason with those types of variations when they come back. Dan: yes, for sure. I think there's a couple places for design to insert themselves here. One is to start figuring out and prototyping and testing what something like this could look like and GitHub Next has already done this, and so they have a couple things in the article that you should go look at. They have a couple of prototypes and some screenshots and stuff of of stuff that they've worked on that is interesting. But it'd be great to see other design ideas for this. And then as you were saying, what is what does this look like applied in Microsoft Word or the iOS operating system or anything like that that is like forward looking or multiple paths or Siri AI. And any of those could be a place to start thinking about how do we have better plans that are more human-centered and not just those that are agent-centered. Is Nik: So there's a lot here. I would definitely recommend check out the show notes and go check out Maggie's page on this. I'll also say that the page itself is actually really beautifully designed. It's really a wonderful way to go through reading and the slides that are associated with this. it's actually excellent like layout and sort of interaction to kind of go through Dan: for sure. this is a meaty article that's definitely worth your time. Goes back to ancient navigation systems of the South Pacific Islanders and l lots lots in here for sure. well that's our show for this week. We'll be back next week with more AI, more design, and we'll talk to you then.