How to close the gap between discovery and delivery | Jenny Martin (The Collaboration Coach & Author of OOPSI)의 전체 녹취록

00:04:43.040 And this is kind of starting from the outside in, so that's where the outputs process comes in. 00:04:43.040 --> 00:04:46.319 So starting from the outside, focusing on the outcome. 00:04:46.319 --> 00:04:55.839 But I guess the problem it was trying to solve was trying to do trying to balance the right amount of thinking with virginity. 00:04:55.839 --> 00:05:16.879 So, how do you do enough thinking and enough collaboration at the right time so you're building the right thing, but without getting paralyzed by analysis and all that bad upfront stuff that came with the the waterfall days, which I'm old enough to have properly lived through. 00:05:16.879 --> 00:05:20.720 Um so yeah, it was it was balancing, it was balancing those things. 00:05:20.720 --> 00:05:38.959 So when when I kind of trended to explain it to you, it's the oopsie is is um it's a navigation tool when you're looking at something big and messy, but it's also a tool for helping you break things down or know that they're small enough to start focusing on. 00:05:38.959 --> 00:05:41.040 So for me, it was joining up the dots. 00:05:41.040 --> 00:05:50.879 The problem it was solving was joining up the dots between other good practices in a way that the team could be really concretely working on something, doing the right amount of thinking. 00:05:50.879 --> 00:05:52.480 That's a long answer. 00:05:52.879 --> 00:05:53.680 No, not at all. 00:05:53.680 --> 00:05:55.839 And I like the the the way you're talking about. 00:05:55.839 --> 00:06:04.480 Then there's a metaphor that you use early on in the book, uh, and it really resonated with me, but it might be because I'm of a certain age as well. 00:06:04.480 --> 00:06:09.279 And use the video game asteroids uh as a key metaphor for this. 00:06:09.279 --> 00:06:13.360 And just explain what why asteroids, what did you mean by that? 00:06:13.680 --> 00:06:15.519 So this it's not my analogy. 00:06:15.519 --> 00:06:16.399 I I've borrowed it. 00:06:16.399 --> 00:06:21.439 So I first saw it in Jeff Patman's story mapping book, but he got it from someone else. 00:06:21.439 --> 00:06:28.240 He got it from a guy called uh NJ, Michael James, who I managed to track down and I asked him, is this your analogy? 00:06:28.240 --> 00:06:35.600 And he said, Yes, he he's 66.66 repairing percent sure because there were three of them, it's probably him. 00:06:35.600 --> 00:06:47.600 So that was his answer, but it kind of explains the way that you well, that you navigate complexity and that you do analysis in an iterative context. 00:06:47.600 --> 00:06:55.040 So the metaphor is if you're as old as me, maybe as old as you, I don't know, and you remember the arcade game asteroids. 00:06:55.040 --> 00:07:03.920 For me, it was meteors because I played it on my BBC Basics, so it was like branded, but my hands go into the into the shift up and down left and right. 00:07:03.920 --> 00:07:14.879 But you're basically trying to clear the screen and blow up all

the asteroids and get to the next level so you can level up, and you start with these really slow-moving asteroids that are chugging around the screen. 00:07:14.879 --> 00:07:26.399 Big chunks, and you'll yeah, and they're going it they're kind of going slow noise, and you're the little prony triangular spaceship, and you're kind of trying to blow it up. 00:07:26.399 --> 00:07:32.160 When you blow it up, it moves, it it breaks into faster moving pieces. 00:07:32.160 --> 00:07:34.079 So they're kind of do-do-do-do. 00:07:34.079 --> 00:07:39.040 And then if you blow one of those up, then they're in really tiny, fast, fast ones. 00:07:39.040 --> 00:07:53.199 And so if you were to just indiscriminately like fire, fire, fire, and blast all the asteroids up, you'd have millions of these tiny ones flying about the screen and you'd just get squished. 00:07:53.199 --> 00:08:18.560 And so the metaphor there is that if you think about the the meaning of the word analysis, which is breaking down, um, you know, if you think about brainstorming user stories or all the old in the olden days, writing that 700-page functional specification, that's equivalent to breaking everything down all at once and being overwhelmed by it and the complexity. 00:08:18.560 --> 00:08:27.600 So, in an iterative, successful agile or product team, you want to be able to clear up after yourself. 00:08:27.600 --> 00:08:33.279 So, so the idea is that you you leave big asteroids unexplored. 00:08:33.279 --> 00:08:39.759 And if you do that, you've got a chance of surviving because they continue to move slowly and you can move around them. 00:08:39.759 --> 00:08:50.559 Uh, so if you just blow up one of those big asteroids and then kind of mine it for all its bits, then you can much more deliberately clear the screen. 00:08:50.559 --> 00:08:52.159 And so that's what the analogy is. 00:08:52.159 --> 00:08:57.600 It's kind of trying to capture this sort of iterative moving in and moving out. 00:08:57.600 --> 00:09:08.639 So it's not just design thinking, it's multiple layers of design thinking as you examine and then come back up again and then examine again. 00:09:08.639 --> 00:09:14.159 And it's the first, so it's the first kind of analogy that has properly clicked for me. 00:09:14.159 --> 00:09:16.320 So yeah, I book I've blurred it for the book. 00:09:16.720 --> 00:09:18.480 It it worked really, really well. 00:09:18.480 --> 00:09:29.279 And one of the things that really clicked for me uh and that we talked about a a few moments ago was the idea that the book is in the framework is very delivery focused. 00:09:29.279 --> 00:09:51.279 Uh, and that feels different than a lot of the other frameworks that I work with, a lot of the people I've talked to about these things, where a lot of time we're focusing on making sure we're solving the right problem, on problem definition, on success metrics and things like that, but not so much on the second half of it, the second 80% of actually delivering value into production. 00:09:51.279 --> 00:09:54.000 And we say it's about outcomes over outputs. 00:09:54.000 --> 00:10:03.679 We say we want to focus on that, but so much of the work that we do is about the storytelling and not the actual delivery element, at least not for product managers. 00:10:03.679 --> 00:10:17.679 Uh, and the best product managers are either ones who have that ability to focus on delivery as well, or make sure that they've got a great delivery manager by their side who and a great engineering lead to make sure that this is happening. 00:10:17.679 --> 00:10:23.840 But I'm curious with your background, is was this focus on delivery really deliberate? 00:10:23.840 --> 00:10:30.960 Or is this something that uh helped distinguish why you put this together rather than using some of the existing approaches? 00:10:31.679 --> 00:10:35.200 I find that question interesting because I don't think of it as a delivery thing. 00:10:35.200 --> 00:10:40.720 I think of it as a as a joining up of discovery and delivery. 00:10:40.720 --> 00:10:50.080 So one of the things with it is that it brings together product, engineering, you know, design folks into the same conversations. 00:10:50.080 --> 00:10:54.639 And that's a gap that I've seen for for 20 years, I guess. 00:10:54.639 --> 00:10:58.480 And so when I was started doing this stuff, it was sort of pre-product. 00:10:58.480 --> 00:11:11.200 So product wasn't even a thing at the time, and I was always motivated by starting the team off, focusing on on the why and the purpose and the outcome. 00:11:11.200 --> 00:11:29.120 So I always I was probably um not a normal thing for a delivery person to be very focused on the outcomes, but uh, but I kind of uh I could see how much difference it made to people's decision making and motivation when we all spent time on that, if you kind of get what I mean. 00:11:29.120 --> 00:11:37.120 And so Boopsy makes sure that the engineering practices are joined up to the product thinking and the outcome-based thinking. 00:11:37.120 --> 00:11:44.000 And it's it's exactly that, it's exactly bringing those things together so that everybody's working on the right thing. 00:11:44.000 --> 00:11:46.960 Because it's one of the biggest problems that I see. 00:11:46.960 --> 00:11:50.320 Well, you know, you mentioned my um Lego series play. 00:11:50.320 --> 00:12:06.159 So I do this uh activity in my training that I've been doing for years, where where I get everybody at the beginning, at the beginning of this three-day training to say, build a plane in your product teams, like build a Lego model that represents some kind of pain. 00:12:06.159 --> 00:12:18.080 And a really common metaphor that people use is uh a customer or stakeholder up on a huge pillar with a with a microphone and a crown. 00:12:18.080 --> 00:13:04.480 So that's a really universal thing, and then and and normally facing the wrong way, and the other thing that people build is um a big wall

with engineering teams on one side and product teams on the other, or business representatives on the other, and then they often build um an idea of what business and product folks want over here, and then a completely different mess over here, and so those kind of um problems are universal, and and folks from different teams, you know, different industries have been building those that same problem, you know, for as long as I've been doing LEGO Series Play, certainly, which is 10 years. 00:13:04.480 --> 00:13:09.039 So it's a kind of attempt to bring those things together. 00:13:09.440 --> 00:13:17.200 Yeah, I don't I don't think uh if I came across how saying this is purely a delivery framework, that's that's not what I meant to say. 00:13:17.200 --> 00:13:24.320 It's the fact that most of the other ones don't uh include the such an emphasis on it as well. 00:13:24.320 --> 00:13:39.519 So this feels more complete that it's got the focus on the what is it we should be doing and why are we doing it, but it also completes the loop and has the delivery element that I'm not always seeing in some of the other things, or it doesn't feel as prevalent. 00:13:39.519 --> 00:13:41.600 Um yes, yeah. 00:13:41.600 --> 00:13:47.919 So before we get into the framework itself, I'm just curious about uh how you actually use this. 00:13:47.919 --> 00:13:55.200 A lot of the examples in the book are feel like they're at specific product team levels. 00:13:55.200 --> 00:13:59.120 So, you know, one team or maybe a couple of squads working together. 00:13:59.120 --> 00:14:10.480 So is that where you usually employ this, or is it a higher level planning exercise, uh coordination exercise across uh maybe a division or an entire company as well? 00:14:11.360 --> 00:14:14.320 The the answer to that, I'm really sorry, is it depends. 00:14:14.320 --> 00:14:18.159 It it depends on your it depends on your starting point, right? 00:14:18.159 --> 00:14:28.000 So the thing it's it's like I was talking about with the boulders, the the thing with analysis is that the lens that you start with can be any level. 00:14:28.000 --> 00:14:36.639 So it could be a really big, could be a really big thing, in which case um oopsie is more kind of in the large, like navigating the asteroids. 00:14:36.639 --> 00:14:50.720 Or it could be um, and this is like some how some folks have been using it already for the last sort of five or six years, is it could be more at the um delivery end where somebody takes something off the backlog. 00:14:50.720 --> 00:14:55.759 And in that case, oopsie helps you split stories or it helps you have the right conversations. 00:14:55.759 --> 00:15:09.519 But because of the nature of it, you could either be taking the pen in a room just and stealthily saying to a stakeholder, tell me about tell me about the outcomes, okay, tell me about the outputs. 00:15:09.519 --> 00:15:24.240 It could be as as light as a conversation structure, or it could strategically help you uh deliver an entire big initiative or portfolio of initiatives. 00:15:24.240 --> 00:15:33.360 So you can because it's analysis, it's starting with breaking down and exploring, it can it can start at any at any point. 00:15:33.360 --> 00:15:35.360 And I'll go into that in the book. 00:15:35.360 --> 00:15:45.759 Um I have a mirror board uh a mirror verse template as well, which shows a kind of in the small and an in the large example. 00:15:46.080 --> 00:15:46.639 Fantastic. 00:15:46.639 --> 00:15:49.679 Okay, we'll definitely put a link to that in the show notes. 00:15:49.679 --> 00:15:53.919 Um let's let's go through each of the stages of this, each word of this. 00:15:53.919 --> 00:15:57.679 So the the first one is uh the O for outcomes. 00:15:57.679 --> 00:16:12.159 And this is gonna sound really dumb, but I want to hear what your definition of an outcome is because there's so many times, I can't count the number of times I've worked with companies, worked with people, where their definition is a number on a spreadsheet. 00:16:12.159 --> 00:16:16.240 We want to get to 100 million ARR or something, something like that. 00:16:16.240 --> 00:16:20.480 What do you count as a good definition of an outcome? 00:16:21.120 --> 00:16:26.000 So for me, an outcome is a change in the world. 00:16:26.000 --> 00:16:39.679 It's it's a change, a benefit, something that has got better, faster, less friction, more is a word that has a has a change associated with it. 00:16:39.679 --> 00:16:48.320 So yeah, uh a meaningful change in the real world that that creates value for somebody, for more definition of an outcome. 00:16:48.720 --> 00:16:51.679 And specifically, is there an anchor there? 00:16:51.679 --> 00:16:54.080 Is it who are you trying to create value for? 00:16:54.080 --> 00:16:56.000 Is it the value for the company? 00:16:56.000 --> 00:16:57.440 Is it value for the customer? 00:16:57.440 --> 00:16:59.279 Is it mutual value? 00:16:59.279 --> 00:17:03.600 Uh is there a right and a wrong for that? 00:17:04.240 --> 00:17:10.799 So typically it would be value for a customer, could be an internal customer or an external customer. 00:17:10.799 --> 00:17:18.480 Yeah, so that's the that's the same framing, the same way that you would start in the product space when you're looking at um opportunities. 00:17:18.480 --> 00:17:36.960 Um and so it's it's very much influenced by because of what I was saying before, Haran, so Teresa Torres's prioritize opportunities over solutions, it's kind of taking that bit of product thinking at the beginning and hooking it into the lower-level engineering practices. 00:17:37.519 --> 00:17:46.720 Yeah, like I said, this is incredibly basic, but the fact is, there are so many times I go into places and they're missing this as the most basic element. 00:17:46.720 --> 00:17:51.759 Is it is that something you see commonly or yeah, yes, totally. 00:17:51.920 --> 00:18:05.519 So, and and it's actually when I'm when I'm talking about um

IPSI and I'm in a room, the first thing I ask the audience is what do you think the difference is between an outcome and a deliverable, or an outcome and an output? 00:18:05.519 --> 00:18:09.680 And and there's always different answers in the in the room. 00:18:09.680 --> 00:18:15.599 And it can be quite a so that change to outcome-based thinking, I think, is really important. 00:18:15.599 --> 00:18:32.160 And and in the various environments that I've worked in when I've been trying to make these kinds of changes, it, you know, it might not be that I'm in there implementing oopsie, but I'm trying to get there, you know, trying to get them working better together and with better outcomes and accelerated delivery. 00:18:32.160 --> 00:18:38.960 The first thing is for me is about getting people to think in outcome-based language, think about outcomes. 00:18:38.960 --> 00:18:44.160 So people you have to keep, keep, keep reinforcing it, reinforcing it, reinforcing it. 00:18:44.160 --> 00:18:56.400 Um yeah, and oopsy doesn't really care how you do that, how you explore outcomes and how you prioritize them and how you define them, just that you do it with focus. 00:18:56.880 --> 00:19:01.920 And a number on a on a spreadsheet is not the outcome uh in in and of itself. 00:19:01.920 --> 00:19:04.000 It's that's never going to be the answer. 00:19:04.400 --> 00:19:10.640 No, I mean it might be a chain of whys that where there's a number involved in it, someone. 00:19:10.640 --> 00:19:25.440 But um yeah, it's being able to join up the work through all the levels of the whys to attach the the customer outcome, the change in the real world, and also uh have an understanding of how that impacts the business KPIs as well. 00:19:25.440 --> 00:19:37.119 So you would you would want to understand how how this customer level outcome is is affecting the bottom line, how you know what what the things that you're going to measure that make sure that you're doing that work. 00:19:37.680 --> 00:19:37.920 Okay. 00:19:37.920 --> 00:19:40.880 So going to the next level, that's that's outputs. 00:19:40.880 --> 00:19:50.400 Um, and you're using that to help focus the team on what are the things that they actually need to accomplish to to achieve that outcome for the customer. 00:19:50.400 --> 00:19:54.480 But how many outputs do you try and focus them on? 00:19:54.480 --> 00:20:01.759 Are you trying to do just the most common scenarios, or are you looking at edge cases and the full the full uh sweep of things? 00:20:01.759 --> 00:20:03.839 Or how do you balance this? 00:20:04.559 --> 00:20:08.799 So each layer of oopsie is an opportunity to prioritize. 00:20:08.799 --> 00:20:13.599 So, you know, I believe in in the the Parito principle at play. 00:20:13.599 --> 00:20:27.359 So kind of, you know, you you might have a load of different outcomes that you want to you want to explore, but there's going to be kind of 20% of those that deliver 80% of the output, of the impact, not the output. 00:20:27.359 --> 00:20:29.759 Same is true at the level of output. 00:20:29.759 --> 00:20:36.640 So this is quite this is quite unique to oopsie is the outputs level, and is it can be where people trip up. 00:20:36.640 --> 00:20:57.279 So by outputs, I mean the not the outputs of the process, of the software delivery process, or or the things that the that the developers are doing or that the team are doing, it's the the things that you verify that are happening as part of the product or system along the way that deliver the outcomes. 00:20:57.279 --> 00:21:08.000 So, for example, if you were implementing a an ERP system, a really important output would be a pay slip. 00:21:08.000 --> 00:21:13.519 Or a really important output would be that the bank balance is updated. 00:21:13.519 --> 00:21:37.359 And so when you collaborate around those and explore them and look for them, you can see this is the systems thinking part that informs oopsie, that there's real value there in those things that that you can interrogate as a customer or a user, or or they're an important part of the system's behavior that you know that it's working. 00:21:37.359 --> 00:21:41.839 So when you when you hunt for outputs, you find valuable things. 00:21:41.839 --> 00:21:53.039 And to answer your question, 20% of the potential outputs that you could produce probably deliver about 80% of the value. 00:21:53.039 --> 00:21:59.599 So depending on how big and complex the thing is, and how you want to use oopsie. 00:21:59.599 --> 00:22:12.480 You might be doing something very new and you might be very uncertain about it, in which case you'd look for a really small number of important outputs to get started. 00:22:12.480 --> 00:22:22.480 If you were doing something that you'd done before, you'd done it loads, it was well understood, you're making some changes, you might take a lot because you already understand them. 00:22:22.480 --> 00:22:34.720 So it depends how you're using Oopsy and the complexity of the space, whether you're probing and prototyping or whether you're getting on with something that's well understood. 00:22:35.039 --> 00:22:44.880 And it sounds like this is the level where you'd use a lot of the prioritization tools or impact mapping and things like that in conjunction with it to help decide. 00:22:44.880 --> 00:23:01.599 It also sounds like this is something that really works well based on top of uh what Jeff Patton does with story mapping, and this is almost the chapter headings or a certain level of uh of uh visibility for of the tasks. 00:23:02.480 --> 00:23:06.799 Yeah, so it it oopsie has got those things embedded into it. 00:23:06.799 --> 00:23:17.440 So at the outputs layer, at the outputs layer, that's when you break them down into opening game, mid-game, and end game, like in Jeff Patton's story mapping. 00:23:17.440 --> 00:23:22.720 And if you do it with oopsie, you actually avoid a load of the rework and the waste. 00:23:22.720 -->

00:23:26.559 So it's is like pre-story mapping. 00:23:26.559 --> 00:23:31.680 We don't need to do story mapping because the oopsie kind of takes takes care of it. 00:23:31.680 --> 00:23:48.720 So yeah, it's like story mapping with um outputs, it makes sure that each horizon of value, if you're using it to navigate something big and complex, each horizon of value tells a complete story that you're kind of growing over time, if that makes sense. 00:23:48.720 --> 00:23:52.000 It's hard to explain in a few minutes, isn't it? 00:23:52.480 --> 00:24:07.359 It is, but it's also it makes perfect sense in terms of concept-wise, but it as always the trick is in the details of implementation, but yeah, relating it back to these tools that that many of us have used before, it it does make a lot of sense. 00:24:07.359 --> 00:24:12.880 Okay, so the thing that uh that levels up into outputs are the processes. 00:24:12.880 --> 00:24:17.519 And process is one of those words that can mean a lot of different things. 00:24:17.519 --> 00:24:25.599 And when I first look at it uh before I read the the chapter, I was thinking, is this the software development process? 00:24:25.599 --> 00:24:29.839 So let's talk about what you mean by process in this case. 00:24:30.480 --> 00:24:43.440 So by process in this case, I mean the interactions that the the customer or the user of the product would have in order to generate the output. 00:24:43.440 --> 00:24:48.480 So one of the worked examples in the book is a shopping app. 00:24:48.480 --> 00:24:59.519 And one of the outputs of a shopping app might be the search results that you see on the screen that the user sees on the screen. 00:24:59.519 --> 00:25:07.599 And that's an important output for the user to see because it's it's saying, Okay, I'm gonna make a decision now about what I put in my basket. 00:25:07.599 --> 00:25:10.640 So a search results as an output is important. 00:25:10.640 --> 00:25:21.200 The process would be open your mobile app, uh type search for a product, receive the search results. 00:25:21.200 --> 00:25:27.519 Do you sort of mean so that the the process is the set of interactions that generate the output? 00:25:27.519 --> 00:25:34.000 If it if it was a back-end system without that much user interaction, there might be a number of process steps and then the output. 00:25:34.000 --> 00:25:48.079 So it might be in your ERP system, there might be run payroll, um, evaluate uh tax rules, um, check to see if Jenny's paying a pension. 00:25:48.079 --> 00:25:53.440 There might be a whole load of number of different processes, and then here's the pay slip. 00:25:53.839 --> 00:26:03.119 If I want if I wanted to relate that back to to uh the agile process, this feels very much like the story level of what we're doing. 00:26:03.119 --> 00:26:05.039 Is that a fair analogy? 00:26:05.519 --> 00:26:08.160 It's a very it's an absolutely fair analogy. 00:26:08.160 --> 00:26:12.000 So that's where the thing is looking most

like a user story. 00:26:12.000 --> 00:26:21.839 And this is one of the things, one of the reasons that oopsie came about, because the problem with user stories is that you don't know what level of detail they're at when you look at them. 00:26:21.839 --> 00:26:24.960 And so it could be much too big. 00:26:24.960 --> 00:26:35.279 And oopsie helps you split it down until you've got those discrete process steps, and then each one of those is useful as a user story. 00:26:35.279 --> 00:26:57.200 But because you've come at it through outcomes and outputs, there's some really big meteors, asteroids that you haven't explored, and now you're getting to the now, you're getting to a level of detail where you can actually build something that works and test it and prototype it without having explored everything. 00:26:57.200 --> 00:26:59.680 So it's exactly that. 00:26:59.680 --> 00:27:04.480 Then and what in fact, one of the ways of story splitting is by process. 00:27:04.480 --> 00:27:09.680 Another way of story splitting is sometimes referred to by outputs. 00:27:09.680 --> 00:27:17.279 So it's so oopsy is like enforcing that breaking down in good practice so that you end up with something that's a nice vertical slice. 00:27:18.079 --> 00:27:25.599 I love how you're talking about this as I was reading and as we've talked in the past, this is not a new thing. 00:27:25.599 --> 00:27:35.119 This is just a refinement of existing practices so that you're doing it in terms of best practice, uh following the way it was always intended to be done. 00:27:35.119 --> 00:27:40.880 But a lot of us slip up along the way, we drift and we we lose sight at times. 00:27:40.880 --> 00:27:46.000 So this is uh just helping to uh enforce better ways of doing it. 00:27:46.400 --> 00:27:47.839 Yeah, that's exactly it. 00:27:47.839 --> 00:27:49.920 It enforces principles. 00:27:49.920 --> 00:27:56.640 So with other frameworks, you could be doing all of the ceremonies, right? 00:27:56.640 --> 00:28:19.200 You could be having your daily stand-ups, you could be using Jira, you could be, you know, creating your backlogs in a certain way, you could be having all of these uh ceremonies and all of this mechanics, and still not know what you're working on, and still not know how to get started quickly, and still not how to split work up in a way that gets you started and gets the and enforces the feedback loop. 00:28:19.200 --> 00:28:29.680 So I was really frustrated by all of that kind of religious, agile with no without any conversations about outcome or value. 00:28:29.680 --> 00:28:32.640 I found that really enormously frustrating. 00:28:32.640 --> 00:28:35.200 And so um, yeah, oopsie. 00:28:35.200 --> 00:28:44.400 If you follow oopsie, you can't miss those principles because it's embedded into it, but it's still flexible enough for you to apply it in any environment. 00:28:44.799 --> 00:28:46.400 Okay, so we've hit oop. 00:28:46.400 --> 00:28:48.400 Uh, let's get moved on to C. 00:28:48.400 --> 00:28:55.440 And so the the uh the SNC is scenarios, and that is uh a lens that you're seeing this through. 00:28:55.440 --> 00:29:02.319 But one of the things that struck me about this is you talk about how it's related to behavior-driven development and things like that. 00:29:02.319 --> 00:29:13.680 What is the difference between what you're talking about scenarios and what everyone else is talking about these days as evals for for AI-driven product development? 00:29:13.680 --> 00:29:15.920 Is there fundamentally any difference there? 00:29:16.799 --> 00:29:18.400 So I I see a difference. 00:29:18.400 --> 00:29:21.200 I mean, they're both they're both of our kind of testing, right? 00:29:21.200 --> 00:29:33.279 So Boopsi is trying to use specification by example and BDD practices to bring forward thinking around detailed examples and tests so that you get it right at the first time. 00:29:33.279 --> 00:29:45.759 So conversations about examples, conversations about concrete examples that help you do the right thing and consider the complexity with with uh collaboration where it's needed. 00:29:45.759 --> 00:29:46.720 So where that's needed. 00:29:46.720 --> 00:29:52.799 Oopsie kind of gets you to do that, which is TDD, BDD, Pad informed stuff. 00:29:52.799 --> 00:30:05.759 And evals uh in terms of AI uh includes all sorts of checks and evaluations and criteria that you could say are tests. 00:30:05.759 --> 00:30:12.799 They often are happening after the event, but you could use those evals to inform development as well. 00:30:12.799 --> 00:30:26.720 So I guess the thing is that the way I see it is mostly when we're thinking about e-valves, we're talking about thinking more about ways of testing and evaluating more qualitative type procedure. 00:30:26.720 --> 00:30:47.279 So, for example, to deal with how empathetic a response is or whether the language was useful or not, because AI always comes out with a different answer, there has to be uh ranges and scorings and things like that for happening. 00:30:47.279 --> 00:30:58.720 Um, in terms of the scenario chance and the influence of BDG in the specification by example, is particularly useful when it's more quantitable. 00:30:58.720 --> 00:31:13.200 So, where you know the thing that can't have a range, like the the amount of your pain right in in the ERP scenario, there's there's not like kind of range of suitable answers for that. 00:31:13.200 --> 00:31:37.039 It's like either right or wrong, and it so that's where it's the expected result that says, and this is what you expect, and so in that way, AI Evals could benefit from that a lot because you know there's some evaluations are qualitative and we care about scoring and how confident we are in the the answer. 00:31:37.039 --> 00:31:45.440 But the really, really important stuff will be in that concrete uh quantitative data. 00:31:45.440 --> 00:31:52.079 So that's where kind of that's where the S of boop C comes from.

00:31:52.079 --> 00:31:58.799 Mostly talking about data-rich logical systems that you can define in advance. 00:31:58.880 --> 00:32:04.960 So they're it has to do when it's deterministic in or non-deterministic. 00:32:04.960 --> 00:32:22.559 And yes, yeah, so you need to use the evals when it's non-deterministic, but you're trying to put something uh um determinant uh deterministic ranges around non-deterministic outcomes, which is probably a terrible way of saying it, but I think we we know what we're trying we're trying to say there. 00:32:22.880 --> 00:32:24.079 Yeah, yeah, we do. 00:32:24.079 --> 00:32:33.519 And and Ipsy um you know, defining all of those test conditions up front wouldn't be beneficial in that scenario, right? 00:32:33.519 --> 00:32:36.000 If for for that type of eval. 00:32:36.000 --> 00:32:47.119 And so, and so yeah, you you would, I guess, use a mixture of eval-based ARI tests and the best part of um. 00:32:48.799 --> 00:32:49.920 Makes sense. 00:32:49.920 --> 00:32:52.720 Um, so okay, so the last one is inputs. 00:32:52.720 --> 00:33:00.720 And we talk about outputs and outcomes and things like that all the time uh in my world, but inputs less so. 00:33:00.720 --> 00:33:03.440 So, again, what do you mean by inputs here? 00:33:03.440 --> 00:33:05.279 What's the what's the secret sauce? 00:33:06.319 --> 00:33:12.880 So um by inputs, I mean the concrete test data that normally comes with a test. 00:33:12.880 --> 00:33:33.839 And so testers will be familiar with this when I talk about specification by example, because often we leave things really ambiguous, and uh because we're we're kind of abstracting the rule of whatever it is that we're building, and that that might exist in an acceptance test, some kind of um user story acceptance test. 00:33:33.839 --> 00:33:48.160 But oopsie encourages you to actually collaborate around the specific input data and preconditions that may need to be arranged to make the thing work as expected. 00:33:48.160 --> 00:34:18.880 And from it, so in my early years, I was a tester and a test consultant and a test manager and just have that lived experience of all the assumptions around the really ambiguous requirements documents, and then it's only when you get to use realistic data and try something that you realize nobody's thought of a whole load of things or made massive assumptions about the structure of the data that's being used to drive uh the test. 00:34:18.880 --> 00:34:25.599 So it sounds like a lot of it sounds like um a lot of detail, but it's it's not somebody always has to do the thinking. 00:34:25.599 --> 00:34:48.079 And if you if you choose, if you choose important examples, you know, using this kind of hunt the value sort of lens that we've applied at all the different layers, uh, then then this bit of collaboration around exactly what this functional test might look like gives you a really clear target for development. 00:34:48.079 --> 00:34:52.639 So you're kind of going, it's kind of narrowing right into something very specific. 00:34:53.039 --> 00:34:54.719 I'm gonna go back to what I said earlier. 00:34:54.719 --> 00:35:01.840 This is the the S and the I, the scenarios and the inputs, talking about the criteria for testing and everything else. 00:35:01.840 --> 00:35:07.920 A lot of the time when we talk about product world and design world, we talk about falling in love with the problem. 00:35:07.920 --> 00:35:17.519 And people get very romantic about that, but they're not always falling in love with solving the problem for people, just the possibilities around it. 00:35:17.519 --> 00:35:22.639 The actual build and delivery of stuff is messy, it is difficult, it is fiddly. 00:35:22.639 --> 00:35:37.119 And the fact that you're talking about uh all this from the perspective of someone who comes from a testing and building and delivery background, but who also is focused on why are we doing this, who are we trying to impact, what what did they get out of it? 00:35:37.119 --> 00:35:48.960 I think that's squaring the circle and bringing it all together in a way that a lot of the time we stop at the point of, okay, now we're gonna hand this off to other people to actually ship. 00:35:49.440 --> 00:35:50.639 Yeah, exactly. 00:35:50.639 --> 00:36:04.079 And uh one of the principles of of oopsie about using examples to guide conversations is trying to include those examples at every layer so that you're able to visualize your understanding of what that means. 00:36:04.079 --> 00:36:08.320 And that's that's why there's lots of illustrations and lots of collaboration. 00:36:08.320 --> 00:36:13.519 And yeah, and I mean that's how that's how I worked when oopsie was developed. 00:36:13.519 --> 00:36:31.280 So I, you know, I used to work in a in a startup uh going back kind of 10, 12 years, and we weren't using oopsie per se at that time, but we were building brand new capabilities, totally green field in the retail space. 00:36:31.280 --> 00:36:54.639 We were defining our APIs and how our product would work, and we were uh we actually defined the exact test data we were going to use in the example, showed it to the customer, big retail customer, and said, if we test this and show you this output, if we do that in a couple of months' time, will that be the right thing? 00:36:54.639 --> 00:37:06.239 And they they saw an error, a fundamental assumption in in our model, a piece of data which they didn't have available to us, which was wholly fundamental to our understanding. 00:37:06.239 --> 00:37:11.360 And if we hadn't done that, we would have wasted three or four months' work. 00:37:11.360 --> 00:37:24.800 And then the other thing is when we were showing we, you know, we're going to produce this type of example, they'd say, Oh, um, but if you use this different example, that would be better because it will help us get confidence in your progress. 00:37:24.800 --> 00:37:32.719 And so often we assume that there's a too much

detail for business folks or too much vision stuff for engineers. 00:37:32.719 --> 00:37:43.360 But the value of having everybody, just a select number of people, you know, collaborate at key points through the product development process just is really powerful. 00:37:43.840 --> 00:37:46.159 So, Jenny, this has been fantastic. 00:37:46.159 --> 00:37:48.559 I think we've got time for for just one more question. 00:37:48.559 --> 00:37:55.840 And uh it goes back to the the adoption of this, because this is all makes perfect sense. 00:37:55.840 --> 00:38:01.360 This sounds just elementary and sensible, and I know it is it isn't. 00:38:01.360 --> 00:38:10.000 Uh so which step or what part of this do you see most teams get wrong or have the most trouble making sure that they're uh doing it well? 00:38:10.719 --> 00:38:22.000 I guess with with the oopsie framework itself, the output bit takes a little bit of thinking in terms of the actual output they're seeing rather than a process step. 00:38:22.000 --> 00:38:32.000 But this the problem in the world, particularly with AI, everybody going fast in the wrong direction is the outcomes, the outcomes thing. 00:38:32.000 --> 00:38:36.320 So, yeah, the the the outcomes has the biggest impact, I think. 00:38:36.320 --> 00:38:41.280 The outputs is a really useful uh opportunity to prioritize. 00:38:41.280 --> 00:38:51.039 But in terms of the implementation, this is the funny thing, is that um I was originally doing this oopsie thing kind of 10 years ago. 00:38:51.039 --> 00:39:00.159 I was on the BDD conference scene, I I wrote a couple of blogs, I did lots of different um conferences and meetups, and then I got involved in other things. 00:39:00.159 --> 00:39:14.480 And it was actually um Jeff Patton who who DM'd me and said, my big client in Estates is using Oopsie to underpin their whole development framework. 00:39:14.480 --> 00:39:16.000 And I was like, what? 00:39:16.000 --> 00:39:29.920 And at the same time, I heard that it was being used in a big global consultancy and another big, huge tech giant company, and they've just latched onto it because it's easy. 00:39:29.920 --> 00:39:34.480 And it's it's the thing that stuck with them. 00:39:34.480 --> 00:39:39.440 So this is why I wrote the book, is because it's been, you know, it hasn't left me alone. 00:39:39.440 --> 00:39:43.360 So it keeps coming, rearing its head again. 00:39:43.360 --> 00:39:52.639 And then when I started to, when I started to do some digging, you know, I found that it was being adopted all over the place just from this one book that I'd written. 00:39:52.960 --> 00:39:54.000 That's fantastic. 00:39:54.000 --> 00:40:14.159 And I want to encourage you, everyone listening, because I had something like this happen to me this past week where something I'd written a while ago uh that I assumed nobody was using, a couple of people came up to me in the past week and told me they were using it, not at the scale that people were coming back to you and Jeff Patton was not uh DMing me about about my thing. 00:40:14.159 --> 00:40:23.039 But if you don't assume that just because you're using somebody's framework, that they know about it, definitely drop them a line. 00:40:23.039 --> 00:40:29.280 It is such a a wonderful thing to hear from people that uh and we're all very open to it. 00:40:29.280 --> 00:40:42.079 If you can reach out to Jenny if you're using Oopsie, reach out to me if you're using my stuff, reach out to anybody if you're using their stuff, because it is so reassuring, it is so welcome, and it's so it's just a wonderful thing to do. 00:40:42.320 --> 00:40:43.760 So yeah, it blew away. 00:40:43.760 --> 00:40:44.880 Yeah, it really did. 00:40:45.679 --> 00:40:47.280 Jenny, thank you again so much. 00:40:47.280 --> 00:40:48.559 This has been fantastic. 00:40:48.559 --> 00:40:51.840 I love the book, I love the approach, I love how complete it all is. 00:40:51.840 --> 00:40:54.239 And uh, where can people get the book? 00:40:54.239 --> 00:40:57.760 Anywhere in any good bookstore or special places? 00:40:58.480 --> 00:41:00.320 Um, so it's on Amazon. 00:41:00.320 --> 00:41:03.280 Um, I think it's on Amazon everywhere. 00:41:03.280 --> 00:41:11.599 Uh, if you're in the UK, then if you go onto the Oopsie website, which is oopsie.org, um then you can get a copy from me. 00:41:11.599 --> 00:41:12.880 It's the same price. 00:41:12.880 --> 00:41:16.159 The the copy from me is slightly better quality. 00:41:16.159 --> 00:41:18.960 Um that's the only that's the only difference. 00:41:18.960 --> 00:41:20.880 It's the exact same book. 00:41:20.880 --> 00:41:22.159 But only in the UK. 00:41:22.159 --> 00:41:23.679 You can only check it in the UK. 00:41:24.000 --> 00:41:24.239 Fancy. 00:41:24.239 --> 00:41:26.480 So, Jenny, thank you again so much. 00:41:26.480 --> 00:41:28.400 Uh thank you for coming on. 00:41:28.800 --> 00:41:29.440 You're welcome. 00:41:29.440 --> 00:41:30.880 Thanks for having me. 00:41:30.880 --> 00:41:37.519 The product experience hosts are me, Lily Smith, host by night and chief product officer by day. 00:41:37.920 --> 00:41:40.639 And me, Randy Silver, also host by night. 00:41:40.639 --> 00:41:46.639 And I spend my days working with product and leadership teams, helping their teams to do amazing work. 00:41:47.039 --> 00:41:50.960 Lou Ran Pratt is our producer, and Luke Smith is our editor.