WEBVTT
00:00:00.160 -->
00:00:04.272Lily, do you have any idea how many people listen to this podcast every week?
00:00:05.581 -->
00:00:14.308I think there's probably a minimum of five, because there's me and you and maybe the guest listens back, and Luke, the editor, and Luaranda producer.
00:00:14.308 -->
00:00:15.653So yeah, minimum of five.
00:00:16.742 -->
00:00:18.829It does feel like that sometimes, doesn't it?
00:00:18.829 -->
00:00:23.949I mean, we record here in the privacy of our homes and we have no idea who's out there listening.
00:00:24.900 -->
00:00:30.096Oh well, I know for a fact that we had more than that for this week's episode, even before we put it live.
00:00:30.231 -->
00:00:36.066Oh, that's true, and I promise you are analytics proof that we have way more than that every week.
00:00:36.066 -->
00:00:43.863But this one was special because, you know, as an experiment, we recorded it live at the Mind the Product conference in London.
00:00:43.863 -->
00:00:45.927Did you see what I did there?
00:00:46.548 -->
00:00:50.923Yes, yes, randy, I caught that, oh dear.
00:00:50.923 -->
00:01:00.972This week's chat is indeed all about experimentation and we had two very special guests and a whole bunch of special friends who joined us during the lunch break at the conference.
00:01:00.972 -->
00:01:07.352We had Iqbal Ali, an experimentation coach, who's been supporting my product team at BBC Maestro.
00:01:08.141 -->
00:01:15.487And we had Faith Forster, vp of Product at Dext, who shared an amazing approach to experimentation that I've since used with some of my clients.
00:01:16.412 -->
00:01:19.602It was a lot of fun doing it live, and maybe we should do more of these.
00:01:19.602 -->
00:01:22.950So if you want us to come to your town, give us a shout.
00:01:23.691 -->
00:01:26.626Meanwhile, let's actually get to our chat with Iqbal and Faith.
00:01:26.626 -->
00:01:34.152On with the podcast, the product experience is brought to you by Mind the Product.
00:01:34.152 -->
00:01:38.268Every week on the podcast we talk to the best product people from around the globe.
00:01:38.891 -->
00:01:45.516Visit mindtheproductcom to catch up on past episodes and discover loads of free resources to help you with your product practice.
00:01:45.516 -->
00:01:53.864You can also find more information about Mind the Product's conferences and their great training opportunities happening around the world and online.
00:01:54.486 -->
00:02:03.921Create a free account on the website for a fully personalized experience and to get access to the full library of awesome content and the weekly curated newsletter Mind.
00:02:03.921 -->
00:02:07.614The Product also offers free product tank meetups in more than 200 cities.
00:02:07.614 -->
00:02:09.602There's probably one near you.
00:02:11.888 -->
00:02:15.276Right, good afternoon folks, welcome, welcome, thank you for joining us.
00:02:15.276 -->
00:02:16.300My name is James.
00:02:16.300 -->
00:02:19.992I'm going to be your host for this afternoon in the Ideation Station.
00:02:19.992 -->
00:02:24.127That's an easy word to say after a mouthful of chicken adobo, right.
00:02:24.127 -->
00:02:30.645So welcome to this fantastic live recording of the product experience podcast.
00:02:30.645 -->
00:02:33.211We've also this afternoon got some other stuff happening here.
00:02:33.211 -->
00:02:38.616Come back in the afternoon break for a karaoke, powerpoint karaoke.
00:02:38.616 -->
00:02:41.425Oh yeah, if anyone's seen that before, it's a whole world of fun.
00:02:41.425 -->
00:02:44.256You should do it, absolutely, yeah.
00:02:44.256 -->
00:02:56.055So the other great thing that's happening this afternoon is our friends are, optimise Lee, are giving out free ice creams down at their stand, so get down there, if you can do so.
00:02:56.055 -->
00:03:05.186In the meantime, I would like to introduce you to Craig from optimise Lee, who's going to say a few words, and, if you wouldn't mind welcoming him to the stage, the man who's buying you all a cornetto.
00:03:07.651 -->
00:03:08.391What an intro.
00:03:08.391 -->
00:03:10.443Thank you so much and so welcome.
00:03:10.443 -->
00:03:11.889Thank you very much for having us here today.
00:03:11.889 -->
00:03:15.723As mentioned, this podcast today has been sponsored by optimise Lee.
00:03:15.723 -->
00:03:18.752Hopefully the name should ring a bell with most people.
00:03:18.931 -->
00:03:24.396If not, we are one of the leading AB testing and experimentation platforms globally.
00:03:24.396 -->
00:03:34.044We work with massive brands such as ASOS, sky, farfetch, bbc and the like and really helping them experiment their way to product success.
00:03:34.044 -->
00:04:07.942So a lot of the work that we do nowadays is bringing experimentation much earlier within the software development life cycle helping product teams and engineering and development teams set up things like feature flagging, feature management, feature rollouts, feature variables all the features and product experimentation really helping them to those kind of teams get more insights into what's working within their digital products and helping them influence where their product is going to go next and getting the best business value and metrics and KPIs and all those good things.
00:04:07.942 -->
00:04:10.667So thank you very much for having us.
00:04:10.667 -->
00:04:13.894As mentioned, we do have the stand downstairs.
00:04:13.894 -->
00:04:15.145Do come for ice cream.
00:04:15.145 -->
00:04:17.216We also have a live product demo happening as well.
00:04:17.216 -->
00:04:24.382So if you want to find out a little bit more, you can come and talk to us as well, or you can find us at optimiseecom, and with that I'll hand you back to James.
00:04:28.649 -->
00:04:29.050Fabius.
00:04:29.050 -->
00:04:32.213Thank you, greg, right, okay, on to the session.
00:04:32.213 -->
00:04:34.384So a few bits of housekeeping.
00:04:34.384 -->
00:04:46.642This is a live recording, so if anything goes wrong which I'm sure it's not going to, but in case it does, we might have to stop things and get the recording back on track and all of that kind of stuff.
00:04:46.642 -->
00:04:51.725So it's really because it's really important that we can put this out as a wonderful episode.
00:04:51.725 -->
00:05:00.399But so, yeah, so what we're going to do the other thing let you into behind the scenes of all of the entertainment industry, right, what we need to do at the beginning.
00:05:00.399 -->
00:05:02.346Has anyone seen live at the Apollo before?
00:05:02.346 -->
00:05:10.151Yeah, that kind of thing, right, so we need a huge round of applause, whooping, hollowing, throwing things in the air, whatever you want to do, basically.
00:05:10.151 -->
00:05:17.824So I'm going to go one, two, three, and then you're going to go mental for our wonderful panel and then they're going to kick off the recording.
00:05:17.824 -->
00:05:18.687Does that sound good?
00:05:18.687 -->
00:05:22.637Yeah, nods, fantastic, all right, okay, only marks.
00:05:22.637 -->
00:05:24.502One, two, three.
00:05:39.411 -->
00:05:40.776Okay, that is really weird.
00:05:43.791 -->
00:05:44.012Yes.
00:05:44.810 -->
00:05:48.918We normally do this from home late at night on a Monday or Tuesday.
00:05:48.918 -->
00:05:51.264It is very strange to do this with people in the room.
00:05:51.264 -->
00:05:52.632Thank you for being here.
00:05:52.632 -->
00:06:01.036And actually we want to shout out to one person sitting here especially who's not even looking up at the moment Lu Ron, raise your hand at least.
00:06:01.036 -->
00:06:03.286Lu Ron, it works for my new product.
00:06:03.286 -->
00:06:04.129He is our producer.
00:06:04.129 -->
00:06:08.560He is the man who makes all of this run smoothly, and we couldn't do it without him.
00:06:08.560 -->
00:06:09.543So thank you, lu Ron.
00:06:16.211 -->
00:06:28.459So today we're going to be talking about Experimentation and we have some great guests with us today and, in the plastic podcast style, I'm going to ask them both to introduce themselves.
00:06:28.459 -->
00:06:29.601So let's start with you.
00:06:29.601 -->
00:06:31.047Faith, faith forster.
00:06:31.047 -->
00:06:32.129Vp of product at Dext.
00:06:32.129 -->
00:06:34.831Give us a quick intro shop.
00:06:36.031 -->
00:06:38.216So I've been doing product now for about 12 years.
00:06:38.216 -->
00:06:47.764I started out, I was originally a management consultant, created my own startup to solve the problems I saw in consulting, so started as a founder, actually did that for about five years.
00:06:47.764 -->
00:06:52.300Learn everything the hard way was very naive going into it.
00:06:52.300 -->
00:06:56.610This is now, with Dext, my fourth SMB product.
00:06:56.610 -->
00:07:00.925I tend to focus on SMB products because I've run my own business twice.
00:07:00.925 -->
00:07:07.983The tech business is an e-commerce business, so I I find I relate to those problems much more and so I feel like I'm a better person for it.
00:07:08.990 -->
00:07:09.451Amazing.
00:07:09.451 -->
00:07:10.112Thank you faith.
00:07:10.112 -->
00:07:16.915And we also have Igbal Ali, our experimentation coach at the BBC, bbc maestro, where I work.
00:07:16.915 -->
00:07:19.454Igbal give us a quick intro.
00:07:19.995 -->
00:07:24.887Yeah, so I've been doing experimentation now for about like over 10 years.
00:07:24.887 -->
00:07:27.156Fell into it totally by accident.
00:07:27.156 -->
00:07:34.021I used to do UX and Development, so now I'm an experimentation consultant and coach.
00:07:34.021 -->
00:07:44.675Last permanent position was head of experimentation at train lines and now I'm kind of work with lots of different companies and Teams and stuff like that.
00:07:44.675 -->
00:07:46.552So yeah, Amazing.
00:07:46.591 -->
00:07:48.416Thank you so much and it's great to have you here.
00:07:48.416 -->
00:07:52.410So let's start off with a down to the basics.
00:07:52.410 -->
00:07:56.814How do you guys define experimentation?
00:07:56.814 -->
00:07:56.855I?
00:07:58.891 -->
00:08:02.500Feel like he's more of the experimentation, except the way I tend to look.
00:08:02.500 -->
00:08:06.490Experimentation, the UX I've spent quite a bit of time in early stage startups.
00:08:06.490 -->
00:08:12.956It's finding finding ways to find data and help you make decisions, particularly when there isn't any available.
00:08:14.350 -->
00:08:18.904Yeah, I think I pretty much agree with that definition actually, so I've the.
00:08:18.904 -->
00:08:30.863The way I define it is a Systematic, data-driven approach to making decisions by mitigating risks, to optimize user experiences or whatever the goal is.
00:08:31.370 -->
00:08:34.855So yeah, so a B tests, that's it right.
00:08:35.336 -->
00:08:40.350So, yeah, yeah, it's all it is is just splitting traffic and then just seeing what works.
00:08:44.538 -->
00:08:49.830What is the most common mistake you see people making when they are experimenting or possibly when they're not experimenting?
00:08:49.830 -->
00:08:53.158The common thing.
00:08:53.879 -->
00:09:19.109I've been seeing that as it pertains, as it's relevant to this group maybe because I think this is more of a product sort of group is the fact that Experimentation is often seen as a bolt on, like you do everything as per usual delivery, design, development and then just at the release part say oh okay, let's just validate that all this stuff that we've done actually has the impact that we expected.
00:09:19.109 -->
00:09:36.403So, yeah, I see that problem so often and it's because there's yeah, there's a perception problem with With testing that it's just a B testing and I guess there's something that's not really a problem that it's just a B testing and I guess there's something we'll probably get into a little bit more.
00:09:38.552 -->
00:09:39.315I could list a few.
00:09:39.315 -->
00:09:44.649Yeah, not being clear on what you're trying to learn, not measuring properly.
00:09:44.649 -->
00:09:48.841So we've run experiments before and then, four weeks in, realized no one actually set up the measurement for it.
00:09:48.841 -->
00:09:58.309Not ideal Using experimentation to try and work out whether or not you've built the right stuff Without really understanding the problem in the first place.
00:09:58.309 -->
00:10:14.409From an early stage perspective, one of my big rights with experimentation was people say you know we need to run this, for you know X number of weeks and spend X number of hundreds of thousands or whatever to to get the Sustitutical significance, to know whether or not this is true.
00:10:14.409 -->
00:10:21.897It's like from an early because I've always focused on B2B and I've often been nearly stage Any data is better than no data.
00:10:21.897 -->
00:10:23.361So you'd like just run it.
00:10:23.361 -->
00:10:28.302Even if it's 60%, it's better than 0%, and so that's one of the things I get quite hung up on.
00:10:28.302 -->
00:10:31.316And people start telling me that you know we need to have X number of traffic.
00:10:32.270 -->
00:10:47.301Yeah, and actually just to just to jump in there as well, like one of the other problems I see is like there's an in, there's such a focus on statistical significance and calculations and the major problems are often much, much earlier in the funnel.
00:10:47.301 -->
00:10:56.057So it's, it's it's crap in, crap out, and if you're just focusing on the crap that's out, then yeah, that there's a serious problem there.
00:10:56.177 -->
00:11:13.863Yeah okay, so how do you, how do you work with teams and how do you work in your business to set things up in the right way so that you, you do have the right kind of Foundations in place, like in the culture and in the business and in your processes, to do experimentation well.
00:11:15.312 -->
00:11:17.081So one of my previous products want to.
00:11:17.081 -->
00:11:18.990It's an analytics tool for small business owners.
00:11:18.990 -->
00:11:21.755We we didn't have product market fit.
00:11:21.755 -->
00:11:22.798We knew that.
00:11:22.798 -->
00:11:30.370We knew the product experience wasn't quite right, but we needed to demonstrate progress against two key metrics to get the funding for an x-round, basically.
00:11:30.370 -->
00:11:37.769So it has six months to prove we could have a positive impact on these two metrics, and so we used experimentation as a way to do that.
00:11:37.769 -->
00:11:39.529So we we ran an ideation session.
00:11:39.529 -->
00:11:41.575We came up with a bunch of ideas of things that could do.
00:11:41.575 -->
00:11:44.842We filtered that by what would have the most impact on those two metrics.
00:11:44.842 -->
00:11:50.139We set up a system so every week we would decide which was the next experiment we'd run.
00:11:50.139 -->
00:11:52.309We'd work out what we're trying to learn from it.
00:11:52.309 -->
00:11:54.855What's the simplest, easiest way to test that?
00:11:54.855 -->
00:11:58.143So it might be just simply putting a button in and seeing if people click it.
00:11:58.143 -->
00:12:09.481It had to be able to be built within a week and then, depending on the nature of the experiment running, we would run it for anywhere from one to four weeks and we, just, we just kept doing it, kept doing it, kept doing it.
00:12:09.481 -->
00:12:09.932We.
00:12:09.932 -->
00:12:17.941So within that six-month period, we got a seven times multiplier on those two key metrics, like Our funding board, were astounded by what we achieved in that time frame.
00:12:20.852 -->
00:12:24.990Now we'll talk about it, I think, in a second, but we're taking very different, different approach.
00:12:24.990 -->
00:12:28.820This is a much larger business, so it's harder to mobilize around that.
00:12:28.820 -->
00:12:32.059You've also got, you know, a roadmap.
00:12:32.059 -->
00:12:35.779This is the stuff that you need to build right, and so we we are.
00:12:35.779 -->
00:12:37.870We are treating experimentation at different levels.
00:12:37.870 -->
00:12:42.370We have certain teams whose job it is to run experiments to optimize certain things.
00:12:42.370 -->
00:12:49.070We are actually in the middle of a big pivot as well, and so we are looking at how do we use experiment to de-risk the pivot that we need to make.
00:12:52.399 -->
00:13:02.936So the question was how do you set the business up for success with experimentation, like what are the kind of things that you need in place to make experimentation work well?
00:13:03.136 -->
00:13:03.597in the business.
00:13:03.597 -->
00:13:04.527Ah, yes, yes.
00:13:04.527 -->
00:13:46.748So I think the core things are that understanding that experimentation is a process it's very much systematic, Going back to the definition is a systematic data-driven approach, and so just setting up the systems and just training the individual teams, people like designers, developers in their specific role in that experimentation process and, with the product owners, just kind of getting them to understand the process itself and why we're doing certain things.
00:13:46.748 -->
00:14:05.739And also that it's not a fixed sort of, you know, a fixed process but, depending on what your goal is, thinking about, the strategy for using the tool sets that you've got available to be able to run the experiments and find out answers and stuff that you want to find out.
00:14:06.807 -->
00:14:09.278I'm curious, so I think you're preaching to the converted.
00:14:09.278 -->
00:14:14.907With the audience in the room today and our podcast audience, we love experimentation.
00:14:14.907 -->
00:14:16.486Yeah, yay, okay.
00:14:16.486 -->
00:14:21.548One of the challenges I've seen, though, is the people that we work with.
00:14:21.548 -->
00:14:27.099They say they like this, and we say the words fail fast.
00:14:27.099 -->
00:14:37.116What we actually mean is learn fast, but when the experiment doesn't work, or works in disproving something rather than proving it, how do you message that?
00:14:37.116 -->
00:14:44.518How do you create the culture of experimentation when, sometimes, the experiments don't immediately yield good results?
00:14:46.645 -->
00:14:48.912I think it comes back to the purpose of why you're experimenting.
00:14:48.912 -->
00:14:53.592So I talked about DeX at the moment, where that's still a big pivot.
00:14:53.592 -->
00:14:58.769For those of you who don't know DeX, we're an accounting software.
00:14:58.769 -->
00:15:06.840It's historically came from an amazing AI engine AI engine we have that reads receipts and then pulls that data and puts it into the accounting software.
00:15:06.840 -->
00:15:18.710So that's our history, and they've gone to markets through accounts and bookkeepers the whole time, and we're about to pivot, because of the commercial model change, to actually going to markets through small businesses as well.
00:15:18.710 -->
00:15:20.134But that's going to be our primary audience.
00:15:20.254 -->
00:15:30.080And so, if you think about our website as an example, the purpose of the website was always to drive traffic to our sales team, who would then contact the accounts or bookkeepers and follow up on that.
00:15:30.080 -->
00:15:32.090Where are you designing the website?
00:15:32.090 -->
00:15:37.110And we're going to make that now about the small business owners, and so our poor sales people are freaking out.
00:15:37.110 -->
00:15:46.101Their pipeline of leads is potentially going to die, and so we're using experimentation not as a way of telling us what the answer is.
00:15:46.101 -->
00:15:52.592We've already made that strategic decision that we're going to focus on small business owners, but how do we de-risk that for the purpose of the sales team.
00:15:52.592 -->
00:15:57.732How do we make sure that we manage that transition in a way that doesn't kill off their part of the business as well?
00:16:00.280 -->
00:16:36.086I do think when you talk about culture, it comes back to mindset, genuine curiosity and trying to learn, but also it's about metrics, right, and so the starting point that we are, when we are starting, is actually re-looking at our metrics and working out whether or not we're focusing on the right things, and so we're in the middle of doing a review around our definition of the customer lifecycle what the most important stages are and what is good and what is great for each of those stages and have a conversation about that first, so we can then work out what are the implications and therefore, what do we need to do and what do we experiment and learn from and how do we make sure that we've achieved something that's worthwhile.
00:16:37.288 -->
00:16:56.775And, I suppose, like to kind of continue that discussion of like what happens when there's lots of failures or, like you know, negative outcomes from the experimentation, of like, hey, we've done all this stuff, none of it worked, or like none of it kind of moved the metrics in the way that we were expecting it to or had the outcome that we expected it to.
00:16:56.775 -->
00:17:19.671So I think that's a good point for you If you then, you know, do have a positive outcome, but people are sort of like biased in the team towards a particular way of, you know, introducing a feature or whatever it is Like is there ever a challenge of getting people to listen to the data and to listen to the output of the experiment?
00:17:19.671 -->
00:17:25.452It's like, here we are, we're going to get the data, and people are like, yeah, yeah, whatever, I want to do this thing.
00:17:27.964 -->
00:17:29.970Yeah, that's certainly something happened to us.
00:17:29.970 -->
00:17:32.355I find the quicker you can do it the better.
00:17:32.355 -->
00:17:48.853So if you've invested months into running an experiment and then it fails, people find it very difficult to let go of that because you've already invested a certain amount of time, whereas if you can just find, like I said, a Vonto, there was a particular feature we were looking to build.
00:17:48.853 -->
00:18:01.398Experiment was actually a discovery exercise, and so they didn't get any results for two and a half months, and so by that time people found it really hard to let go of that as an idea.
00:18:01.398 -->
00:18:13.769Instead, we started doing like let's just build a button and see people click on that button, so you can build that in a day and then you get the results within two weeks and it's very easy to then go okay, fine, that didn't work, let's try something else.
00:18:14.050 -->
00:18:32.771I think reframing the conversation from being about conversion rates and certain sort of business critical metrics to insights and learnings is really key.
00:18:32.791 -->
00:18:41.575So there's an argument to say that there is never really a failed experiment, and I know that a lot of experimenters would have this argument.
00:18:41.575 -->
00:18:46.751But in fact, there is a failed experiment is where you haven't learned anything.
00:18:46.751 -->
00:18:53.633So the key thing to really focus on for the experiments is this is what we've learned.
00:18:53.633 -->
00:18:56.260This is what we're going to do.
00:18:56.260 -->
00:19:05.058Next in terms of this is how it's helped our decision making process and, if you can, if you can, we focus in that way.
00:19:05.058 -->
00:19:13.388It kind of takes the mind and the edge off of the fact that you know maybe it didn't have the conversion rate that you expected.
00:19:13.388 -->
00:19:23.674And also, like going back to what Faith said the more investment you put in, the greater importance everyone places on the outcome.
00:19:23.674 -->
00:19:38.964So the way to fix that is to you know, yes, have this big change in mind, but break it down so you have smaller risks, and design experiments in a way that you can learn something from them.
00:19:40.509 -->
00:19:40.911One of the.
00:19:40.911 -->
00:19:47.588If anyone was at the leadership summit yesterday, I'm going to share one of the greatest things that was said there yesterday.
00:19:47.588 -->
00:19:53.273It was Georgie Smallwood, who's now at Moon Pig, talked about changing the language around some of this stuff.
00:19:53.273 -->
00:19:58.830So when she talked about experiments with people in her business, they didn't really get it.
00:19:58.830 -->
00:20:04.291When she changed the language to doing due diligence, they got it.
00:20:04.291 -->
00:20:10.738And exactly what you're going to do with de-risking and everything else exactly the same thing, just changing the terminology and how you communicate.
00:20:10.738 -->
00:20:13.748But both of you have now talked about de-risking things.
00:20:13.748 -->
00:20:20.891Faith, you shared a while back a two by two matrix with me that I've used a lot since you shared it.
00:20:20.891 -->
00:20:21.974Can you tell us about it?
00:20:21.974 -->
00:20:25.346You had risk as one of the axes, but I'll let you explain.
00:20:25.729 -->
00:20:26.792Yeah, sure.
00:20:26.792 -->
00:20:43.952So when I so I've been at TechSnap for about two and a half months when I arrived, we had a bit of a culture clash going on internally because historically, it's been all about this AI engine and automation, and so there were differences in opinion on how you should approach certain problems.
00:20:43.952 -->
00:20:48.167The tech guides would want to start with a working prototype or investigation of AI documentation.
00:20:48.167 -->
00:21:05.102Design guides were wanting to start with clickable prototypes and research, and there was this conflict around what's the right way to go about doing it, and so when I arrived, I was kind of going this is crazy, like it depends on the problem.
00:21:05.102 -->
00:21:06.446There's different ways to solve things.
00:21:06.446 -->
00:21:28.179It depends on what it is you're trying to solve, and so one of the ways that we've fixed that now is, as part of our road mapping exercise, every initiative that we're planning to do needs to be mapped against this two by two matrix, so that the vertical axis is about the clarity of the opportunity and the horizontal axis is about the risk of getting it wrong.
00:21:28.179 -->
00:21:37.368And so if you've got and so that was sort of I got some input from fellow product leaders around like have you guys used this before?
00:21:37.368 -->
00:21:44.840How would you give guidance around this and sort of shared the results afterwards the way that we looked at that.
00:21:44.840 -->
00:21:48.338So the clarity of opportunity is like do you actually understand the point you're trying to solve?
00:21:48.338 -->
00:21:49.243Do you know the solution?
00:21:49.243 -->
00:21:55.278Do you know who you're building this for, like you know all the sort of typical things we would ask it.
00:21:55.826 -->
00:21:58.721Some of that stuff tends to come a bit more inherently to product managers.
00:21:58.721 -->
00:22:07.131The risk of getting it wrong is where I think it's probably less natural, and so the way that I shaped that was around the DVF model.
00:22:07.131 -->
00:22:11.988So desirability, feasibility, viability and trying to understand which was your greatest risk.
00:22:11.988 -->
00:22:22.555So you always have to look at all three but which one's the biggest risk, and so you should start there first, and so that then, if it's a desirability is the biggest risk, then you start with the design-led approach.
00:22:22.555 -->
00:22:28.805If feasibility is the biggest risk, then you start with the tech-led approach, and if viability is the biggest risk, then you start with experimentation.
00:22:28.805 -->
00:22:30.692And so that's then guided.
00:22:30.692 -->
00:22:43.238The conversations that all product teams are having between the product manager, the designer and the tech lead within their team around what's the right approach for this particular initiative that we're doing and sort of trying to create that sort of triangle of trust again.
00:22:51.724 -->
00:22:52.346Optimize-ly.
00:22:52.346 -->
00:23:02.597Feature experimentation helps teams build features with more impact by combining feature flagging, feature delivery and experimentation into a single easy workflow.
00:23:02.597 -->
00:23:08.617It's the only solution on the market that combines feature flagging and experimentation in one platform.
00:23:09.345 -->
00:23:19.337Product teams can test features, identify winning configurations for customers and remove the guesswork of interpreting experiment results without sacrificing statistical rigor.
00:23:20.185 -->
00:23:21.636That was tricky to say, but anyway.
00:23:22.688 -->
00:23:23.048It was.
00:23:23.931 -->
00:23:33.134Every release can be high quality and high impact by quickly and safely validating code and features with real users as part of your delivery process.
00:23:33.775 -->
00:23:40.808With feature experimentation, your product team can be confident about findings without needing to involve analysts, speeding your processes up.
00:23:41.651 -->
00:23:48.469So try Optimize-ly feature experimentation for free and start shipping confidently with rollouts.
00:23:48.469 -->
00:23:53.659Optimize-ly's free feature flagging and experimentation solution.
00:23:53.659 -->
00:23:58.099Create a free account at optimize-lycom forward.
00:23:58.099 -->
00:24:26.191Slash free Dash feature Dash flagging and I think well, I know, Iqbal, you've been working with one of the product managers in my team to do some kind of scoring or analysis of, like, all of the stuff that we want to learn and to understand, like what we should run tests on and what we should prioritize Do you want to talk a?
00:24:26.211 -->
00:24:33.557little bit about that and how that works, because I think it's kind of like assessment at a sort of different level, but still assessment.
00:24:35.106 -->
00:24:35.527Yeah.
00:24:36.568 -->
00:24:48.837So, going in, we've got a set of problems that we've identified that users have, and sometimes these problems might be an assumption that we've made that the problem exists.
00:24:49.285 -->
00:24:53.536Sometimes it'd be verified through user research and feedback surveys etc.
00:24:53.536 -->
00:25:21.685But essentially, knowing what those problems are, listing them out, putting a priority next to them in terms of this is the order in which we're going to take them off, in terms of I think this or we think this is really important, because this has come up in multiple times, and then kind of ideating on solutions in terms of trying to solve that problem.
00:25:21.685 -->
00:25:32.166So, in terms of prioritization, the using this approach, the problem has a criteria score next to it.
00:25:32.166 -->
00:25:51.170So, in terms of, like, we've got priority matrix with lots of criteria to which to score specific tests, problem statements should be like a magnifier or something to say okay, this is a serious problem that we need to pick up first, and then these are the different sort of solutions that we can approach.
00:25:51.170 -->
00:25:58.582And in terms of to start off with a new problem, hey, maybe we need to validate that this problem is actually a thing in the first place.
00:25:58.582 -->
00:26:06.078So then there's a kind of like a whole stream of testing to do with each problem.
00:26:11.228 -->
00:26:19.894So, while we have this live audience, I'm curious who does AV testing on a very regular like runs more than I don't know, five AB tests a week.
00:26:19.894 -->
00:26:25.471We got one.
00:26:25.471 -->
00:26:42.336I remember having a conversation with a guy, a product manager, who worked at Cookpad and they were being coached by no less than Mr Kagan himself and I think they were trying to run about 20 like AB tests or like experiments.
00:26:42.336 -->
00:26:52.038Maybe they weren't AB tests actually, but experiments every single day and I was like, wow, that's insane levels of experimentation.
00:26:52.038 -->
00:27:14.292And and then I've recently had a couple of discussions with people where they felt their teams were sort of paralyzed by experimentation and there were obvious things that could be solved in obvious ways and like experimentation shouldn't be done, they should just, like, make the user experience better.
00:27:14.292 -->
00:27:17.402What's your kind of take on that Like?
00:27:17.402 -->
00:27:22.153Is there a time when we shouldn't be doing experimentation, when we should just be?
00:27:22.153 -->
00:27:24.800Is it JFDI?
00:27:25.061 -->
00:27:30.675JFDI yes, when your boss walks in, says we already know the answer, just do it.
00:27:32.820 -->
00:27:37.979Yeah, we had that conversation last week actually about something is yeah, that they wanted to.
00:27:37.979 -->
00:27:40.567We're currently revealing our onboarding flow.
00:27:40.567 -->
00:27:43.395It's trying to create the baseline that we'll then experiment off.
00:27:43.395 -->
00:27:46.711But the question around what is the baseline?
00:27:46.711 -->
00:27:52.923They weren't going to build any personalization off the informational capturing in onboarding and I was like, why would you not do that?
00:27:52.923 -->
00:27:54.953We know you should do that, like, just just do it.
00:27:54.953 -->
00:27:58.829It's so simple, it's like that's not even a hard build, just just get on with it.
00:27:58.829 -->
00:28:00.636There are.
00:28:00.978 -->
00:28:09.910When I talked about the experimentation framework we set up at Fonte, there was definitely things there that we experimented with that we knew we needed to do.
00:28:09.910 -->
00:28:17.089One of them was around personalized push notifications to give, to share data from the platform to draw them back in.
00:28:17.089 -->
00:28:43.288So we knew we needed to do it, but we still did it as an experiment because we needed the quick results, because we had to demonstrate positive momentum within six months, and so even to actually build that as a feature properly was going to take a good two or three months, and so we just ran it as a series of really dodgy experiments just to get the hit on the metrics that we needed and then knowing that we would then build it properly after we got over this hurdle with funding.
00:28:44.392 -->
00:28:52.040So I think it's never we talked earlier about absolutes like there's never an absolute time where it's the right or the wrong time.
00:28:52.040 -->
00:28:56.757It's about understanding what are you trying to prove right now, and they're working out the best way to do it, I think.
00:28:58.586 -->
00:29:06.667I mean, there's some very specific situations where you wouldn't test like defects, just you know, like I said, just do it.
00:29:06.667 -->
00:29:22.269And also, when you can't maintain the data integrity of a test, then that's probably a, and I've had situations like that as well, where we just kind of go okay, we need to find another solution to this.
00:29:22.269 -->
00:29:33.287But I think in general, there's actually a lot of situations where you think you don't need to test, where actually there is a test in there somewhere.
00:29:33.287 -->
00:29:47.672And if you think about it in terms of what you're trying to do is to de-risk what you're trying to, the goal that you're trying to achieve, and if you think about the evidence hierarchy I don't know if you guys are aware of the evidence hierarchy pyramid.
00:29:48.794 -->
00:30:04.358Yeah, evidence hierarchy pyramid is a sort of like a categorization of evidence in terms of their value, in terms of rigor, very, very much used in the medical, pharmaceutical and other industries.
00:30:04.358 -->
00:30:07.467So subjective opinion is very low down.
00:30:07.467 -->
00:30:09.692Ab testing is really high up.
00:30:09.692 -->
00:30:12.238User testing is a little bit lower down.
00:30:12.238 -->
00:30:50.865So, basically, the goal is to try and get as high up the evidence hierarchy as possible to be able to de-risk what it is that you're trying to do, and also understanding that AB testing is a research tool and reframing experimentation as a learning framework, then that kind of opens up a whole range of possibilities in terms of how you use experimentation in the first place as a kind of like a research tool to get the answers that you're actually to the questions that you're looking for.
00:30:50.885 -->
00:30:57.429Okay, we've got time for one more question from us and then we're going to throw it open to Q&A from the audience.
00:30:57.429 -->
00:30:59.272So think of questions.
00:30:59.272 -->
00:31:10.992But last question from us, starting with you, Iqbal what's one piece of advice that you would give to people who are trying to improve the way that they experiment?
00:31:12.156 -->
00:31:28.759I'd say, introduce experimentation much earlier in the process than you think, and it's always easy to think that it's as early as possible in the flow, but actually when you break it down you can get even further.
00:31:28.759 -->
00:31:37.969It's like if you break down things in terms of assumptions, there's tests you can do to run and validate those assumptions with well-designed experiments.
00:31:37.969 -->
00:32:04.217So I think, yeah, just looking, focusing on the top of the funnel in terms of from the experimentation sort of process way, and focus on improving the experiment design and that's not necessarily the UX design, but the design of experiment and how that relates to what you're trying to find out and why you're running the experiment in the first place.
00:32:04.704 -->
00:32:05.346That's cheating.
00:32:05.346 -->
00:32:09.215That's two pieces of advice I'll let you off.
00:32:09.215 -->
00:32:09.978Go.
00:32:09.998 -->
00:32:10.178Faith.
00:32:10.178 -->
00:32:13.557I would say experimentation to me is all about speed.
00:32:13.557 -->
00:32:16.028It's about speed of getting data to make those decisions.
00:32:16.028 -->
00:32:19.523So I think it's like find a way to just do things really, really quickly.
00:32:19.523 -->
00:32:22.692Like don't fall into the trap of building features, thinking that's an experiment.
00:32:22.692 -->
00:32:30.950Just find the easiest possible, quickest, simplest way you can work out whether or not something will work, and just do that and get the learnings.
00:32:32.173 -->
00:32:32.673Amazing.
00:32:32.673 -->
00:32:33.516Thank you so much.
00:32:33.516 -->
00:32:41.698We're going to take it to the floor with our handsome assistant While he is looking for people with their hand up.
00:32:41.698 -->
00:32:47.803I just want to ask another question, like how many people experiment like run?
00:32:47.803 -->
00:32:50.609I'm just going to talk specifically about AB testing, right?
00:32:50.609 -->
00:33:00.276How many people set up an AB test once a week, once every two weeks, once a month?
00:33:03.265 -->
00:33:05.109How many don't do AB testing at all?
00:33:05.109 -->
00:33:07.394Okay, Okay.
00:33:07.434 -->
00:33:08.798I should have asked that in the beginning.
00:33:10.205 -->
00:33:11.816What about any other form of experimentation?
00:33:11.816 -->
00:33:15.770Because it's not just AB testing, so they wouldn't do anything else.
00:33:15.770 -->
00:33:16.112I guess.
00:33:16.112 -->
00:33:16.732Ghost mode, I guess.
00:33:22.404 -->
00:33:25.074Yeah, you mean like a sort of Wizard of Oz.
00:33:25.394 -->
00:33:30.750No, Not instruction, basically.
00:33:30.750 -->
00:33:37.868So running it it's not affecting any live traffic, it's just getting results.
00:33:37.868 -->
00:33:42.498But it's not affecting anything live, it's just returning you some like dummy data.
00:33:44.846 -->
00:33:46.308Okay, yeah, yeah, I see what you mean.
00:33:46.308 -->
00:33:54.180So you're kind of simulating, like how people might or how traffic might respond to the how are models with people?
00:33:54.180 -->
00:33:58.135We would get that without actually impacting any production results.
00:33:58.135 -->
00:33:58.458Got it.
00:33:58.458 -->
00:33:59.823Got it Interesting.
00:33:59.843 -->
00:34:01.307There was ghost mode in the book, didn't you?
00:34:01.528 -->
00:34:02.612Yeah ghost mode.
00:34:02.972 -->
00:34:03.614Never heard of that.
00:34:06.541 -->
00:34:08.246But what's your name and where are you from?
00:34:08.246 -->
00:34:09.789And then ask I'm Jasper.
00:34:09.809 -->
00:34:20.179I'm from the Netherlands, so my question is regarding as being a PM, would you think it's helpful or detrimental to have domain knowledge for experimentation?
00:34:21.885 -->
00:34:25.304So is it helpful or hurtful to have domain knowledge for experimentation?
00:34:27.389 -->
00:34:32.722I'm going to say helpful and because, for probably the wrong reason, I think you can.
00:34:32.722 -->
00:34:40.996If you've got the domain knowledge and you've got the experience of having done this before and what happened, you can skip the experiment quite often and just move on.
00:34:40.996 -->
00:34:48.014But then I guess the downside of that you've got to be careful, not making assumptions and every situation is different.
00:34:48.014 -->
00:34:59.777But I do think having domain knowledge does speed up the process quite significantly know what you shouldn't bother even worrying about and just crack on with things that will have more potential for impact.
00:35:02.545 -->
00:35:05.679Yeah, if it depends on your definitional domain knowledge.
00:35:05.679 -->
00:35:11.487But if your domain knowledge means the experimentation process, definitely yeah, it's very beneficial.
00:35:11.487 -->
00:35:17.213Hiya, my name is Vincent.
00:35:17.213 -->
00:35:21.983I have a question around how much room or time you should give an experiment to run.
00:35:21.983 -->
00:35:26.199Yeah, I might just leave it up to you.
00:35:27.731 -->
00:35:30.317The age old question.
00:35:30.317 -->
00:35:33.282It's not as simple.
00:35:33.282 -->
00:35:36.349It's different depending on the company.
00:35:36.349 -->
00:35:46.711There are actually three variables, well, four variables really, to kind of dictate how long an experiment should be running the traffic volume.
00:35:46.711 -->
00:35:55.184The conversion rate for the chosen metric that you're going to be using to determine whether a test is a successor or failure.
00:35:55.184 -->
00:36:10.222Seasonality, that's another one, and then the conversion rate difference that you're actually going to see at the end, and then, obviously, the conversion rate difference that you're going to see at the end is the unknown variable here.
00:36:12.273 -->
00:36:16.083So there is like a process that you go through which is a minimum detectable effect.
00:36:16.083 -->
00:36:32.289We basically kind of go through and kind of go okay, if I run this test for a week, if the result is higher than, say, five percent, then that gives me that is a detectable effect.
00:36:32.289 -->
00:36:40.324I can detect an effect of five percent after running the test for a week, for instance, or running the test for two weeks or whatever it is.
00:36:40.324 -->
00:36:42.614But yeah, there's no.
00:36:42.614 -->
00:36:56.284And also it really depends on the level of certainty that you're after, and then that is dependent on what it is that you're testing, how important it is that you have a high level of certainty.
00:36:56.284 -->
00:37:01.117So yeah, it's not a simple sort of answer.
00:37:01.117 -->
00:37:04.862But yeah, there's a framework in place to try and answer that question.
00:37:07.210 -->
00:37:18.306I was just going to say, and if some of that went over your head, it's also really helpful having a data expert or statistician on the team who can help you answer that question.
00:37:20.512 -->
00:37:22.436I tend to be more pragmatic, I guess.
00:37:22.436 -->
00:37:28.324For me it's about what result are you getting, and do you need to let it run longer to get a better result?
00:37:28.324 -->
00:37:32.940So if it's clearly one is winning over the other, you can just cut it short and move on.
00:37:32.940 -->
00:37:37.349If it's a bit tighter, then you might want to let it run for longer and see what happens.
00:37:37.349 -->
00:37:41.378So I think it also depends on the nature of what you're experimenting with and how high risk it is.
00:37:45.873 -->
00:37:46.514My name is Georgia.
00:37:46.514 -->
00:37:48.336I work for NHS Education of Scotland.
00:37:48.336 -->
00:37:55.309I wondered if you had any top tips or insights of how experimentation changes throughout the product lifecycle.
00:37:55.309 -->
00:38:00.942So thinking about in discovery phase and then when you've got products are more mature, yeah, just any thoughts on that.
00:38:01.389 -->
00:38:01.911Good question.
00:38:04.811 -->
00:38:04.971Okay.
00:38:04.971 -->
00:38:13.438So we had a situation where we wanted to re-platform a website.
00:38:13.438 -->
00:38:23.766In terms of re-platforming that website, it couldn't look exactly the same as it did in the first in that A variant.
00:38:23.766 -->
00:38:54.545So what we did is break down the assumptions and if there was anything that this component definitely has to change in our incumbent, the new version, well then, how much do we know about this, what impact, how users behaving with this component, run some tests to learn about that, run some tests to see what would happen if we remove it, if it's changed, etc.
00:38:54.545 -->
00:39:01.416And then, using these kind of satellite tests, you can kind of just de-risk some of the.
00:39:01.416 -->
00:39:05.460Again, it's all about de-risking some of the big sort of changes.
00:39:05.460 -->
00:39:21.119So, yeah, it's all about breaking it down and kind of using it as a learning research tool and sending out satellites in terms of tests to try and discover and mine for insights.
00:39:21.119 -->
00:39:23.704Basically, does that help at all?
00:39:23.704 -->
00:39:24.184I don't know.
00:39:26.190 -->
00:39:27.413We do have that problem at the moment.
00:39:27.413 -->
00:39:33.782Actually, we're about to do a big redesign of our product as well.
00:39:33.782 -->
00:39:37.208The product wasn't really designed for a small business user.
00:39:37.208 -->
00:39:38.309It was designed for the accountable keeper.
00:39:38.309 -->
00:39:58.244We know it's not right for the small business user, but we're trying to drive product-led growth directly now, and so there's a bunch of work going on to try and optimize the user journeys, focus on activation, the conversion at the front end to the funnel, and then churn as well.
00:39:58.985 -->
00:40:00.387The problem is like metric.
00:40:00.387 -->
00:40:01.309Like churn, it's an outcome metric.
00:40:01.309 -->
00:40:04.097There is just so many variables that impact on it.
00:40:04.097 -->
00:40:08.349We know we haven't got the product or the proposition right for this audience.
00:40:08.349 -->
00:40:11.514So until we get that right, churn is not going to fix itself.
00:40:11.514 -->
00:40:15.278And so there are still things you can learn.
00:40:15.278 -->
00:40:28.777It's still worth doing the work to understand, kind of are there some things that we can do to have a positive impact on the churn while we buy ourselves time to sort out the proposition and the product?
00:40:28.777 -->
00:40:36.349And that learning we can still use in terms of how we then take this to market next time or what we're doing with that redesign.
00:40:36.349 -->
00:40:47.905So there is opportunity still in doing those experiments, but it needs to be understood, the context within which you're trying to do it, and that's not going to solve the whole problem.
00:40:49.192 -->
00:40:55.253Okay, we've got, I think, five more minutes, so we've got two questions left, so please, Hi, I'm Emma.
00:40:55.713 -->
00:40:56.635I work for Accelerant.
00:40:56.635 -->
00:40:58.442That's a B2B SaaS company.
00:40:58.442 -->
00:41:08.097We launched our website on Monday, so we don't have many customers yet.
00:41:08.097 -->
00:41:14.184What number of customers can you like is the minimum tab to start with experimentation?
00:41:17.516 -->
00:41:17.777One more.
00:41:18.871 -->
00:41:19.672I would say five.
00:41:19.672 -->
00:41:27.023Actually you can go in and see that from least five, but you want to talk to those five people, you don't just rely on the data.
00:41:27.023 -->
00:41:32.516But yeah, five is the absolute minimum to learn something, because you need to identify patterns.
00:41:32.516 -->
00:41:39.246If you've got one, two, three there and it's like you can't establish a pattern, anything beyond that obviously is far better.
00:41:40.512 -->
00:41:49.068So it's important to understand the different sort of research tools, like if you think A B testing is a research tool, user testing is a research tool, etc.
00:41:49.068 -->
00:41:49.309Etc.
00:41:49.309 -->
00:42:00.246So and then, depending on what stage you're at, if you're looking at doing much more user testing, for instance, then what is user testing good for?
00:42:00.246 -->
00:42:15.885So user testing is good for figuring out problems with your usability, understanding, perception, issues with your website and also task based stuff like given the task, how long does it take for a user to complete a task, etc.
00:42:15.885 -->
00:42:26.974So those are things that you can absolutely start with and for those are the things that you don't need much traffic for, as soon as you've got like a decent amount of traffic.
00:42:27.134 -->
00:42:44.237And what that decent amount of traffic is is unfortunately it's not like a very clear sort of answer but you want to try and introduce A B testing at some when you hit that mark, and that A B testing will give you a different answer, a different question.
00:42:44.237 -->
00:42:45.498Which is it answers?
00:42:45.498 -->
00:42:52.309What will happen if I launch this thing to all of my traffic, which user testing cannot tell you?
00:42:52.309 -->
00:42:58.918So so, yeah, it's basically when you're, when you're small, it's the you know.
00:42:58.918 -->
00:43:14.246A B testing perhaps isn't the right thing to do, but you can use a test to do risk and then, once you've grown to certain size, you need to A B test, because now you got a decent enough amount of traffic that it can damage your bottom line if you get something wrong.
00:43:16.175 -->
00:43:17.139And last question.
00:43:18.331 -->
00:43:18.952Thank you very much.
00:43:18.952 -->
00:43:20.394My name is Silas Parek.
00:43:20.394 -->
00:43:29.349So my question for today is, like you know, let's say you have a product and there isn't much usage user base into the product right now.
00:43:29.349 -->
00:43:36.289So as a PM, I am thinking that you know, if we introduce some discounts that can be an increase in ads and rate.
00:43:36.289 -->
00:43:44.989So my actual question is no like, without doing all of these things, how do I test my assumptions so that you know I'm making the right decision?
00:43:44.989 -->
00:43:48.934Thank, you.
00:43:50.838 -->
00:43:55.807Sorry, that was how do you if you don't have that much traffic going to your site?
00:43:57.056 -->
00:44:05.420Yeah, so the solution is you know, I'm planning to introduce some discounts to the users, but I don't want to do it, you know, before knowing that.
00:44:05.420 -->
00:44:07.483You know this is going to link traffic.
00:44:07.483 -->
00:44:08.947Yes, so how do I know that?
00:44:08.947 -->
00:44:11.099What kind of experiments I know I can do?
00:44:12.016 -->
00:44:18.384Yeah, so the because the discounts are going to change the behavior of your customers, Ekbal, this is a good one for you to take.
00:44:18.384 -->
00:44:19.579Yeah, yeah.
00:44:20.335 -->
00:44:22.039So it's, it's.
00:44:22.039 -->
00:44:35.088It's always very difficult to answer without getting into specifics and you know really digging deep, but there's, you know you don't always have to look at the bottom line conversion rate metrics.
00:44:35.088 -->
00:44:42.282There are other metrics that you can look at earlier on in the funnel which might be determinants to what's going to happen later on.
00:44:42.282 -->
00:44:53.204If you've got an AB testing tool already set up, there's no harm in just running an AB test and finding out can we learn something.
00:44:53.204 -->
00:45:13.846But what you want to be doing is you want to be much more focusing on the secondary metrics, all the metrics around your final conversion metrics, because your conversion metric you might not have enough traffic to get a read on, but the other metrics will tell you something about the user behavior and then that might be something useful that you learn.
00:45:14.375 -->
00:45:19.007So it, yeah, without getting understanding much more detail.
00:45:19.007 -->
00:45:25.206It's, it's hard to explain, but how to kind of provide an answer for that?
00:45:25.206 -->
00:45:35.005But yeah, there's always some way to de-risk and essentially what you're trying to do is to design some solution to try and figure out how do I de-risk this thing?
00:45:35.005 -->
00:45:35.947Yeah.
00:45:37.438 -->
00:45:38.882Thank you all very much.
00:45:38.882 -->
00:45:44.342If you have any further questions on this, if you want us to do another episode on the topic, get in touch with any of us.
00:45:44.342 -->
00:45:49.164If you want us to do another episode, get in touch with Lillie Rye and let us know.
00:45:49.164 -->
00:45:52.021I want to hear a big thank you, please.
00:45:52.021 -->
00:46:08.106First off to our guest, teh Iqbal, and Teh To James, who's running the mic, to the amazing sound crew who made this all happen and you for showing up and making this great.
00:46:08.106 -->
00:46:08.789Thank you so much.
00:46:09.914 -->
00:46:10.155Thank you.
00:46:22.327 -->
00:46:24.750Hey Lily, you're a senior product leader.
00:46:26.052 -->
00:46:32.543Why yes, randy, how kind of you to notice I may have led just a few product teams in my time.
00:46:33.414 -->
00:46:39.021So you know how important it is to hear stories and get insights from other product leaders facing similar challenges.
00:46:40.177 -->
00:46:41.322Oh my golly gosh.
00:46:41.322 -->
00:46:47.005Super important and, to be honest, not so easy to find that these days.
00:46:47.005 -->
00:46:49.742So much has changed in the past couple of years.
00:46:50.556 -->
00:46:51.360Oh God, I hear you.
00:46:51.360 -->
00:47:00.996Well, the reason I bring it up is because Mind the Product have just released a series of brand new case studies from senior product leaders, all about leading product teams through change.
00:47:02.114 -->
00:47:02.797Tell me more.
00:47:03.514 -->
00:47:19.983Well, in this brand new and totally free resource, mind the Product, explore the stories of five product leaders who have effectively navigated change near current and past positions, unveiling crucial lessons and sharing the principles you need to embrace in order to tackle challenges in your role as a product leader.
00:47:21.034 -->
00:47:22.099Very interesting.
00:47:22.099 -->
00:47:24.161So who are these product leaders?
00:47:24.875 -->
00:47:25.860Well, they've got some goodies.
00:47:25.860 -->
00:47:43.222There's Dave Walsh, who is the former CPTO at Zupla, kate Lido, who's an amazing product leadership executive coach, and Navya Rahani Gupta, cpto at Peak, for starters, and there's even a bonus tip section on how to look after yourself as a product leader and Oodle's more further reading suggestions.
00:47:44.195 -->
00:47:45.641This sounds amazing.
00:47:45.641 -->
00:47:47.018How do I get my hands on it?
00:47:47.976 -->
00:47:51.257Oh, that's the easy part Just sign up at mindtheproductcom.
00:47:51.257 -->
00:47:53.141Leading through change.
00:47:59.074 -->
00:48:04.900The product experience is the first and the best Podcast from Mind the Product.
00:48:04.900 -->
00:48:09.724Our hosts are me, Lily Smith and me, Randy Silver.
00:48:09.724 -->
00:48:13.764Lu Run Pratt is our producer and Luke Smith is our editor.
00:48:14.394 -->
00:48:18.264Our theme music is from Humberg based band Pow, that's PAU.
00:48:18.264 -->
00:48:26.822Thanks to Arnie Killer, who curates both product tank and MTP engage in Humberg and who also plays bass in the band, for letting us use their music.
00:48:26.822 -->
00:48:34.603You can connect with your local product community via product tank Regular free meetups in over 200 cities worldwide.
00:48:35.355 -->
00:48:38.644If there's not one near you, maybe you should think about starting one.
00:48:38.644 -->
00:48:41.936To find out more, go to mindtheproductcom.
00:48:41.936 -->
00:48:43.739Forward slash product tank.
