How Big Tech Makes Sure You Can't Put Your Phone Down

WEBVTT
00:00:04.080 --> 00:00:08.880
A lot of companies are resistant to experimentation because involves humans letting go of control.
00:00:09.439 --> 00:00:13.839
I definitely do that where I pick up the phone and then minutes later I'll be like, what was I even doing here?
00:00:14.480 --> 00:00:16.239
We shouldn't be afraid of gathering data and learning.
00:00:16.239 --> 00:00:18.960
What we should be afraid of is thinking we're right.
00:00:21.199 --> 00:00:23.920
Hello world and welcome to the Tech Overflow podcast.
00:00:23.920 --> 00:00:25.280
I'm Hannah Clayton Langton:.
00:00:25.838 --> 00:00:30.239
And I'm Hugh Williams, and we are the podcast that explains to curious people.
00:00:30.719 --> 00:00:31.039
We are indeed.
00:00:31.039 --> 00:00:38.079
And today we are coming to you from Hughes Place and So I'm still down under and still loving life.
00:00:38.079 --> 00:00:42.320
Today I tried my first magic, which is a type of coffee.
00:00:42.320 --> 00:00:45.118
I don't know if I'm gonna get in trouble for saying this.
00:00:45.118 --> 00:00:47.039
It was just like a really, really good flat white.
00:00:47.039 --> 00:00:47.679
Is that fair?
00:00:48.158 --> 00:00:48.640
I think that's fair.
00:00:48.640 --> 00:00:57.520
The way it's done is you uh you grind the coffee, put it in the machine, and just run it into one cup when you'd normally it between two, and you kind of stop it halfway and then so get the best bits.
00:00:57.920 --> 00:00:58.079
Oh, yeah.
00:00:58.320 --> 00:00:59.920
And then you know, put in the milk, the coffee art.
00:00:59.920 --> 00:01:00.719
Life's good.
00:01:01.039 --> 00:01:01.359
It was great.
00:01:01.359 --> 00:01:02.880
So, what are we talking about today?
00:01:03.439 --> 00:01:06.640
We're talking about Hannah, why you can't put your phone down.
00:01:07.120 --> 00:01:07.840
It is true, guys.
00:01:07.840 --> 00:01:12.078
I can't put my phone down, which my husband and my sure, can attest to.
00:01:12.078 --> 00:01:14.640
But I'm hoping we're gonna learn that it's not just me.
00:01:15.120 --> 00:01:15.760
It's not just you.
00:01:15.760 --> 00:01:17.280
I pick up my phone way too often.
00:01:17.280 --> 00:01:23.200
In fact, I think I've got to a stage in my life where I pick it up to do something and I find myself just drawn to an app.
00:01:23.200 --> 00:01:29.438
I open up the app, I go deep into this app, and then a of minutes later, I can't remember exactly why I picked my up in the first place.
00:01:29.438 --> 00:01:32.718
But phones are super, super addictive, and we're gonna really dig into that topic.
00:01:33.280 --> 00:01:43.519
Yeah, I'm not gonna reveal my screen time numbers not proud of them, but I definitely do that where I pick up the phone and then five minutes later I'll be like, what was I even doing here?
00:01:43.519 --> 00:01:52.159
And sometimes I'll close an app and I will literally navigate to it, like Instagram or maybe Slack if it's if work-related.
00:01:52.159 --> 00:01:57.519
And I've just closed it and I'm my brain is like looking for more stimulation or something.
00:01:58.159 --> 00:01:58.480
Yeah, absolutely.
00:01:58.480 --> 00:02:05.760
And look, the companies that build these apps are trying to you to engage with their app, stay engaged where they're at.
00:02:05.760 --> 00:02:17.840
And today we're gonna chat a little bit about the the methods they use to do that, some things around sort of and how they try and improve their apps and and their web as well, and really explain that it isn't all about you, Hannah.
00:02:17.840 --> 00:02:23.199
It's not that it's not some character failing, it's uh you big tech, and big tech is winning.
00:02:23.840 --> 00:02:23.840
Wow.
00:02:23.840 --> 00:02:29.199
Okay, and is are we gonna learn something like it's just like tip-top product management that's drawing me back?
00:02:29.598 --> 00:02:38.400
Yeah, there are lots and lots of bright people working in large companies, and their mission is to drive engagement and figure out ways to keep you engaged, Hannah.
00:02:38.960 --> 00:02:39.520
I am very engaged.
00:02:39.520 --> 00:02:49.120
Okay, so Hugh, as a product management professional, if after my quote unquote engagement, like what behavior are user?
00:02:49.919 --> 00:03:02.960
So I want you to use the app or the web experience in a in a way, and that's gonna depend a little bit on whether it's like a Facebook or an Instagram that's more of a social or something like Google Maps, you know, guiding you from A B or Google Search.
00:03:02.960 --> 00:03:06.479
So what engagement means is is going to vary between the apps.
00:03:06.479 --> 00:03:12.718
But inside of these companies, they are definitely in some way and they're doing some aggregate counting.
00:03:12.718 --> 00:03:18.878
So, for example, a lot of these big companies track what's DAO, which stands for daily active users.
00:03:18.878 --> 00:03:27.598
And so they're very, very interested in the number of people use the product every day, and they've probably got a goal driving up DAO over the year.
00:03:27.598 --> 00:03:38.000
And so all the product managers, all these super smart who are working in these teams, they've probably got DAO and they're probably really trying to make sure that whatever it is that they build drives a daily habit.
00:03:38.479 --> 00:03:50.080
Okay, so just thinking back to one we've talked about So for a like a social network type app like LinkedIn or engagement is basically me scrolling, maybe hitting a like.
00:03:50.080 --> 00:03:59.680
Whereas for an app like Uber, which is or any ride-sharing which is much more transactional, the engagement's about me like converting, requesting a ride.
00:03:59.680 --> 00:04:00.080
Yeah.
00:04:00.080 --> 00:04:00.800
Buying something.
00:04:00.800 --> 00:04:02.000
Makes sense, makes sense.
00:04:02.000 --> 00:04:13.280
Okay, so engagement is one metric, or maybe it's a suite of metrics that might look slightly different for different But can you give us some other metrics that these companies are gonna be tracking?
00:04:13.840 --> 00:04:14.240
Yeah, absolutely.
00:04:14.240 --> 00:04:15.360
Great question, Hannah.
00:04:15.360 --> 00:04:17.360
So Mao is another one.
00:04:17.360 --> 00:04:18.240
You can guess what that is.
00:04:18.560 --> 00:04:19.278
Monthly active users.
00:04:19.278 --> 00:04:19.838
That's it.
00:04:19.838 --> 00:04:20.560
Ding ding ding.
00:04:20.560 --> 00:04:21.360
Ding ding ding.
00:04:21.920 --> 00:04:24.480
And so, you know, it's gonna be a much bigger number than right?
00:04:24.480 --> 00:04:27.360
So DAO is the number of users that are using you on a day.
00:04:27.360 --> 00:04:33.199
Mao is the number of users who've used you in a particular So you're typically gonna get a much, much larger number.
00:04:33.199 --> 00:04:37.439
What often companies are interested in is the ratio between two things, right?
00:04:37.439 --> 00:04:44.000
So if you think about what percentage of the daily users are you monthly, that's pretty interesting, right?
00:04:44.000 --> 00:04:46.079
So what they call the Dow Mao ratio.
00:04:46.079 --> 00:04:58.079
So take the number of daily users, divide it by the number of monthly users, and you get kind of the percentage, if you like, of folks who are using you as a daily habit, who are in your user base that are using you monthly.
00:04:58.079 --> 00:05:05.680
And so what you'll often find in these companies, and the social companies, is that they're trying to drive up the mal ratio.
00:05:05.680 --> 00:05:12.240
So they're trying to take that latent user base who are using the app on a monthly basis and trying to activate them to be daily.
00:05:12.240 --> 00:05:15.040
So the down mount ratio is one of the other things that they
00:05:15.519 --> 00:05:29.838
And presumably the mal user base is attractive because they you know you've used them before, you might have lapsed as a sort of more loyal user, and that's a sort of lower hanging fruit than having to bring someone brand new in, get them register, get them to create the habit.
00:05:30.480 --> 00:05:30.879
Exactly, Hannah.
00:05:30.879 --> 00:05:34.959
And you know, when I was working on Google Maps, we had about a billion Mal.
00:05:34.959 --> 00:05:39.838
So about a billion users used us monthly, but not that many used it daily.
00:05:39.838 --> 00:05:45.519
So what was what was happening there was when a user know how to navigate from A to B.
00:05:45.519 --> 00:05:52.000
So you're going to somewhere unfamiliar, you'd you'd Google Maps, you'd love it, it'd help you get to a place, but it wasn't a daily habit.
00:05:52.000 --> 00:05:56.000
So it wasn't something that you were always using to guide you through the world.
00:05:56.000 --> 00:05:59.199
It was something you were using when you didn't know how to through the world.
00:05:59.199 --> 00:06:01.278
And so we're pretty interested in the down mount ratio.
00:06:01.278 --> 00:06:02.319
How do you drive that up?
00:06:02.319 --> 00:06:05.360
How do you turn Google Maps more into a daily product?
00:06:05.920 --> 00:06:06.240
That makes sense.
00:06:06.240 --> 00:06:08.160
Because I don't need Google Maps for my commute.
00:06:08.160 --> 00:06:16.240
Maybe if I'm looking at if the trains are on time, if I'm a like more exploratory travel on the weekend, then that's first thing I fire up.
00:06:16.560 --> 00:06:16.560
Yeah.
00:06:16.560 --> 00:06:28.720
And look, they did a lot of work on Google Local, probably after I left, to be honest, where they really did a lot on opening reviews, photos, the latest photos, getting people to take more photos.
00:06:28.720 --> 00:06:31.759
So trying to give you a real feeling of the vibe of places.
00:06:31.759 --> 00:06:34.399
And so that helped it become much more of a daily habit.
00:06:34.399 --> 00:06:36.000
It wasn't just a driving app.
00:06:36.480 --> 00:06:36.480
Okay.
00:06:36.480 --> 00:06:41.759
So for all apps, you have your user base and your frequency of use.
00:06:41.759 --> 00:06:48.240
And then I assume for Amazon, it's going to be about ads to basket, that sort of thing.
00:06:48.240 --> 00:06:54.399
For Instagram, it's when I like or maybe send a post to or I hit up a story or something.
00:06:54.399 --> 00:06:56.000
You know, I add something to my story.
00:06:56.000 --> 00:07:00.000
And then Uber, it's going to be about actually ordering the Uber WhatsApp.
00:07:00.000 --> 00:07:02.160
I mean, do WhatsApp even have to track?
00:07:02.160 --> 00:07:04.879
Like, isn't everyone just using it all the time?
00:07:05.360 --> 00:07:08.160
Yeah, well, I guess they've got, you know, they've got some competitors.
00:07:08.160 --> 00:07:12.399
You know, we see Apple has the iMessage ecosystem, Facebook True.
00:07:12.399 --> 00:07:14.160
There's lots of people, other things that people use.
00:07:14.160 --> 00:07:23.040
So certainly if I was at WhatsApp, I'd be, I'd be tracking Mao, I'd be tracking Dow, I'd be tracking the ratio between and making sure that I was continuing to drive that up.
00:07:23.360 --> 00:07:23.600
That's true.
00:07:23.600 --> 00:07:34.399
There's this really weird phenomenon where like I can be to the same person on Instagram, WhatsApp, and then like Facebook Messenger, probably less so because I've aged that a bit.
00:07:34.399 --> 00:07:40.959
But it's like a WhatsApp conversation is different to an Messenger conversation, even though it's both via my phone.
00:07:40.959 --> 00:07:41.600
That's like my boss.
00:07:54.240 --> 00:07:54.240
Yeah.
00:07:54.240 --> 00:07:57.439
But certainly, you know, these are the kinds of things that all these teams are tracking.
00:07:57.439 --> 00:07:59.600
And of course, there's a competitor release as a feature.
00:07:59.600 --> 00:08:05.600
So, you know, if we're working in one of these companies release a feature that's really compelling for our users, you know, the user base will shift slightly.
00:08:06.160 --> 00:08:06.800
Yeah, yeah, for sure.
00:08:07.278 --> 00:08:11.680
And uh, you know, our job is to respond to that and make sure that we continue to drive up our metrics.
00:08:11.680 --> 00:08:13.920
I'll throw one more metric in the mix too, Hannah.
00:08:13.920 --> 00:08:16.639
Uh, there's a thing called 3d7, 4d7.
00:08:16.639 --> 00:08:18.160
Have you heard of that one?
00:08:18.560 --> 00:08:21.920
Okay, no, but I'm guessing it's three days out of four days out of seven.
00:08:22.319 --> 00:08:22.319
Absolutely.
00:08:22.319 --> 00:08:25.680
And so you're interested in users who are fairly engaged.
00:08:26.319 --> 00:08:28.319
Oh my god, I'm I'm 7d7 Instagram.
00:08:28.319 --> 00:08:29.920
I'm 8d7 Instagram.
00:08:31.278 --> 00:08:39.840
Well, they will love you because I bet they're tracking 3d7 4d7 to see if people are, you know, pretty engaged, using it of the time throughout the week.
00:08:39.840 --> 00:08:41.360
Um, so that's another popular metric.
00:08:42.080 --> 00:08:51.600
I once dated a guy who worked for a dating app, which is a meta, and he was an analyst there and he was telling me the metrics that they were tracking.
00:08:51.600 --> 00:08:57.519
And I know that Hinge says stuff like our main objective that you delete the app, but that's obviously just marketing.
00:08:57.519 --> 00:09:04.720
And he was talking about like it was like messages exchange, and then he was like, Oh, we can tell when you've you've a phone number or something like that.
00:09:04.720 --> 00:09:06.399
And it was it was pretty interesting stuff, actually.
00:09:06.879 --> 00:09:07.200
Yeah, absolutely.
00:09:07.200 --> 00:09:08.720
Actually, dating's a hard one.
00:09:08.720 --> 00:09:11.440
I, you know, I worked at Tinder for a little while, and right.
00:09:11.440 --> 00:09:23.120
I mean, people are super, super engaged, and that's fantastic, and you want them to be engaged and using the product and know, carrying out the tasks, meeting people, that kind of But then when they get into a happy relationship, they vanish.
00:09:23.120 --> 00:09:24.080
You hope.
00:09:24.080 --> 00:09:25.840
Yeah, you hope they do.
00:09:25.840 --> 00:09:27.120
Yeah, happy relationship.
00:09:27.120 --> 00:09:28.480
You hope they uh they vanish.
00:09:28.480 --> 00:09:37.519
I guess we should be a bit more uh open-minded about it, but vanish, and then of course, you know, if that relationship work, they appear again and uh become super, super engaged.
00:09:37.519 --> 00:09:41.200
So often, you know, users are active for a while, inactive for a while, and then they come back.
00:09:41.519 --> 00:09:48.080
I was gonna say that must be difficult to predict, but then I bet you have people who disappear off for like three weeks and then they're back again.
00:09:48.639 --> 00:09:48.960
Yeah, absolutely.
00:09:48.960 --> 00:09:50.399
There's a lot of that going on.
00:09:50.399 --> 00:10:00.480
And uh we used to have a sociologist that Tinder and her job to really think about people and dating and definitely had people in the team who really thought about dating from a that wasn't just tech.
00:10:00.960 --> 00:10:04.558
This ultimately sounds like it's this is an example of product management, right?
00:10:04.558 --> 00:10:12.320
Like if the goal of Instagram or WhatsApp is to have me their app three or four D7, then they are absolutely that goal.
00:10:12.720 --> 00:10:13.918
Yeah, they must love you, Hannah.
00:10:13.918 --> 00:10:16.240
You're even uh an advocate for their ads.
00:10:17.120 --> 00:10:18.399
Yeah, it's pretty sad.
00:10:19.278 --> 00:10:26.798
Uh so you might ask the question though, like how are they are they driving up 3D7, 4D7, the down mouth ratio, whatever is they're driving up?
00:10:27.038 --> 00:10:28.558
And the answer is they're running a lot of experiments.
00:10:28.558 --> 00:10:33.038
So a lot of a lot of experimentation is really the the if you like.
00:10:33.440 --> 00:10:50.639
Okay, and so experimentation, as I've understood trying out different features of a product with different user groups, and then basically looking at how users in their behavior and basically finding out which size of button or color of a button people are responding best
00:10:51.120 --> 00:10:51.519
Yeah, absolutely.
00:10:51.519 --> 00:11:10.639
And you know, folks like Google and Meta who own Facebook and Instagram are running tens of thousands, if not hundreds of of experiments every year and trying to highly optimize things that, you know, are as simple as you say, as you know, fonts, sizes of buttons, whether the buttons have rounded all these kinds of things.
00:11:10.960 --> 00:11:22.720
And I'm sure that if I picked up a version of Instagram was like two or three years old, it would just feel really But it's all these tiny little changes that over time evolve the product.
00:11:23.200 --> 00:11:23.600
Yeah, absolutely.
00:11:23.600 --> 00:11:25.840
And this is a relentless machine, right?
00:11:25.840 --> 00:11:28.080
Like this is a machine that's just running the whole time.
00:11:28.558 --> 00:11:34.960
And if they've got millions or billions of users and have a super rich data set across which to run all these
00:11:35.600 --> 00:11:35.600
Absolutely.
00:11:35.600 --> 00:11:44.320
And you know, if you worked on Google search, you can probably get a signal within, you know, seconds, if not minutes, uh, you?
00:11:44.960 --> 00:11:55.278
Here's a question, because I've got a brother that China, and sometimes I use their e-commerce websites if I'm like sending them a gift and it's very different UX.
00:11:55.278 --> 00:12:05.360
So does something like Instagram have different UX for its or Asian users versus its European users, or is it trying to optimize across both?
00:12:05.759 --> 00:12:21.278
Uh I can't speak to Instagram specifically, but what I you is that particularly large alphabet languages, so we're about things like Chinese, Japanese, and Korean in particular, the experiences are very different for users who you know and type these languages.
00:12:21.278 --> 00:12:24.000
And one of the reasons is it's quite hard to type.
00:12:24.000 --> 00:12:24.399
Yeah.
00:12:24.399 --> 00:12:29.200
So, you know, when you've got hundreds or thousands of characters, it's much, much harder to type.
00:12:29.200 --> 00:12:33.360
And so you'll find often that their experiences are much if you like.
00:12:33.360 --> 00:12:34.480
That's how it feels.
00:12:34.480 --> 00:12:34.798
Yeah.
00:12:34.798 --> 00:12:38.720
There's a lot more things to click on because it's a lot to click than it is to type.
00:12:39.278 --> 00:12:40.080
Ah, that's interesting.
00:12:40.080 --> 00:12:40.558
Okay.
00:12:40.558 --> 00:12:40.879
Yeah.
00:12:40.879 --> 00:12:49.519
So my husband is a native Arabic speaker, and when he types to his family, he uses the English alphabet, it replicates sounds of the Arabic script.
00:12:49.519 --> 00:12:53.678
But he says it's much easier than typing in the Arabic on WhatsApp.
00:12:54.240 --> 00:13:04.798
And you know, think of, I mean, Arabic and and Hebrew are but Chinese, Japanese, and Korea are in a whole different ball game because the alphabet is so, so much larger, which isn't case with with Hebrew and Arabic.
00:13:04.798 --> 00:13:20.080
So when I've worked in these large companies, um, I when I was at eBay, we used to just let the Korean team do Korea because certainly our Western intuition around what the should be, the way we experimented with it, how we thought it, just did not sell properly.
00:13:21.200 --> 00:13:27.360
That makes I I always had noticed that the Asian websites are, as you say, like they felt noisier to me.
00:13:27.360 --> 00:13:37.120
But when you break it down to the fact that it's easier to than to type in the large alphabet languages, that's just like the most obvious thing I never would have thought of myself.
00:13:37.120 --> 00:13:45.759
So there's different versions sensibly as you broke it of an app, depending on where you're from and what UX will best suited to you.
00:13:45.759 --> 00:13:46.480
But how does that work?
00:13:46.480 --> 00:13:53.519
Like my brother lives in China and he would download in Mandarin because he speaks Mandarin, and then he comes holiday to London.
00:13:53.519 --> 00:13:55.038
Does his app change?
00:13:55.600 --> 00:13:56.480
No, no, not usually.
00:13:56.480 --> 00:14:02.158
So typically what an app is built for is what we call a country language pair.
00:14:02.158 --> 00:14:11.600
For example, if um we've got listeners in the US, they get a US experience, but you choose whether you get a US experience or a US Spanish experience.
00:14:11.600 --> 00:14:20.158
So probably when your uh brother downloads the app in China, probably got a Chinese Mandarin experience.
00:14:20.158 --> 00:14:20.720
Yeah.
00:14:20.720 --> 00:14:26.639
And when he comes to the US or he's over in the UK, probably that way.
00:14:26.639 --> 00:14:30.558
So unless he changes his country because he's permanently to be there.
00:14:30.558 --> 00:14:41.278
So Amazon's probably a better example where you know, if come to Australia and you open up your Amazon app, you're going to be shopping in the UK unless you actually go and change your country to Australia.
00:14:41.759 --> 00:14:49.360
Okay, so this has happened to me, and it will surprise for me to tell you that I clicked on an Instagram ad yesterday because they they promoted a dress that looked great.
00:14:49.360 --> 00:14:56.240
And I clicked on it and it said, Oh, are you in Australia So it must have been twigged because I have the location on that I'd moved.
00:14:56.639 --> 00:14:56.960
Yeah, absolutely.
00:14:57.600 --> 00:15:07.840
So we've done a little bit of a a deep dive on UX there, I know that UX is heavily guided by experimentation, and we're gonna maybe call it A B testing.
00:15:08.320 --> 00:15:13.440
And just to state possibly the very obvious, that's you have one group A and you have one group B.
00:15:13.440 --> 00:15:13.678
Yep.
00:15:13.678 --> 00:15:23.038
And you're pushing out a different feature to group A and B, and you're looking at your metric success in group A group B, and that's how you decide which is the better
00:15:23.678 --> 00:15:24.080
Exactly that.
00:15:24.080 --> 00:15:24.798
Exactly that.
00:15:24.798 --> 00:15:39.600
And there's a lot of art and a lot of science in doing that, a little bit more science than art, but these large companies have huge experimentation test bed environments where you can actually go and do this in a pretty scientific way and actually figure out what's the right experience for your users.
00:15:39.600 --> 00:15:49.038
And when you're letting mini flowers bloom, whether it's A, or A, B, C, D, E, and F, you're gonna pick the winner and gonna make the winner the default for everybody who's out
00:15:49.440 --> 00:15:49.840
And doesn't this?
00:15:49.840 --> 00:15:50.720
I have a few questions.
00:15:50.720 --> 00:15:54.000
So I'll throw them all out and then you can answer them in order that makes most sense.
00:15:54.000 --> 00:16:02.960
But like, would you notice if you're part of an A B Because sometimes you sign up to be in a beta version of app, but A B testing would you notice?
00:16:02.960 --> 00:16:07.840
And then, like, are they selecting user groups based on attributes?
00:16:07.840 --> 00:16:14.639
So we want to test this feature on users who are 2D7 because we want to get them 4d7.
00:16:14.639 --> 00:16:17.519
And Hannah doesn't need any testing because she's already
00:16:18.480 --> 00:16:19.440
A couple of great questions there.
00:16:19.440 --> 00:16:28.639
I'd say you can figure out that you're in a test group by down and looking incredibly carefully at your experience versus say a friend's experience.
00:16:29.200 --> 00:16:31.840
As long as they're on the same operating system, right?
00:16:31.840 --> 00:16:40.240
Because we we talked last season about Android versus iOS and how those experiences might be different purposefully or one's ahead of the other.
00:16:40.558 --> 00:16:41.120
Yeah, yeah, exactly.
00:16:41.120 --> 00:16:50.158
And so, you know, I'd say if you open up your iPhone and a iPhone, you're both sitting at the kitchen bench and you sit and you really study this carefully, you will be able to spot some differences.
00:16:50.158 --> 00:16:59.120
And so things to look out for are things like colors, fonts, of fonts, those kinds of things, sizes of images, typefaces, kinds of things.
00:16:59.120 --> 00:17:01.360
And you you'll be able to spot some differences for sure.
00:17:01.840 --> 00:17:01.840
Interesting.
00:17:01.840 --> 00:17:06.798
And and they are choosing user groups based on what they're looking to drive.
00:17:07.598 --> 00:17:09.118
I'd say that's probably the edge case.
00:17:09.118 --> 00:17:13.920
I'd say in general, what they're doing is they're trying to a representative part of the population.
00:17:14.480 --> 00:17:14.480
Okay.
00:17:14.880 --> 00:17:22.160
So they'll typically randomly assign some part of the to be A and then randomly assign a different part of the to be B.
00:17:22.160 --> 00:17:31.440
And they'll know that if there's a difference in engagement those two, that that's a statistically significant we call it, that they could, you know, then ship to everybody.
00:17:32.000 --> 00:17:35.838
And I know from working for a B2B to C tech company.
00:17:35.838 --> 00:17:41.358
So we sell a B2B product, but it's for the businesses we work with to interact with their consumers.
00:17:41.358 --> 00:17:42.798
So B2B to C.
00:17:42.798 --> 00:17:48.078
That can be tricky because you're asking your clients you their customers for experimentation.
00:17:48.078 --> 00:17:57.038
And we don't always get the level of excitement or as we'd want to improve our product because our clients are super protective of their customer base.
00:17:57.519 --> 00:18:01.358
And I'd say probably also they don't realize the magic of right?
00:18:01.358 --> 00:18:06.720
I think you have to see it up close inside a tech company to just how powerful this is.
00:18:06.720 --> 00:18:08.160
I mean, I'll tell you a quick story.
00:18:08.160 --> 00:18:18.960
The the most important, significant A-B test I've ever been in was slightly increasing the size of the images in the search results page at eBay.
00:18:18.960 --> 00:18:23.920
Instead of just showing users, you know, small thumbnails, showed them medium-sized thumbnails.
00:18:23.920 --> 00:18:30.480
That experiment alone put more than 400 million US dollars of revenue into the city.
00:18:30.640 --> 00:18:31.038
That's crazy.
00:18:31.838 --> 00:18:31.838
Right.
00:18:31.838 --> 00:18:32.960
And it's such a small adjustment.
00:18:32.960 --> 00:18:34.720
It wasn't even technically very hard.
00:18:34.720 --> 00:18:35.118
Yeah.
00:18:35.118 --> 00:18:35.440
Right?
00:18:35.440 --> 00:18:37.118
We're just showing slightly larger images.
00:18:37.118 --> 00:18:48.400
And I think when you've seen that up close, when you've just how powerful it is to be able to measure changes on the base and understand a change in engagement, then you want more of it.
00:18:48.400 --> 00:19:00.798
And so I'd say a lot of companies are resistant to because it involves humans letting go of control and letting have control and also being more about testing any idea not right.
00:19:00.798 --> 00:19:02.640
Companies get quite resistant to that.
00:19:02.640 --> 00:19:05.680
But once you've seen it up close, uh, you just want to do more of it.
00:19:06.078 --> 00:19:06.078
Okay.
00:19:06.078 --> 00:19:16.558
And I imagine it's very rare that you actually like lose 10% of your users overnight because you've changed the size of a font and people have just deleted the app, right?
00:19:16.558 --> 00:19:20.880
Because if you're a company that's worried about I presume that's the worry, right?
00:19:20.880 --> 00:19:23.519
That you're gonna lapse a bunch of users because you made the wrong decision.
00:19:24.000 --> 00:19:24.240
Yeah, absolutely.
00:19:24.240 --> 00:19:25.680
And I think you have to have a budget.
00:19:25.680 --> 00:19:32.240
So you have to have a think about kind of if we're gonna put our users in different treatments, how much are we willing here to do that?
00:19:32.240 --> 00:19:35.358
And you also want to be monitoring these metrics all of the right?
00:19:35.358 --> 00:19:43.598
And making sure that if treatment D is not a great and we're statistically sure that that is a poor treatment, we just turn off D as soon as we possibly can, right?
00:19:43.920 --> 00:19:48.558
Okay, and when you say budget, that's not budget for like to make changes to the platform.
00:19:48.558 --> 00:20:00.798
That's budget because if your conversion drops by not point two percent because of the way you formatted the button, you want to have planned for that eventuality rather than lose money.
00:20:01.278 --> 00:20:01.598
Yeah, absolutely.
00:20:02.240 --> 00:20:09.358
And I assume this is kind of like we talked about in previous episodes about how reported behavior is very different be very different to actual behavior.
00:20:09.358 --> 00:20:15.118
Like we think or maybe want to behave in a certain way, and then our habits shine through with the reality.
00:20:15.118 --> 00:20:28.798
And I guess that's why emotionally led decisions around UX be pretty flawed because you think you want a button that's this color in this corner of the page, but you've not actually tested that that drives the results that you're after.
00:20:29.278 --> 00:20:38.240
Yeah, and I've had lots of arguments with UX people where, you know, they've said, hey, it doesn't fit our sort of, you know, corporate ethos, the style guide, whatever else it is, and I want to do that.
00:20:38.240 --> 00:20:40.880
And I always say, like, let's just run the test.
00:20:40.880 --> 00:20:50.240
And then when we've got some data, we can have the about who's right, who's wrong, whether the style guide needs changing, whether you're right, that we shouldn't be afraid gathering data and and learning.
00:20:50.240 --> 00:20:53.358
What we should be afraid of is thinking we're right.
00:20:53.838 --> 00:20:53.838
Okay.
00:20:53.838 --> 00:20:56.480
And is there sometimes like a change fatigue?
00:20:56.480 --> 00:21:05.759
Like people famously don't like change, and then Apple their iOS operating system and like I can't find the button on the phone book.
00:21:05.759 --> 00:21:10.720
But is am I really gonna come round to loving it, or do they can you sometimes get it wrong?
00:21:11.118 --> 00:21:12.160
You can sometimes get it wrong.
00:21:12.160 --> 00:21:16.160
Where experimentation is powerful is when you're optimizing existing experience, right?
00:21:16.160 --> 00:21:24.798
So we're talking about things like changing colors and font button sizes, colors, these kinds of things, sizes of boxes, how large an image is.
00:21:24.798 --> 00:21:26.400
Experimentation's wonderful for that.
00:21:26.400 --> 00:21:31.200
And that'll help us find the the local maxima, best of the existing experience.
00:21:31.200 --> 00:21:34.160
If you want to go do something completely different, useless.
00:21:34.720 --> 00:21:35.759
I feel like Apple do this a lot.
00:21:35.759 --> 00:21:38.480
Like they're always changing my operating system, and can never find anything.
00:21:38.960 --> 00:21:40.000
And they're not a data company, right?
00:21:40.000 --> 00:21:42.160
They are a hardware company, right?
00:21:42.160 --> 00:21:49.278
So if you look at the largest companies by market cap, you you've got folks like we'll skip over NVIDIA and we'll skip over Apple for a second.
00:21:49.278 --> 00:21:52.160
You've got Meta, Google, Microsoft, Amazon.
00:21:52.160 --> 00:21:56.798
Those folks are consumer companies that are all about data software.
00:21:56.798 --> 00:21:58.640
Apple is not one of those.
00:21:58.640 --> 00:21:59.598
Apple is a hardware company.
00:21:59.598 --> 00:22:01.200
That makes wonderful, wonderful hardware.
00:22:01.200 --> 00:22:04.720
I love their ecosystem, but they are absolutely not a data
00:22:05.278 --> 00:22:05.278
Okay.
00:22:05.278 --> 00:22:07.920
And that's why I sometimes find their updates
00:22:08.480 --> 00:22:08.480
Yeah.
00:22:08.480 --> 00:22:13.118
And so the designers run Apple, not the experimentation
00:22:13.598 --> 00:22:13.598
Okay.
00:22:13.598 --> 00:22:14.640
That makes sense.
00:22:14.640 --> 00:22:20.720
So assume in your time at eBay and Google Maps, et cetera, done your fair share of experimentation.
00:22:22.160 --> 00:22:30.240
So what would you say is like the biggest surprise or biggest failure or most counterintuitive experiment outcome came across?
00:22:30.798 --> 00:22:38.798
Yeah, so obviously we talked about the experiment of the size of uh images at eBay and that being certainly the successful uh experiment that I've ever been involved in.
00:22:38.798 --> 00:22:46.960
I should say to finish out that story, um, you can guess what idea the product manager had who was working on that, the idea they had.
00:22:47.440 --> 00:22:48.078
Make it even bigger.
00:22:48.720 --> 00:22:49.118
That's it.
00:22:49.118 --> 00:22:53.118
And that uh delivered even more gains to the company so it did.
00:22:53.118 --> 00:22:53.519
Okay.
00:22:53.519 --> 00:22:54.960
So you can guess what they did after that.
00:22:55.278 --> 00:22:58.400
No, because I'm gonna say then you're gonna have like one hit per search.
00:22:59.118 --> 00:23:05.440
Yeah, and eventually the images got so large that as a team we had to say, look, you know, we need to we need to stop increasing the size of the images.
00:23:05.440 --> 00:23:10.078
So then the PM went and worked on uh increasing the quality the images, which also mattered.
00:23:10.078 --> 00:23:10.640
Okay.
00:23:10.640 --> 00:23:12.798
Um, so this this was a gift that just kept on giving.
00:23:12.798 --> 00:23:17.680
Turned out larger the images, the more people shop, higher quality of the images, the more people shop.
00:23:17.680 --> 00:23:22.720
And it was one of the most important things in the period of uh eBay when when I worked there.
00:23:23.200 --> 00:23:25.200
Okay, any other cool experimentation stories?
00:23:25.838 --> 00:23:26.400
I can tell you a ton.
00:23:26.400 --> 00:23:30.960
I'll I'll tell you one where I uh I kind of came last, as So another eBay story.
00:23:30.960 --> 00:23:37.038
So we were thinking about there's a button on eBay that says So we obviously, you know, bid on an auction, right?
00:23:37.038 --> 00:23:40.720
And we were sitting there one day and we thought, I wonder if bid's the right word.
00:23:40.720 --> 00:23:42.480
Like, should we say bid now?
00:23:42.480 --> 00:23:43.598
Bid soon.
00:23:43.598 --> 00:23:48.160
Um, should it say bid and maybe it has a price that you could bid at?
00:23:48.160 --> 00:23:49.838
So you could say bid at$4.20.
00:23:49.838 --> 00:23:52.640
It's like, oh, this is an interesting theme.
00:23:52.640 --> 00:23:56.160
How do we get people bidding more on auctions and and do that?
00:23:56.160 --> 00:24:01.519
So we came up with, I think it was five different ideas of what the bid button could look like.
00:24:01.519 --> 00:24:03.598
And I was sure I was gonna win this.
00:24:03.598 --> 00:24:07.278
I know so much about eBay, I've been involved in this part of eBay for ages.
00:24:07.278 --> 00:24:09.278
My treatment's gonna win for sure.
00:24:09.278 --> 00:24:10.720
I came stone cold last.
00:24:11.200 --> 00:24:11.598
Oh no.
00:24:12.160 --> 00:24:12.558
Stone cold.
00:24:13.038 --> 00:24:16.318
I thought you were gonna tell me that you you nailed it and you guessed the winner now.
00:24:16.640 --> 00:24:16.640
No.
00:24:16.640 --> 00:24:23.838
And look, the takeaway for me is what we should do is have many flowers blooming, lots of different ideas, make easy.
00:24:23.838 --> 00:24:38.078
As long as it's legal, it's not deliberately destroying the you know, whatever it is, we should get these ideas out as as we can, as frequently as possible, and measure what happens, and then you know, let the data speak and then let the winner win.
00:24:38.558 --> 00:24:39.278
I'm loving these stories.
00:24:39.278 --> 00:24:40.720
Have you got one more for us?
00:24:40.720 --> 00:24:43.759
Because there's they're they're really uh they're giving me pause for thought.
00:24:44.078 --> 00:24:44.078
Yeah.
00:24:44.078 --> 00:24:50.000
My good friend Michael Cameron, uh, who I've worked with at you know, he went off and founded a company called Rome to
00:24:50.400 --> 00:24:51.200
Oh, yeah, I do know that one.
00:24:51.358 --> 00:24:52.000
I love Rome to Rio.
00:24:52.000 --> 00:24:52.640
Melbourne Company.
00:24:52.640 --> 00:24:53.598
Um is it?
00:24:53.598 --> 00:24:54.960
Yeah, it is indeed.
00:24:54.960 --> 00:25:02.798
And uh, I was working with Michael a few years ago when he was still at Rome to Rio, and he told me a little bit about some that they ran.
00:25:02.798 --> 00:25:04.640
And look, maybe just we just dive in.
00:25:04.640 --> 00:25:06.558
I'm sure many of our listeners have used Rome to Rio.
00:25:06.558 --> 00:25:07.598
It's a fantastic product.
00:25:07.598 --> 00:25:10.400
Great for getting from A to B when A and B are a long way apart.
00:25:10.400 --> 00:25:11.519
It's basically a short summary.
00:25:11.519 --> 00:25:15.358
So if I want to go from Melbourne to Manchester, I want to out what flights I can take.
00:25:15.358 --> 00:25:16.960
Like, you know, do I take a train from Heathrow?
00:25:16.960 --> 00:25:18.480
You know, whatever else it is.
00:25:18.480 --> 00:25:20.078
So fantastic product for that.
00:25:20.078 --> 00:25:28.558
The way they make money is uh they get a referral fee if you some accommodation or you book a car after you've done that through something like booking.com.
00:25:28.558 --> 00:25:37.358
So Michael and his team highly, highly incented to get to click a little part of the screen that said book or book car rental.
00:25:37.358 --> 00:25:41.278
And they played around with a bunch of different treatments of how do you say that?
00:25:41.278 --> 00:25:42.960
Like, how do I say book accommodation?
00:25:42.960 --> 00:25:49.038
Should I say book a hotel, book accommodation, book a room, a room?
00:25:49.038 --> 00:25:51.598
Um, what about best priced accommodation?
00:25:51.598 --> 00:25:53.680
Best priced car rental.
00:25:53.680 --> 00:25:58.078
So we can we can sit here and brainstorm lots of different That's exactly what they did.
00:25:58.078 --> 00:26:06.240
And then they tried some different treatments, ran those as a big A B test, and they actually increased this is a small startup, right?
00:26:06.240 --> 00:26:14.400
They increased their revenue by over $250,000 a year by the text on the part of the screen that said book accommodation.
00:26:15.278 --> 00:26:19.038
Because it just was more uh engaging unconsciously to users.
00:26:19.680 --> 00:26:20.000
Yeah, absolutely.
00:26:20.000 --> 00:26:22.318
Turns out colours of buttons matter too.
00:26:22.318 --> 00:26:25.200
If you go and look at a lot of the travel sites, you'll find all the buttons are green.
00:26:25.200 --> 00:26:26.078
Not a coincidence.
00:26:26.078 --> 00:26:28.400
Green signals go.
00:26:28.400 --> 00:26:34.558
Green signals go, and so you get these sites that are blue or purple or whatever else it isn't all the buttons are
00:26:34.960 --> 00:26:34.960
Huh.
00:26:34.960 --> 00:26:35.680
Do you know what?
00:26:35.680 --> 00:26:44.640
When I was a kid, I remember being told that McDonald's red and yellow because it made you eat fast and got you like out of the restaurant so more people could come in.
00:26:44.640 --> 00:26:48.720
And I guess this is the virtual equivalent of that sort of behavioral nudging.
00:26:49.358 --> 00:26:49.680
Yeah, absolutely.
00:26:50.240 --> 00:26:50.400
Oh, wow.
00:26:50.400 --> 00:26:53.598
Okay, so colors, sizes, placement.
00:26:53.598 --> 00:26:55.038
How you say things.
00:26:55.038 --> 00:26:56.000
How you say things.
00:26:56.640 --> 00:26:56.640
Absolutely.
00:26:56.640 --> 00:26:58.640
It all matters and it all drives engagement.
00:26:58.640 --> 00:27:11.118
There's an army of incredibly smart, well-qualified, young product managers in all of these companies who are incredibly hard to drive up whatever metric it is that they want to drive up.
00:27:11.118 --> 00:27:15.358
And so, Hannah, it's not really your fault that you can't put your phone down.
00:27:15.358 --> 00:27:19.759
It's kind of you versus an experimentation engine that's by lots and lots of smart mods.
00:27:20.160 --> 00:27:24.640
And is there also an element here of like the of stuff?
00:27:24.640 --> 00:27:37.200
So Google Maps is maybe not going to be as personalized to but like my Instagram feed or TikTok feed, if I had I keep going back to it because the content, it makes me It's something that I'm interested in.
00:27:37.200 --> 00:27:53.838
And I assume that's good like processing of all the data that I'm feeding in with my use and then applying that to the and images that are in the app and then pushing me the ones to keep me scrolling, to keep me reopening the kind of thing.
00:27:54.240 --> 00:27:54.240
Absolutely.
00:27:54.240 --> 00:28:00.318
And look, one of the biggest areas of experimentation in these large companies is search and recommendations.
00:28:00.318 --> 00:28:05.680
And the news feed or the feed in any social app is a is a of that.
00:28:05.680 --> 00:28:15.038
And so you tend to get the brightest set of folks working on search and recommendation style problems and feed style And you know, their job is to make sure that you keep on
00:28:15.838 --> 00:28:30.558
And back in the early days of Instagram, I remember that hashtag was really big or or and maybe Twitter as well, it was like you hashtag your content, and that's how the app or the algorithm knows what you're pushing, and then it link you to relevant like recipients of your content.
00:28:30.558 --> 00:28:32.078
But that they're not really used anymore.
00:28:32.078 --> 00:28:33.358
So how are they doing that now?
00:28:33.358 --> 00:28:36.720
They just know that it's a picture of like, I don't know, on toast.
00:28:37.278 --> 00:28:37.680
Yeah, absolutely.
00:28:37.680 --> 00:28:41.920
So, you know, today things are a lot more sophisticated than used to be.
00:28:41.920 --> 00:28:50.558
And so things like processing the images as they're trying to identify objects that are in them, and also other of them, you know, are they a pleasing image?
00:28:50.558 --> 00:28:51.838
Is it well focused?
00:28:51.838 --> 00:28:53.200
Those kinds of things.
00:28:53.200 --> 00:28:55.838
Is um there a single object in the image?
00:28:55.838 --> 00:28:59.200
All of these kinds of things are the sort of metadata that can extract from it.
00:28:59.200 --> 00:29:10.558
And again, you can put this into the algorithms, into the AI and the AI system, you know, will be able to serve the that are, you know, probably great for all customers, great for you.
00:29:11.118 --> 00:29:12.798
Does videos make that easier or harder?
00:29:12.798 --> 00:29:24.480
Because like what was it a few years back, Instagram moved being very much a still image-based app with still content to videos and reels and everything's moving all time.
00:29:24.480 --> 00:29:27.118
Is that going to be easier or harder to identify that
00:29:27.440 --> 00:29:31.278
Uh it's definitely a little bit harder, but there's lots and of work that's gone into video over the years.
00:29:31.278 --> 00:29:34.400
So, first of all, what happens with video is you divide it into shots.
00:29:34.400 --> 00:29:39.519
And so if you're watching a video, you'll find, you know, that cuts between scenes in different sorts of ways.
00:29:39.519 --> 00:29:43.920
So there's, you know, hard cuts, there's fades, there's wipes, there's all kinds of things that happen.
00:29:43.920 --> 00:29:52.160
So you basically cut the cut the video up into all of its scenes, and then you can basically process the scenes to kind of understand what's in those scenes.
00:29:52.160 --> 00:30:02.640
So it's definitely more work than a still image, but you can of a video as a series of still images, and particularly when you break it up into scenes or shots, um, you know, it's a lot easier to process and try and understand.
00:30:03.278 --> 00:30:10.000
Okay, so it's a combination of relevant content and user experience.
00:30:10.798 --> 00:30:11.598
Yeah, exactly that.
00:30:11.598 --> 00:30:15.519
And lots and lots of experimentation to mine my That's it.
00:30:15.838 --> 00:30:20.318
Okay, so I have revealed my cards as an avid Instagram and user.
00:30:20.318 --> 00:30:33.358
And I was trying to think about like if I could only keep apps in my phone, and I'm not including here like my banking, because I obviously need my banking and my like multi-factor authentication, because as we know from our privacy episode, that's just good sense.
00:30:33.358 --> 00:30:37.118
But I'm thinking about the like more optional social apps I'm using.
00:30:37.118 --> 00:30:43.200
And I've decided that my top three that I couldn't live are Instagram, WhatsApp, and Google Maps.
00:30:43.200 --> 00:30:45.838
Like, I I really don't think I could get anywhere without Maps.
00:30:45.838 --> 00:30:47.519
But what would you say yours are?
00:30:48.000 --> 00:30:48.558
That's a great question.
00:30:48.558 --> 00:30:52.480
I know if I had my my screen in front of me, the answer's too hard.
00:30:52.480 --> 00:30:54.400
I mean, first of all, I'm gonna agree with you on Google Maps.
00:30:54.400 --> 00:30:58.318
I'm a little bit biased, I guess, since I worked on it, but uh, but I love that product.
00:30:58.318 --> 00:30:59.838
Yeah, it's such an amazing product.
00:30:59.838 --> 00:31:01.680
The team's done so much amazing work.
00:31:01.680 --> 00:31:04.960
Look, sadly, I've got Gmail pegged to the bottom of my screen.
00:31:04.960 --> 00:31:07.440
I think that's a fantastic email app.
00:31:07.440 --> 00:31:09.838
I'm still primarily an email kind of person.
00:31:09.838 --> 00:31:11.519
I have five different email accounts.
00:31:11.519 --> 00:31:15.200
So, you know, that's something I just can't do without to of function.
00:31:15.200 --> 00:31:19.358
And then maybe my third one's a little bit more fun, is I I my Whoop.
00:31:19.358 --> 00:31:23.440
Um, and Whoop takes up, I don't know, like a quarter of my my home screen.
00:31:24.000 --> 00:31:27.598
Okay, so Whoop is like a wearable fitness tracker, right?
00:31:28.000 --> 00:31:28.318
Yeah, exactly.
00:31:28.318 --> 00:31:31.358
So, you know, you can think of it as a fancy heart rate monitor.
00:31:31.358 --> 00:31:38.240
So it's always tracking my heart rate and what's called heart rate variability, which is your ability for your heart to pace.
00:31:38.798 --> 00:31:45.200
Okay, so you're a big wearables fitness data guy, which we probably do an episode on at some point because people are into it.
00:31:45.200 --> 00:31:45.759
I love my garment.
00:31:45.759 --> 00:31:49.358
It's I didn't make my top three, but I do love the the fitness tracking.
00:31:49.920 --> 00:31:55.200
Let's get somebody on from that part of the world because I our listeners would love to know how these wearable devices
00:31:55.519 --> 00:31:55.920
And definitely.
00:31:55.920 --> 00:32:01.598
I think it's safe to say that your Whoop app improves your more than Instagram improves my life.
00:32:02.078 --> 00:32:02.318
I don't know.
00:32:02.318 --> 00:32:07.598
Whoops got me being very, very religious about when I go to and when I get up in the morning, which probably make me a lot less fun.
00:32:07.920 --> 00:32:08.720
I can attest to this.
00:32:08.720 --> 00:32:11.759
Hugh is all over his uh his fitness optimization.
00:32:12.480 --> 00:32:12.960
Yeah, yeah, yeah.
00:32:12.960 --> 00:32:13.680
It's so important.
00:32:14.240 --> 00:32:18.078
Okay, so there's a few more insights into us as co-hosts.
00:32:18.078 --> 00:32:21.038
That's been a super awesome episode.
00:32:21.038 --> 00:32:23.358
I don't know that it's gonna help me stay off my phone.
00:32:23.358 --> 00:32:26.960
Well, do you know what's gonna help me stay off my phone is the sunshine and your pool.
00:32:26.960 --> 00:32:28.640
That's what's gonna help me stay off my phone.
00:32:28.640 --> 00:32:35.519
But once I'm back into the cold depths of the English I think I'll be uh straight back on my addictions.
00:32:36.078 --> 00:32:39.358
I'll give you one life hack though, and it works for me is put your phone away every Sunday.
00:32:39.358 --> 00:32:44.880
Just don't take it out of the cradle and uh pick it up again Start with uh 1D7, Hannah.
00:32:45.440 --> 00:32:46.880
But what about my Google Maps?
00:32:46.880 --> 00:32:48.960
I guess I gotta keep it local on a Sunday.
00:32:48.960 --> 00:32:51.118
I know where the farmer's market is and I know where the gym is.
00:32:51.358 --> 00:32:52.798
Yeah, get somebody else to navigate for you.
00:32:53.038 --> 00:32:53.038
Yeah.
00:32:53.038 --> 00:32:54.318
I'm also, do you know what though?
00:32:54.318 --> 00:32:55.278
Apple Pay.
00:32:55.278 --> 00:33:02.960
I should have put I don't know if I would put that in my top three or if that's just a sort of table stakes, but I I never hold a physical card anymore.
00:33:03.598 --> 00:33:03.838
Maybe on Sundays.
00:33:04.318 --> 00:33:04.640
Maybe on Sunday.
00:33:04.640 --> 00:33:05.358
Yeah, fair enough.
00:33:05.358 --> 00:33:10.798
Okay, well, I will take my New Year's resolution with me, we will see you next time, guys.
00:33:10.798 --> 00:33:16.558
If you have enjoyed this episode, please like, subscribe, and rate the podcast.
00:33:17.278 --> 00:33:27.598
Yeah, and you can find us at techoverflowpodcast.com where got all the episodes from season two and season one, and we're also on LinkedIn, Instagram, and X.
00:33:28.400 --> 00:33:28.400
Awesome.
00:33:28.400 --> 00:33:37.838
And I think we should go and take a look at the UX of here, because maybe we can drive some addictive behaviors getting uh some of those buttons in a really cool place.
00:33:38.318 --> 00:33:39.680
Yeah, I think I'll just make the images bigger.
00:33:40.000 --> 00:33:40.000
Yeah.
00:33:40.000 --> 00:33:42.240
All right, guys, we'll see you next time.
00:33:42.240 --> 00:33:43.038
See you next time.
00:33:43.038 --> 00:33:43.358
Bye.