Dollars to Donuts

By Steve Portigal

About this podcast   English    United States

The podcast where we talk with the people who lead user research in their organization.
In this podcast



Machine generated. There may be errors. Report errors to us.
May 10, 2016
In the final episode of the season I speak with Monal Chokshi, Head of User Experience Research at Lyft. We discuss traditional paths to a user research career, creating routines for meeting different types of users, and the emergence of leadership roles in user research. As researchers we love having curiosity and it’s fulfilling to do the hands-on work. That’s what makes us passionate about research, but in a business one of the dangers is getting too far into that fascination, but then not taking that next step. Maybe that’s where managers can help with making sure that the data and the analysis and then these insights and actionable recommendations are affecting product. We’re not just doing the research and being fascinated for our own curiosity and fulfillment, we’re making sure it’s put to good use. – Monal Chokshi Show Links Monal Chokshi Follow Monal on Twitter Lyft Lyft partners with Didi John Zimmer and Logan Green Commodore 64 VIC-20 Trilogy John Morkes Jakob Nielsen Sun Microsystems SoundCloud Terry Winograd Don Norman at UCSD General Assembly Lyft’s partnership with National MedTrans Network Follow Dollars to Donuts on Twitter and help other listeners find the podcast by leaving a review on iTunes. Transcript Steve Portigal: Thanks, Monal for being a guest on Dollars to Donuts. Monal Chokshi: Yeah, sure. Well first Steve thanks for having me on your podcast episode. Really excited to talk more about user experience research at Lyft. So my name is Monal Chokshi and I’m the head of user experience research at Lyft. Steve: And if I didn’t know what Lyft was, what would you explain to me? Monal: Yeah, so Lyft is an on demand transportation service. So for those not familiar, basically all you have to do is download the Lyft app, open it and tap a button and basically you have a driver ready to drive you wherever you want within minutes at your doorstep. Steve: Where does Lyft operate? Monal: So the headquarters are in San Francisco. We also have offices in Seattle, Nashville and New York. Steve: And where do you have drivers? Monal: Drivers all throughout the country. I believe we are now in over 190 cities. Steve: This is an American operation? Monal: Yes. Steve: So drivers in the United States right now. Is there an international goals for the company? Monal: Well right now we recently in the last 6 to 9 months announced some partnerships with international ride sharing companies. For example with Didi which is the largest ride sharing company, similar to Lyft, in China. And Grab which is in Southeast Asia. I believe they’re based in Singapore. So the idea being that travelers from the U.S. would be able to travel to one of these places, open the Lyft app and call a car. It wouldn’t be a Lyft car. It would be one of these folks from our partner companies coming to pick them up. So we haven’t launched that yet, but that is a partnership that we’ve announced and we’re really looking forward to getting that going. Steve: It makes me think of like code sharing and airlines where you can purchase through the sort of front end that you’ve always used, but the service is delivered through a partnership with somebody else. Is that a, is that a – does that comparison apply to something like this? Monal: I think it might. I think it’s a pretty good way to think about it from an analogy. Steve: I’m sure there’s probably better analogies for how products and services in one market end up in another, but that’s the best I could think of on the fly. That’s really interesting. Monal: Yeah. Steve: And I’m sure there’s going to be design challenges – what does that experience look like in that different situation. Monal: Yeah, we’re really excited about designing this new experience just because of the various challenges that come with different cultures and different transportation schemes for various cities, especially international. As we all know the cultures, the modes of transportation are different from city to city in the US and then overseas there are various challenges with language as well. So there’s a lot of really exciting things to think about. Steve: I love like the really specific use case you’re looking at now which is I’m from the United States, I use Lyft here. I’m in a different context so how can I bring access to that service with me. It makes me think about what people that use Netflix complain about it or I think I hear this with the iTunes store. For example, you can’t get the same access when you’re in a different place. So this gives you – I’m sort of imagining it gives you – you can sort of bring your Lyft travel experience with you into these other environments. Monal: Yeah. And people use and are comfortable with Lyft here. We want to be able to make sure that they have the comfort of the same kind of service over there, something that they can trust and feel good about. Something that’s familiar in an area where everything seems unfamiliar to them. Steve: Can you talk about the work that you do here and what is research like at Lyft? Monal: Sure. So research at Lyft has always been part of our core DNA in a sense. The founders, Logan and John, are really focused on treating people better and that includes all of our users, our drivers, our passengers and so really understanding who these people are and figuring out how to create the best user experience for them has always been top of mind here. Before I joined we did do various research activities and there were several initiatives happening at Lyft. Once I joined, I joined to sort of make that part of our regular product and design processes and so the kinds of research that our team does really runs the whole gamut of what UX research can do. We do everything from field studies and surveys to user and usability testing of new designs and products. So yeah, it’s been a really fun place to work, just also because user experience is so well regarded and supported here. Steve: When did you come to Lyft? Monal: I’ve been at Lyft for about a year. Steve: So you’re describing a little bit about how – you know what you were brought in to do, to sort of – I don’t think you used the word formalize, but that’s sort of the word that comes to mind. I guess is the question about – you know what that change has been that you’ve helped drive. Monal: Sure. I think a lot of the research previous to when I joined hadn’t been formalized in a sense that now it’s much more a part of the regular, daily, weekly process. And I’ve also joined to basically build a world-class team of user researchers so that – you know Lyft has grown so much in the last year and we are continually growing at a very increasingly rapid rate. So there are a lot more needs that we have from a research perspective and I think as we grow the company there are more needs for insights. So, bringing that to the table and helping build a company that’s very data and insights driven is a priority for the company. Steve: I’ve seen sort of two different modes of how insights – I’m sure there’s many more, but two kind of primary modes in terms of how insights can impact decisions. One is sort of the reactive and one is sort of the proactive. And you know where reactive is somebody comes to researchers with a question and says ‘Hey, can you help us make a decision about this?’ and the other is researchers often who have done research are evangelizing those insights to try to bring them out to find the people that can use them to make decisions. I don’t know if you even buy into that kind of framework, but I’m wondering a) does that ring true and b) if so like how does that map to what you’ve seen in your work? Monal: Yeah, sure. I think we do a mix of both. When I first joined I think it was more the former in the sense that people have specific questions and we’re always aiming to try and answer business questions using various methodologies, whatever is going to work best to get those questions answered. But there are definitely times when we actually see something that is interesting and we’ll probe further on it in order to say actually there’s an opportunity here, or actually there’s some pain points or something that we need to look at further here and we’ll bring that to the attention of stakeholders in the company for further investigation. Steve: So it sounds like part of that evolution, the growing of the company and the growing of the role of what you and your team are doing is – I mean developing the ability and developing the context to have those conversations. Yes we can answer the questions you’re asking and hey everybody else here’s things that we’re already doing or can do to support you. Monal: Um-hmm, definitely. Steve: So maybe that’s – maybe what I sort of asked inarticulately a moment ago, maybe that is the vector of a growing research practice is that you shift from reactive to more kind of reaching out. Steve: Okay. So we’ve talked a little bit about sort of some of what you’ve been working on in the last year, but maybe we can just go back, back, back and maybe talk about how did you end up being a researcher? What are some of the things that you’ve done that have led to today? Monal: Yeah, definitely. It’s funny, I often joke that my background and my entrance to becoming a UX researcher is very non-traditional just because it is in fact on paper very traditional. I’ve been focused on this area since the beginning of my career, even back in my days at Stanford as an undergraduate. So I think that’s pretty rare for folks in our area. So yeah, when I was growing up I always had a fascination for computers and technology and played a lot of video games as a kid. Started programming on my own for fun when I was in middle school, on my Commodore 64 – dates me a bit. And when I went to school in the mid-90s I was lucky enough to be at Stanford where a lot of stuff with the Internet was just coming, seeing light, and getting out into the world. So very lucky to have been there and found my major which is called symbolic systems. So symbolic systems is an interdisciplinary major made up of core classes from computer science, philosophy, psychology and linguistics. And I graduated with a concentration in human computer interaction. So a lot of what I do today is rooted all the way back from that time and one of my first – I guess the – my start of my career, I’d say I did – I was more of a – what we would call today a UX generalist. So back in the dotcom times, as you remember – sorry to call you out… Steve: Hey, I had a VIC-20. I had an older Commodore computer than yours. I’ll own it. Monal: Cool. So as a UX generalist – I’d maybe call someone like this today – I did everything from front end coding and UI design to information architecture and user research. And back then, I mean I was really seeking more objective ways as a designer to know how do I go about my design. And unfortunately I think most user research at that point was limited to usability testing. So while I had done a summer internship while I was in school at Boeing in a usability lab, I grew more interested with various kinds of methods of research. Like what else can we do, how else can we learn about users? And I was very lucky to have had a mentor at this company called Trilogy – it was my first company that I worked for – someone named John Morkes who now has a consulting company in Austin, Texas. He’s had it for many years now and back in the late 90s he actually worked with Jakob Nielsen at Sun. So he was probably the best mentor I could ask for and we together built a usability lab at Trilogy and he mentored me while I learned more about doing things like contextual inquiries, focus groups, all kinds of methodologies and that’s what got me really excited about focusing my career in this area. And I had some really big successes from doing multiple method research in terms of helping the company understand what was actually happening with users, even at that time and this was back in like 1999. So I saw the power and the value of what user research could provide. From thereon out it got a little more challenging as the fall of the dotcom era occurred and at that time user experience research or usability they called it – I guess UX was sort of coming to bear around that time, the term. It was much harder to find a job doing this because at that time it was seen as somewhat of a luxury kind of a position. Companies didn’t necessarily see the value in having it beyond something that if they could afford to do it, great. It wasn’t table stakes. So I focused on continuing with design, but I kept coming back to research in my mind, this is what I want to do. So I eventually went back to grad school to focus on understanding more different kinds of methodologies, how can I apply this? How can I become a UX research as for a career. So I went to UCSD and studied cognitive science there and eventually started working as a UX researcher and I have been ever since. So prior to Lyft I was at SoundCloud in Berlin for 2 years, starting a research team there. And prior to that Intuit and Yahoo for several years. And you know as I mentioned around the dotcom time I worked in some various startups. Steve: You prefaced this whole story with – that your story, it’s a traditional path which makes it sort of – as you said, that’s the exception. Monal: Exactly. Steve: So I was listening as you were talking and think well what is the traditional path. So I don’t know. I’ll just throw it out as a question. What about your story and the path you’ve gone through, what makes it traditional? Monal: Traditional on paper. Steve: Yes, I’m sorry, yes. That’s an important disclaimer. Monal: Right, right. Well it’s because I actually, you know back in 1994 I took a class at Stanford called introduction to Human-Computer Interaction with Terry Winograd where we were actually designing and looking at user interfaces for software. So this to me is something that most people, maybe going back that far, didn’t necessarily look at at that time. And even when I talk to folks today, you know 20+ years later, folks are still like wow, what you do is really cool. And I say yeah I know. They’re like how do I do that? How do I get into this industry? I want to do that, but I don’t have the background. I didn’t go to school for it. Should I go to school for it? Should I take classes? How do I break in? And I think there are a lot of folks who – most folks I know who are researchers today, even designers, people in the UX field, they’ve come from all different paths and I think that’s part of what makes it a really interesting set of folks because we are so diverse and we have lots of different paths that we come from, but all of ‘em include doing something with people, trying to understand people and enjoying that. Steve: So what’s your answer to those people that ask you how do I get into this? What would you tell them? Monal: Well I think it depends on the circumstance. My advice – I mean, and it depends on what kind of career they’re looking for, but there are more classes today which I encourage people to take. There’s lots of stuff online and I mean obviously the best thing would be to just get the experience, but I think it can be very difficult to do that cold. I know there have been folks who have just gone out and done research on their own and said hey, look, I’ve done this about your company. I’ve used your app. I tested it. Sent it over to the company and sometimes with really great success. I’ve even had some people send me stuff like that about Lyft. So it’s really interesting to see how folks are basically trying to break into the industry. But I mean definitely a traditional on paper route is very helpful, but I don’t think it’s necessary. Steve: So in your traditional on paper route – it sounds like I’m talking about a paper route – I mean, you know the – it seems like there was this culmination for you with UC San Diego. It seems like that’s where you said oh I’m going to go become one of these things and that’s the way that you went about doing it. And of course you had all this background, going all the way back to Terry Winograd’s class, but like a graduate degree in – what was it, cognitive science? Monal: Um-hmm. Steve: See, I don’t know. My perception, that seems not traditional on paper to me. That sort of seems like more in the HCI realm and less in the – although having said that I’m not sure is it a social science degree or a design degree. Like what’s my perception of the traditional on paper thing? So the fact that you and I don’t even share definitions of traditional on paper even though – we have – I’m a little older than you but we have – we’ve seen some of the same eras of this work. I don’t know I think it only goes to bolster your really important point that the diversity of backgrounds and disciplinary orientations and all that that make up the work is just part in parcel of what this practice is about. Monal: Um-hmm, definitely. I mean I chose to go to UCSD because – well first it was on my radar because of Don Norman and you know the Design of Everyday Things, previously called the Psychology of Everyday Things, really resonated with me and that was sort of my bible for – for you know when it first came out. And then there were multiple labs there that were focused on using ethnographic techniques in order to design technology for everyday use and so that was one of the things that really resonated with me in terms of going there. It wasn’t so much the cognitive science academic courses. It was more about the folks there and the opportunities to do research there where people were thinking very similarly in terms of you know how to do research and it was very applied in that sense. So yeah, so I – even though it does sound somewhat orthogonal to UX it was actually very close to it. Steve: Especially once you explain it. Yeah, seeing an academic use of ethnography kind of in terms of using technology that seems very – that’s still cutting edge for the work world and it’s extremely cutting edge for academic back a few years even. Monal: Yeah and back then I think in the early 2000s – I mean when I graduated from Stanford I always wanted to go back to school ‘cuz I’m always like wanting to learn and a student at heart and I wanted to work first. And so I already had on my mind getting a graduate degree and craving the desire to go back to academia at some point. I just wanted a break and some real practical experience, but that said it wasn’t that I hadn’t tried becoming a researcher. Back then you really needed an advanced degree to get into that field because I had tried. And part of it was the fall of the dotcom era. So it was really difficult to have those opportunities, but the companies that were still looking for folks like that, like some of the bigger folks – I mean I don’t even think Google –Google definitely didn’t have folks then looking at that. But maybe Yahoo or eBay. That you definitely had to have more of an academic background to enter. Steve: Right and such a contrast to now where as you said there’s a lot of classes out there where these classes, and General Assembly springs to mind, but I think there’s probably dozens and dozens of other examples, but they seem to be maybe the most common brand that I come across where that – like a UX Intensive from General Assembly, like that’s the that’s sort of the symbol of qualification where an advanced degree was something. So that’s – if you just compare and contrast what it takes to get an advanced degree vs. what these kinds of programs are providing and sort of the coin to the realm has really shifted. I don’t know part of me things the population has – of researchers has just exploded between the time that you’re talking about and now. I’m now sure what the – do we know what the cause is and what’s the effect? Or what has led to what? Monal: Yeah, it’s a good question and actually before I left for Berlin in 2012 I really feel like there’s been a shift. I mean – when I finished my degree and started working again I thought wow I’m always going to have to work at a big company because I still had this mindset of what I explained back in the dotcom days where this was a luxury. You’re the last to come into the company and you’re the first out. But that’s no longer the case and I’ve thought about this a bit and I sometimes think that maybe, you know back then it was all about the technology – wow, this is really tough to build. There’s a lot that needs to go into this. Just getting that done was a really big accomplishment. And I think now – not to say that building code is easy, or writing code, but building an app it doesn’t have to be that difficult. And so it’s become really important in the marketplace to have a great experience. It’s not just about having the functionality now, it’s the form as well. It’s a lot of times a differentiator as to why people choose your product vs. another. And I think that’s part of the reason why people are jumping on the bandwagon of oh yeah, actually this is really impactful. We can learn how to get our product to be successful in the marketplace. Steve: And that think about the importance of the experience, I agree with you totally and it’s – we keep getting it demonstrated to us over and over again – I mean as consumers I guess we see this. It’s really hard to do that. And you know we have lots of good examples of a good experience and it’s just amazing to see how it’s impossible to sort of replicate. I think it you look at sort of the battles between mobile phones over the last 8 years or something, like it’s hard to – just because someone does one that has a good experience you just see these terrible things come out of other companies. And if we were two people wearing – you know with design in our title, we might talk about that issue in a certain way. I wonder if you have a perspective, wearing a research hat, like why do you – what’s the conversation a researcher should be having about why it’s hard to – we know that a good experience is important, but the creation of that experience is like amazingly elusive. Monal: Yeah, I don’t know if it’s necessarily something that researchers can always pinpoint too. I think a culmination of factors have to come together. Various things and some of them are just very emotive and users can’t describe them. Researchers might be able to observe it happening, but I think it takes a really good researcher to be able to dig in and understand all those things that are happening in this person’s subconscious so to speak. Steve: Yeah. That makes me think of a comment you made early on where you used the phrase ‘world-class research team’. Can you define what that is? Monal: Well I’ll tell you a little bit about what I look for in a researcher. So there’s three main traits that I sort of boil down to what I think are most important for researchers – user researchers to have. First people need to be empathetic, so have user empathy – and part of that is being a really good listener. So being a good listener, being a very observant person, also being personable and approachable because any time you’re with a user or someone that you want to learn from and about they need to feel comfortable in your presence. And along with the user empathy piece and the definition of being empathetic is you need to be able to look at things from that person’s perspective. So if someone is really judgmental that’s going to detract from their ability to be a researcher just because you have to be able to consider their point of view. The second thing is analytical thinking. So obviously strong one. One of the things that I think can be the most impactful for the quality of research is being able not just to collect all the data, which you might use the empathy skills to do in a sense, but to then take those insights – well take the data and the findings and turn them into insights and actionable recommendations that will drive good results for the business and for product design. So there’s a lot of other areas also where analytical thinking can come into play, but I think that’s what’s unique to this field and this profession. And lastly, this one is probably overlooked by many, but I would say having really great project management skills and being organized because oftentimes as researchers we are managing so much when it comes to participants, sessions, multiple projects and you have to be organized and keep all of your data very organized. It’s just part of the territory in terms of having those skills to manage all of that. And oftentimes manage the teams and stakeholders that come along with that so that you can ensure that people are there to observe, to participate. You know there’s some administration work – administrative work going on here, but a lot of it revolves – your success as a researcher resolves around you being able to stay on top of it and be very detail oriented. Steve: So those are things you look for, right. How do people exhibit those characteristics in a way that you can assess? Monal: Um-hmm. I mean it’s definitely hard to do in a single interview. During the interview process I definitely like to be able to see them in action. We have folks do – typically do like some kind of homework during the process and I often like – like very many companies like to have them do a challenge in the office where we recreate, you know kind of a real life scenario, some role playing, and so we can kind of see how they tackle problems as well as interact with folks and understand how they may work with stakeholders. I mean teamwork, all these other things are also very important, but the three that I pointed out were specific to, I think, user experience researchers that are not necessarily applicable to other fields for specific reasons. Steve: I like that your three kind of span a research project, for lack of a better noun. I mean you’re not looking at people’s data gathering abilities. Or not just that, but what you do with the data and then how do you help it – how do you help that result succeed in the environment in which you’re working. So sort of managing the logistics of your stakeholders and their participation and kind of their engagement and so on. I like your list quite a bit. Monal: Thank you. Steve: I think those three things make sense, but is there anything controversial there or anything missing or where you would dispute how other people approach this? Monal: There’s definitely a lot missing from the list. I think these are just three things I keep in mind, but of course there’s plenty of other traits that are important. You know, as I mentioned, teamwork, being a fast learner, being adaptable to change. So many things that we probably look for on a holistic level. You know if you want somebody on your team you want to like them. There are lots of things that I think can be applied to many different roles when you join a company. I mean at Lyft we definitely get excited about folks who are also excited about our vision and the vision at Lyft you know is something that attracted me at first. Having lived in Berlin for 2 years, coming back to San Francisco I didn’t know much about Lyft, just because we didn’t have it over there and I’d heard a lot of friends and other folks talking about this company. So I finally decided to give it a try when I was on a visit here and thought it was so interesting, like the whole sharing economy and talking with my driver and I started thinking about Lyft as a company and realized I didn’t know much about them, let me check ‘em out and searched them online and started reading about their story and that’s part of what got me really excited because our founders were really interested – they weren’t just interested in building a better taxi. They were really interested in kind of the long term, big picture vision of creating better communities and reconnecting people through transportation. You know the long term vision is redesigning cities. It’s really cool. I mean this is such a hot space right now and we’re working on so many exciting projects as a company. One of the things I love is that we’re constantly executing. We’re getting things out the door. There’s always something big happening, but the vision is always what keeps us focused on what it is we’re trying to do. And that vision is something that I really can get behind because it makes my job feel a lot more meaningful. I mean I think as UX researchers, you know having empathy is something that we just innately have within us and we desire helping others and so when you look at the meta, for this company that’s what we’re aiming to do as well. So, you know we’re looking at reducing the carbon footprint, getting fewer cars on the road, putting more people in vehicles, making rides cheaper through ride sharing, Lyft Line, and building communities through that experience. I’ve had so many amazing conversations with other passengers or drivers in the car and I always just leave with a smile. So it’s really fun to work on something in the office, take a Lyft home, and actually talk to my driver about oh, wow, you have this issue, we’re actually fixing that right now. I can’t actually tell them that, but it’s really neat to know that we’re making a difference on a daily basis. Steve: It’s fascinating to hear you talk about the relationship between, especially between you as a researcher, but with your researcher sort of identity on, between being a UX researcher and the vision of the company. So you talked a little bit earlier on about the different users. You kind of said, like you talked about passengers. You talked about people using the app which I guess might not be the same as the passenger. You talked about drivers. Tell me about, who do you study? What do you look at? Monal: Well as you said, definitely passengers. People who are looking to get a ride, who are using Lyft to take rides. And then the drivers who give rides. So currently the Lyft app that we have today, the single app, serves both of those types of folks. And for people – most people probably don’t ever see the driver’s side, but once you apply to be a driver, are approved as a driver, there’s a whole nother world on the other side of the app. So we look at both those users and more recently we announced at the beginning of this year a partnership with a company called National MedTrans Network. So their users are a whole new set of users for us to investigate and learn about. I can tell you a little bit of a story about them. So they – National MedTrans Network is a company in New York. They work with insurance companies like Medicare and Medicaid whose patients are these elderly folks who basically need routine medical rides, rides to doctor’s appointments. A lot of times – all the time non-emergency rides. And they often don’t need to be transported in an ambulance or any special kind of vehicle. So National MedTrans has all these relationships with various car service companies, including taxis in New York City. One of the interesting things that came to bear was that whenever they have a taxi or car that doesn’t show up for a patient or cancels for one reason or another, which is pretty common, it takes – in New York City it takes them about 45 minutes to get that passenger, that patient, another ride through the taxi company. And so they started using Lyft and found that wow you can get a ride in just a matter of moments using Lyft and get this patient home so they can take their medication, so they can do whatever it is they need to do and not have to wait there on the street or whatever. And these folks have a lot of times more special needs then some of our normal types of passengers. So the great thing about this is in terms of building this new product research was highly involved. And so I went over to see National MedTrans, brought a designer with me and as anyone who’s done contextual inquiries know there’s so much more to learn when you’re in that environment and learning who the users are, what their daily tasks and processes are and the entire environment. And this company where we visited them was like people in cubes functioning as a call center. They’re scheduling rides. They’re working with taxi companies. They’re talking to patients, caretakers, people who are managing transportation needs for these patients and it was really, really insightful for us to be there and use the insights there to figure out how to build the best possible product for these users. It’s something totally different than what we have in terms of our native app offerings today. Steve: And who will- within that use case are there multiple users there that have different experiences with Lyft? Monal: Well the primary user in this case is the person at Nat MedTrans who is requesting a ride and managing the ride. So right now a lot of these elderly folks, they don’t even have smartphones. So it is a little bit of a juggle in the sense that Lyft wasn’t necessarily designed for this use case from the first and so we’re getting to the point where how can we build the best possible product. There’s the driver, there’s the passenger and now there’s this middle person who functions as if – kind of like the app does for you or me who might be taking a ride today. Steve: It just speaks to a really, really (in capital letters for me) BIG idea. A lot of the technologies that we see are apps and there’s interesting – I don’t know let’s just call them democracy or democratic access I guess is probably a better word for this. If you don’t have the income or the technology, sort of know how to use those things you’re shut out of a whole bunch of ways to participate. And what excites me about what you’re talking about is that it’s a fascinating research question then is how do we make non-mobile device users Lyft users? Those principals can apply to anybody but you guys are digging into that. It’s really fascinating. Monal: It is actually really fascinating and just being there and thinking through these whole – you know this set of issues and this problem, it really brought me back to my roots where this feels like a true user experience problem to solve, you know. Lots of different players, various tasks and processes, how do we make this work? Steve: I hear the sort of analytical aspect of it as you talk about it, but I also hear your sort of – your point one about empathy. Like there’s something in your voice which won’t come through in our transcript, but there’s something in your voice as you talk about it. It seems like this touched you or this struck you in a certain way. Monal: Yeah, definitely. I mean the thing that I think struck both me and the designer when we were there was like wow we’re there in this call center, someone called in, a patient who had been waiting and they couldn’t get a ride and how quickly they were able to resolve the issue with just using Lyft. And it just blew our minds. We were like wow. It was just really cool. Like we made this person’s day so much better by just getting them home. Steve: So that project sounds like a sort of a special one, maybe sort of outside. Or it’s a new area for you to be looking at beyond the traditional set of users that you’ve been thinking about. I don’t know, in the life cycle of doing the research, whether it’s getting your data or helping the teams make use of it, I don’t know is there any spots that you find particularly challenging that you’ve had to dig into? Monal: Well another thing that actually drew me to Lyft was the fact that I think there are a lot of really interesting research challenge with our product. We’re really on the cutting edge with regard to mobile devices, being completely mobile, having to try and do research in a very constrained environment which is a car. And I think that there are a lot of methodological challenges that come with that. So for example if you think about it the best research comes from real life situations. You want to be able to observe – from of course from life qualitative user research. You want to observe what’s happening in the real world, or simulate it as closely as possible. And if you want to find out what’s happening in ride experience, if you’re in the car you’re already biasing what’s happening there because you need to announce yourself as the researcher. Everyone knows you’re from Lyft. People are going to be a little bit more cautious about how they act. Let’s say take the researcher out of the car. Let’s just put a camera in there. Well then there’s legal constraints as well. You need their permission and if you have their permission then of course they’re again going to be a little bit on guard and this will really bias the results if this is what you’re interested in learning about. So we’ve had to come up with some creative and innovative ways to study some of the things that we’re interested in learning more about and I think along with that you know recording, conducting research, and huge amounts of data are always challenges for researchers across all industries. So specific to Lyft though, and specific to this area in particular, I think we have some really interesting challenges and it’s fun because it’s – you come to work every day thinking about – you’re really stretching your brain and you feel really good when you come up with something new and get to be creative in those senses. Steve: Are there any process innovations or anything that you can share? Monal: Not exactly (laughs). I can tell you about one of the first programs that I started when I joined. So we wanted to ensure that we had user testing for most of our projects coming through on a regular basis. So I started a program called Drivers in the Office, so D-I-T-O, which we call “deeto”. And along with that PACSITO, which is passengers, which we often abbreviate as PACS, in the office. So DITO and PACSITO, it’s like a weekly user testing cycle and it’s been a lot of fun. We have people come into the office here in San Francisco. Occasionally travel to other cities and occasionally do some remote studies wherever appropriate. But what we do is we have, and especially for drivers, for DITO, what we’ll do is we’ll actually have drivers come and bring their car, park in our lot and then we’ll have them use a prototype of a new design or product that we are testing and this allows us to capture everything that’s going on while they drive. So it’s one thing to be a passenger, in your house, on the street, looking at the app. You can stop, you can look at it for as long as you want in a maybe normal situation let’s say. But as a driver you have so many other distractions and concerns. I mean it’s a very cognitively complex activity. You are doing so many different things at once and now we’ve just added one more thing to pay attention to which is the app. So it’s a huge design challenge for our team and also from a research perspective it’s really important for us to be able to get the data in the right environment, in the contextual way that they will be using it because they’re using the app at an arm’s length. They can only look at it upon glance, most of the time when they’re driving for safety reasons. And we have to be really cautious about ensuring that it’s not overly distracting. So one of the things that’s been really great about this program is that times when we’ve tested like more low fidelity prototypes where we can’t go in the car, versus going in the car, we’ve learned quite a bit. Steve: There seems like there’s an interesting, maybe sort of a hybrid – there’s like a semi-contextual aspect here where they’re not in a lab, there’s a high fidelity simulation of what the thing might be. You’re dealing with their actual car, you have realistic use cases, but you’re not also sending them off into the wild to live with this thing for a day. It’s a constrained experiment, but you’re sort of creating I guess as much context within an experiment as you can. Monal: Um-hmm, right. And I think that’s important for us to – it’s a big part of how it’s going to be used in the real world so we need to capture that – how they do so in that environment. Steve:What about other examples of kinds of, within what you can talk about, what are some other kinds of research that your team has done? Monal: We basically try to do as much mixed methodology as needed. We do surveys, as I mentioned contextual inquiries. We’re basically employing everything that we need to design the right kinds of studies to answer the business questions. You know we have some user interviews coming up. We have – what we also try to do is work very closely with our analytics team. Steve: Can you talk about anything that has – that people that use Lyft has experienced that’s changed as a result of some of the research you’ve done? Monal: Definitely. One of the most exciting things that I feel user research has impacted was our Lyft app redesign that launched at the very end of last year. So we did a lot of iterative user studies, user testing especially around various versions of designs and the product moving forward and most of the product and design decisions relied heavily on the research that we did. So by the time we launched the app, the new redesign, we felt and our team felt very confident that it would be successful once it launched and it definitely was. We’ve seen really great results and have heard a lot of great positive reviews and comments about what we did and I think a huge part of that, I feel really proud to say, was due to the research that we did. So whenever you can affect the flagship app on such a big scale with this design you can feel really good about it. Of course we don’t take all the credit. There’s such incredibly talented people here that you know we were just lucky to be part of that process. Steve: I think even just this phrase about sort of being proud of what you accomplished, it makes me think about how do you lead a research team. I mean giving out kudos, I think that’s one part of it. I’m just projecting on you here a little bit, but what are other things. Talk to me about being a leader in research. What does that mean? Monal: Yeah, I mean it’s interesting because I think until recently there weren’t a lot of leaders in research. Primarily I feel like research has had a lot of individual contributors and that’s how we’ve been seen in a lot of ways, people who are providing data analysis, insights, and hey, great, thanks you did your job. And now with the growth of research and especially user research we’re seeing more roles where there are leaders. And I think it’s great for all individual contributors because a lot of times – and this still happens today – the individual contributors who have worked as researchers in the past haven’t always reported up through someone who understood what they were doing. And so it’s great to see that that’s changing to a degree now and I think just in that sense of having the empathy (back to empathy again) for understanding like what these researchers are trying to accomplish, what their personal goals are – I think a lot of what researchers in general – I’m just thinking back through my career as well – have felt is, I mean we really want to make a difference. We really want to have influence on the product or the design. We did this great research, we want to make sure that it gets in there. And this hasn’t always been the case for me in past companies. And one of the great things here at Lyft is that what we’ve produced has been so highly utilized, probably more than any other company I’ve worked at. So coming back to your question in terms of leading, I think ensuring that the work that we do does get implemented or is seen as respectable, credible, and that people understand what we do and how we do it. And bringing them along for the ride too, it brings us to a point where within the organization you’re seen as oh wow, like they do really great work and they’re going to affect really positive change for us. So sort of helping the team be known and understood as experts and ensuring that the work that they do has a positive effect on the company and for themselves. Steve: And you’re right. That fell on the individual contributor to sort of – you had to do all that you had to do and you had to be able to do that. Monal: Um-hmm. Steve: And so you’re talking about like what a good manager does. They sort of, they advocate for their team, they… Monal: Um-hmm. Steve: …protect them and champion them. Monal: Definitely. Steve: You talked early on about this really important mentorship relationship that you had in your career. Are there commonalities between what you experienced kind of as the recipient of mentorship and what you try to provide as a leader now at this point? Monal: Definitely. I mean John who was my mentor back then, he gave me so much and I feel like I’m at the point in my career where I definitely can do that for other folks as well. So right now it’s primarily here at Lyft, but I’d love to do that outside of Lyft as well. Steve: How do we define mentorship? We talked about leadership, but what’s the difference between mentorship and leadership? Monal: Yeah, I think mentorship is a little bit more focused on – well in my mind helping folks develop new skills as well. Leadership doesn’t necessarily have to do that. I think leadership can do that. I mean if there was a Venn diagram it would probably be overlapping in the circles, but I definitely believe that a good mentor will help someone on a personal level develop into whatever career goals they have. Helping them along that path to get to where they want to be. And for me I learned a lot of methods from John. I think I’m trying to do that as well on a mentorship level, but also just be someone always who’s there to talk through, you know, career related growth. Steve: You know in some ways – we’ve talked a little about sort of the evolution of the field and it makes sense that at that period of time developing methods, not that that was all you were learning, but that was kind of a key. And I think about your three areas – I mean the first one you listed, you kind of titled it as empathy and then you listed like 18 other, like really important sort of soft skills that are crucial to being a researcher. So when you’re there for people as you say I think that’s starting to model, yes the methods, but also now these other pieces that are so important to being a good researcher. Monal: Um-hmm. Steve: That are just very, to me very human skills that we all can – we all can do to work on. Monal: Um-hmm, definitely Steve: Are there other things about you and your background that we should understand to know what kind of researcher you are? Any weird dots you want to connect for us? Monal: Yeah, actually I think one of the things that’s really unique to me is that I previously had another life as a professional athlete. So I was very active in team sports and that’s a big part of like my philosophy too. Very team oriented. I started playing soccer when I was six years old and I got a scholarship for running when I was at Stanford. So that’s been a really big part of my development and who I am and the really cool thing about running at Stanford is that when I started there and started running on the cross country team there we weren’t nationally ranked. And by the time I was a senior my teammates elected me as the captain of the team and we won the NCAA national title that year which was less than four years later and so it’s been really interesting to draw a lot of comparisons and parallels to my athletics career and how I’ve envisioned like sports and teamwork and accomplishments and my working career. And it’s funny, I don’t even realize I’m making these analogies. I started talking about it to a coworker the other day and I’m like wow I do this actually quite a lot. But there are so many parallels to draw with regard to pushing yourself hard, getting to the next level, goal setting, you know the competition and teams, relating to people, supporting one another. And one of the reasons why I think we won the national title back in 1996 was because we had such a tight, supportive team and I really feel that we have that now at Lyft as well. User research is part of the product design team and we are such a close team. Actually we just came back from a weekend retreat in Tahoe. So we’re kind of like a tight-knit family and I really feel that that support and that kind of environment fuels – helps fuel really great success. Steve: So some of these parallels you’re describing, they seem about what athletics teaches you about working and collaborating. Are there any threads that you can pull from athletics that go into things that are specific to UX research? Monal: That’s a really good question. I think I need a minute to think about it. It’s something I haven’t thought about before. I’m sure there are. I think I need to noodle on this, but I like the question a lot. I think because a lot of what – the parallels I’ve drawn have been more meta, like high level. Yeah, nothing springs to mind, but at some point I’ll get back to you. I’ll be like ‘Steve guess what, I figured it out and now my life’s questions have been complete Like I know. I see the light!’ Steve: Do you have questions for me? Monal:Yeah, actually, so I was curious because you’re one of the few UX research veterans in the field and I was curious what kind of trends you’re currently observing in our little field that’s growing? Steve: Yeah, I think – I liked what you said about mixed methods. I think that seems to be, you know, where traditionally on paper there was a lot of us and them. I feel like I’m hearing more harmonious stories and I think it behooves us as a practice to tell harmonious stories, but because that speaks to a successful, integrated, mature, not a diva, you’re wrong, this focus group sucked, blah, blah, blah. Like that’s sort of – that’s what we’ve been saying and I think true or not it’s sort of disharmonious and I feel like the stories of how groups have come together, to me that’s like oh we’re a little more mature as a world now because we can coexist well with everybody. And the qual/quant things used to be cats and dogs, but as you said you guys integrate well. I’ve heard that from lots of groups that I’ve talked to and I’ve seen that. You know clients I work with start to say like oh yeah, we have this group, this group, this group and we can kind of poll lots of – so that’s mixed methods. That’s also, you know – I don’t know sort of mixed disciplines I guess. So it seems like that – to me there’s just a maturity level there with that. Yeah, that’s my big one I’ll leave as the answer for that. Monal: I agree and I really think that in order to have the most robust insights we do need a good triangulation of different types of data to be able to tell that story. I was also curious, I mean as we talked about UX research has come a long way from you know the mid-90s and where the worldwide web was at that time. I mean just technology has come a long way and UX research has alongside of that. So that said, where do you think UX research needs to develop most given our current landscape? Steve: These are the questions I should be asking you but I didn’t. Monal: The tables have turned. Steve: Yes. Yeah, what’s the sort of opportunity? And this came up as a topic in an earlier episode of this podcast series where I don’t know that we answered it, but we talked about just this idea of research sort of evolving to the point where it disappears. That research right now there are leaders that are trying to develop in a business, but – and you’ve talked about the successful collaborations and so on. Is this an activity or is this a discipline. You know I hear so much about teams trying to – I mean one way to deal with the overwhelming demand of research is that research leaders are empowering teams to go do their own research and so then they have to deal with this question of well what do we do as the researchers vs. what do we kind of let everyone do. I mean that’s the consequence of the evolution of research and we’ve been saying, as part of that evolution, like everyone should be doing it, everyone can do it, right. There’s books that teach people how to do research, like that’s a change and so we – I think in the early days we struggled with do we let go of this or do we hold on to it and we chose to do a bit of both. So how does that play out I think is an interesting question? How should it play out? And research doesn’t exist in a vacuum, so how technology has developed. You know we are in a lot of the way – in a lot of environments that research is being used it’s a close colleague of design, right. I mean you talked about being part of the product design organization, not some other – part of the market research organization. That’s who our sort of, our teammates are. And design is changing and you talked about this already too, right. That companies used to just do technology and sort of experience was secondary. So I think where research goes we’re sort of subject to these larger changes in how – in how experiences are created. So I don’t have an answer to that and I’m not sure it’s a need. I don’t feel like oh we better get this sorted out or we’re dead. I just feel like there’s some interesting, sort of existential questions to reflect on. You know I mean over the course of our careers you and I have seen what kinds of organizations and what kind of roles are there for us. It’s shifted a lot and as you said there weren’t leaders like yourself a few years ago and not in the numbers that there are no. So even just for our own careers, like it’s going to be different in a few years. I don’t know. It’s tough to predict the future, but I think those are the – those are interesting, if abstract areas for me to keep looking at. Monal: Um-hmm, definitely. I mean I think it’s – I agree with you and I also think it’s really great news that the discipline is growing because there is a lot of value that we can provide. And you know the Lyft research team is also growing so I think it’s a really good sign of, you know how much we’re able to impact the organization in a positive way and I love seeing now, after having come back from Berlin – previously it was like oh we can only work at big companies. And now I feel like every startup has a researcher. Maybe not every startup, but it’s becoming more and more common which is amazing. Steve: You know the designer as founder sort of thing, it’s like a – it’s a thing that gets recognized and talked about and I feel like maybe we identified with – I feel like I was on some Slack channel or some list and people were saying well when do we get our first researcher as founder. I’m probably misquoting someone that said that to me and I apologize for that and that company may exist, but certainly that idea is nascent whereas designer as founder in the startup world at least is well established. Maybe that’s the, you know sort of next wave for us. Monal: Um-hmm. Cool. I have one more question for you. Steve: Alright. Monal: So I’m really curious, and I know you’ve had a lot of questions about this podcast, but I was curious what you’ve found to be the most striking similarities between all of whom you’ve talked to so far. Steve: Don’t you love when you go out in the field and the person at the end says like “hey what else have you heard?” Or “am I normal?” They want you to kind of normalize them as part of it. That’s not exactly your question here, but yeah, what do researchers have in common? I think the emphasis on product is really, it’s like a thing that we should be proud of as the field and it goes back again to your list of three things that you talked about. As you kind of researchers needing to do good data collection – I’m paraphrasing you here – make sense of it and then sort of help it live in the organization. I am definitely twisting what you said, but that’s sort of thrust I took away from it. And I don’t know, I carry some scars from my early days of coexisting with people that – where research was their end – they loved – and you know I guess scars, but I’m also definitely guilty of this myself. Like research is fascinating. If you do it well the experience is great and you learn things that have no relevance. And I mean I’m a big fan of not knowing if it has relevance, but you learn things that you don’t know if it’s what you’re going to do with it, but you also learn things that just are great and they’re not applicable, but are part of what you have to do on that journey. And then you make sense of things and you find some really interesting takeaways, or things that are potential takeaways that aren’t necessarily usable by the company. Of course then you find here’s the things that we learned and here’s what we need to do about it and here’s all – but all the way along there’s a lot of fascination, right, like… Monal: Um-hmm. Steve: …it’s super engaging and to me very creative and I don’t know. I can be seduced at many points along the way. I mean, I don’t know – I think I’m good at making sure I get to that end point, but there’s a lot of seduction along the way with just stuff that is super inspiring, super thought provoking, that you kind of dig into and start synthesizing and organizing. So the scars are around, you know maybe sort of early days in the field, or my early days, where that’s where people kind of dwelled and someone had to come along and kind of pull them and say like okay you can’t deliver that. You have to show actionability. That sort of discussion was needed. And I mean it makes sense that the people that I speak to, like you who are leaders – I mean in order to be a leader you have to be beyond that. There may still be practitioners and contributors who kind of need to learn more of that, but I feel like if I take the temperature of the practice in terms of how it’s being led it’s about what we do with research. Monal: Um-hmm. Steve: And that is a massive change again over when I started in my younger days. And so to see that in common – again, maybe you go well of course Steve look who you’re talking to and what kind of jobs they have and look at what kind of companies they work for. You know again, yeah, you might find people at that skill level, like yourself, and experience level – if we were to like shift back 10 years you would be maybe working in an R&D lab or an advanced products group or something and not shipping stuff. So yeah, I think one of the commonalities is like these are people making products, or helping companies make services, whatever the thing that their company does they’re doing that and obviously that’s – I think that’s really, really good. Monal: Um-hmm, yeah. One of the things that struck me about what you’re saying is that fascination and I think as researchers like a lot of what we love is having that curiosity and it’s so fulfilling just to do the hands-on work. My role is managerial, but I’m always still – you know I do some hands-on, individual contributor work as well. I just love doing it, right. That’s what makes us passionate about research, we love doing it, but I think in a business one of the dangers is getting too far into that fascination, but then not taking that next step. And maybe that’s where managerials, or managers can help with making sure that the data and the analysis and then these insights and actionable recommendations happen and are affecting product. That we’re not just doing the research and being fascinated for our own curiosity and fulfillment of that, but that we’re actually making sure it’s being put to good use. Steve: I think that’s really important and it reminds me of just a period in my own career where I was managing my own team in my consultancy and I think like – we always had many different projects for different clients and I want to say maybe for a period of 18 months, which is a long time in sort of consulting years right, I didn’t go in the field at all. I was doing what you’re talking about, right, sort of managing the problem, helping with the synthesis, kind of trying to tie everything back. Sort of a creative lead and it was really cool to sort of – you know you remove one part of the puzzles and the other parts – I guess the other things kind of rise in prominence. So it was real interesting for me to step out of certain tasks and step into other tasks in a larger way. And I actually remember the first time that I went – it just worked out that like oh yeah I was going to do some of these interviews and I almost had a panic attack. Like it had just been so long. And I was not on my own. I just remember like kind of coming up to the door of somebody’s and just about freaking out. And you know the way I work now I’m sort of – I do more of everything, but having had that experience of sort of again playing just part of the role was really, really interesting. It’s good to hear that you – it sounds like you’re in the right spot. You still have your hands in things, but you can lead and kind of manage in that spot. Monal: Um-hmm, yeah. Steve: Alright well that’s maybe everything we have time for to talk about today. But this was really interesting. It’s one of these conversations where now I’m curious about a million other things. I’ll have my imaginary next 90 minute conversation with you. But yeah, thank you very much. You’ve really shared a lot today. Monal: Thank you Steve. This was a lot of fun and yeah, I really enjoyed it.
April 26, 2016
In this episode I speak with Kate Lawrence, Vice President of User Research at EBSCO Information Services. Our conversation covers where to place user research in the organization, emotions in fieldwork, and empowering others to advocate for information literacy. The ability to write, synthesize, distill and synthesize findings into a report that is digestible and spot on is very challenging and there are people who can do it well and I really strive to find those people for the team. So great writing skills – and part of that writing is storytelling. It’s a combination of communication, analytical skills and research skills. – Kate Lawrence Show Links Kate Lawrence Follow Kate on Twitter EBSCO Information Services EBSCO Industries Ipwich, MA Mark Zuckerbeg’s hoodie Shonda Rhimes’ Year of Yes Qualtrics UserTesting Pizza in Japan Cruella De Vil Affectiva Eat This Not That User Research War Stories Deirdre Costello Kristen Arakelian Flipster Follow Dollars to Donuts on Twitter (and Stitcher) and take a moment to leave a review on iTunes. Transcript Steve Portigal: Welcome, Kate, to Dollars to Donuts. Kate Lawrence: Thank you for having me. Steve: Why don’t you begin by introducing yourself? Kate: My name is Kate Lawrence and I’m the Vice President of User Research at EBSCO Information Services. We are the leading provider of research databases and other content to libraries and institutions all around the world and we’re headquartered actually out of Ipswich, Massachusetts – a beautiful town on the North Shore of Massachusetts, north of Boston. And our parent company is actually EBSCO Industries and they’re headquartered out of Birmingham, Alabama. And EBSCO Industries is one of the 200, one of the top 200 privately held companies in the United States and EBSCO is actually an acronym. EBSCO stands for the Elton B. Stephens Company and it’s a family owned company since the 1940s and again it’s one of the largest privately held companies in Alabama and one of the 200 largest in the United States and what’s interesting is one company, many different businesses and so the largest part of EBSCO Industries is EBSCO Information Services where I work in Ipswich, Massachusetts, but what’s interesting is EBSCO Industries has lots of other businesses. For example EBSCO owns the company that produces the most fishing flies of any manufacturer. They have real estate holdings. So it’s quite diverse in that way. Steve: Does that create a certain culture and not so much EBSCO specific, but companies that have diverse holdings – and I don’t know if you can even answer this, but you know we hear a lot about you know you work at a start-up, you work at this kind of company, but just you work at a very kind of unique organization. It’s part of a very diverse, larger company. Kate: It is and what makes it fun and unique to me is every day brings something different. So the part of the company where I am focused is the Information Services division. So we’re – we work on products and services that provide essentially information, premium information, to information seekers around the world. And occasionally I will get pulled into another project. So if there’s – we have a – there’s furniture company that makes work stations and so we did this whole research project looking at what are the furniture needs of the new library – you know the new information commons? What is that collaborative furniture that is needed, that’s different from the study carrels of yesterday? So we occasionally get pulled into projects and I’ve actually done a small research project on game cameras for one of our other outdoor product products called PRADCO. So you do get these projects peppered in that keep it very interesting. Absolutely. Steve: It seems like the first example leverages an area that you’re focused in, that the information work – the information, information user is already focused on the library and people using the library and the work that they’re being done and if I’m guessing the game work is much further outside that. Kate: It’s true. The game camera work was outside of that. What we do on a day to day basis is our research team focuses on studying the path and the search process for information gathering, information seeking. And like you said libraries are customers and think about libraries and think about library patrons. You have your academic market. You know you have students as young as age 6 and 7 and 8, starting to work on research projects. I mean it might just be the history of the Lego Corporation or finding out about Harriet Tubman, but that’s research. And so we have young students, elementary school students, all the way up to 95, 98, 101 year old public library patrons. So all of us interact with the library in some way and that’s why you see EBSCO products all over the world. Steve: You know when you describe the young kids doing the Lego Corporation history research project it makes me think that there’s some major shifts that are going to impact the business that you’re in and what your research is going to uncover. Education continues to but what are some of these larger changes that we as a culture are going through that are driving changes in your product and things that you’re looking at in research? Kate: Changes are because of Google. Google has changed everything about information seeking and information gathering and search. And Google being described by many of our students, they’ve described Google as their oxygen, their mother, their water, their soul mate. And they are completely serious and it makes perfect sense why they would describe Google that way, but Google has been this seismic shift in the way that people conduct searches for information. So when you think about, Steve, when you – when the two of us were in college, when we had a research paper to do we went to the library. We worked on it there and do you remember – everyone remembers the smell of the shelves. You know you went in and you – the library had a certain smell to it. It put you in a whole different zone when you were doing research. Research is different today. It’s a lot of guided self serve because you don’t have to be – because of electronic resources and you can access these materials from home, you can log into your library website from Starbucks, anywhere. You can conduct research on the go and so you’re not necessarily within sight of the librarian. You’re not necessarily even on campus. It’s a different experience and it actually starts in Google. Steve: So – and you speak about this in a way that makes me think you’ve talked with a lot of people about this over the years. Kate: Students are an area of tremendous focus for the user research group at EBSCO and as I was saying we have students all over the world. And students by the way is that third grader all the way up through graduate and doctoral students and fellows who are doing intensive research. So you have people who are doing research for a basic introductory class and they’re not particularly invested in the topic – may be in their first semester in college. And then you have your doctoral level researcher. So when we set out to design products and experiences and we think about interface design we have to think about this range of abilities and at EBSCO our customer and our user are not always the same person. Steve: So the people that are making choices about what we call information products, information systems… Kate: That’s right. Steve: …they are choosing something that’s going to be rolled out to students, I guess, and other users as well. Kate: Students with a very different approach to the usage and very different knowledge of the topic of information seeking and very different level of searcher. Sometimes I equate it to trying to appease the parent through a great experience for the child. So you think about parents buying products for kids and then you think about who are you trying to suit? Who are you trying to serve? And what we’re trying to do is create an experience for research that is delightful for students and then as a result also… also pleases the librarian, but pleases the librarian customer because it’s serving students so well. Steve: So that’s an interesting area to think about. If as you said Google has really, really changed things – you know I’ve done a little bit in this area so I have a little knowledge which is dangerous – I mean when you describe these two groups, the people providing the service – the facilitators, you kind of used the metaphor of parents – librarians and people in that role. You know my casual observation is that this shift – you know the shift that’s Google and kind of everything that falls under the umbrella that you’ve related, how they’re experiencing it, and this is just natural – how they’re experiencing it as professionals and people with a lot of history with tools as they’ve evolved, and how these students are experiencing it, they’re looking at that shift very differently. Kate: They do look at the shift differently and it reminds me of when – I remember when Mark Zuckerberg put on that hoodie and started wearing that outfit to work, it was upsetting to people. There was a certain generation and a certain group of people who are really offended that someone who runs a company is going to show up in a hoodie. And I think like – it’s kind of like when parents say “oh the kids today.” There’s a sense that we don’t have to do it the way our parents did it. We don’t have to kind of treat research like the Oldsmobile that my parents had to drive. Like I don’t – you know look Mom, I don’t have to do research the way you did it because I don’t have to be in the library and you had to be in the library. So I think we’re all still adapting and coming to terms with the new rules of research and the new way that students are getting through and studying in college. It’s very different and when you said it sounds like you’ve studied a lot, we do go out and interview users. We do employ ethnography and we sit with students on the floor of the coffee shop in Las Vegas, or right outside their community college. Or we sit with them after swim practice in Texas where they’re in a breezeway in a library trying to do a little research on the go right after swim practice. And so we sit and we watch their organic search process because until you see it, it’s hard to believe that it is the way that it is and it’s piecemeal and it doesn’t necessarily have a solid strategy, like this is how I’m going to go in, this is how I’m going to do it. There’s a lot of trial and error. There’s a lot of trial and error and it absolutely starts in Google because they feel safe there and starting your research in Google by getting this foundation of your topic, it reduces anxiety. And research can be, for some students, a very anxious process. Steve: I’m thinking sort of the network of – I want to use the word stakeholders in the – in the broadest sense possible. You’ve talked about students. Certainly for younger students there’s the parents. There’s the people that are providing them – let’s just call them librarians, is that the right term? People that are providing them with these tools? Kate: Yes. Steve: And those are kind of all outside and then inside your organization there’s you and your folks that are going out and talking to these people. There’s somebody that is taking what you’ve learned and acting on it in some way. What – very generally what does that node look like? Kate: That is, at EBSCO what we have is we have evidence based product development and we’ve embedded user research into the product management organization for this reason. So what will typically happen is we have achieved in the last – I would say 12 to 18 months – we’ve been able to achieve alignment. And so we understand, for user research, what is coming 12, 18 months from now. So we’re in front of it, we’re doing the research now. And so that was one of my key learnings when I first started the group and I first started doing user research. I did every single project that came my way and we joke about the year of yes. You know Shonda Rhimes just wrote “The Year of Yes” and I said I lived that year. I said yes to everything just to get the demand so that I could get the resources. And what happens when you do that is then you’re doing research on everything, but strategic alignment is critical. And so now we’re very much aligned with what the product development cycle is and we’re able to do the research at the outset instead of what used to happen – and this is typical of many companies. Years ago it was let’s do usability testing before we launch and that is a tough situation for anyone to be in because then you’re this kind of glorified version of QA and you’re just – you don’t have any chance to make an impact if something isn’t testing well. Steve: Can we pause that thread and can we go back to something? I want to talk more about how this has become adopted and integrated. But can we go back to this earlier point and sort of who’s involved in it to me seems really challenging. So you are sort of uncovering new paradigms. I think there’s – they’re kind of, they’re transgressive paradigms (my word not yours) – but they are changing the way that we think about how people are searching, what outcomes look like for information problems and so on. You’ve established how that works between different parts of the organization internally. And again I’m projecting here, but if I’m in the business of selling products to these librarians and these products – it’s what you said before about trying to please these different audiences. These products have to – they reflect a new model for how people are going to use them, but that new model might be a little – I don’t know, how is that new model being received by the librarians, by the people that are sort of selecting and rolling out the kinds of tools that you have to offer? Kate: I have been so encouraged by the librarian reaction to the research because one of the talks I give, one of the themes of a talk I give on a regular basis is called Student Researchers – The Reality Show, or the Realities of Student Research – because it’s very different and it’s librarians who come up to me afterwards and say thank you, you’re validating what I’m seeing and now I have greater context. And also many of our findings support the role of the librarian in an active partnership with faculty. Let me give you an example. You are sitting in your Intro to Philosophy class. You’re a freshman at University of Virginia and you’re there and all of a sudden you hear I’m getting a research assignment. And so this kind of fills you with a little bit of anxiety and the professor is giving information about you have to use scholarly sources, things have to be peer reviewed. You’re getting kind of the elements of the rubric. So New York Times, Time Magazine, these are not written for academics. These are not written for an academic audience. They’re not peer reviewed. They’re not considered scholarly peer reviewed materials. So that’s another point of anxiety for students because they have to go to materials they may be not be familiar – as familiar with. Something amazing happens when in addition to the faculty member, your professor assigning this work, there is an amazing phenomenon that occurs when a librarian is brought in to the classroom – okay, taken out of the library. He or she leaves the library and is in active partnership with the professor, talking about this is how to do this, this is how you navigate the library website to find these scholarly databases from EBSCO, etc. It’s the equivalent of when your parents – both your Mom and your Dad – say Katherine, we’d like to speak with you. Something in you is queued up to listen and you’re more apt to learn it and you’re more apt to learn it and absorb it. As opposed to the professor saying well next Tuesday you’re going to meet in the library and the librarian is going to walk you through X or Y. So what happens when we talk about this is some of these findings they empower librarians to advocate for a more active role in this process of teaching students information literacy skills. And that’s exciting to see when they say this is going to help me help my students and they appreciate that. And I’ve been – I’ve been overwhelmed by just how wonderful the librarian community is and how supportive our customers are of our work. Steve: That’s really fabulous. There’s something too – can you expand a little bit on this notion of – you’re talking about kind of the librarian in the library and then the librarian in the classroom. Kate: Yes, it’s so interesting. Steve: There’s some kind of frame shift that’s happening there. Kate: There is. And so think about – think about the amount of effort it takes you when you’re in the library to get up, leave your laptop unsupervised, walk over to the librarian desk and ask a question. That’s actually harder than any of us might imagine. And so also when you’re in the library, what we’re hearing from students is often they are collaborating or they’re having a meeting and they’re with other people. So to interrupt that and get up and go ask for help is hard. And other obstacle is some students don’t know how to ask. And I have a customer I’ve worked with over in Europe who said that they actually post signs at the reference desk that say – you know how to ask. You know ways to frame your question because that’s an obstacle. And then we’ve had students say to us you know I want to – they actually feel very positively about the librarians. They’ll have Live Chat and they said oh the librarian came to my class. He or she is so nice. But there’s still – are you using a library? Are you using librarian services? Are you going to the reference desk? No, I’m just asking my friend or I’m asking my roommate. And the reason that is happening is – goes back to what I said about a lot of this searching is guided self serve and the guided part is – and the self serve part is it’s 11 o’clock, they’re home, they’re on the couch, they’re not in the library per se. And so who is nearby? That’s your go to. Steve: Can you talk a little bit about the, I don’t know, mechanics, logistics challenges – you know these kinds of, the people that you’re doing research with and then you described some really interesting contexts that you’re in as well. Are there things that you have had to figure out to make that kind of an established process for your group? Kate: So working with students is fascinating. We also research physicians. We research nurses because EBSCO has medical products. We research, again, the public library patron which is all of us from the time we’re two, or two months old, to the time when we’re a hundred and something years old. We have relationships with our public library. And we also research corporate users. So we’re – EBSCO’s user research team – in our team we are researching people all the time and I always say that our most important job is matching method to question because everyone gets really excited about big contextual inquiry, ethnography studies, but really that’s not appropriate in every scenario. It’s more of a tool when you’re examining markets. Like let’s study the public library patron, let’s study college students, etc. But you know what I find – one of our greatest challenges is we have to know where to spend our time and so we rely on – we rely on this technique of – that I talked about with the strategic, strategic alignment to understand what’s coming, what are the priorities and how can we do research to best support that. And we have some great product managers and product directors who help – who always come and knock on our door and say I’ve got this thing coming up, I’m going to need research for it. And so we’re – culturally we’re at that point where we’re not having to chase people down anymore. They’re actually coming to us and we have a backlog and we have a wait and we conduct probably close to 80 studies a year and we hope to increase that with every new person we add to the team. Steve: So, I mean your point that not everything needs to be a contextual research study, what are some other methods that you rely on? Kate: We have used – we’ve used lots of different methods. So we’ve used traditional surveys. We subscribe to Qualtrics, and we also – has been a great resource for us because it’s allowed us to, you know just do more work in a shorter amount of time and it’s a way to kind of expand the group without adding resources. I will tell you though some of our most fun work has been methods that were new to the group, like for example we did a modified video diary study because we study students and we study students who are under the age of 18 and so we deal with legal issues and consent issues and one of the things that’s challenging about researching 14 year olds, middle schoolers and early high schoolers, is that especially 14/15 year old young men, boys, they don’t necessarily, they’re not the most talkative group in the world. And so I’d read somewhere that Google had put video cameras in envelopes and mailed them off and asked students to – younger kids to track themselves searching, etc. So we basically sent off these video cameras and I think I only spent $50 on each one because we weren’t sure we were going to get them back. So we sent them off in these padded envelopes with some tasks for users to complete and it was great. And it’s a method that we want to use again because we got the quiet 14 year old boys, we got them talking. And they told us lots and lots about how they conduct research and what they’re looking for and they evaluated screens for us. And every single video camera came back. Not a single one was damaged. It was a great method and it was so much more successful than it would have been if we’d had them in a room, asking questions. Steve: Because they had sort of their own space to respond. Kate: Yeah and it was this – it’s the selfie generation. And so it was just this kind of extended selfie and it was just extended selfie and it was really precious. It was precious seeing the things – just – they took pride in it. They created a little movie of their life and how they conduct research and we got to be privy to that. I mean the challenge is if you put those 14 year olds in a room, because they’re under 18 we have the parents in there as well. So you’ve got the researcher who’s like a second parent and then another parent, and it just – it’s not an opportunity – it’s not a situation where they’re very talkative in person, but we did manage to get some information out of them through the video cameras. Steve: You point of the selfie generation reminds me of this great video from a couple of – I want to say there were two Canadian students that were living in Japan. I think they were undergraduate students and they were doing these sort of mini auto ethnographies, for lack of a better term, where they did one video where they ordered pizza and they showed what the interface looked like and you know – I mean Japan is such a great place to talk about how things are different than what your normal reference frames are. So they kind of called them all out and then when the pizza came they sort of showed what the food looked like. They went through sort of, like the whole user journey, to use some jargon – they went through that whole journey in a very delightful, YouTubey kind of way and it made me think about maybe self reporting is – culturally can take on more of a role and you just proved my hypothesis with your camera study. Kate: Well you know the raw self reporting, that’s what I love, because so often when you’re filling out a survey you’re self report… Our self reporting – I mean this is why we go study users in the wild because people do things differently than they say they will. And I always tease my husband because he’s the only person when you go to the physician for your annual physical – you know the physician will say how many times a week do you exercise, like how many – do you eat all your vegetables ,etc. And he tells the truth and I said no one every tells the truth. No one ever self reports accurately. And they account for that. But he does. And this is why we study people in their environments. This is why when we go out we always come back with new information that’s so fascinating to folks because within the walls of our company in EBSCO, here in Ipswich, you know we make assumptions like everyone does and then you go out and you get this raw data and it’s exciting. So I like the idea of an ethnography app and I like the idea of the raw data capture because I want that. It’s those words and those truths I think from users that they pivot products and they really launch products that are so much better targeted and better suited for the users. Steve: How do you help these folks feel comfortable or feel like they’re doing the right thing? Kate: The same way I work to make people successful and then comfortable in an in-person interview. So let me give you an example. Years ago I had a boss who said to me the reason you’re good at usability testing Kate is you serve it up cold. You don’t give any emotion through the process. So if someone is doing something right or someone is doing something wrong they can’t read it on your face. And I was proud of this for about 15 minutes and then I started to realize over the years that that Cruella De Vil kind of cold fish approach to usability testing and user interactions is actually awful. And the reason it’s awful is because you need a genuine connection with people for them to tell you their truths and I have learned – I learned that the hard way as a researcher. And so when you connect with someone in person it’s that warmth. It’s that finding common ground that’s important so you can get – you can get information from that person. They can tell you what’s painful about the research process? Or they can tell you oh I’m trying to book an online cruise and this is what I don’t understand. And then you can actually take those words and take that wisdom and go back and tell the product team, like this is how you can really help these users, these customers. But you can do the same thing if it’s not in person. If you have a self reporting tool – like when we did the video diary study we wrote instructions like we were talking to 7th graders because if you send them a packet that reads like tax filing instructions, you’re not going to get any answers and you’re not going to get an video cameras back. So we put ourselves in the shoes of the user. What is their reading level? What is – how are they likely to talk about this? What terms – I mean they’re not talking about information literacy. They’re talking about finding articles for a paper. So we try very hard to speak their language and I think that’s one of the skills I value, it’s one of the traits I value most in our researchers is the ability to adapt and understand and listen and then adjust accordingly. Steve: Which you can’t do with the instructions. You have sort of one shot at that. Kate: You have one shot at the instructions and so you have to write on their level. You have to consider yourself as the user. And you may have to do some sort of pre-survey or pre-work in order to get – in order to get a sense of how they’re going to process the information. For the video diary study our researcher Lin Lin and myself we actually tested the testing instructions. We went through several rounds with people saying, you know do you know what to do here and you have to. So often we’re testing the survey or testing the test instructions to make sure that we’re being clear. Steve: Right the research – research artifacts are, they’re a designed tool. Kate: They’re a designed tool. Steve: And you have to test them just like anything else that has to be tested. Kate: Yeah, and I think that – I think that we can all get to a point where we feel comfortable. Like okay I’ve done this kind of test before and this was understandable, but you know recent experiences – I’ve had a couple of surveys that were challenging – we got feedback after the fact like this was challenging or – and so now we’re trying to be better about understanding how long the survey should take someone and then adjusting compensation accordingly. Because we had a survey that we thought we – I think we gave a $50 Amazon gift card for, but it ended up taking people almost 40 minutes. And so we realized we have to be better about understanding the amount of effort and how we’re going to compensate for that. Steve: Can I go back to your Cruella de Vil? Kate: I know, do you like that? Steve: I do. So – ‘cuz I think this is an interesting personal style of fieldwork thing. Kate: Yeah. Steve: I’d love to get you talk a little more about – you sort of learned what not to do and I wonder if you could characterize what your approach is now in terms of common ground and so on. Kate: So when we were talking about the usability testing and serving it up cold the whole approach was very kind of generic and you know show no emotion. So people are succeeding or failing. It was very metrics based, like oh did they succeed in this task to add X to their cart, or whatever it was. And so kind of I think as a person you evolve to start to understand that it’s about the holistic experience. It’s not about this one slice of the UI, it’s about this journey of I was in online travel at the time when that happened and it’s about this journey of how do I start to think about what trip I want to book for my honeymoon or what flights I want to book or what vacation package destination I’m thinking about etc. So as you start to think about the whole journey you realize that you’re not just, it’s not about just succeeding or failing. And so the warmer tone – you know you kind of go from Cruella de Vil to I guess more – to more of like the Dr. Oz who’s asking the questions about why and trying to unearth and dig at the whys. And trying to unearth those pain points and trying to look at it from a standpoint of empathy and understanding and not just let’s see how long it takes this user, this participant – how long it takes this person to find the shopping cart button. It’s really about what are your struggles? If you could change any one thing about this process or this product what would it be? You know and when you’re – we always say to users, we always say to students when we study them today, we say the process – how does it make you feel when you get a research paper assignment? How would you complete this sentence – the process of you know finding research for my paper makes me feel – and we ask them to fill in the blank. And through those words and the way they complete the sentence we start to get that picture and I don’t think it’s a coincidence that EBSCO and our product development team and as a company we’re starting to consider emotion when we think about not only the process of buying products as a librarian, but the process of using them as students and other end users. It’s not a coincidence that our research group is all women. I think that we’ve brought that emotional dimension into the discussion in a way that had not been present prior. Steve: So what I heard is that a way to bring that emotion in is in the questions that you’re asking. Kate: Yup. In the questions that we’re asking and also we’re going to be exploring later this year emotion analytics. There’s a company right in our backyard here in Massachusetts called Affectiva and they’re out of the MIT labs. It’s an emotion analytics product and it’s something we’re very interested in because for many people conducting research – we have this wonderful researcher on our team, our lead qualitative – our lead researcher. Her name is Deirdre Costello. She’s a librarian and a researcher and she likes to say conducting research for many students is like eating your vegetables. And so you don’t necessarily look for the emotions of they’re jumping up and down or they – like you’re not expecting people to look like they just hit the winning lottery ticket. But you’re looking for certain emotions to register on their face as they’re finding the path forward toward these scholarly resources. And so this is something that we as a team are going to be exploring later this year is how do we – how do we capture and quantify those emotions and what about the emotional state changes with certain design changes? Or changes when you’re looking at different products? We’re going to be exploring that and that is, that’s really new research for our group. We’re excited about it. Steve: This is a tremendous evolution from this Cruella de Vil era of sort of looking at metrics. Kate: Metrics. Steve: Looking at a lot of other kinds of things. Kate: Well you know that same boss said to me – and actually he’s a really – he’s a wonderful old smart guy. We were all learning at the time. He was the one who said to me I want the findings written up as a white paper, really in white paper format. And so I did that and I would pass people in the hallway and say hey did you read that report? And they’d say yeah, yeah, looked great. And I could tell that they never read it. And so the next time I did a usability study I said to my boss at the time I will write the white paper, but I’m also going to create this parallel version. And I actually based my reports – I did them in PowerPoint but I made them very visual and I based them on the format of that book written by the editor I think of Men’s Health or Men’s Fitness Magazine called “Eat This Not That” because someone, somewhere in my life had that book at their house and I was leafing through it and it was these visuals like look if you go to Panera don’t eat this, eat that. You know this is healthier for you. And it was just very visual. And so in 10 minutes you get a sense of alright if the only option is Jack in the Box this is healthy option at Jack in the Box. And so I did the report – like look users don’t understand this, but they understand that. And I did it in a very visual way, modeled after that book, and all of a sudden we got traction. All of a sudden the people who weren’t reading the reports were calling me into meetings and saying I want more of this, how can I get more, I love this. And so creating results that tell a story in a visual way is very important. Having said that we also in our group now, we do write the white paper and we do speak at conferences and do different things with our findings because we need a variety of different deliverables to satisfy our customers. Steve: Right. You have different audiences… Kate: We absolutely do, yeah. Steve: Can we talk a little about – you made the comment it’s not surprising that your group – did you say it was all women? Kate: We are. We’re all women and we have a male intern in the summers who is wonderful. So we are certainly welcoming of – we’re welcoming of women and men, but at the moment we are all women, yes. Steve: I mean what do you think that’s about? ‘Cuz I think if you look at the field of user research in general, and this is just anecdotally for me, it’s you know I think predominantly women is my guess. And someone may be rolling their eyes at that, but that’s my sense of it. I don’t know, what do you think is happening there culturally? Kate: Half of my group also have librarian degrees and I think librarians you find mostly women. So I do think the qualities I look for in a researcher, you know we are of course gender blind and so I think these qualities can exist in anyone. I think – you know some of the qualities I look for – we want people who are highly adaptable, curious, great communicators is key. And I was mentioning Deirdre previously. Deirdre used to be a writer – she taught writing at Boston College. So the ability to write, synthesize, distill and synthesize findings into a report that is digestible and spot on, that is very challenging and there are people who can do it well and I really strive to find those people for the team. So great writing skills and part of that writing is storytelling. So I’m not sure if you said to someone who do you think has better communication skills, men or women? It’s a combination of communication, analytical skills, research skills. I think it’s just coincidence that we have all women, but boy we have a terrific group. I mean I always say I’m loving this part of my career. I’m loving it because of the chemistry we have in the group and I just – you know sometimes when you’re in a job that’s frustrating you think well maybe someday and I finally found my someday. So it’s nice to know that it’s out there. Steve: I finally found my someday – seems like that’s a song they should play at somebody’s wedding. Kate: I know, but I finally found that environment and – but you know part of that is you have to build the culture that respects user research and you have to be – you know my boss is very supportive of our work and if he wasn’t we would go nowhere. I mean you have to be in the right place in the organization working with and for the right people and you have to hire well. You know we actually have an opening right now and we’ve decided that we’re going to just wait for the right person to find us because when you go out and you recruit the person you want isn’t necessarily available at that moment and you may not find that person and then you end up with a pool of six people. You narrow it down to three, and then two and then you end up choosing that person who maybe in a different world you wouldn’t have chosen. So we just have this fantasy that the right person will find us. Steve: Well and I think – you know your catch phrase is I finally found my someday, but you’re talking about building it. I think you built it. Kate: I built it and so when you – when I started at EBSCO it was – let’s see it was almost 5 years ago and so when I started at EBSCO user research really hadn’t emerged to be its own entity yet and I remember my husband – I came via the information architect path, up through user experience, and my husband would always say to me why don’t you focus on the research? It’s what you like the best. And at the time research was – there was a lot of usability testing, or at least my version of it and I just said I don’t that I could do that all the time, but he said but you love it and I said you’re right I do. So I took a job at EBSCO and my first job here I was focused on usability, but I was also helping to write requirements. Like I was a requirements analyst for these products and platforms. And I remember at the time – I conducted I think two usability tests and I realized there was this big need – there was a major need for someone to focus exclusively on usability and user research and previously this had fallen to multiple people. It was a dimension of many people’s jobs. And I went into my boss at the time, named Ron, and I said Ron I would like to focus just on the user research piece and he said well I can’t let you do that because I still need you to be able to write requirements if I need you to do that and I said okay I’ll make you a deal – if you let me do more usability and user research I will write requirements whenever you need me. And he said yes, ‘cuz you’re definitely not going to have enough work for just the usability piece. And I said okay, deal. So we had a deal and I never wrote another requirement because the demand, that was my year of yes. I said yes to every project that came my way and then I was in technology at the time and then I moved over to product management and things really took off because then I was embedded in the product development life cycle. Steve: Is there a point at which doing research sort of morphs into leading research? Kate: There is and that is how a bill becomes a law or how a girl becomes a woman. You know it’s really – it’s quite a transformation. It is. Like all of a sudden it’s – it’s growth, it’s evolution and what starts to happen – I remember you go from doing testing and doing research to talking about it, to evangelizing for it, to finding it a home within the organization. And that’s where the leadership engine kicks in. And I think one of the reasons I feel well supported at EBSCO in that people value the work of my team and when they promoted me to Vice President that was really exciting for me personally and also for the research in the organization to be given that level of visibility and that was exciting. And it’s, it’s – it means a lot to our customers that we have prioritized this as a function within our company and our organization and so when they buy our products, or they invest in a partnership with us, they understand that we are looking at our users 24/7. We have a relentless and a passionate focus on our users and customers. Steve: And you described those different milestones in that evolution and was becoming part of the product management organization a significant point along those different stages? Kate: Absolutely because products – this is where the products – this is where the products live and this is where they’re created and this is where they are evolved. And so when we were located in technology that was fantastic because we were where things were being built. But a lot of the upfront research needed to happen – it needed to happen sooner than we were getting it in technology. And so moving us over to product development was a real culture shift and it was an important one in our growth. And something in user experience, I constantly had that debate of should I be in – at the time at other companies it was should I be in marketing? As a UX person should I be in marketing or should I be in technology? And actually at the time I always opted for technology. And so it was a shift for me to be at EBSCO and say you know what I think we should be in product management and I think that’s the right place for us because my whole career I’d actually fought as a UX person to stay in technology. But at EBSCO it made more sense for us to be in product management. And now, again, we can achieve that alignment a lot better because we’re where those discussions are happening. Steve: Right. I want to follow-up on something else you said as well. You described these qualities that are just so essential, that you look for in researchers. How are those – how would someone make those qualities visible? Kate: That is a great question because how do you – how in an hour interview do you say oh well I’m curious, I’m a great communicator, I can distill down findings. I always say the great researchers are like Navy SEALs. They go in, they adapt, they accomplish their mission, they leave without a trace. Because you want to be efficient. You want to go in, you want to get the job done and you want to come out and you want to leave people kind of unruffled and let them go on with their lives. And so I look at communication skills before I even look at research skills. So for example you wouldn’t believe some of the cover letters we get or some of the emails and you know I have very – I actually have – I can’t say I have a low tolerance – I have no tolerance for things like bad writing and bad grammar and so that, because our work is published externally, that’s just a deal breaker for me. There’s a curiosity and a work ethic that we look for. I just wish you could sit in our team meetings and meet some of the researchers and – it’s funny my husband actually calls Deirdre, our lead researcher, he calls her The Oracle because she knows everything. I mean you can sit – you can sit her with a fifth grader to talk about research and she can make a connection and that yet she’s amazing with 60 year old esteemed department heads of you know medical or surgical departments when we’re doing physician research. So she can – she can do it all and my friends – I remember my friends and I, when we were all dating before any of us got married, we had this joke about we’re looking for the guy who’s tent to tuxedo. You know he can go and rough it in nature and then he knows which fork to use at the black tie dinner. And I think as researchers that’s what I appreciate. I appreciate the person who can kind of rough it in the wild and then be completely comfortable at a black tie dinner and everything in between because we research everybody and you have to be able to connect across that continuum. Steve: Can you talk a little about – we’ve talked about your team and sort of growing your team and a bit about the history. What can you say about kind of the make up of the team? Things that have changed or new ways you’ve found to expand the team? Kate: We actually just added a user research recruiter and this has changed my life in such a positive way. In fact one of our researchers, Deirdre, said to me have you ever had something amazing come into your life and you didn’t – and it solved a problem that you didn’t even really realize that you had. So we have a new research recruiter whose name is Kristen Arakelian and we brought her on for one purpose and that is to help us curate higher quality samples. And when I say higher quality participant samples I don’t mean okay every person on the panel has to be an Einstein. I mean sometimes we – you know we’re sending surveys to people who aren’t being responsive or we have these dead email addresses or we wanted medical nurses and we ended up with a surgical nurse. We just need to make sure that when we have a study ready to go we have our – we’re building out these participant panels and these participant – uh, these databases that are going to support our work and it has been incredible. And so now as researchers we get to focus on the research and she is doing all of this terrific work building our ground game, making sure she’s connecting. I mean here we are in Boston and we have so many universities and so many hospitals in our backyard and so she’s making these connections and she’s on social media and she is making lists and adding these people to our panels so when we have a research study and it’s ready to go we can actually launch it that day, or we can start it that day. That’s been a life changer. I wish I knew to do this sooner because my solution was always to add more researchers and there’s a lot of stress around recruiting. I used to say recruiting was our largest pain point. Pain point solved. It’s been wonderful. Steve: What do you think is her secret superpower? Or not-so-secret power that makes that work? Kate: What makes it work is I didn’t go looking for a user research recruiter. And in the instance of our recruiter Kristen had all the skills that I need someone who is polished. I need someone who is a great communicator and has those writing skills and understands how to kind of reach out to someone she’s never met before. And she has all the qualities that we need and she had no experience doing user research recruitment and I didn’t care because I knew she’d be right for the job. And I knew her prior to EBSCO and so I took her out for lunch and I said I have this opportunity to you and she said well you know do you want to give me a description and I said no because then you’ll think too much about it and I need you to come and start and see how you like it. And she did and she jumped right in and she’s made our – she is curating these samples for us that are yielding tremendous feedback and we don’t have the trial and error of okay we sent this survey to 20 people and only 6 responded and who knows someone who is a teacher in a K-12 school? You know we don’t have to go the friends and family route and we don’t have to bootstrap it anymore. We have a recruiting operation and it’s absolutely changed our lives for the better. Steve: That’s fabulous. Are there – you mentioned earlier that the things that you do – the work that you do is published in public and you go to – you give talks about it. Are there stories or recent successes or things that you can share with us? Kate: There are and one of our researchers named Lin Lin is located in Shanghai. She’s based in Shanghai, in China, and she conducted research on Chinese students and how they were similar or different then many of the US based students that we study on a frequent basis. I’m really impressed by her findings because Lin is this researcher – you send her out and she comes back with gold every single time. And it’s impressive because when you research students in China what’s missing from their research experience is Google because of government sanctions. And so what’s interesting is Chinese students have a similar approach in that they get very anxious about research at the outset, upon receiving the assignment. But research for Chinese students is episodic and each of those episodes of research it’s a combination of open web resources. So China doesn’t allow access to Google. They have something called Baidu instead. So there’s open web resources and there’s these episodes of research that happen. So over the three weeks of the assignment there’s these episodes of research and in each of the episodes it’s a mix of open web and scholarly resources which is through the library website and also consultation with the professor and perhaps a librarian is interspersed, is intermixed in there. That’s very different. The researching episodes is very different than what we’ve seen and what we’ve documented in US students. In US based students what we see is research in two microbursts. There’s a flurry of activity when the research assignment is made – is given. And the first microburst is that activity related to sizing and scoping. And so you’re on Google. You are looking for – you’re trying to understand your topic better and that Google search leads you to Wikipedia and there’s a lot of open web searching that’s happening. Then there’s a period of dormancy and we assumed that during that dormancy it’s this procrastination and there’s kind of shame and bad feeling about that because – and dread and oh-no. But no, there’s not. It’s about efficiency and students are – they’re locked and loaded because they did their open web. They understand alright Google and Wikipedia, like I have a grasp of my topic. And then there’s the second microburst and the second microburst of activity is right before it’s due and that’s when you go into the library website and you access the scholarly resources and you find those articles and you find those eBooks and you find those videos that are peer reviewed and scholarly and meet the criteria that your professor has set forth. So if you’re a librarian the reason that that information is valuable to you is because you understand that your opportunity to influence the student research process in the US is at the outset, is during that first microburst of activity. Is getting into the classroom with the professor and you know introducing the research topic and the scholarly resources that are available to them because most likely at the tail end of that, during that second microburst, the US based students are – it’s that flurry of activity related to finishing the project. It’s not about I’m going to go consult with the librarian per se. So we love having an international research team because we have the opportunity to do these cultural comparisons and they’re fascinating. Steve: It seems like part of – there’s many elements there. It seems like one element is that by looking at what the behavior is for a Chinese student it highlights behaviors in the American use case that you might not have thought to reflect on until you see how it plays out differently. Kate: It’s true and that’s the kind of research that helps us gain an understanding of students. You know students in China and students in the US, but it also helps us advocate for and help our – advocate for our customers and it empowers our customers to advocate for themselves. So that’s the kind of research that matters because our customers walk away and say great I can now evangelize to have myself as part of the assignment process or in the classroom because I’ve got this research that’s been documented. So we do a fair share of – we write the white paper or we’ll give a talk at library conferences or we’ll – we do a lot of traveling to our customers and talking about our findings and actually collaboratively conducting research with them. One of our favorite activities that we’ve been doing more of is to travel to customer sites and invite in three or four students, or three or four nurses or whatever product we’re looking at, to come in and demonstrate their search process live and we – it’s very interactive. Like people will say well did you know you could do this? Or what about this? And it brings that – again it’s the reality show of how users are interacting with different products and different services and it’s fascinating to watch and usually that exercise garners some great enthusiasm because it opens people’s eyes. And I will tell you something. I have been very – I have been really pleasantly surprised by the great questions that many students ask around participant privacy and around sharing of information. And you know I think many people think millennials are just posting their whole lives on Facebook and Instagram and they’re very knowledgeable about privacy and I’ve been very impressed. And we’ve had to adapt. We’ve had to make sure that we talk about how this information will be shared. We used to kind of ask casually like can we take a picture of you so as we’re thinking about these results, blah, blah, blah – sometimes we will ask that, but certainly not as much as we did because we’ve had a lot of students say no thanks, or I’m not on Facebook anymore and I – you know when they talk about social media it’s not about this mass sharing. It’s they’re really kind of curating their online profile, their social media presence and they’re not careless about it. They’re very thoughtful and they care about privacy and they read the small print, as they should. I’ve been very impressed with how seriously our participants take that and as a result we are very specific in our releases. And we deal with children so when we talk about what are the qualities you look for – certainly good judgment, ethical research practice is critical. It’s the most important thing at the top of – at the very, very top of our list because without that good judgment I’m putting way too much in your hands that you’re responsible for and if I can’t trust you there’s no job. Steve: Wow, that’s really good. Can you talk a little about your own background? Kate: My background is actually – so I studied sociology as an undergrad and I loved, loved, loved studying sociology and I loved the research element of sociology. And I went to graduate school for public health and public health – and epidemiology being part of public health is fascinating because you’re looking at what programs, what interventions have an impact on a population. So unlike medicine which is the study of – you know is curation – you’re about, it’s curative – public health and epidemiology is about prevention. And so you’re looking at behaviors and you’re looking at interventions and you’re tracking those things over time. So the research element is very heavy in that. So I like to say that chapter one of my career was in healthcare and my job understanding end user needs and building products that met end user needs, there was a health reporter for the Boston Globe named Betsy Lehman and she was a cancer patient at Dana Farber in Boston and because of a medical error she actually passed away because she got four times the dose of a medication that she should of and she did not survive that. And so this was heavily covered in the Boston area and my hiring was a part of the staffing up that happened following Betsy and one other person was severely injured as a result of a medication overdose. So what they did, the Partners Healthcare system decided every unit is going on this online order entry. And so I was part of the staffing up that happened following that incident and our job was to understand okay this online order entry, we’re replacing the pen and paper – the clipboard at the end of the bed – it’s different in the labor and delivery unit. It’s different in an emergency department. And so my job was to understand and then – understand the user needs and translate that into system requirements. And it was just an amazing job because we did nursing shifts so I would work 11 pm to 7 in the morning on a labor and delivery unit and see the workflow there and understand how that product needed to be customized. So that was really fantastic work in the healthcare field. And then I went into online travel for the second chapter of my career. And online travel is similar in its complexities because you’re thinking about booking cruises, booking vacations and the complexities involved in that. And so booking engine, it was all about conversion and getting people into the booking engine and helping them shop and compare. I was at TripAdvisor for a little while and primarily doing information architecture and usability work and that was – online travel is fascinating and to this day I think about some of the things I learned about you know why Kayak made some of the decisions they made about their filters and things like that. It’s, it’s – I still refer to some of those points because it’s a complex product and you winnow down these results by using filters and things like that. And then I was in the UX field like I said up through information architecture, into the broader UX and then I – I fell in love with user research and I carved a path – carved a path forward to do just that and it’s been incredibly satisfying. Steve: Is there something about you as a researcher, whether it’s maybe the product of those experiences or anything else? Like what’s – I think the Cruella de Vil anecdote is one – that’s sort of an evolution. I don’t know, anything else that would characterize – we talked about leadership a bit, but I’m thinking about doing the research. How do you – what’s… Kate: So doing the – yeah, you know I love research because I love hearing people’s stories and I only – it’s funny I only read non-fiction and so I’m just very curious. I love hearing – I love new experiences and I love hearing people’s stories and I also love distilling that down, finding – you know finding those nuggets. And I love the moment – you talk about this actually in your book, Steve, but you talk about getting from question/answer to question/story. That is such a satisfying moment when it happens in an interaction with a user. And I never get tired of that. And in fact every time – you know we can be working on a large ethnographic study and you think okay you know we’ve got 25 public library patrons or we’ve got 22 physicians we are going to be speaking with and I always walk out of every session – you know you’re drained and you’re exhausted, but it’s that feeling of you just – you just ran a marathon and it’s the greatest feeling in the world because people just shared their stories with you and you get to take those stories – you’re obligated – I mean that’s such a privilege to have someone share that information about how they interact, how they live their life. And then you get to come back and represent that through their words and through their stories to product developers and owners. I just think that’s the best work in the world. Steve: I love how you describe it and the “it’s hard” but you feel good after. Kate: And you know it is true and that’s why we joke – you know as a researcher you’re like a Navy SEAL, I mean minus the incredible – minus that torturous training process, but we have different training we go through, but you have to kind of – you need to be able to be dropped into any situation, adapt quickly, find your way, get your answers by making people – you know you have to make them comfortable and then you’ve got to go and you’ve got to go back and you’ve got to deliver what you said you were going to deliver. And so we work hard to find those – we work hard to excavate and to uncover what’s there. And I’ll tell you something, my family and I are in the process of moving now and as I think about it – I always say that moving is an analogy for these design excavations and these changes, these product evolutions, because when you move every single thing in your house you either have to – you have to hold it in your hands and you have to make a decision about it. Does it stay? Does it go? And when we think about changing products and moving – you know evolving platforms, like we have to look at every aspect. So the research is comprehensive. We have to look at every single piece of it. We have to understand users in all aspects of their lives and we have to make decisions about does it stay, does it go, does it change and how? Steve: So what else should I have asked you about? Kate: One question that we are asked often is are you guys the user experience group? Are you UX? And we say we’re user research. And I think user experience is a different configuration at every company. And we at EBSCO – we’ve tried a couple of different versions and the version that works now is that user experience, because of our diverse product set – you know we have corporate products, and medical products and school products and we have you know a digital magazine product called Flipster for the public library. All these products are so different and these services are different and so there is no universal user experience because the user is so different. So what we do is user experience is a responsibility that falls to the product managers and product directors within each of those product pods. But user research is central. So user experience is not centralized because of the differences in all of the products. But user research is centralized because I will tell you having done the research 60 year old physicians and 16 year old students they’re searches and their strategies for searching are not as different as you might believe there are. So there are similarities and when we learn about people’s process we can apply – we can extend some of those findings across different markets, I think more than you can when you’re thinking about a product experience. Steve: So is user experience part of the product management organization? Kate: User experience falls within product management meaning it’s a responsibility, it’s not a centralized group. It’s decentralized within the product pods. And I think over time, you know these models with evolve, but that’s where we are now because we’ve tried on a couple of occasions to have it a little more central and it just didn’t take off as it should. So I think over time we will continue to revisit that, but right now user experience falls as a responsibility, as a product experience within the product pods. Steve: I think you know these conversations are – they often cover a key point that you made which is like here’s where it works for us right now and these good things that are evolving, but I like how you’re talking about – you know there’s a responsibility, there’s an action, there’s a task and then there’s a team or a job title or a department and that I think sometimes we conflate the two of them when we talk about who does what, how is it structured? There’s activities that have to take place and then there’s well who does those activities and what do we call them? Kate: Yeah and there’s a sense of – I think the challenge in centralizing UX is you’re trying to be everything to everyone and that gets challenging and then you’re not as close to the products as you might be. But when we’re – in user research when we conduct research we have the product managers with us. I mean they’re the co – they’re working on that research with us. They’re sitting on the floor of that hotel room in Las Vegas where the student is conducting research and they’re watching it right along with us which actually brings up a great point about user research. When you are doing user research in the wild be safe. Make sure you build in safety measures for your team. I mean we have to meet strangers and so be safe about that and that’s something that we talk about and it’s something we accommodate for. Because we’re going to our users so we don’t go alone. And we make sure that we’re safe about it because I’ll tell you something – even – I had moments where conducting usability tests, you know you’re conducting these sometimes after work and it’s dark and then the usability test staff at the testing center has gone home and you’re in this testing studio and these participants you’ve never met before are coming in. That’s something – I want to write some form of a blog post or an article about that because it’s something that I don’t hear talked about a lot, but safety in conducting user research should be paramount for all of us. Steve: One of the things that I’ve struggled with and I think I wrote about this in interviewing users is the difference between being unsafe and feeling uncomfortable and that – I mean my suspicion is if you don’t kind of check yourself it’s easy to think that one is the other. Kate: Agreed. Steve: And not to minimize being unsafe in any way, but there’s feelings of being out of control. There’s feelings of just being out of your comfort zone. If you’re not comfortable being uncomfortable you might not know the difference between being uncomfortable and being unsafe. Kate: Yeah, and as I said part of research is new experiences, new people, environments that are not your own, not your own native environment and so certainly if you’re someone who’s easily thrown off your game user research might not be for you because you’re put in all sorts of circumstances and people will say the unexpected. There have been – I mean we always talk about this. You know you’ll get asked for a job – I’ve had researchers who’ve been asked for a date. I mean it’s really – I mean the unexpected does happen – does and will happen. Steve: Yeah, I’ve wondered a lot about the experience of being a woman in user research and the experience of being a man and that if there’s an aspect to that that like men just couldn’t possibly believe. And maybe this is about being a woman in our society also that men couldn’t possibly understand, but – that example of being asked for a date seems like oh yeah, that happens. Kate: Sure. And you know when – I’ll be honest with you, when we were in Las Vegas doing the interview I had recruited the student I was meeting with off of Craigslist and so I was meeting a stranger that we’d recruited off Craigslist for a research interview when I was out at a library conference in Las Vegas and I asked my colleague Dave to come with us and Dave is very fit and very muscular and I asked because he was part of the project, but I also asked him because he’s strong. And I was – I had recruited someone off Craigslist and that was deal with my husband. He said I’m nervous about this. I said well I’m going to bring this guy Dave and he said okay, well I feel a little better about this. But you have to think about safety and in that situation I wasn’t – I wasn’t protecting against feeling uncomfortable. I was protecting against you know all the crazy stories you’ve heard about Craigslist I guess. Steve: Is – yeah, is being asked for a date an unsafe experience? I guess it depends on how, but – I’m not even sure how to ask this question, but sort of separate from the Craigslist experience, how does – how do women in the field deal with the set of things that may – the type of content or conversations that might come up with – how do we even frame that? Kate: I know. It’s a great question and think about it. Think about part of the Cruella de Vil, like serving it up cold, no one’s asking Cruella de Vil on date. Do you know what I mean? I mean she’s like the ice queen, so who is going to ask this icy researcher who’s not even giving a sense of like am I doing the right thing on this, am I passing this test on the website. So part of being comfortable with people is being comfortable but still having boundaries. Still being the researcher. And part of it is not being alone in a circumstance that would make you uncomfortable, or make a conversation like that too convenient. And you know I think personally I can sense when we might be getting off topic, or we might – or the conversation might be getting too comfortable for the participant and so then I just move it forward. But I think this is an area where some sharing of these thoughts and experiences would help the larger research community to think about these issues and hopefully not have to be in a situation where they’re uncomfortable or unprepared. Steve: I’m going to pitch hard for the war stories – having people talk about for the last few years. With the aim – I mean there’s many more circumstances that come up than just the one that we’re talking about, but my goal is yours. To hear these stories starts to, you know let people if nothing else be aware that they can – these are the kinds of things that can happen and so, you know, there doesn’t seem to be one size fits all right way to handle it, but to – like you said the Navy SEALs have to adapt. Kate: You have to adapt and you have to adapt and you have to know your boundaries and you have to be comfortable with uncomfortable conversations. And so I was just having this conversation with my husband the other night about being comfortable with conflict and we talked about that as a trait of good management and I said you know as a researcher you have to be because sometimes you’re putting a product or something in front of a user and it may evoke some emotion, they may be upset about it. They may make that personal against you. So as a researcher it’s a requirement. You have to be comfortable with some level of conflict because not everything is smooth sailing in these research sessions. Steve: And so conflict could simply be asserting boundaries and it may feel – if you are someone that’s kind of – that doesn’t – that is trying to avoid conflict, drawing a boundary which could be – I liked your example before of just moving things along. That’s asserting a boundary and that may feel like conflict to someone. Conflict is let’s get out of here – I’m going to punch you in the face – it doesn’t have to be dramatic. Kate: No, and the conflict – right, the conflict could be – the conflict or the confrontation could be as simple as you’re not being responsive to a direction they’re taking the conversation in and you know I think as we move away from methods like the focus group – you know the in-person focus group where you have – that’s not a method that we use often because I think there’s so many challenges with it. So you’re moving to these individual – you know you’re playing Charlie Rose and someone else is sitting across from you and you’re understanding their lives, right. You’re kind of moderating the narrative of your user’s, your participant’s life and their experience. And as we move to these one on one, the deeper exploration, it gets more personal and the stakes are higher and being boundaried is important. It’s important as a researcher. It’s important as a person. It’s important for the participant to understand the parameters of the session. Steve: And that you as the researcher have permission to continue to reinforce those boundaries. Kate: As researchers one of our challenges is trying to figure out what should our temperature be and the temperature being about that level of interaction with, empathy with our participants, our users. And you know you don’t want to have yourself dialed all the way down into the freezing zone where you’re disconnected from the user and you’ve got the Cruella de Vil phenomenon happening. But you also don’t want to be so over the top connected and warm that it becomes like a friendship conversation and you have difficulty stepping back and still having those boundaries as the researcher. So this question of temperature, this question of you know how accessible are you, we haven’t solved it yet, but if I had to look at – if I had to look at a temperature scale I’d say it’s in the – the medium to warm is the ideal. The approachable, medium, warm temperature. Steve: If I can pile on your metaphor a little bit, there’s your internal temperature and there’s your external temperature. And so you know I mean I’ve interviewed people about sad things. I imagine you have as well. Kate: Yes. Steve: Talked to people about situations that are sad. There’s a difference to me between how you feel and then what – and then how you do or don’t exhibit that feeling. Kate: That’s a great point because part of our research has been focused on physicians and nurses and you know because EBSCO has strong presence in the medical market. So when we did a large study of physicians, what we ended up doing was talking to physicians in all different specialties and one particular physician stands out in my mind because she – she saw patients she – she delivered end of life care and her stories were unbelievably touching and several were very sad. And it was hard not to cry because some of these stories were really quite sad and I think what she saw – so inside, inside you’re really feeling the pain of what she’s describing, about families saying goodbye to other family members at the end of life, but to the outward – to the participant, to this physician, this really lovely, just wonderful, compassionate human being , to her I think the researchers, we just looked somber. You know we looked like she had told us something serious, which she had, and we looked somber and we nodded and we were respectful of that. But I don’t think she needed us to cry to know that we had feelings. I think the way we handled it inside, I think we had stronger feelings that we’d led on, but we treated the information soberly and somberly because it was – you know it was sad. It was sad. And that’s part of uncovering people’s truths that is so incredibly rewarding. So if I go into a session like that, and you know I use that approach that was recommended to me years ago, the like serve it up cold, I don’t deserve the privilege of being in that room. I don’t deserve the true honor of speaking with these participants who are helping to evolve our products and you know show us their lives. You don’t deserve the job of a researcher if you’re not truly empathetic to your user. Steve: Sure. I think what that story illustrates, that seems very important, is that you had the feelings in the experience with the end of life professionals. You had the feelings inside. You acknowledged for yourself that you were having those feelings, but you chose in the context of research how to act on or how to express those feelings, and that you made a different choice that you would make in a social situation – you know anything that’s outside the context of doing research – you chose how to react differently than you would. Kate: Yes, and that is key because the participant – the person that you’re speaking with needs to feel like the researcher is in control. So if someone starts sobbing uncontrollably all of a sudden the physician might be worried about the researcher and then it creates this shift in the dynamic that is hard to recover from. And so it’s the same as if someone tells you a joke or says something funny. If you laugh so hard that you can’t regain your composure it shifts the dynamics of the interaction. So it’s – that’s where having the boundaries – and it’s not so different than being a teacher or being a parent that you know your kid says something silly, or does something, and you think okay well do I want to reinforce – do I want – how do I need to react in this moment to convey the message that okay we’re going to continue – we’re still going forward or we’re still going to have a nap even though you just said this thing that’s really silly and we could laugh about it for 2 hours. So you have to keep the research – you have to keep the research moving forward, but you have to convey that when someone – you have to convey that you’re listening and you’re processing what they’re saying. So when someone is telling you a very sad story, when you’re somber and when you are listening intently and making eye contact that says I’m here with you, I hear what you’re saying and it’s registering and I understand that it’s serious. I get it. Steve: If you – I think it’s different with a professional, but if you – I mean this has probably happened to all of us. We’ve been someone who’s had something difficult happen to us and we talk to other people about it and they say oh, I don’t know what to say or something and that we end up in the role of trying to help them. Kate: Yeah, we end up in the comforter role. And keep – you know something I always say to other researchers is you have to be very clear that the research session is a judgment free zone because one of the challenges we have is when you’re researching and interviewing students sometimes they fear that there’s a right way to do what you’re asking them to describe and they don’t know what it is. So you don’t want people to feel that they have to say well I’m sure there’s a better way to do it, but this is how I do it. You want to be – you want them to feel comfortable and you know show us how you do X, instead of – and you don’t want them to be apologetic and you don’t want them to feel that what they’re showing you isn’t good enough. And often during those – I mentioned those live student demonstrations that we do with some customers that are so enlightening. We start of and we talk about you know your act of conducting research – your act of information seeking is uniquely you. We want to learn about it. There’s no right, there’s no wrong, whatever works for you is what we want you to show us. So we really spend a lot of time on that at the beginning of those sessions because we have a room full of librarians. We have a room full of library staff watching someone conduct research. And so we don’t want them to feel intimidated. Steve: I had a client that I worked with that said to me that sympathizing is judging. And we think of judging as sort of the right or wrong, but she was saying you know if you say oh that’s awful – I’m not going to do justice to how she put it, but it’s taking a value framework that is yours and putting it on their story. Kate: Right. You’re handing your world view to someone and saying oh it’s okay. Or you’re making a judgment and there’s no place for that. And so we hear it all. I mean that’s why we talk about you know anything can happen in these research sessions. We hear it all and we hear – you know we hear students talk about how they conduct scholarly research and sometimes the techniques they’re talking about aren’t necessarily scholarly. You know if they’re talking about looking on Wikipedia which isn’t a peer review…which is not a citable source – and so we don’t stop them and say well did you know, because we’re not there to fix anything. We’re not there to teach. This is not a teaching moment. This is understanding and this is actually exploration and one of the things we try to do in our group is we try not to say – when people come to us and say okay we want to validate – we don’t go out to validate. We go out to explore and in that process of exploration and excavation, you know we’re lucky enough to unearth findings, but we do so because we’re listening and not putting our own framework on the participants or the users. Steve: Well Kate, let’s wrap it up there. It’s been a really excellent experience to talk with you and thanks so much for everything that you shared with us today. Kate: Steve, thank you and I look forward to seeing you in Boston someday soon. Steve: I look forward to that too. Thanks. Kate: Okay, take care.
April 12, 2016
This episode features Pree Kolari, the Senior Director of Design Strategy and Research at eBay. We talk about the career arc of a researcher, having impact on the product, and breaking down organizational walls. We can make it a little better in terms of customer experience where they buy with one less click. We can also change the dollar amount eBay makes by .1%. That’s a win and it’s very easy and it is very, very tempting for everybody on the team to go “Yes that’s great, let’s go do it!” But then you ask the why questions and go for deeper insights, not just the surface level insights. You go deeper. You find out that it’s a lot more and you will have small things that you could do on the product that will give you wins two years from now. – Pree Kolari Show Links Pree Kolari Follow Pree on Twitter eBay eBay Classifieds StubHub eBay Motors eBay Fashion local maximum stack rank Moto X The Design Sprint Kinect Zune Metro Karmann Ghia A Pattern Language by Christopher Alexander The Designer is Present Product Management + User Experience Follow Dollars to Donuts on Twitter (and Stitcher) and take a moment to leave a review on iTunes. Transcript Steve Portigal: Thanks, Pree for being a guest on Dollars to Donuts. Pree Kolari: Absolutely. Looking forward to talking. Steve: Let’s do the basic intro part of the interview. Tell me something about you and your work and kind of why we’re here to talk. Pree: Sure. I’m Pree Kolari and I lead the Design Research team at eBay. It’s actually Design Research and Strategy team that I lead at eBay. And we’re a pretty sizable team, about 20 people across the US and Europe. So we have people in San Jose, San Francisco, Portland, Germany and London. So pretty sizable team. We work across all of the products on eBay. So we work within of – we work with the product teams and work on both new products as well as existing products. Steve: So if I’m not an industry insider and I’m listening to this and I think eBay – eBay has multiple products? Like can you explain you know eBay a little bit for those of us that aren’t in the know. Pree: Sure, absolutely. So eBay is interesting because you have a couple of different kinds of users. One is the seller and then another one is a buyer. So both of them see different sides of eBay. And they both interact with eBay via desktop, mobile, you know multiple devices as well. So every one of those interaction spaces are product areas from our standpoint. And so that’s one. Then eBay as a company plays in many, many different spaces. One is eBay the marketplace itself. Then there is a bunch of other places which is Classified, StubHub and all of those things. Those are separate teams by itself and generally speaking we don’t necessarily work across as much as we should, but we do. So that’s – does that that give you… Steve: So where does something like eBay Motors fit into that? Pree: Absolutely. Steve: Is that like a StubHub? Pree: It is – no, no, no eBay Motors is part of our team. Those are all things that are considered verticals. Good point. So within of – the shopping side of things – there are certain areas that have specific verticals and eBay Motors, or Fashion, you know those are the spaces that are a part of our team. Steve: If you’re an eBay shopper do you experience those things – if you just go in and search do you know when you move between the verticals? Pree: Ideally if we did our job well from a design perspective, as well as from a research perspective, you shouldn’t notice it. But sometimes we do, you know because it is two different teams that are delivering the same product, right, and so that’s one of the big challenges of doing good research because as a researcher you’re thinking end to end. You’re thinking the customer journey. You’re coming at it from a user standpoint and users don’t see your org structure. They see the product. The moment the users see our org structure we have a problem and that’s exactly what we shouldn’t be doing. Steve: So it sounds like the experience today for someone, there’s lots of points where you wouldn’t see the structure and some points where you would. Pree: Absolutely, absolutely. Lots of places where you shouldn’t see it, but some places you see it and that’s kind of why a bunch of us have jobs. Steve: To try to… Pree: Try to fix it. Yeah? Yeah, it is about fixing. Some of it is about fixing. Some of it is about like coming up with new things that will make the older behavior that we have started kind of become back – you know take a back seat. Steve: It makes me think about what – you know how research fits in, which I think we could probably spend a long time talking about – I mean I think sort of the counterexample is always the – you know there’s a question that comes from a certain team and they’re really only interested in the answers that affect their decisions within that. Pree: That’s right. Steve: Whereas, as you say, as a researcher you’re looking kind of end to end. Pree: Yes. Steve: So the question is you know what – you know how does the work that you do – how do you structure it to touch different teams at different points so that those decision can be made. Pree: It’s a good question. So one of the things that our team, the whole team does is talks to our counterparts, our colleagues or people who are part of our core teams. Yeah? And the main point there is understanding the goals of what we are trying to achieve. The secondary part there is to train people not to look for local maxima. We have – we have a habit – and this is just a bad habit that’s come over time, of looking for small wins, local maximum. Instead, how can we focus on long term thinking? So there is a little bit of bringing people along that happens. And during that time the framework that we end up using is just a simple one of generative, iterative, evaluative. So here’s the different kinds of spaces that we can look at research. The first part is generative – that’s about generating ideas. That’s about diverging in from an ideation standpoint. Next one is iterative – this is where you’re continuously testing things out to learn quickly and make this product much, much better. And then there is evaluative. Now we’re pretty close to ship. How do we make sure that we ship something really amazing? And so that’s the three step kind of a thing. If you look at a waterfall process, you look at an agile process, pretty much those things fit right in. Yeah. Steve: Can you say a little more about local maxima? That’s a great term that comes from say like some kind of number theory something. Pree: It’s a statistics kind of a – it comes from statistics. It’s used a lot in the stock broker side of things, right, and because you’re always – there are people who are there for the short term, they just are day traders that just made money. They want to sell it off and then they want to move on to the next one. So if you look at today’s stock you probably see an up or a down and then the up could be a good up and it could be the maximum up that the stock has ever gone, or you could step back a lot and see that the trend has been going up for years – 25 years or 30 years. And then you suddenly think, okay you know what if I hold on to the stock it’s going to do very well. So now you look into a product or a research side of things. You start to see that we’ve made – we’ve taken a certain path. Let’s say with eBay we’ve taken a certain path with respect to how we make a buyer go through the categories, go through the shopping experience. And we can fix it a little bit. We can make it a little better in terms of customer experience where they feel like hey you know I got to buy with one less click. We can also change the dollar amount by which eBay makes out of this about .1%. Now that’s a win and it’s very easy and it is very, very tempting for everybody on the team, including us researchers, to go yes that’s great, let’s go do it. But then when you ask the why questions and then you ask – you go for deeper insights, not just the surface level insights – you can see patterns, but you have to see the insights. You go deeper. You start to find out that it’s a lot more and you will have small things that you could do on the product that is – that will give you wins two years from now. That’s what I mean by local maxima. Steve: So I’m thinking about – we won’t get into any specifics, we’re talking about this very generally – I think about things like you might have other kinds of maxima like how often they’ll come back? Pree: Right. Steve: But then you might have sort of larger issues like the meaning. Like if you want to change this site – and I’m really just making this up – you want to change this from a transactional partner to something that becomes a support system. Pree: That’s right. Steve: Well now that’s about longer – that’s about shifting the overall meaning through a lot of design choices and you have to measure that very differently than did the amount in the cart go up. Is this what we’re talking about? Pree: You’re absolutely right. It’s not just looking at a financial benefit that you can quickly calculate now. It’s about looking – it’s about taking some amount of risk which are long term financial gains because it will come back, you know, and the team will start to have an intuition of the fact that it will come back. And there is a certain – you know we’re a profit making company, right, and at the end of the day we want to make profit every day, yeah. And so it’s not about just putting – taking all of our eggs and putting it for long term wins. It’s about being very intention and saying hey you know here’s a few things that we’re going to do where it is high risk/high reward – here’s a few things we’re going to do which is absolutely beneficial and we can see the benefits right away. Steve: You’re describing a strategy. Pree: Basically. Steve: Or a road map. Pree: That’s right. Steve: Like here’s where we want to go with this. Pree: I am. I am describing a road map and a strategy. Steve: And you said right at the top that strategy is in the title of your group and you’re talking about research as a strategic tool… Pree: Absolutely, right Steve: …very much. Pree: Right, right, right. So as in people on the team come from different kinds of backgrounds. There are people who have no research degrees, if you may. There are people with MBA degrees, right, and that’s intention. When you put these kinds of – different kinds of people together there is – there is a radical collaboration if you may. And that kind of collaboration happens because people are coming at it from different mindsets, but they respect each other’s mindsets. Yeah. And when you start seeing that happen other people also join in and I want the engineers, I want the designers, I want the product managers joining in and pitching in and making it their own idea. And that’s how you take it to ship. Steve: So in this – this maxima example I think is a really great one. We’re sort of avoiding the pitfall of a local maxima I guess maybe is the theme here. What kinds of conversations are you having with people and how are you helping them – I’m assuming you are helping them to see that as a choice or a different path. Pree: Yeah, good question. So you have – there is – this is where it’s a little bit of an art more than a science. And it’s a little bit of wisdom more than here’s 5 things you can do. What I look at is I think of – you know I don’t do as much – personally don’t do as much research as I used to do out in the field. I do very, very little. But what I end up doing is I end up doing research on – in the corporate side of things if you may. It’s about understanding the organizational psychology. It’s about understanding what are the needs, what are the pain points, what are the motivations of these people who are working inside of an organization. And once you understand that and you map it you can think about like why is this person thinking this way and you can still figure out that there is a motivation that works with what you’re trying to communicate and it becomes very easy then. And so I am always looking for win/win situations. It’s a negotiation at the end of the day too. So it’s a win/ win situation. Let’s figure this out and make it happen. So I don’t think I have a specific answer there, but more of a generic answer. Steve: You make me think about my own work. I’m someone that works outside the organization. So I come in and the best engagements for me are ones where I’m partnering with somebody who has that insider knowledge and has sort of worked over time to try to find out how to be successful so that I’m not blundering it and saying I know the best way to do it. It always has to be adapted to the context. And there’s often this interesting tension – maybe tension in a good way – when you come in – and you’re newer in this role so you probably have a perspective on this. When you come in you see everything that’s wrong that needs to be changed. And then when you spend time you choose your battles. You figure out your strategy. What I see, because I’m always in the newer side of things, is you also get – I don’t want to say co-opted, it’s too strong a word – but you become part of the system. And so I have these situations where I feel a deep passion, like this is what these people need and people that are my gatekeepers are like yeah there’s maybe not an appetite for that. And it’s not to say that we then fall flat, it’s just – it’s where we put our energy. So I love what – I mean I think it’s – I love what you’re able to do. You have to be embedded. You have to build relationships. You have to kind of go over time. Without being kind of confrontational to you, do you think, have you seen it in yourself, that you’re now part of the system – you can in to sort of drive change, but then you become part of the system. What does that arc look like? Pree: Very good point. So I’ve done like consulting work like you have for a long time. And then I moved into the corporate side of things – 2008. So been doing this for 8 years. Big change. Big difference. The advantages and good things that corporate or the consulting side comes with is they come with a fresh perspective. And you come – like when you come and work with us you come with a fresh perspective that we never would have thought of. You also come with an expertise. Yeah. So the hierarchy that has already been established within the corporation doesn’t apply to this consultant who has just come in. And so that’s an awesome thing. Yeah. However, these – the guys who come in also do not know the culture of the organization, or do not know the history of the organization. Now within the corporate structure there is researchers inside the corporate who are embedded and they work really, really hard to build the relationships to make sure that they are doing the right thing. And sometimes yes, getting embedded for too long will make them feel like hey you know I’m part of it and I cannot make any changes. But we end up, we end up really working hard not to let that happen and to make that we go back to – like within our team we have a – we build together the team with a set of principles and these are principles that everybody in the team worked together and we just kind of articulated. One biggest principle that we have is about impact and you know we write down like hey what does impact mean? The other one is about transparency. And it’s about transparency with respect to communication. Yeah. And as soon as we end up being extremely transparent we break down walls that otherwise exist within the corporation. Yeah. And it’s part of our jobs to break those walls. Yeah. And we – within my team we are not measured by how many reports you have written or whether you have finished so many projects. You are measured by the impact on the product. Yeah. So it is super important to work with the product manager. It’s super important to work with the business person. It’s super important to work with the designer to make sure that the impact is clear. And that’s when it changes our own behavior a little bit to not conform to the culture in the way you described it. Does that make sense? Steve: It does. I feel a little skeptical and that’s not a skepticism towards you. It’s just in general sort of having observed this because in order to be successful there’s a certain amount of conforming that has to happen. Pree: Absolutely. Steve: It doesn’t mean that – you know what I mean, you have to kind of conform to figure out how to choose the battles. Pree: Yup. Steve: And there’s battles that you’re not going to choose now because you’ve found ways to be successful. And think about yourself in 2007 – I’m going to guess, you would come in and look at yourself doing this job and like shine the flashlight in some corner and say like guys you’re denying this. And now you, 2016 you is kind of saying like well no you know we have these principles and we’ve had impact and we have this track record and like we have lots and lots of success and then – you know I’m doing a little puppet show here – 2007 you is kind of saying like no, no, no. Like over here, this thing. That’s the tension that I think is… Pree: Absolutely. So I agree with you. So if I look back in the 2008 me I would look at the 2016 me and say hey look, you’re not listening to me. You’re not listening to this part of the equation. And that’s the reason why I think – you know it’s learning, right, from a life perspective. Absolutely, so I prioritize certain things to make an impact on? I do. And everybody on my team does. And we do it because we believe in the fact that making impact is more important than making impact on all the 10 things. So let’s imagine there is a study where we saw 10 different things that are very important things to fix, okay – an evaluative study. We might choose to pick only two of the top things and some of it might be confrontational, but we might pick only two of the top things because we know that if we pick those two and go really, really hard it will help us instead of diluting our impact with going with 10 different things. Yeah. Steve: Um-hmm. Pree: And that’s the choices we end up making, but the choices are super important because it’s exactly based on impact. Doing the right research means talking about all the ten because those are all observed and those were patterns that were observed. At the end of the day we are not doing research, we’re doing applied research, you know, and we’re working within a product company that ships products. You cannot ship a research report. You can ship a product. So that’s what we’re going after. Steve: Yeah, that picking the two of the ten I think is really… Pree: It’s an art. It’s hard. Steve: …really interesting. And it’s not like you haven’t learned those other 8 or haven’t articulated them. I don’t know do those float around kind of in the ecosystem somewhat? Pree: They do, right. As in they might come back again and this is where – so I joined the eBay team only less than a year ago actually and our team is quite amazing. Like have been doing amazing work for quite a few years, since ’99. The research team has been there and have been doing like quite, quite a lot of good work. There were challenges in how this work was heard within the organization. There were challenges with respect to resonance on some of the outputs. And so we had to stack rank everything. So this is the engineer from me talking here. I’m all about stack ranking, drawing a cut line and saying hey, here’s the few things we’re going to go after and we’re going to go after really hard. Steve: Is that a fancier way of prioritizing? Pree: Basically. Steve: Was that what you’re saying or is there like a methodology? Pree: It is prioritizing. Steve: Okay. That’s a great phrase though. Pree: Yeah – stack ranking? Steve: Yeah. Pree: Yeah, it is. It’s probably engineering talk, but yes. I would rather you know – because at the end of the day we are – again, concentrating on shipping, yeah. And so again I think that those are things that are coming from my background as an engineer. As a person who has like shipped multiple products. As an engineer, as a start-up person. As like – you know – coming back to if we did amazing research and there isn’t a follow through afterwards the amazing research just sat on shelf and nobody benefited from it. Because I know that in my heart, and I know that a lot of people who do this, we’re doing it not to just do research. We do it because we enjoy seeing the output of this which is the experiences that have changed. And that’s what gives us satisfaction and that’s the long term view. Steve: Even just the idea of sort of the output of – these are significant efforts. A “research project”, whatever it looks like, it’s a significant effort, right? Pree: Right. Steve: And that even to get what you learned down to 10 things or 10 big ideas or 10 opportunities that takes a lot of pain. You have to leave a lot of things on the floor and synthesize and organize. And then – I’m just sort of toying with this idea you’re describing that says okay team we’ve kind of got these 10 things, let’s go at them with two. It’s very different than – and I’m sort of thinking about the consultant cycle which is, you know, it’s hey the consultants are back, here’s their thing, it’s a significant effort. It sort of ends the engagement. We have to impart as much as possible because we’re going to leave. So to get it down to 10 is really hard, but it’s like I think about in certain circumstances like what would happen if we said okay there’s two things. Yes we have a lovely Executive Summary that says like here it is on one slide, but just to really shout loud about two things. Pree: Sure. Actually let me give you an example, okay, which is from my previous work at Motorola. So here’s the interesting thing that happened. So Google had just acquired Motorola and that’s when I joined Motorola as part of the Google, when they acquired, around that time. And one of our – one of the challenges that we got from – during that time it was like a bunch of the Google executives who joined Motorola and it was clear there was a mandate, said let’s reduce the number of phones that we’re making to very few and whatever we are making let’s make something amazing. So we did a bunch of ethnographies where we saw people taking pictures. That’s what we were looking after. We were looking at images and people taking pictures. And one thing we observed over and over again was how hard it was for people to quickly capture an image. And what all things people were doing. And this is where – you know you’ve seen this in the field. Mom holding a kid and trying to take a picture and a selfie – there’s a diaper bag in another hand. It is, it is tough, And how do you allow for that to happen? And we could have written reports and reports about this, but we went in with one phrase. We said, you know, pocket to picture in 4 seconds and one hand. That was our output from a research study that took 2 months if you may. Steve: And you learned, just to – you learned lots of things… Pree: We learned lots of things. Steve: Tons of improvements that could be made… Pree: Absolutely. Steve: Lots of opportunities for fixing, creating new. Pree: Absolutely. And when we know that we want to get from a pocket to picture this much time, okay, and one handed. Now you suddenly, suddenly have, have something that the whole organization can rally around. There is a purpose. It’s a measurable goal. And what happened was with the MotoX we had a gyro – every phone has, you know has, has a sensor in it. We put in a low power sensor as well and people could just twist twice and with one hand take a picture. We even changed the UI. In any big company you could think about like how difficult it is to remove the interface. And we removed the interface to make the whole screen a tough target and you could take a picture. And suddenly the user experience completely changed. And so could we have gone with like hey the pictures that were taken in a dark bar were too dark? Sure. We had a whole bunch of things, right, but we didn’t. We made a choice and we made multiple choices afterwards and so the quality of the picture came up after and we worked very hard to change the quality of the picture and we have specific things about how you can change tones based on the picture, like a skin tone vs. food vs. the sky and make the perception of that picture to be higher quality. And so what we’re talking about is very data driven design. And so when you have data and you’re very specific about how we can use the data to change the design it always works. Steve: And I think it’s interesting that you research output, if you will, was sort of a vision statement… Pree: That’s right. Steve: …you know for a design brief. And that’s different than personas or journey maps. It’s not about the format of the output. It’s not even “Here’s what we learned about people.” You’re kind taking it all the way to say that we did this research and it tells us that this is what the product needs to deliver. Pree: Yeah. It’s an objective. It’s, yeah it’s an objective. Steve: It seems like there’s something interesting about how teams are structured, or how these programs kind of run, that lets a research project output a product decision. As opposed to researchers learned this, they talked to designers, they ran a session, the designers decided – you’re sort of putting it all under the same umbrella. We did this activity, the thing that we learned is the product should do “x”. Pree: Yup. Steve: So how do you structure – how do you bring the right players together and sort of have the whole endeavor such that that can be the output? Pree: Everybody who’s on the team is assigned to a product area. So they have a core team that they work with and it works very well in an agile kind of an environment in eBay because otherwise you’re a service organization within – of a big company. We’re not a service organization within a company. We are a product organization. So what that means is let’s imagine a person on the shopping experience. They’re part of the shopping experience team and are embedded within the team and they work with the team everyday on daily stand-ups. Yeah. And they figure out, they have – they maintain a roadmap of what research has to be done. They bring up things within the team and talk about what are hypothesis, what are myths, what are things that we know, what are data, what are insights that we know, that we really need to go after. And so all of those things are super important coming from the researcher’s standpoint. Then the next step is about participation within the research. So I’m a firm believer that research isn’t done only by researchers. Everybody is a researcher. It’s like everybody is an artist. Like you know we all have this innate ability to observe and make conclusions. That’s part of what we do. Some people just have been trained in this and are probably much better at the structure side of it. Now you have –when you have researchers who are catalysts within the core group, to understand more about the user so that we can make products that are absolutely relevant to the need of the user. The need may be even unknown. You know it might be a blind spot. They might not even know that they need it. Fine. But building that sensitivity for the whole team is part of the researcher’s job and then it is also part of the researcher’s job to collaborate, to get to an output. When I gave you example of phrase that came in the end it is not the researcher who comes up with that kind of a thing. It happens because the designer, the engineer, the PM is all part of that discussion. They own it. They co-own it. And that’s what – that’s when you get successful products. Steve: That’s a good explanation of sort of how you set up those conditions to make those kinds of big impact. I mean it’s back to your point of impact earlier on. If you can come up with that phrase as your output you’re setting yourself up for impact in a really… Pree: Absolutely. Steve: …massive way. Pree: And that goes a little bit – you know with research we end up – we end up thinking a lot about outputs from only one research report or one research study. That’s I think history. I really believe that it’s about building criteria, putting together principals that kind of collect all the knowledge that we know from multiple studies. And that’s part of the job of an in-house researcher, right, which is different from somebody who is coming at it from a consulting standpoint. And what that does is it helps make good decisions throughout the product development process. We’re making – you know I really believe that the intuitive decision making is bullshit. It’s either data driven decision making or principle based decision making. And so we have to give the ability for people to make the right decisions and that’s – you know it’s getting the data and putting together principles that help us make the decisions. Does that answer your question? Steve: Yeah. So to try to tie together a couple of things that you’ve mentioned. This idea of everyone is a researcher and then different decision making styles, I’m curious about, you know there’s people that are researchers for a reason. They have kind of an approach. Can you compare and contrast maybe the mindset of someone that has researcher as their title versus. these other people on the team that they’re working with? Pree: Sure. So I think one of the big things as a researcher, you know we are curious. And the curiosity is what makes a good researcher. The hunger – it’s a sponge. It’s like wanting to learn. And we’re paid to be unbiased. Versus let’s talk about a product manager or let’s talk about a designer. They’re paid to be opinionated. And having opinions and having ideas of where they want to go is what they’re paid for. And now there is a tension. And both people are coming at it from different angles. Both want to understand what the users want, yeah, because we want to finally make a product for our users, but are coming at it from two different angles. And so that tension is what I think is the root of the issue and then like figuring out how to manage that tension – it’s never going to go away. It’s managing it and directing it in the right direction. Steve: And I think that tension – as you described sort of how that team, your team, came up with – now I can’t get it right – what was the – from pocket to photo in… Pree: It doesn’t matter. Steve: …four seconds. Whatever. So that is a product of those different perspectives, right. Pree: Right. Steve: So I’m going to just pick on you slightly. I mean you used this phrase which we hear a lot when we talk about research which is like what users want and that’s sort of shorthand, but I think in the examples that you’ve given I think you’re really saying that research is identifying something else. And maybe this is just me being semantically stubborn or something, but no one asked you – and we know this, right – no one said hey I wish I could do this with my phone, right. Pree: Yeah. Steve: When you say what people want it implies that the researcher is taking requests, but what you’re describing is something much – it’s that hunger and that curiosity and that – and that synthesis to pull those principles out. Pree: Absolutely, as in we’ve been doing this kind of work for a long time and we always go for these unarticulated needs. What people say, you know, is very easy to get to. And a lot of times what people say is like they’re lying, you know, and they’re saying what you want to hear. It’s getting to the deeper insights by looking at patterns, looking at behavior, looking at what people are making. And so I go back to Liz Sanders who is a mentor from a long time ago. What people say, what people do and what people make. You start to see that each of these areas give you different kinds of data and triangulating between that data takes a lot of synthesis, takes a lot of analysis. And then you get to the core of it and articulate what it is that our users want. And the point that I think is most important and is lost is it is not two researchers going off for two months, coming back with this magic. This analysis doesn’t necessarily happen in a closed room. The right way, the new way to do it is the analysis happens in an open space, happens with others. It is co-created so that at the end of the day our products are – the objectives for our product is co-owned by everybody building the product. Steve: So let me ask somewhat of a more tactical question about that. You know you’ve talked about some people – some people are paid to have opinions. Pree: That’s right. Steve: And so how do you help people be effective in synthesis when what they’re coming in with is what they’ve been working on, what their vision is, what their intuition is, whatever that is. How do you help them sort of embrace the messiness of the data and be in that synthesis process and – I don’t know. I’ll just stop. Pree: So some of it – so we’ve been doing a lot of what we call iteration zero sprints. We just started doing a lot of those things. Basically it’s design sprints you know. And Google’s done a good job of writing a little bit about the design sprints. You must have seen some of the design sprints work that Google Ventures has put out. A good amount of work which is really good and – and that’s a very tactical way to get a bunch of people together and not necessarily throwing them right into the middle of swimming pool, but letting them wade in the kiddie pool. Now what happens is everybody gets a sense of what the water is like. How cold the temperature is? What it is like to float a little bit? That kind of a thing. The researcher does most of the heavy lifting, or a bunch of researchers do most of the heavy lifting. Then when we all come together towards the patterns everybody is on board. Otherwise you know this is a challenge that our industry has faced for quite awhile, right, which is trying to explain qualitative vs. quantitative research as well, you know. And we do a good amount of both kinds of research within our team and I see a lot of it where – and I’ve seen this in multiple companies where time it’s like 10 users, even though those 10 users were observed for, each of them observed for a few days maybe. Oh, it’s only 10 people. So do you think we can put a million dollars on this or not? And that question always comes up. And so that’s where getting people really connected with the stories, the real people, is super important. Storytelling is a very important part of design and make good design decision. So it comes back to that part of design that we plug into research. Steve: So how do you answer that question? Are these stories sufficient to commit an investment? What’s the response? Pree: It usually ends up being yes. So there’s two things that help quite a bit. If you look at venture capital money going into startups, or you look at like companies investing into new products, there is one which is very good data that’s driving certain directions. Second, conviction. When the team sees certain things and they believe in it there is a certain level of conviction. And you cannot necessarily explain it in words, but you can see it. Us as researchers, what we do is we change that part for the whole team. And that’s the value-add that we bring in. It’s quite amazing. Steve: I feel like yes and no. I mean yes I know what that feeling is like and I know what that conviction feels like and I know when you just – you know that here’s the opportunity because you’ve been out, you’ve been with these people… Pree: That’s right. Steve: …you’ve experienced their stories. You’ve breathed the same air as them. Pree: How do you then translate it to conviction for everybody else? Steve: Yeah. Pree: You know as in like – as a consultant I’ve done immersion rooms. I’ve done videos, video clips and like it’s not enough. And all of those things help. Now what we’re doing a lot more of is we’re actually taking people into the field. As part of our sprints we have users – we have two sets of interactions. We have interactions in the beginning of the sprint with the users. We have interactions at the end of the sprint with users. And through that interactions, first interaction is about understanding the ecosystem, the domain, the landscape. And the next interaction is about evaluation if you may. It’s more concept evaluation. And so everybody has been part of this journey and it’s not just theater. It is actually immersing yourself in the people’s lives. Steve: I think you’re describing a really nice version of this – we’ve been headed towards sort of everyone being a researcher, research is integrated, everyone participates. And I think there’s a – we keep talking about tensions. There’s an interesting tension there where this thing works best when everybody plays as deeply and richly as possible and yet it’s not – it’s a profession not a practice. What am I trying to say? There’s a better way to say that. You know filling out expense reports is not a job function at a corporation. It’s an activity that everyone that works for that company does. Research is a job. There’s leaders that have that function. I mean I wonder if this is a point and a transition where – because you’re describing something that’s just really fully integrated at its ultimate version. But they need us to facilitate it, to kind of guide it. Pree: Yeah, and I’m describing an ideal situation. As in it’s called work for a reason. As in it’s not all like easy and simple. We’re learning and we’re making it happen every day. So it is not – it’s not that everything is like done and like flower petals are put in front of us for us to come there. It is definitely a struggle. At times – like and different teams are different. There are day to day stresses and tensions and those are places where again if we believe in where we want to go as a team I think it makes it much easier because again it comes back to our talk about the local maxima as well. You know even within our profession. Steve: Could we switch gears a little bit? Pree: Absolutely. Steve: Talk about your choice to join this group and take on this role. You’re describing the reality and the reality is of course everything is a mix of wonderful and challenging and that’s not – I don’t think that’s unique to your situation, but something was compelling to you about this opportunity and can you talk about what you were looking for in… Pree: Absolutely –So eBay – so my background is engineering and design. Have worked in design research, innovation consulting for a good while. Then moved into Microsoft, was part of the small team that invented Kinect and also was part of the team that did Zune – all that kind of work, like some of the initial Metro designs. So moved to Motorola afterwards to lead the design research team right when Motorola was changing quite a bit. You know Motorola had gone down in terms of how it was doing and there was only one way to go once you go down that much. It’s like up, or disappear. And my personality overall is more – I guess I’m a betting man. I take risks in spaces – calculated risk of course, but I like that excitement. And so I joined. After a few years some of the work was nicely done already. There was things that were in place. So personally I was looking for other possibilities and this thing came along and I again looked at what was happening. eBay as a company is a 20 year old company, has established itself as a leader within of this marketplaces. Makes a lot of money by doing this. However if it continues to do the exact same thing that it did it probably doesn’t have as much of a growth trajectory. And so how – so overall there is a challenge for the company to reinvent, to think about new possibilities. And for an outsider, for me I was like looking at it and I was thinking hey, you know, that sounds like an exciting opportunity. And a place where somebody like me can make impact. The team was amazingly good. The culture was very good. So I felt like hey you know I could work in this culture. And it felt exciting to move at a time when eBay was just celebrating its 20th anniversary, just split from PayPal as eBay by itself and there is a future that we all can create together. Does that answer your question? Steve: Yes. So if I’m a researcher and I’m looking at different organizations, because you’ve mentioned a number of different orgs that you’ve kind of come into, you know what are things that you would advise me to look for to try to assess – I mean a good… Pree: A good fit. Steve: …a good fit for researchers I think is different than other kinds of professions. Pree: Yeah. Steve: You know what do we look for? Pree: I think – it’s a very good question and I haven’t thought much about it to be honest. You have researchers at various levels. So there is an entry level researcher who’s done a good amount of academic work let’s say, you know, or has gone through some amount of training. And then there is a researcher with like mid – you know two, three years of experience and then there is the researcher with about ten, fifteen years of experience. Or plus, like much more experience. And then there’s also researchers at various kinds of levels. There are researchers who are IC, individual contributor kind of researchers. And then there are managerial kind of researchers. Different places I think offer different things. So there are certain places – like if you look at the Googles and the Facebooks of the world, if you may, there is a lot of opportunities to learn. And you can have a researcher who is just starting out, a younger researcher who is wanting to learn a lot of things. It’s an awesome opportunity. Even companies like – the bigger companies are really good places for you to learn. eBay is one of those. A new person comes in who has no experience and we’ll train them and they get to observe. They get to be part of a pretty big organization and everybody in the organization comes with a different skill and they all are willing to teach. And they’re willing to learn from this new person as well, you know, and so that automatically makes that kind of an environment better. I would say for a mid-level kind of a person it might be valuable to go to a smaller company, a start-uppy kind of a company where you are probably the only person. You can build out the team. You can figure out what – you know build a network, build a network of mentors, build a network of people that have done this before who can become your personal advisors if you may. And then you can go next step. And as you get more senior there’s only a few places that will hire you as well. Because you’re too expensive for – you know it’s just – or it’s where – or you move into different areas. I know of researchers who moved into product management. I know researchers who moved into design. And you move into lateral areas and grow that way. And all of it is learning and it’s something that as researchers we enjoy. Steve: So maybe a related question, when you look at people that might join whatever team you’ve been on over the years, what are some kinds of things that you see that kind of – that make your antennae go up. Like what are some good, strong signals that say there’s something here with this person? Pree: Sure. I think it comes, comes down to passion and you can – again, like when you see people really – and I’d say passion translates into several things. It translates into a real hunger for learning. It translates into caring about the product and the user. And you can see the decisions being made that go towards caring about the product and the user. And I’ve noticed that researchers at a certain point they been taught research so much that they sometimes think of research I’d say in a little bit too pure kind of a way and aren’t thinking about it as a tool, a means to an end. And I see some researchers who have gone past to where they have learnt the tools to cause damage. When I say cause damage as in like they start to think about the product. They start to think about… Steve: The good kind of damage. Pree: …the good kind of damage. Steve: It’s back to your impact point, yes. Pree: Yeah. So that’s what attracts me to like okay saying hey here’s some people that have potential because they care and they care about the product. They care about the user and they’re very passionate about it. Steve: That’s great. Alright, so just picking up on this notion of passion, we’ve talked, as of course it’s the focus here, we’ve talked about your work and a little bit about your background and kind of how you got to where you are. I’m wondering about other things that you have passions for that reveal something about how you – what we’ve talked about today? Are there any kind of connections or cross pollinations between research, it makes you do these other things, or these other things kind of impact how you think about research? Pree: Oh yeah, yeah. No, I’m – I have too many interests as in I’m a Dad. I enjoy being a Dad. I am a bicyclist. I bike. I bike commute to work, plus I bike on the weekends anytime I get. I love cooking. That’s kind of a place where I can calm down. Design, generally speaking like design, designed objects. Well-designed objects, well – you know even furniture making, those kinds of things. I’ve been doing this for years and I do a little bit whenever I get time. Life hacking – it’s just more about productivity for me, as well as just like peace of mind, all of those kinds of things where I could probably get more out of less if you may. Steve: Um-hmm. Pree: I also recently got back to this. As a kid I grew up in a – I grew up in a small little town back in India and I come from a farming community and I’ve learned – like a lot of the farming side of things very early in life and one of the things was bee-keeping and I keep bees and that’s something again like you know, backyard honey. I enjoy doing it. It’s a – maybe there is a pattern that goes across all of these too. Have – love old air-cooled Volkswagen Karmann Ghia. Again, the engineering side of it, you know fixing things, making it work again. Those kinds of things excite me. All of it, I think the pattern that goes across is it’s all intense stuff and I like the intensity. Intensity kind of calms me down, and that’s why I end up doing it. And you can see some of these things where I bring them back to work. You know if you think about bees and you think about how it works, the bees as a colony is a brain. A single bee by itself will not sting you if you meet the bee somewhere on the road. You know unless you go and try to squeeze it or something like that. It will never sting you. That’s the last thing it wants to do. But as a colony they work together and one of the ways you can get stung is if you try to steal their honey. And anytime they get a notion that you’re going to steal their honey it’s going to sting you. And they go after it together and there is a whole lot of science there. One bee stings, there’s a little bit of smell that it leaves and then every other bee wants to attack that particular thing. And so now again, simple rules and everybody executes. I bring some of those things back to work. I bring principles and principle-based decision making from cooking. Cooking is an interesting thing. I never cook with a recipe. I cannot cook with a recipe. But I cook with principles. There are certain things that go well together and those have been learned over hundreds and thousands of years. Different cultures have learned this many, many ways. I do read books about food. So principles at work for me come from principles in cooking. Cooking is an interesting thing because if you don’t cook with recipes you kind of borrow principles that help you cook well. It might be about like how you put the right amount of garlic and ginger in your food. Or it might be about like how you pair a certain kind of vegetable with a certain kind of meat. And it’s that connection and it sets – it’s a pattern that works well and if you – you know I read a good amount of books about food. I don’t read about recipes, I just read about food. And I love to read about the history of food as well, as in like how food changed over thousands of years. And those kinds of things intrigue me just because of the fact that there’s this knowledge that has been passed on generation to generation – not necessarily written down, but passed on and it’s a set of principles. Steve: And so now I’m thinking about what you’re describing and my own inept attempts to cook. If you think about how much garlic or how much ginger to put in you can do it, but if you’re a novice, if you’re like me, the only way you know if you’ve done it right is to taste it, at which point it’s too late. Pree: That’s right. Steve: Which I think is similar to some of the work things we’re talking about, like how do you create something and how do you evaluate it. You know you sort of invest – you invest time or building effort… Pree: You do, right, you do. And that’s also like – here’s an interesting thing as in like from a cooking standpoint. Like my Mom was explaining to me when to add salt to vegetables that were cooking. She had no idea about the science of it, but she was right. You know she was talking about like adding the salt just a little bit after it boils. Right when it boils add some salt and add a spoon of water so that it starts to boil again. That’s what she told me. And I started researching – this is the researcher in me. I started researching why are you saying this? What does this have to do? And then I read up about how the salt added at the right time either makes the vegetable mushy or the skin of the vegetable stays intact, the inside gets cooked. And so it’s a science. And some people have learned it by experience and are passing it on to generations via patterns. And that’s why it’s quite amazing. And architecture is another one like you might have heard of Christopher Alexander and the pattern languages. You know that side of architecture always amazes me. It’s interesting because there is patterns to be seen and patterns that have been happening for years. You know we took the pattern language in the software and a lot of the software now is based on this, you know. So it’s quite amazing. Steve: You made this point early on about what’s an art vs. what’s a science. Maybe we can take some of what you’re describing with cooking and architecture. Can we talk about just field work? I mean we’ve talked a lot about the context in which research gets applied, but if you think about just doing – field work I think is maybe a rich place to look at just for a minute or so. I don’t know – you know can you draw – I’m going to just throw it back at you. You were sort of talking about cooking as a science but I think there’s an art aspect to that as well, architecture. And so how do some of those principles, do those things apply to field work as an activity in either an art way or a science way? Pree: Oh absolutely, right. So – and so let’s go to one other passion which is parenting. And I struggled with this, which is being there. Being here now. You know being present. And the mobile phone, the computer, all the screens kind of distract me. And I’m sure a lot of people have the same thing where my kids are wanting my attention and I don’t give them as much attention. Now on the field side of things, right, and this comes to bee keeping. It comes with the mechanical work that I will do on my car. You know all of those things, the attention that you have to give is exactly the attention you have to give while you are interviewing somebody. You have to be there. And you have to feel like that’s the only thing in the world for you at this moment. Is it a skill? Is it learned? I don’t know. But that’s something that everybody – you know – and I know you’ve written a whole book about it. It’s crazy. Steve: That’s great. That’s a wonderful thought. How old are your children? Pree: 8 and 9 – you have kids too? Steve: I do not have kids, no. What should we have talked about that we didn’t get to? Pree: I think that’s an absolutely good question. That’s a question that everybody should be asking at the end of the interview, yeah. Steve: That’s right, that’s my stock – it’s one of my stock ending questions – everybody’s stock ending question. Pree: Let’s see. What should we have talked about? I think we covered a lot of things. I’d say I think we might discuss a little bit and I want to ask you your thoughts on this which is where is design research going long term? And I think a lot about is this field a temporary field that was right for a certain time period and will it just disappear? Or is this a field that will morph into new things. What do you think? Steve: That’s what I was driving at a little bit when I ineptly compared research to expense reporting. I mean I’ve thought for years that there’s a potential for this to be an activity not a role and it’s interesting – I don’t know, it’s very exciting to hear how the activity of research has really, as you’re describing it in these ideal situations, it’s really grown. Like what we mean by research and what it entails and the skills and all that. It’s – there is a point, and you probably remember this from your early days, where, you know buying a video camera was like, that was sort of the step into doing research. Well we bought a video camera. And it was sort of a naïve time, but I love hearing you describe kind of the – I guess the maturity and sophistication and the capability that this set of activities has. But I have just been saying for years like is this just design? Or should it just be design? And I feel like – I don’t know if this is the same trend or an adjacent one. Part of that sort of questing that you and I and people in our industry have been doing, sort of evangelizing, advocating, hoping, wishing, wanting, has been to see research more fully integrated, more fully adopted and I think with that has come a couple of things. There’s sort of a good and bad and as you said there’s an ideal and you’ve laid out the vision for the ideal and you strive every day to achieve that ideal. I also think, and hopefully I’m not just being grumpy about it, but there’s a lot of commoditization of research that’s happened at the same time. Pree: Oh absolutely. Steve: Where there’s lots of junior people coming in. They’re not always getting the mentorship that you’re describing. They’re being asked to kind of execute closed end – not explorations. They’re barely doing evaluative. They’re swimming in the local maxima to mix metaphors. And that some people are kind of you know dusting their hands off going well we did it. We hired a researcher, we’ve got research being done. So… Pree: It’s a checkbox, yeah. Steve: Yes, yes. I guess that isn’t new. I guess I just see more versions of that where there was a time where research was ignored and now it’s sort of pulled in, kind of kept in the corner and not doing very much. And I feel like oh we wanted everyone to do research and now they are, except they’re not always doing – they’re not always kind of doing it as a leadership, strategic activity the way you guys are and so many of our peers are and so sometimes I wonder like well is this the consequence – is this the downside of us wanting this so much that it has been adopted but not in the way that we had hoped for. Pree: Yeah. And I think you mentioned something very interesting which is this is an activity that everybody does or it’s just one of those activities. It’s not necessarily a discipline. It’s not a profession. It’s a very interesting thought because I have a sense that you know over the last few years, like it’s been 20-25 years that we have had kind of a new discipline come about which is the product management discipline. You know it used to be more program management before or it used to be more like putting – connecting all the dots, rather than owning the product. And I also wonder out aloud if we somehow connect with the product management discipline somehow or the design discipline even more so. And I don’t have any conclusions. It’s one of those things that kind of keeps me up and I think about like where is this going? Steve: I’ve said this before on this podcast, it seems like research follows design or follows user experience and that I think some of the things we wring our hands about are things that that profession, that community, was wringing its hands about three years ago. And so the move – I hear UX people say all the time that product management is kind of, is the opportunity for UX to grow. Rosenfeld Media did a virtual conference recently that was like a PM plus UX event. So there’s these signals that these are the conversations that are happening. And so where’s the research side of – ‘cuz sometimes research is sort of like hidden under UX a little bit. Pree: It is, but if you see some of the latest things that are happening, I feel like research is actually leading design and the reason I feel research is leading design is because of a few things. One is how connected we are with users? And how that gap between a user having a need to a solution that can be shipped is kind of reducing. As that happens you end up having a lot more of this need gathering. A lot more of this co-creation if you may of the end product. That is important. And we’ve always been at the leading edge of this. We’ve been pushing design from design being just inspiration and like let me make this up, to design that is like intentional. And we continue to push this discipline. So I would any day call it as leading the discipline rather than kind of following it. Following it maybe from a time standpoint, maybe. Yes I see where you are going with this, but yeah. Steve: So that dynamic is interesting. And maybe design is maturing to not be decoration. I mean it’s been maturing that way for a long time and that’s – I’m just reflecting what you’re saying – that starts – as it matures to be sort of creating experiences, creating value, then the hook for design to do that is research which then goes back to your question, which we’re not going to be able to answer, which is should it just go away or will it just go away? Pree: I don’t know. Like – another interesting development that’s happening with the design side of things is patterns. So over and over again now we are starting to see design patterns that are used multiple places. And so suddenly the value of having to design individual things go down. And it happened in software a few years ago. As in some of the very good software programmers know that the best thing to do is to have a collection of all of these things that they use over and over again and they just pull it in, cut and paste. And most of the code is not written. It’s just cut and pasted together. I’m not reducing the value of a person – of software here. You know I can code and I know it’s a lot of work – patterns and knowing when to pull those patterns together is the skill. So you mature the discipline to something else. And we are very good as a research discipline to figure out those kinds of things, the needs or what patterns to pull together. So I think we’re set for good success. That’s what I’m thinking. Steve: I love that. Maybe we should leave it on that wonderful high note and thinking about the success we’ll continue to have as a discipline. So thanks very much, Pree. It was a wonderful conversation. Pree: Absolutely. I enjoyed it. Thank you so much. Steve: Thank you.
March 15, 2016
This episode features Gabe Trionfi, the Manager of Research at Pinterest. We discuss the evolution of user research, collaboration between disciplines and the journey versus the destination. Great researchers really know the on the ground, in the trenches, skills, tools, things they have to do to move forward with the group, but they also have to play this unique role where they pop out and get the bird’s eye view or the broader context and think about that, but not in a way that distracts the rest of the team. And then they port back down to the blood and the guts of what we’re doing every day in the trenches to figure out how can I take those observations that are abstract and make them useful for the team? So all research conversations for me waver between the two. They’re up in the air, really philosophical, abstract, but then also down on the ground, really practical, really technical. – Gabe Trionfi Show Links Gabe Trionfi Follow Gabe on Twitter Pinterest Joe Namath guarantees it Joe Montana goes to Disneyland Failure to Lunch June Jo Lee Michael Graves and Target Tiffani Jones Brown Sherlock Holmes Evan Sharp Ben Silbermann Social graph/interest graph Altay Sendil Larkin Brown Cassandra Rowe Jolie Martin Paul Pattishall Julia Kirkpatrick Burlington Country Style Donuts Tim Horton’s Millionaire Donut Shoppe Follow Dollars to Donuts on Twitter (and Stitcher) and take a moment to leave a review on iTunes. Transcript Steve: Let’s start with the introduction. Tell us who you are, what you do, where we are today. Gabe: Yeah, so my name is Gabe Trionfi. I’m the Manager of Research at Pinterest. I’m a researcher. It’s what I do. It’s what I want to do. It’s not the journey, it’s the destination for me. I’ve been doing research for about 10 years. I’ve been at Pinterest for about 3 ½ years now and we’re in the office, third floor, new headquarters for Pinterest. And we’re in Alberta, a Canadian province theme named room. Steve: Very appropriate. Gabe: Very appropriate. Steve: We have two Canucks talking research. Gabe: Absolutely. Steve: So you made an interesting comment about it’s the destination, not the journey. Gabe: Yeah. Steve: So could you explain that? Gabe: Yeah, I think for me when I think about research it’s my career. It’s what I want to do and I always want to be doing research in some way. I have the opportunity now at Pinterest to manage a team. It’s my first time managing a research team. I’m always going to want to work as a researcher in some way. I’m not looking to become a PM or a head of product. I want to be involved with research. I also think it’s interesting to think about a career path. Like I’m looking for a positive trajectory for my career, but whether it’s being a manager or an individual contributor, that’s like not the most important thing. The most important thing for me is are we working on something that’s interesting? Is there an interesting research question to be have? Is there a product that people are going to be using? That’s really important. Are we shipping stuff that’s making people’s experience better? I want to be a part of that. Steve: But some of that isn’t research. Gabe: Absolutely. Steve: Some of that is the thing that research informs. In the context in which you’re doing research you want to have a good question, but you want someone to make stuff out of it. Gabe: Yeah, so I think sometimes when we talk about – in design or creative disciplines like writing and research, people are like well you’re a manager now. Are you always going to be a manager? Or does it mean less if you are not the manager? I would much rather be – go forward and have a role where I was a researcher on something interesting that people were using vs. always choosing to be a manager on something that people weren’t using or was potentially not that interesting from a research perspective. Steve: You mentioned to me before, outside this interview, that you have a history in theater. It’s almost like you want to direct, but you don’t want to give up your acting jobs? Gabe: Yeah. Yeah. Or it’s all part of acting. It depends on how you think of it. I think it has – there are some really interesting parallels. So I had never managed a team or built a team. At Pinterest I was the first researcher. There was like an opportunity where they said do you want to build a team? I said I want to try that. I didn’t know what it would be like and there’s been a ton of learning. Obviously when you’re first out of the gate you don’t know enough to be great, or even good, but you learn over time. The one thing that was interesting, taking it back to theatre, was I found that building a team reminded me a lot of casting a play because when you’re really putting together a great research team it’s not just skill sets. It’s not just experience, but it’s also understanding how will these people interact with each other when we’re building knowledge together? How will they be perceived as a group, as individuals? It’s like casting a play, like how will this actor relate? Or to this other actor that we’ve already cast? Do they have chemistry? Do they represent the relationship of the characters well by the way they look and what they sound like? And that was really surprising to me. It’s been a really fun part about building the team at Pinterest. Steve: I want to follow up on that, but I also want to go back. Let’s just go back to a few different things here before we lose them. You opened with this big idea about the journey and the destination so now I’m going to pick it apart until you wished you’d never said it to me. Gabe: Okay. Steve: Because I – I mean I feel like you’re describing problems that are sort of journey problems not destination problems if that’s a way to slice it. Like you cast a play, you can be expert in it and you can know all these actors and know what their strengths are and you can do the right kind of – what’s that audition where you see people play off each other. Like a screen test or whatever that is. You can do all those things, but it’s still an emergent thing. You don’t know. Gabe: Yeah, I think for me I think of research as a destination as quite large. That there’s many things to be learned. I still think of research as early days, particularly in the context of technology and start-ups. And so I think it’s fast and deep and interesting and sometimes I may talk with people who seem like – well this is how I’m getting into tech. So I have a social science degree and a little bit of background in psychology. I’m going to be a researcher for awhile, but really the destination is like to be a product manager or to be a VP or head of – and for me the actual research part is so interesting and there’s so much to learn and do. I definitely feel like it’s the beginning and that’s ironic of course ‘cuz you’ve been doing this for much longer than I have. When I worked at previous places like IDEO, they have people who have been working in research since, you know, the early 60’s in the context of… Steve: I have not been doing this since the early 60s. Just for clarification.. Gabe: But in the context… Steve: But the field… Gabe: Yeah, the field goes back. You can even go back to World War I or World War II cockpits in air planes. The sort of human factor, early ergonomics. That’s sort of the seed of a lot of this stuff. ’63 is when the Royal Design Conference, right, in England – the Royal Academy of Design – they start talking about research for the first time. Engineers start talking about design and research and really what it means. So it’s a discipline that the idea has been germinating for awhile, but I think in practice every day it’s still fairly new, especially when you get outside the context of physical products and service design and you get into like pure software technical. The majority of people I work with here have never worked with a researcher before. And so that’s a challenge because you’re sort of still at the beginning and sometimes you can do really incredible work and people may not know that it’s that good because they actually haven’t worked with anyone before. And that’s sort of humbling and interesting and it’s a challenge at times, but it’s also a huge opportunity. So that’s interesting. I often talk about research being early like – you know the difference between Joe Namath and like Super Bowl 3 I think – who’s like a big time football player. He is the quarterback. He’s sort of glitzy and everyone knows him. He’s the underdog. He says they’re going to win. They win. He’s like celebrated on a bunch of magazine covers. But it’s like in the 60’s, the NFL. So like he goes home and has a Schlitz or something, right. Like he kind of has a regular life. Fast forward like to the 80’s. It’s Joe Montana, right, for the San Francisco 49ers, who wins a bunch of Super Bowls. He then – he’s a big deal, like Joe Namath, but like he goes to Disneyland afterwards and he gets like a lot of money and he’s super wealthy and commercials and whatever it is. The difference between Joe Namath and Joe Montana, we can argue about it as athletes, but the main difference is timing. When were they the quarterbacks of the Super Bowl winning teams and sometimes timing, when you’re at the beginning that’s just a constraint. It’s just part of being at the beginning and I think that research has a long way to go as a discipline and there’s a ton of opportunity, but like if you’re a researcher 10 years from now you’re going to be standing on the shoulders of schlubs like ourselves who had to do it in the early days. It will be way better then because people will understand it and also the market will shift in a way that will require research or something like research to inform product development. Because it will be more competitive. There will be more services and apps within the same space that need to differentiate from each other. So timing, where we are. Steve: Where are we between Namath and Montana? Gabe: That’s a good question. Steve: Is there a visual continuum here and we could get the whiteboard out? Gabe: Yeah. Steve: Let’s just challenge that. A recent issue of the New York Times Magazine was all about work and so this article, this article, this article. There was one – it was really about people eating at their desks, but the side bar piece – unfortunately I don’t remember the name of the person, but the side that kind of goes with the article that supports this photo essay says so and so, an ethnographer, is like the first thing. And it was not an article about oh these people are like anthropologists in safari hats going into corporate America. They didn’t go that thing that journalists were doing 3 years ago and 15 years ago which is ooh, look how weird this is. They were really just – it was sort of incidental and characterized the fact that like hey there’s an expert who’s watching people eat lunch at their desks. To me the cultural moment there is like this is a thing that we don’t have to just point to it and go oh-ah, we can use it then to tell another story. So that seems we’re – I don’t know if we’re taking this person to Disneyland, but we’re not having Schlitz. Gabe: Yeah, so I think this is an important point which is – like in research we often pay attention to context, so my context today is within tech, within start-up land. I think when we look at research trajectories in terms of its development as a discipline it actually tracks differently across different contexts. So I think when you look at physical products, service design, I think that industry what’s happened there, that collection of industries with research is far ahead of where we are within tech. Steve: You think tech is behind? Gabe: Well I think tech is tracing the trajectory. To me it looks the same and I think it’s headed in the same direction, but I do think it’s at a different point much like just the industry, the current modern industry, like post-Internet, apps, software design – it’s fairly new itself, right. And so with a lot of new industries, especially industries that have access to a lot of data, right, even the history of psychology itself has had moments like this where people think it’s all about this type of data, this quantitative data, right. That’s how we can – we’ll find the future opportunities within that and then we realize like well actually one type of data may not be enough. We need different types of data. And then we might switch and say like well I’m going to do this other thing, or I’m going to do this other type of data acquisition or method, right, and then really mature fields, they come to usually a realization which is like yeah there’s no one thing that’s going to do it for us. We need multiple things to try and understand complicated phenomena and then within that, especially within the world of business and tech, then you’re still going to have make a risky call because again none of this data can tell you the future. It can paint out trajectories and possibilities, but you still have to take a risk. I want to work at a place where someone’s going to be able to understand the data, triangulate and then make a strategic decision that’s going to lead to a better outcome for the people using it and the people working on it. So I think that when we get over – again, some disciplines, some industries in the beginning there’s things that seem easy that are a lot more complicated than they seem. The example I always give is like the common thing is qualitative versus quantitative data which of course they both have subjectivity embedded in them. They both have trade-offs. They probably should be used appropriately which is what’s your question, then let’s choose a method that fits the question. But the metaphor I always say is like I would never hire a general contractor to build my house who is like I only hire carpenters who use hammers. There’s no screwdrivers on my site. It makes no sense because what should determine the tool is the rest of the context. So let’s not use a hammer on a screw. Let’s use it for nails and let’s use a screwdriver for screws. And that’s the same thing for me about methods which is it doesn’t matter what the tool is. What matters is the context and what is the appropriate design to get at understanding the question and also the outcome goal. What is our goal? What are we going to do with the answer? Those are the things that are important to me. Steve: So you could start to sort of map this trajectory of research and tech and look at well what were the predominant tools – if the tools are a proxy for what questions are we asking, we might just have people showing up with hammers. I remember the point at which Microsoft got usability. Like that was going change – and this was I want to say early to mid 90’s and there was a lot of press about, like the physical rooms they were building. I think I went on an interview there and went on a tour there and you realized oh they’re just going to like run, like huge – I mean they’re going to do usability testing at scale. You know the word that we like to say in research or everyone likes to say in tech I guess, about at scale. They were really going to scale the hell out of this and sort of usability their way into something. And there wasn’t a lot of – I mean that was leading edge thinking back then. And I think as you’re describing this trajectory or this changing context it’s about the tools and the tools are the thing that we can see more easily. I don’t know what design questions were being asked of Microsoft, but I can infer based on the fact that this is where they put all their money. So I think you’re sort of describing a long time ago already, you’re saying now we’re like starting – we’re at a point where we need these tools and we need to be asking these questions. And so – I don’t know. Is that fair, the recap? Gabe: Yeah. And also sometimes the context is like within software and apps the marketplace actually comes to bear on the types of disciplines you need to invest in to differentiate. So if we’re in a marketplace where everybody has a really high quality engineering team, everybody has a really high quality design team, and we start stacking the disciplines up and you’re going to enter in sort of the messaging as UI which is really popular amongst a suite of apps today, right, where you’re just – it’s a concierge service or an on demand service that is using the interface of like a traditional SMS app. There’s a bunch of them out there, right. They all have great designers. They all have great engineers. They have great marketers. They need to understand how to differentiate from the other 5 apps within their space and they’re going to have to invest in something. They could invest in partnerships with other companies. They could invest in celebrities, right – to sort of like build their awareness. They could invest in researchers to try and understand what is most meaningful about this experience that they’re building and what are the opportunities to separate themselves from the other within the space. So some of it is disciplines and methods over time. Some of it’s the market itself becoming more competitive and sort of more compressed in terms of offerings out there. Traditionally design has been used in recent years to separate or differentiate, right. And I think research follows design and if you don’t have a designer it’s really rare that you have a researcher because design is working on the experience on the front end of software that people see and think of when they think of an app or service. If you’re not investing in that you’re probably not investing in like a deeper understanding of your audience’s needs as a way to differentiate. But like the classic examples – like Target hires a famous designer to make like toilet bowl brushes that are really pretty. That’s a way to differentiate from others within their space who are smell – who are selling commodities like that. I think research can – will and is having the same sort of role within tech as more and more apps and start ups start to fill into the same spaces. It’s not about building the infrastructure, say plumbing the Internet with social connectivity. It’s about like okay there are 5 financial service apps now that are redefining the loan space, how are they different from each other and how can we ensure that ours is the most relevant to our audience? Let’s get a researcher in here to help with that. Steve: I think what I heard you say though is at a strategy level there’s a lot of different things that we could do if we’re trying to differentiate our financial app. So we could try to get Leo and his posse to use that. Or whatever sort of way to generate attention. We could have a controversial CEO that makes outrageous comments about people. Or we could change our pricing model. Or we could – there’s a lot of things that don’t necessarily come from a nuanced insight about the world. It’s just sort of differentiation for its own sake. Or there’s a strategy there and one of those out of – let’s say there’s 20 obvious ones – and I can only come up with 2 right now – I’m just parroting you back. So one of those might be let’s invest in experience which means investing in design, investing in research. Gabe: Also writing. Writing is a similar discipline to me than research within tech now. It’s newer. Usually it’s working across many different teams within a tech company usually. It’s critical, like what is the voice of your brand, but also of your product itself. And I see a lot of – I work with a lot of writers here. Tiffany Jones Brown started as the lead of the writing team here, has since become a Creative Director. She has a great team of writers that I’ve worked with before. Building a relationship with that team, I feel just a commonality or similarity in terms of where writing is as a discipline. And the – if you invest in writing it says something about your strategy when you have a tech oriented app or software service. It says like hey we want to make sure that people feel differently about our brand or about our product. And you see a big differentiator in lots of tech companies now in terms of how do they talk to their users through their product. And so again it’s just another strategy that you could invest in. And research, and a suite of different research disciplines is part of that. Also data science can be part of that. Business analytics can be part of that. Ways to, you know, bank capital, not just monetary but like understanding. Like you can bank a lot of knowledge as capital to inform your strategies or to help move your product forward. Steve: If I take a naïve look at what you’re describing I feel like there are some disciplines that are output disciplines and there are some that are input and by that I mean, you know, writing is crafting the words that go on your website. Engineering is the code. Design is the pixels and every designer – everyone is rolling their eyes at that. That’s a really, it’s a very broad brush, but research is an input to decisions. Business analytics is an input to decisions. I don’t know if that’s a fair – if it’s that binary or not. Gabe: I think about it similarly. So I think about it as a suite of disciplines that are about indirect influence. So I’m a psychologist. I’m not going to build something with regards to the software that’s going to get into the world, but what I’m building is understanding or knowledge, but also sometimes creating experiences that are inspirational. I talk a lot about information and inspiration as part of the research role that is going to indirectly influence that thing that enters into the world. Sometimes there’s a one to one. You can just say like hey that decision there was based on this work over here. Sometimes it’s much more diffuse in that the whole host of teams that are actually contributing to the output – the most important thing is I want to work in an environment where research is a part of that, whether indirectly or directly, and it’s not something that’s on the side or separate. I often talk about like Sherlock Holmes and Dr. Watson. Like I’m very happy to be Watson to someone’s Holmes because I think that’s a real role and those Holmes stories wouldn’t be the same without the indirect influence of Watson, right, sort of guiding, informing, being a foil for Sherlock as he goes ahead and builds the solution to the mystery. And I think indirect influence is really interesting. Really hard to do. It – sometimes it’s easier just to like come in and say we should do this and then people do it, right. But to sort of like change people’s perspective on something they think they know very well, but actually might be very different outside of their own social circle or their own sphere of experience – let’s say on the other side of the planet there’s a different culture, different country, a different type of person who’s using your product and perhaps they just have a different mental model that you didn’t even know existed. Bringing that back, but also not just a shock, but to potentially influence, like oh hey perhaps we should think about this a little bit differently. Or perhaps it’s another way to think about it, that there’s not just one way to think about our product, there’s actual multiples that we should keep in mind. And that’s a good thing because it means the product is relevant to lots of people and complex and so it’s not just a simple like P2P commerce, like I’ve got this thing, do you want to buy it? Yes you bought it. Cool, transaction, safety, all that sort of stuff. It’s more complicated than that in some cases. Steve: So I think research has suffered from the sort of the naïve Dr. Watson – well not that Watson himself is naïve, but if you take a shallow look at those stories Watson is merely the chronicler. He’s the sidekick. He doesn’t do anything. He and I – you know, you do the close reading that you do and you see what he enables and how he kind of supports and I think that’s the beauty of those stories. You kind of go at them a little bit deeper and you realize there’s a complicated thing going on and that it – the more you invest in learning what those stories are about the more they kind of pay off. So just to bring it back then to research – research has maybe historically been viewed at the way we naively view Watson, that research is just asking people what they want. Sort of chronically things and reporting them back. And research as you’re articulating it, and of course I agree, has the potential and does more often now play more of that influential, maybe Holmes is the one that gets brought to Disneyland. I’m going to crush all metaphors together. Like Holmes is the one that gets carried on the shoulders but the researcher is – ugh, I can’t untangle this here. So I guess I’m trying to ask in a really twisted way here is about you know what are the things that you’ve seen or that you’ve practiced to give you the chance to play Watson, to enable things, as opposed to being relegated to a mere chronicler? Gabe: So, I think this is an important question. So for me I had one experience. My sort of first design research job was at IDEO and IDEO is sort of at one end of the scale design thinking to an extreme. Huge opportunities to do different sorts of work and to really be part of a deep, embedded process. Then I worked at Facebook which is an engineering company and it has a very strong engineering first perspective. So when I was thinking about my next destination I was thinking about what lies in between? What are the sorts of things that are out there? And there were a few companies and I think at the time the commonalty is they all had founders who were designers. And I think when you have a founder who’s a designer there’s an awareness of what research might be. There’s an awareness of the Watson role and its power in terms of creating new ideas or – and I think you want – I wanted to try and see if I could work at those places. Pinterest was one of those places. So Evan Sharp is a designer. He’s one of the co-founders along with Ben Silbermann. And so Evan knows about research. He’s not an expert in it. That’s important, but he’s open to it. He’s interested in it. And so that’s why they brought me in. I was about – there was probably about 70 employees when I joined, which at the time some folks thought well that’s early for a researcher, but if you understand what research can do it might not be that early and in fact the broader field we see now – I see startups all the time with a handful of people who are bringing researchers in when it’s relevant to what they’re doing. So to tie it back to the metaphor, to me it’s important to be at a place where the Sherlock wants the Watson to be there because a lot of times that may not be the case. And so Watson’s ability to contribute is limited by the partner’s desire for input. I think at Pinterest there’s a big desire for that. It’s a really interesting place to do research. Very open, very focused on the audience that we’re trying to reach. In part it’s probably due to, you know companies have lots of values. Evan will talk about this. I think the one, the value that I think is really unique at Pinterest, and differentiating, is something called “knit” which is how we talk about interdisciplinary collaboration and respect. It’s understanding that there are many people who can contribute to successful outcomes at the table and we should understand each of their disciplines, strengths and weaknesses and we should leverage their strengths. So “knit” is very Pinterest, knitting right, like weaving together disciplines. There are a lot of starts where that sort of multidisciplinary approach is not part of what they do, culturally or strategically, and it’s a huge part of what’s happening at Pinterest, whether it’s Community Ops, Design, Writing, Brand Design, Marketing, Research. There’s a whole bunch of disciplines – Data Sciences, Business Analytics – that are all working together very closely – Project Management, Product Management – to sort of put together what it is Pinterest does. So that context, that environment that’s willing to accept what you have to offer, but also try to like embrace and really use it. That’s important for research and I think that’s a choice you have as a researcher. If you seek those places out you are in part prototyping or designing your own experience. Versus going to a place where you might know from the outset they want a research, they don’t know what researchers do, they’re not certain that they will use the research. And that’s usually pretty clear in the hiring process or early on. And I think that’s not to say that’s a lesser than experience, but it certainly is one you’re choosing as a research practitioner in terms of your daily experience. Steve: Or you should be aware of that? Gabe: Yeah. Steve: And you’d do well for yourself to know that that’s what you’re choosing. Gabe: Yeah. Steve: I think I saw an artifact of the knitting culture. In the men’s room there’s some Python script coding advice hung over the urinal and the title for it – I don’t remember what it said, something about like python knitting or something. Gabe: I think it’s called – officially those posters are calling knitting in the bathroom. And there’s someone – an engineer here is – there’s I think a couple of them working together to sort of pass on tips and tricks for certain software languages by putting those up in the bathroom stalls. Steve: It’s not the only company where I’ve seen like technical advice in the bathrooms. Gabe: Yeah. Steve: But the knitting – I didn’t understand the knitting reference until you kind of explained what that means culturally. Gabe: Yeah, absolutely and I think it’s – you know again there’s a lot of great – we have other values at Pinterest that are great. But I think that one is really unique. Again this idea of – for me it hearkens back to some of the experiences I had at IDEO. This emphasis on multi-disciplinary approach to design challenges. I think it does lead to different outcomes. My background, I have an undergraduate degree in theater and I care a lot about process and how we’re doing what we’re doing together and so again that part of Pinterest and being a researcher at Pinterest is real exciting. Steve: So we never resolved this. Again, I was kind of picking at you at the beginning when you talked about journeys and destinations as things that you – and so I think I understand as you explain more that being a researcher is a successful state for you. It’s not a stage to get to something else. But when we kind of break the world up into journeys and destinations I mean you – theatre is about the journey, right. The payoffs are – and you said someone who cares about process is someone that cares about the journey. So I think I got hung up a little bit on how you were using that phrase and I jumped all over you, but I appreciate finally we have – finally I have clarity. So thank you. Since we haven’t talked about this at all can you explain what Pinterest does? Gabe: Yeah. So if you listen to Dollars to Donuts you’ve heard that Pinterest is the world’s largest catalog of ideas. The question is like what does that mean? And I think at that – at the heart of that is like an interesting conversation for researchers, but also an interesting characterization of the opportunity and challenge Pinterest has. So Pinterest is very different than a lot of the apps or services or companies it’s often compared to. For a long time the press sort of positioned Pinterest as another type of social network. Pinterest is not a social network. It’s a very personal experience for the people who are using it. It involves things like saving things that you want to then go on and do or plan or make or cook or buy. But a lot of – when you do research with Pinterest you are really deep into it. They see it as this very personal moment. Sometimes people will even – again it has to do with discovery of content, the things you care about. Sometimes it has to do with learning. There’s lots of stories here from Ben our CEO talking to people early on where they would say to him anecdotally like I didn’t know I had a style. Like I didn’t think that was part of who I was, but then I started pinning clothes or things for my home and one day I was looking at my board about my living room – you know best living room, or like my dream living room – and I realized like wait a second, seeing all those things together it all sort of snapped for me. Like I’m – this is actually my style. This is what I’m into aesthetically. There are a lot of moments like that I think for Pinners because it’s very personal. Steve: A Pinner is someone that’s using Pinterest? Gabe: Yeah, someone who is using Pinterest is a Pinner. Someone who works at Pinterest is a Pinployee. We’ve got a lot of pin puns going on here, at the Pin Factory, which is what I call it. Steve: We’re going edit all those out. Gabe: So back to the point, I think Pinterest is personal. It’s done in a way where it’s in a public space, but it’s not social in the same way many social networks are about broadcast. Like this is my identity. This is what I’ve done. Pinterest is really about – it’s about aspirations. It’s about anticipating. Like I want to do this trip so I’m planning it. And then it’s about participating. Participation on Pinterest is not within the digital space. It’s within the analogue, the real world. The part where you have this pin for this recipe and then you go cook it. You make it, you serve it. Whereas other social networks are often about broadcasting in the moment, like real time, or they’re about broadcasting this happened. So they’re sort of reminiscing or looking back on the event. Where we see Pinterest is used to plan, you know, in the US Thanksgiving dinner. You start seeing people pinning Thanksgiving recipes months in advance. Halloween costumes is a big – DIY Halloween costumes is a big thing in the US on Pinterest and people start looking for those how to do – you know put together their dream costume, the best costume, the cool costume, like 2 months in advance of Halloween. So that Pinterest is a personal experience. It’s about planning to take action in the future. It hopefully supports you taking that action because you have the information you need. It’s very different then a lot of other services. So what does this have to do with the catalog of ideas? I think the important bit is we see in all of our research, market research, qual, quant, that this idea of a place to find ideas is differentiating. It’s useful, it’s fun and people enjoy that. The challenge is when you sort of – you have to contextualize that. From a marketing perspective you have to say you know okay what about a place to find ideas. A place to find ideas is not enough context. So there’s been different ways we’ve approached that. For some period of time we talked about bookmarks. Some period of time we talked about – more recently we were talking about like a physical thing like a catalog where you go and browse and discover lots of ideas. The key thing is the ideas bit. We know that’s important. The question is how do you contextualize that, not just for a US audience, but actually for a global audience and that’s really challenging. Pinterest itself is based on you know a metaphor of like a pin and board – pinboard. The challenge is that that doesn’t actually internationalize. Not every country has a tradition of a pinboard with a pin. And so how to provide context for what this thing is and it’s not – you know it would be easy if it was just like oh it’s a social network for, I don’t know, saving ideas, right. It’s not that though. That’s not really the core. And so how do you contextualize that? That’s the challenge. So as we think about different ways of messaging to audiences, especially audiences who are not using Pinterest today – there’s a ton of research we’ve done. There’s a ton of research to be done as we look for that really killer way to describe a complicated and robustly powerful product to people who don’t know what it’s about. And that’s an opportunity for research, whether you’re a marketer researcher or quantitative researcher. And it’s part of what the team has worked on and will continue to work on with our partners in writing, marketing and then just even overall strategically, like the decision makers of Pinterest. How do they want to – they have to go and speak to the press. This is something they’re saying all the time. They need to feel that it’s accurate, that it’s representative and that it really gets at the – the core or the central sort of essence of what they think the product is and will be in the future. Steve: So I asked you what the product is and you describe sort of the research and strategy journey of trying to describe it – to answer that question in the way that sort of has the most benefit for Pinterest and current and future Pinners. Gabe: Yup. Steve: But I’m going to ask you again, like – I mean as you say people are out there now talking about it. So you kind of said – you took the phrase catalog of ideas and then you unpacked it into all this wonderful stuff which I think tells us what kinds of things you’re thinking about and what your work is meant to address. But just for that person that’s scratching their head right now and going like okay, but what is Pinterest? Gabe: I gotcha. It is a place to save ideas digitally, to discover ideas related to the things you’re interested in and then to go and do the things that you have saved and discovered. So it’s about saving, discovering and doing. Steve: So what kinds of digital information are people saving? Gabe: Well I think we’re limited to digital today, but it could be many, many things in the future. It could be saving many types of objects. Right now we’re within the space of like the Internet. We see the – the most common use cases are we see a lot of folks use Pinterest for recipes. Planning daily meals is a huge part of what pinners. Also like discovering or trying new recipes, especially if you’re into that. It’s often about doing, right. Another popular use case – trips and travel. Like right now you want to go on a trip so you go to various websites and maybe you have a folder on your computer or a piece of paper – you’re writing down like where do I want to go when I’m in Hawaii? Where do I want to stay? You can save all those things to a board on Pinterest which is similar to a folder, but it’s within the bounds of Pinterest itself. You can save those links. You can find them on Pinterest. You can find them off of Pinterest and save it to your Pinterest. So that you can go to one board and have all the ideas of what you want to do on your trip when you go there. And you can access that when you’re actually on your trip, right. We also see a lot of people who use Pinterest for fashion, whether it’s discovering new trends or thinking about inspiration for what they want to wear. In fact there was a story this week in the – oh gosh, I can’t remember the newspaper – and this writer had talked about – she let Pinterest choose her outfit every day. So what she would do is she would take out of her closet an article of clothing she had – let’s say a pencil skirt. She would go to Pinterest and type in pencil skirt and she would look at the most popular street style or fashion images of people wearing pencil skirts. Then she would find the one she would like and then she would mimic it with her available wardrobe. So every day she let Pinterest dress her. Again, that really critical thing there is 1) she was looking for ideas, 2) she was then doing it and what was also interesting about it is that she was using Search. We see Search as being a huge part of a lot of Pinners experience in order to find those ideas. But Search I would say is a little different on Pinterest. The Search in this case, and what we see across many people using Pinterest is it’s not to find one idea that’s like the right idea. We don’t have a ranking algorithm that puts the right idea at the top. When you’re in a discovery system or a discovery mode you can search for outfits with pencil skirts, but there are many possibilities in front of you. You are looking across this set of data or images that’s about discovery and you’re trying to look at not just the best, but there are some that are good and there are some that are even better to make or to learn or to discover something that you want to then go do. Steve: So you described before this – you know your own professional path a little bit and that you sort of found Pinterest as a company that was in between maybe other places you’d worked at in terms of what was research and how it was understood. But as you’re talking about Pinterest and what’s going on and what you’re interested in exploring, there’s also a unique class of problem that I hear you kind of digging into with joy and I don’t know if that’s the culture or you or what. And that is we don’t know quite exactly what this thing is and the best way to talk about it and it’s changing and people are inventing new things for it and the world is culturally diverse. So the fundamental, like what is Pinterest and how do we describe it and how do we build for it is – like that’s a hard problem. It’s a dynamically shifting problem. I mean did you know that’s what you were getting into? Gabe: Well I have two answers. So the first one is like – the first one is sort of addressing the like what is it, it’s dynamic – like I think there’s something really interesting about Pinterest which makes it unique, which is again opportunity, but also challenge, which is there’s a utility element to this, right. It helps you to get things done. It helps you to find the things you want to do. But it’s not just a utility. It’s not a task based software. It’s not about lists per se. Although you could visualize pins and boards as lists, right. But it has this other element that I think is related and unexpected which is it has this way of conveying or inspiring people to think about possibilities. To discover things that they didn’t know about and to learn something about themselves. And it does this in a way that – we know from research that people really enjoy the aesthetic presentation of this information. They think it’s beautiful. And so, you know people can get – we interview people and they talk about my Pinterest time, right. They’re on their couch on a Saturday morning. They’ve got TV on in the background, maybe some Netflix. They’ve got a big cup of coffee. They’ve got like their phone or perhaps their iPad, and they’re browsing through Pinterest. And some of it’s about what they want to do. Some of it’s like what might I want to do? This sort of anticipation, aspiration. This sort of beginning of the experience. And Pinterest puts together these two things which are traditionally separate, right – the sort of exploration/discovery, aspiration/inspiration with like utility. And so that’s a tension within the product that’s really interesting. I think it draws people in, but it can make it really challenging because it would be much easier to do one or the other and that’s not where Pinterest is. It wants to do saving and doing, but it also wants to do discovery. And of course the more saving you do the better Pinterest hopefully gets at making recommendations to you about things that you should check out based on what you’ve done. That’s definitely an interesting part of the product and the tensions within the product, or opportunities and challenges. Secondarily you were talking about did I know that when I was getting in here? I was super interested in discovery systems. I think discovery systems are a really interesting part of the technology sort of landscape at this point in time. As part of discovery systems it’s like moving away from basic search which again is not basic in any sense, but many companies have been able to perfect the “I need this thing – help me find this one thing.” I call that like the needle use case. Right, I need the – help me find the needle in the haystack quickly. That’s impressive and it’s great, but it’s also now become expected. And so when you move into – it used to be called Search and Discovery. When you move into discovery systems that play up the discovery experience first suddenly, yeah you have to be able to help 5 people find the needle which is like perhaps one small moment within a user’s session, but actually the use case of looking through the haystacks is really interesting to people, right. So they’re looking through like haystacks in a field right and they’re like well what’s in this one? What’s this one about? How is this one different than that one? They’re exploring and discovering things that are interesting to them. They spend a lot of time doing that. And then they get to the needle. That’s something that I was already thinking about before I joined Pinterest and I could see Pinterest as having a unique position within that space. The one thing that’s related to that which we don’t talk to users about and it’s a super – I’m not an expert in this, but it’s an intriguing idea – is the notion of the interest graph. Are you familiar with the interest graph as a concept? Steve: No. Gabe: Okay, so social graph is sort of like, you can think of that as a subset of the interest graph. Steve: What’s a social graph? Gabe: The social graph is basically let’s map out with technology all of the people and their relationships to other people. And what’s interesting is like that’s been a very disruptive technology moment in the world today. But what’s interesting about it is that’s a big task – don’t get me wrong – but it’s finite. There’s a certain number of people on the planet at any point in time. Computers can count them, right, very quickly and probably count them real time if they have the information. The relationships between those people, they’re finite. You can measure them. You can ask people like we think these are people you might know, do you know them? Yes I do. No I don’t. That’s interesting, but again – huge in scale, but finite. The interest graph resides over top of the social graph and it’s this concept or theory that we could map out the relation between ideas, objects and people. And that’s really interesting to me and that’s a place I think Pinterest has a unique place to play in within the context of the other technology companies. What’s interesting about it is that the hallmark of humanity is every day we make new ideas, objects and people. So the interest graph is infinite. And if you had a technology that could map it out, so it could tell you which ideas are related to each other, which objects are related to which ideas, which people are interested in which ideas, right. You can do really cool things that are technically cool, but actually experientially cool. Something that actually would make people’s lives better which is most of the interests I have today are limited by what I’m exposed to. Like I only can know about the things that I know about, but if I had a technology that could tell me about something happening on the other side of the planet – in Germany there’s a new type of music scene happening that I don’t know exists today. I don’t know anyone who knows it, but if the technology knew that I would love it and could tell me about it perhaps I would discover it years before I would naturally. Or perhaps I would discover it when I wouldn’t ever have discovered it on my own. And I think that’s really cool. What Pinterest does nicely in that is that everybody is building this sort of interest graph rocket ship, this technology, but the fuel actually makes a difference. So on Pinterest today when people put things together in collections or boards they save them – it’s them. It comes from them. They’re saying something semantically about the objects that belong together to them and have meaning to them. They name that board of that collection and that fuels 100% pure interest. It’s really what I care about. Whereas other companies may have fuel that’s somewhat representative of that. It may be representative of a lot of other things. It may have more impurities into it. So I think Pinterest is uniquely positioned to sort of deliver a consumer experience on interest graph that I’m really excited about. I think it’s a really interesting concept and so one of the reasons I came here was discovery system, interest graph, recommendations. Like this is a pretty interesting moment in time for that sort of technology and one that actually has a real human experience on the other side. Steve: I love that you ended with the human part. Like that was the last thing that you said in this really amazing kind of romp through some big ideas. And I was sort of reflecting as you were talking that, you know – I mean we’re having a conversation today about research. That’s sort of the context and you’re talking about research, I think, I don’t know, in a unique way. I don’t know if you feel that. I’m kind of laying that on top of you. I mean how is this a research conversation? I’m not forcing you to defend it, but let’s just go meta, like why are we talking about this on Dollars to Donuts? Gabe: Well I will say that for me when I think of research I think of it as a discipline. Like as a blue-collar type discipline. I don’t think about – it’s a set of practices that you can learn and I think those set of practices can be applied to certain businesses or contexts. And so I don’t think of myself as a – I don’t believe in like super empathy powers. I sort of react negatively to that. I do think research is one way, or one discipline that can contribute to product development, design – service design. There are many that can do it. It’s not necessary to do it, but it can be useful. And so it’s a conversation about the context that research can play in, but also just about what are the duties, the set of practices – what are the things that researchers might think about in terms of their contribution and what they bring to the table. Great researchers I think they really know the sort of on the ground, in the trenches, skills, tools, things they have to do to move forward with the group, but they also have to play this unique role where they pop out and get the bird’s eye view or the broader context and thing about that, but not in a way that distracts the rest of the team. And then they sort of port back down to like the blood and the guts of what we’re doing every day in the trenches to figure out how can I take those observations that are abstract and make them useful for the team and actual for the team. So all research conversations for me, like sort of waver between the two, right. They’re up in the air, really philosophical, abstract, but then also like down on the ground, really practical, really technical. And so that’s probably why we’re having a conversation about like how did you choose where to go next and also some of it was like well I wanted to be in a place where they were going to care about the work that I did, but also there was this huge like theoretical thing that I could learn about in the context of their products. Steve: And moving back and forth between the tactical to blood and the guts and the big picture, philosophical. I don’t know, I’m guessing that – and just sort of watching your face as you talk, that not only do you like those end points, but I feel like, and maybe this is my own thing coming in here, there’s something about – and this I guess is the journey thing coming back in as well, but there’s something about the motion between those two kinds of thinking that I think for me sparks some stuff. That we can talk about the big picture – I mean you can really talk about the big picture, but sort of the energy we get from that vibration, from the well what does this mean, how do we deal with the blood and guts kind of – with the team every day. That – I don’t know. I’m throwing that out there. It’s a very leading question for you Gabe, but… Gabe: I think there’s certainly like personality differences that attract certain type of people to certain roles, right. And so there’s probably something about that that appeals to folks who are doing research for sure. I also want to make sure to like be clear which is it’s really important to me that as part of that process we’re shipping something. That I can actually see something in the world that I’ve contributed to. Otherwise I would have then stayed in academics. I would have been working at the theoretical level and for me there was something really exciting about work I did at IDEO where I worked on a furniture project and I’ve seen that furniture in the world many, many times. That makes me feel like there was – it’s not just about ideas, it’s actually about building something and doing something. And that’s I think important for me when I hire researchers or when I think about the type of environment I want to be in. It has to be – it can’t just be interesting. It has to be impactful. It has to be action oriented. There has to be something that happens at the end more than an idea. And so that to me is like I like that fluctuation between the tactical and the abstract, but to an end, and that end is to put something into the world that makes it better for people in some way, whatever the “it” is. Steve: I really like – I agree with you that stuff being interesting is sort of not an end to – it’s really great to be sort of turned on by stuff and like getting new ideas. You know for me trying to put it in a language – I mean what’s the good and bad word now – the A word – actionable. That you know – I don’t know, people go back and forth on whether that’s a good word or a bad word, but I mean I guess in my own career I had – the ship had sailed a long time ago on like measuring my own success by what shipped. And I hear from researchers as well that sometimes it’s frustrating. That – and so I’ve just sort of shifted my own role. Like I’m in the business of helping people see as clearly as possible what decisions they have to make and what actions they have to take in order to ship something, but I can’t – and maybe this is because, you know the difference between being inside and outside of the organization, I can’t measure my own success by what gets shipped or I would just be an alcoholic. But I do measure my own success by – I mean someone said it to me once like oh, what you do is kind of like take someone to the doorway and kind of like open the door and like point them kind of on the way through. Gabe: I’m probably using like very tacky language when I talk about ship, but I – so to be clear, shipping something can be pretty broad. What I mean is there was a decision made. That there was some sort of result or outcome of the research. And so definitely some of the hardest things I’ve worked on were the shipping moment was convincing someone not to do something. Like that is just as important as convincing someone to do something, in fact it may be harder in some cases because they’ve already decided, no. No, we’re going to do this thing, we’re going to make this product and helping them to understand how that might be a destructive decision for their business. That’s a type of shipping, right. Shipping a decision. So yeah, maybe I’m overly emphasizing the analogy. Steve: No, I think I’m just one sort of rung before that which is like I think for me I probably ship influence. I empower people to make a decision, but I can’t actually control if they’re going to make that decision. So sometimes you sort of – you learn something, you package it, you convey it and they need to do something. They need to do something. Like the business is just – the world has changed, the business has changed. They need to do something and the other factors are just so large that sometimes doing nothing is the outcome, but you know that you changed somebody’s mind. You gave them – and I’m not trying to undersell what I do. I mean I’m trying to be realistic like I just don’t get to ship decisions as much as I would like to even. I’m going to go drink after this. That’s what you’ve taught me today. Gabe: Well as long as I’m going with you it’s fine. I think – so one thing you brought up is control. Like I personally have had moments of development where I do not – I try to focus on what’s in my control. I can’t make a team do something, right. They actually have to make the decision. That’s a healthy part of the process and sometimes they will do things that are risky or potentially counter to the recommendations. And there’s a whole host of reasons why that might be. I try to recruit people and hire researchers who are resilient because I do think a huge part about being a great researcher is going out on a limb to say something important, potentially getting knocked down, and getting back up again and continuing to say that. I think really great researchers, in tech especially, they act like it’s a sprint, but they know it’s a marathon and often times at Pinterest we talk a lot about like the research part and delivering that, that’s just the first step. Now the real work is continuing to bring it up with the team. Continuing to make it part of conversations. That takes a long time sometimes. Particularly like if it’s something very meaningful or hard to do. People just don’t, when you deliver like a hey this thing you thought was the case for many years is actually potentially not quite like that. It’s a bit different and you should really think about it differently. Human beings are terrible at that. Like we don’t do that in our personal lives. We don’t just shift our total point of view without some sort of major event, like some sort of like peyote, or some sort of like near death experience. That just doesn’t happen. And so it’s not going to happen at a company or a business and you have to think about that sort of marathon bit, like bringing it up over time. It’s going to take awhile. You have to work at it and you can’t just think you’re going to come in and say no I have the answer, it’s this, and everyone’s going to be like here, here, good job, let’s do it. It’s harder than that. I just think that’s part of the research gig. That’s part of shipping information and inspiration, right. That inspiration bit. It can be very transformative for a designer in a moment, but getting to the long term outcome of that, the result of that usually is isn’t a couple of days. It’s probably a longer period of time. Steve: So tactically when you are meeting prospective research hires for Pinterest, how do you – that’s – and obviously you look at many things, but how do you look for that? Gabe: That’s a good question. I think first of all – how do you look for resilience. I think you talk to them a little bit about their career and I ask people to tell stories about the research they’ve done. You ask them to tell stories about, you know, what’s an example of a project that you did that you think showcases your skill set and really shows the best of you. And then you ask them questions about – they talk a lot about doing the research, then you actually have the real conversation of like so what happened next? What did you do? How did you approach communicating that to other people? What were the challenges you faced? And really – because we want to talk about the researchers, researchers, but again I just feel like that’s the smaller part of what we do in a context of an organization. ‘Cuz that’s what we’re good at and we like and we like that and we want to talk about it and we want to make that the big part, but like great – this is like you can build it and they don’t have to come sort of moment. Or even – I heard this great quote awhile ago which was “you don’t have to make a mistake to lose the game” and that’s about context and timing and sometimes like you have a great insight for an organization but the timing is not correct and it’s out of your control and so then what do you do? Do you pack it up and go away? Do you like start making posters and put them up every month? Do you continue to like any time anyone brings up tangentially relation, like oh that’s like what we found in this research from before, are you aware of it? That’s like part of the story of research and try to focus a little bit on that in conversations. Steve: Is that something you are teaching researchers that you hire? Or does someone have to come in? Gabe: I don’t think of myself as a teacher. I did not enjoy that part of academic life. I think of myself as a collaborator and I think it’s something we discuss a lot. How are we doing this? What are we – you know deliverables – what’s the nature of deliverables? Trying things – like the one thing I do try to ask the team to do, even when they’re tired of me and of the challenge is like well what’s something you could just prototype? Let’s just try to do something very different. And I think the team itself has built a good sort of muscle around like let’s try it differently to see what happens in terms of communicating and continuing to push those conversations forward. I do think it’s hard at a startup like Pinterest. When I joined I was around – I think there were 3 people in my class who are all still here – or my whatever, my cohort. We were like 70, 71, 72 – pick your number. Now there are 700 people and I think what’s really challenging with research is that you can learn something and communicate it to the entire organization and everyone can hold that knowledge. In the case of a startup like Pinterest where you’re growing – like next week 20 new people are going to start. And those 20 people they don’t know anything about that. And so you have to continue to make sure that people understand – they’re going to come in and say like hey how come we don’t do this and it’s easy just to say like well we tried it and it didn’t work. Or well it’s not going to work. Or, that’s easy. The harder bit is like actually here’s what we’ve learned and here’s why we think that may not be the best way to proceed or move forward and so – that’s a real – like it’s an ongoing challenge of any like company that’s growing quickly, but particularly it hurts researchers because the artifact of what you’ve done is knowledge and knowledge you either have to read it, or communicate it. And so figuring out how to like maybe put it in a space or you create a wiki, but then you’ve still got to get people to go there. It’s a real challenge I think for any researcher at a fast growing company. Steve: Right. There’s a lot of talk about – you said at scale earlier in this conversation. That you know dealing with insights at scale is a challenge as companies grow and so on. But you’re talking about sort of insights, like insights in the tornado, or insights when the sands are shifting. Gabe: Yeah. Steve: You can have a great meeting – you can – you know all the good things we do in research. We talk to our stakeholders. We bring them along in the process. We bring them out to meet with people. We involve them in synthesis. But over that – say that’s an 8 week process, that team has changed and then two weeks later, after you’ve had some really great meeting to make some decisions and have a workshop and do whatever things that we do. Two weeks after that there’s a bunch of new people and two weeks after that there’s a bunch of new people and two weeks after that. Gabe: Yeah and also the actual, you know the audience. Like perhaps your insights are stale. The audience has changed significantly, right. Or perhaps the technology, right. The technology itself, there’s like a new innovation that you know we have a new team coming in that’s been acquired and they have like a whole new technology they’re introducing that can really change the context. And so again this notion of in a startup land a research has to again believe – act like – know that it’s a marathon because there’s so much changing and you’re going to have to really stay abreast of all of that and check your own assumptions, check your previous recommendations constantly so that you are really providing the best information you can at that time. Steve: That’s a really good point and I think that’s sort of the researcher arc, right. You come into a situation. You can tell that everybody’s hypotheses are full of crap. And you’re like okay I don’t know what the truth is, but I know that you’re wrong and we’re going to go learn and you come back with a truth, or a set of truths or a set of things that approximate truths and that you advocate for, but that also – then you could risk becoming part of the problem because now you can kind of clench on those things that you discovered through your own sweat and your own effort in the field, but what you’re saying is those things may have a short shelf life given the changing space that a company like Pinterest is involved in and that you have to let go of those things which is sort of counter to the you’ve got to go – the marathon is kind of about persuasion and advocacy and influence and getting out there and giving that same presentation 20 times to different audiences at the same time as also be willing to let go of it. Gabe: Yeah, and I think – so there are – I think that is a challenge – a real challenge. I think there are some ways to guard against it, or some strategies. So one strategy at Pinterest, I – my vision for the research team was an interdisciplinary team. Trying to put together both qualitative user experience researchers, of which there are different kinds of qualitative user experience researchers. They’re not all the same. A quantitative user experience research group, market research. Altogether. There’s an economy here in terms of like actually getting things done, but there’s actually, like what it does nicely is it brings together different data points to build the knowledge that we’re learning together. Steve: Is that knitting? Gabe: That’s knitting within our own research team right. I think a lot of companies you can have, like those groups can get siloed early on and then you have that moment years later where you’re like well we’re working on the same thing, what’s going on? Or we’re competitive with each other. Trying to put those disciplines together and then what allows – it’s like so work that I’ve done in the past that I still think is relevant, well if it’s still relevant we’re probably still working on it and now we work on it with a perspective of multiple methods. So mixed methods. This is a way to sort of take an idea and really, truly understand it deeply because if it’s – we think it’s – we hypothesize something from the qualitative research then we can hand it off to a quantitative researcher who can then quantify that at scale of which you might learn something that is – you think there’s a correlation between a macro question and a whole bunch of smaller questions. Well if you validate that relationship then you can take that macro question and you can put it in all of your say brand equity trackers for market research and all your markets and then you can measure it there and now what you’re doing it you’re building this interlaced qual and quant sort of lineage of knowledge that over time can become very robust and then you can say now that you’ve validated it at scale and you’ve used multiple methods and you’re monitoring it in each market, well now you can say we know this is a real thing, we’re trying to change it. We can continue to measure it over time with a brand equity tracker and market research to see like are we doing better, are we the same or are we doing worse? That’s super exciting and it’s a way to guard against your own bias, like this is true. Maybe when you get to the step of handing off to a quant person – like no this is not true, it’s scale. Or it’s different – it’s often probably true, but it might be different than what you thought it was. So there’s actually three things going on here, not just one. Or you find out when you pass it over to a market researcher who does brand equity or some sort of large survey, they’re like oh this is true for, you know, Western European countries. It’s not true of Pacific Rim countries. And so – again, bringing in more methods, building out – if it’s an important question, building out different levels of understanding can be really robust and guard against your own bias towards – but I said this before and it was true. It’s still true. Steve: So in terms of this multidisciplinarity and you’ve kind of said this how you sought to build the team, maybe you can describe the team more specifically and where are you at? Where’s the team headed? Gabe: Yeah. So we have three qualitative user experience researchers. We have Altay Sendil who comes from IDEO. He has an industrial design background. He was at IDEO as a researcher forever. He brings like a really strong perspective on ethnography and really generative work. We have Larkin Brown who comes – she was previously at Google and she’s like – to me like a real digital UX person and she has a lot of areas of focus. She’s also really into – she serves as a stylist within Pinterest and helps with brand styling and also communication to press around fashion. And then we have Cassandra Rowe who’s – she worked at, previously at Intuit, but really did a lot of international research at Netflix. She was part of their early growth on the qual side. So we have an expert in sort of formative, an expert in digital, an expert in international. Then we have one quantitative UX researcher named Jolie Martin and she is really looking at the intersection of attitudes and behavior for Pinterest. So that’s experiments and – AB experiments and surveys, to truly really understand the quantities and quality of experience. And then we have Paul Pattishall who is on the consumer side doing market research for us – things like brand equity trackers, product surveys, NPS, category deep dives. Like what are the most common interests in France right now for people who aren’t using Pinterest so we might approach them with relevant content. And then recently had Julia Kirkpatrick who joined, who’s doing market research for the business side of the company – Partners Products is what we call it. That’s the people, that’s the disciplines. Again, the important bit is that we’re all working together. We’ve hired people who, they work with a team in an embedded type way, but they also know when – when the team has a question that’s outside their skill set to go to a practitioner within the team to get help and support. We also, you know we’re still – so that’s six people now. We’re adding two more people this year. We’re looking for another quantitative user experience researcher because I think one person is not enough. You have to have two to really be a team so you can play off of each other. And a UX or qualitative researcher to focus on our business or Partners Products. So these are things like the web analytics if you own a domain, or advertising on Pinterest which is something that’s a new area for Pinterest where we can do some really innovative things because of the visual display elements of Pinterest. So we’re adding folks. We’re looking for folks, but the important bit is all those disciplines working together – we’re one team, but you could think of it as three separate teams. But because we’re connected there’s a lot of collaboration and interaction amongst the team members which I think is really – it’s an exciting environment for me as a researcher because people know a lot and you learn a lot and you can pick up a lot of tips and tricks along the way. Steve: So just thinking to wrap up, as you describe the team and you talked before about – I used the word teacher and you kind of shied away from that term. You said you think of yourself as more of a collaborator and given the nature of how that team is structured and kind of what you have created as a collaborative culture and how you’ve inherited kind of – you’re participating in the larger Pinterest culture – I agree with collaborator. And this is just all a preamble to a leading question – the trouble with interviewing somebody that you know a little bit is that there are certain answers you’re trying to draw out of them. So I’m going to try a few different ways. What – you know in terms of your role as a manager, if it’s not teacher, what would the people on the team, what nouns or adjectives would they use to kind of characterize your managerial jim-jam? Gabe: That’s a good question. I mean this is my first time managing. So I think my jim-jam is under development. What I strive for is one where they feel like supported. That they – I will – if people ask me for feedback I will give them deep feedback on whatever it is, their deliverable they’re creating because I know in this environment you might not get deep feedback from other people and I think if you’re doing work you deserve to get that feedback. So supportive, probably into the – like I care definitely a lot about research and what we’re doing. So I talk a lot about research. At lunch I’m like the person who’s like, answers the question about what they did this weekend and then follows-up with the like hey let’s talk about this project that’s happening. And so I’m – they would probably say I’m pretty focused on research. So supportive, into the research. I definitely want to empower them to make decisions. So we have a model where they’re working with teams and they are the ones that need to develop the roadmap with their teams. They need to make the calls. I’m there as support. I can provide context. I do a lot of like I – like showcase their work and constantly talk about their work because that’s part of my job is to like hey did you know that Cassandra just did this in Germany with that team, the activation team. You should be aware of that work. That’s a huge part of what I do, connecting the dots at an organizational level. So supporting, you know advocating. Nerding out on research. Steve: What’s your superpower as a researcher or as a manager? Gabe: I don’t – that’s a good question. I tend to like shy away from superpowers because I think it’s about like training and practice and whatnot. As a manager, well I don’t think I’m there yet. Like I haven’t done this long enough to really know that about myself. And I have a great team. Like I think I have a lot to learn and a huge part about managing probably is like managing through hard times and I probably need some more of those hard times before I really can assess. Steve: Careful what you wish for! Gabe: I know! As a researcher, I mean I care a lot about the work that I do. I think that’s like a double edged sword sometimes because sometimes you have to care a little less. But I think I – it’s a good question. I really – the times I’ve thought – I’ve thought to myself well like I’m pretty good at this thing, but then I’m like that’s just an illusion, right. That’s like I don’t have a special power. I just have a set of practices and things that I try to learn about and do. And so I’m hesitant to answer that because maybe there are no superpowers. Maybe it’s just about hard work and just caring about the details. I can tell you what my weakness is which is I don’t tend to go to conferences enough or get out and socialize enough in the research team. So when someone comes to visit me, to talk about research, or who’s a researcher, that’s huge for me ‘cuz that’s how I try to like understand what’s happening in the world. That’s a moment for me. That’s like my conference. That’s my getting out and interacting with other folks around research and so – ‘cuz I’m pretty heads down. I’m pretty in the trenches, right. And so that’s why I definitely celebrate those moments and I really appreciate when researchers come visit me. I have sometimes students come by who want to talk about their future career or get more information. I tend to like accept more one on one conversations than really approach like group activities, as a way to just get a sense and reflect on what’s going on in the field. Steve: Is there anything that we didn’t cover in our conversation today? I know we could go and go. Gabe: We could. I have a lot of things to cover, but I’ll leave it for another time. I do have some questions for you though. Steve: I don’t even have to ask that! Gabe: No, I’m like ready to go. So my first question is… Steve: Oh your first question! Gabe: My first question is like what’s the biggest surprise – what’s been most surprising to you in terms of the Dollars for Donuts interviews you’ve done? What stands out as surprising that you didn’t expect before you started this? Steve: There’s probably surprises in terms of just having these conversations and there’s probably some surprises about just having this as a thing that I put out in the world. But I also – it seems I push back on the notion of surprise. I think I wrote about this in my book even where I used the phrase bland curiosity. ‘Cuz as researchers you get asked the what’s the biggest surprise question all the time and I feel like well that supposes that you entered into it with something and then like a bucket of water appeared and splashed you and you went oh… Gabe: Okay, this is good. Let me reframe then. What has been the most unexpected theme across these interviews? Steve: I haven’t synthesized them yet. I mean part of – and you know this is sort of the discipline of research like – and you can do it different ways – you can kind of just go and throw yourself into it and then occasionally you’ll be like oh we haven’t seen any themes except blah, blah, blah, blah. Oh that’s a theme. And so I’m just going to keep thwarting you Gabe. I mean sometimes I resist reflecting and just sort of let it be what it’s going to be. So I mean there are things that come up, but also it’s because I ask about them and then – you know how teams form? What people’s career arcs are? These are sort of the topics that I’ve chosen to talk about, so I don’t know that they’re surprising themes. There’s something that we didn’t get into a lot – there sort of a structural issue that’s emerged from the beginning which is – which I remember this being like a design topic from the 90s and probably goes back before then – where do these people sit? Do they sit with their – are they centralized or are they embedded? So it comes up a lot and I think it’s a really interesting sort of managing – what I like is that no one says this is the right way to do it. People have said like this is what we’re experimenting with now and here’s the tradeoff. So I guess that’s a thing that I would never have thought to ask about and that’s also because I don’t have a job inside a company that has a bunch of research people, so it’s outside my own experience. So I’ve learned about that as a topic and it seems to be one that when I talk to people they say like oh yeah, that discussion and this episode was interesting and we’re thinking about how we’re going to do that. Gabe: Yeah I think that – I mean I have that as a listener all the time. How are they doing it? Why are they doing it that way? Absolutely. Okay, speaking of being a listener, my second question is will there be a Season 3 of Dollars to Donuts? (laughs) Steve: We’re going to do it in Oculus. We’re under negotiations right now. Podcasting is over. Maybe it should be – maybe they should be something like – something that’s only – they’re going to be 4 seconds long and then they’re going to disappear. Gabe: I won’t put you on the spot. This is a more serious question. As a young lad growing up in the cold tundra of Canada during the 70s and 80s, who appreciated donuts, there were 2 major chains. Which was your preferred chain – Country Style Donuts or Tim Horton’s? Steve: Well I’m sorry. We grew up in the same town. Gabe: We did. Steve: And I’m older enough than you that I remember Millionaire Donuts. Do you remember Millionaire Donuts? Gabe: No, I don’t. But it was the donut of choice? Steve: Oh I don’t know. I mean Tim Horton’s was kind of the place, ‘cuz Tim Horton’s used to make their donuts in the store. Maybe you’re the one that told me this. They don’t do that anymore. I even went for a job interview at Tim Horton’s as the shift – I didn’t get the job – it was like shift work. You had to make the donuts, like 3 shifts a day. They had these big racks in the back where they would – it was an interesting sort of – I mean I didn’t get to do a lot of research, but I saw how those were made, which maybe everyone knows this, but they like have all the sort of naked donuts there and they like kind of pour over like a big vat of chocolate goo and that’s how they get that sort of drippy thing out their – they just pour it over on these racks. Gabe: So I don’t think we can really reach a higher point then naked donuts as part of our conversation today, so it was like… Steve: Are you – not only are you taking over the interview, now you’re wrapping up. Gabe: How can you beat naked donuts? Steve: Well there’s a visual that no one needs… Alright, I think that is our – I’m going to take your cue… Gabe: Thank you. Steve: … and wrap us up. Thanks Gabe, this was great. Gabe: Awesome. Thank you.
March 1, 2016
In this episode I chat with Elizabeth Kell, the Senior Director of User Research at Comcast. We talk about the growth of Comcast’s user research practice, essential soft skills for research candidates, and putting a human face on the people that use your products. If you don’t have a natural curiosity then I can’t teach you to be a researcher. It has to be ingrained in you. To be a researcher is to want to understand. I was that kid that was 5 years old, like why, why, why, why, why? And you know I’m still like that as an adult. I need to know how things work. So I look for that [when hiring]. – Elizabeth Kell Show Links Elizabeth Kell Comcast Comcast Center Fancast X1 MyAccount app IIT Institute of Design Steelcase Livia Labate Tom Loretan Alan Wurtzel Practical Empathy by Indi Young Hands on a Hardbody Charlie Herrin Tipping from questions into stories Over-the-top Kano analysis Prentice-Hall Information Architects by Richard Saul Wurman School of Visual Arts Erik Spiekermann and the Berlin subway map Joyce Bromberg World IA Day Soft Skills Are Hard Follow Dollars to Donuts on Twitter (and Stitcher) and take a moment to leave a review on iTunes. Transcript Steve Portigal: Alright Liz, well thank you. Thanks for being part of the podcast. Liz Kell: Thanks, Steve. Steve: So I’m going to ask you the question that I always ask everybody at the start of almost every interview about any topic I think it is which is just to give a sort of basic intro to start us off. Liz: Sure. So my name is Liz Kell. I’m Senior Director of User Research here at Comcast in our headquarters here in Philadelphia. I run a team of 11, a diverse team of user researchers and I’ve been doing this for about 8 years. Steve: Another question that I often ask was so people in the US will know the Comcast name, other people might not. So – and probably there’s some – I guess you can explain Comcast and we’ll learn something about the company in terms of how you describe it. So with that long preamble, like what is Comcast? Liz: What is Comcast for the folks who don’t know or don’t have the services at home? Comcast is the largest cable Internet service provider in the United States. We have over I want to say 22 million video subscribers and 23 million high speed Internet subscribers. I may be slightly off on those numbers so hopefully nobody fact checks me. We also provide digital voice service to the home, home security services and yeah, those are our four main business areas right now. And so that’s what we do. Steve: So within your group’s purview I guess do you get involved in all those different areas of the business? Liz: We’re involved in all areas of the business and in the layers beneath too, so all the service layers that go into supporting all four of those products and services that we deliver. So we look at the customer service – the whole customer end to end experience and that includes everything from you know the sales and the marketing and going in and observing what’s going on in the call centers, to looking at the installation, riding along with the agents, watching people try to self install kits. Watching people try to get self-help or call into the call center and troubleshoot and things like that. So in addition to evaluating the product we’re also evaluating the services. Steve: When you say product, for a company like Comcast, what’s a product? Liz: Product – I think of the product as like the thing. Steve: Like a physical thing? Liz: Sort of. Yeah, I mean – I mean it’s a bunch of electrons right, but the television is something that comes to this big screen on your TV and you have the interface that you interact with. You have the set top box that you interact with. You have the remote controls that you interact with. So that’s like kind of thing and we do a lot of evaluating and trying to improve those things. For high speed Internet there’s less to interactive with. That was one of our big insights when we were doing some research on the high speed Internet product. But really how do people interact with that physical gateway which is you modem and router in one. With home security there’s a lot of interesting things to explore around like where do people decide to put the different sensors or what kind of different pieces of home automation are they adopting and why and how are they using them and how are they controlling them, either through mobile or web interfaces. So there’s a lot of like the thing I would call it, like a tangible thing we’re evaluating as well. Steve: So that’s hardware and software? Liz: Both, yeah. Steve: So these questions are naïve ‘cuz I don’t necessarily have the language for them, but so there’s hardware and software and services, but the services I feel like you’re describing sort of are more customer support services. Are there other kinds of services that Comcast delivers that you’re examining or studying, trying to improve, that aren’t about sort of managing my success with the hardware and software I guess? Liz: Really about self-service and getting help and being able to get the support you need for your things. I keep calling them things now. I got that word from you. Steve: So Comcast is kind of a hardware and software company and there’s a service layer to enable hardware and software. I’m sort of thinking of someone that’s like more of – a company that’s a services company like has services as a thing that you would consume, but you’d only consume Comcast services in order to get to a point where you’re having the hardware and software, the thing. Liz: Yeah. I mean people subscribe to our services and they get something for it which I consider – I call product. You know we’re part of a product organization. So people subscribe to our services so it’s really I think about figuring out how to get the help and support you need with the services and then being able to use the products. So we look at both. I don’t know if I’m answering the question or not. Steve: I think we’re both totally confused now. Let’s see if we can get through this without using the word product or service or thing again. So you started off too by describing the size of your team. Can we go back in time and talk about the history of your role? Sort of the history of user research and insights at Comcast. Liz: Sure, of course. So I said I’ve been here, I’m coming up on 8 years. I was actually hired to be the first user researcher on staff when we moved into – we have a new headquarters here in Philly which isn’t really new anymore. We moved in almost 8 years ago. It was a month after I started and when we moved in space was actually built out to have an internal usability lab which was something that the company didn’t have before. So looking to bring some of this capability in-house rather than outsourcing as it’s been done in the past. So I started out as a team of one and when I started in 2008 we really – it was supporting a couple of different websites. So which was originally the portal for our customers, high speed Internet customers to get their email and it was similar to like Yahoo or AOL. And then which was really one of the first destinations to go and watch you know full episodes or full movies online. So it was kind of an exciting space to be in. You know the frontier of like watching TV online. You know in some ways it seems like yesterday and some ways it seems like ancient history. But since then you know just every year something new just kept coming into the fold. So that was 2008. In 2009 we were working more on iPhone apps and the iPad came out. Then we started taking over the TV interface and you saw the result of that now. If anyone has our X1 system that’s the latest TV UI that we worked on for many years. We also took on hardware, so looking at the design of the remote controls, set top boxes, all kinds of things like that. Starting then to move into services, looking at the online, the acquisitions, the self service, the MyAccount app. So the scope of what we touch just keeps growing. You know looking at employee tools and some of the software that our agents use out in the field or helping with an engagement to try to evaluate and improve the tools that our techs use out in the field. So it’s gotten to the point where we almost touch like anything that someone needs to interact with in the company which is really exciting. And I’ve grown this from a team of 1 to a team of we’re up to 11 now in 8 years. So I think that shows really great commitment of the company to how important it is to understand your users and design for them. Steve: So can you go back along that timeline. You’re sort of framing it around the things that Comcast was doing and then the way that you and your team were able to contribute, but something was also happening internally, I’m hypothesizing, that started to say hey this capability, these people that we have can help drive good design or good outcomes for customers. Are there sort of key moments or kind of transitional points that you can think about where wow we’re – yes we can help with that. Yes we’re going to start doing that. How did that come about? Liz: That’s a great question. Well my background actually was more in field research and anthropology. I did the design research program at IIT Institute of Design. So I was more into field research, worked at Steelcase and that’s what they focused on before I came here. But when I got here you know it was really about we need to start up this usability practice and I think the big wins in the beginning were – there was definitely a hunger and appetite for this and the woman who was leading the IA group at the time, Livia Labate along with Tom Loretan who’s my VP of UX. He’s been my boss for the whole time I’ve been here. They had an appetite for this already. So I came in with two champions ready to support me and it was pretty easy. There was just a hunger and people who wanted testing and they wanted to see customers interacting with their interfaces. It was really easy to just get up and running and show some immediate results and to show some immediate impact on the UI. So with that I think the legend of my team started to grow, first within the UX group with designers coming to me saying hey we want to test this, hey we’re working on this, into you know our close knit neighbors. We were underneath the product organization and we still are. So some of the close knit neighbors in product started coming to us and saying hey I want to learn more about this or I want to learn more about that. And then they would invite someone to a readout and next thing you know someone for the business unit would be like hey can we use your group to learn more about our users in this way. So I think it was kind of a bit of an organic growth. And thinking about key moments, that’s a great question. You know we’ve had a few of them. I think you know we did a big ethnographic study back in 2011 that got a lot of attention from our GM for the VBU which is the Video Business Unit and then some of our market research partners and then they started spreading the word and having us put our presentation up in front of other audiences within the company. And one of the people I got connected to – this is kind of a cool story – is Alan Wurtzel who’s the President of Research for NBC Universal and has been for a long time. And of course you know we own NBC Universal, we’re all the same family. And I got to build a relationship with him and got to speak at his offsite and present on some of the really cool research we’ve done and some awesome videos of people navigating our interface and talking about how passionate they are about TV and it led to a relationship there. So it’s just been growing organically, but it’s really just about finding your champions and then spreading the word, and letting them spread the word and taking those opportunities every single time it’s presented to you, to just go get in front of more people and share what you do and how exciting it is. I think one of the keys too is you know when I present I try to bring so much enthusiasm about the subject. To really go in there – I love to be like animated up in the front of the room. I love to show videos and I put a lot of effort and I challenge my team to really be creative with their videos and make sure that they’re really engaging. I think that videos often sometimes have a really awesome impact and help humanize the experience. So I think some of that’s the turning point as well. Just being able to share your work in a really engaging way I think makes a big difference. Steve: I’m going to try a leading question. There’s a little bit of – I mean a good leader is humble about their accomplishments and I think there’s a certain humility in your answer in that you talked a lot about other people’s interest in your work kind of drove the growth of the demand, but I want to push you a little bit and can you talk about things that were – like work is not just work. There’s work that’s relevant. There’s work that points to opportunities. There’s work that connects. You were able to – in order to drive that kind of increasing demand, it’s not just about being animated. Like it’s all the things that you said, but there’s something, I bet, that’s in the content of what you and your team have been producing and I wonder – again I’m totally leading you, but I wonder if you can, with all due humility, reflect on – what about the content of the work that you’ve done that has driven that organic growth? Liz: So with all due humility it’s really about I always say you know our teams’ goal is to really bring – the word empathy doesn’t mean like you feel sorry for someone or you know – it’s really about – I think Indi Young just wrote the book about empathy and I was reading it and I was like that’s exactly what it is. It’s like being able to put yourself in someone else’s shoes, but not necessarily understand it because it’s not your world. And I’ve had stakeholders say to me after a readout, or say to members of my team that you know, not necessarily that we learned anything new or that we didn’t know, but you really humanized it. You really illustrated it in a way that was so impactful that we didn’t realize – you know it caused this kind of emotion in our customers. Or you know the problem was – this was the problem or this wasn’t the problem. Like this is really the heart of it. So it’s about being able to illustrate and bring empathy and just humanize the experience for folks who really aren’t out there necessarily interacting with our customers or users every day like we are. I think that’s just – that’s where the passion comes from. I don’t know if I’m answering your question? Steve: You’re answering it and it’s a wonderful answer. You know I’ve had this experience a lot in years of doing research with – it’s sort of the classic response when you share something that you’ve learned is that we already knew that response. You just kind of flipped it around for me where it’s not – because sometimes that’s sort of confrontational and my response is to get defensive. Well, are you sure you knew it? And I think what you’re sort of talking about – I mean kind of a cultural, organizational value in doing this work is as you said to humanize. And it’s not about oh my God did you know that this thing is going on because that might not – you might not be able to surprise anyone with that, but to bring it back in a way that humanizes. And I think what I’m drawing from what you’re saying too is that’s a powerful way to drive change or to help people take action based on what you rediscover or represent. Liz: Yeah, exactly. It was kind of an epiphany when someone said that to me you know ‘cuz sometimes we would. We’d be like oh well I don’t know if we’re really like bringing any new insights here. They’re things we already knew, but when someone said you know it’s not about necessarily bringing us anything new – and I don’t want to say that we don’t bring new insights because we bring tons of them, I know we do and we’ve affected a lot of change – but there are some things that just the company knows and they know they need to change, but nobody’s really ever told it in the type of storytelling manner that we bring to the table where like the end to end experience with videos of people, showing them their passion, their joy or their frustration and really talking about it in an eloquent way, or even things like when we go in the homes you know we often try to get multiple family members in the interview because that’s when you can really get the genuine emotion and confusion and frustration out. You know I’ve shown videos of husbands and wives getting into arguments over like how to actually use an interface. That tells a way better story then I could ever tell. Putting together a presentation that shows like for example a matrix of tasks and who succeeded and who failed. It’s a far more powerful story to say like hey this is really confusing and I’m going to show you a video of a husband and wife arguing about it because one understands it and the other one doesn’t. Steve: This makes me think about documentary films a little bit. There’s documentary films that I like and ones that I sort of almost tolerate. I will bring this back to what you just said. You know there are those documentary films where it’s about a topic and the topic is so amazing that it seems like what the filmmaker has done is sort of open the lens and sort of be there and then kind of chop it up in a way that follows a linear narrative and you’re like I can’t believe these people are trying to win a truck by standing on it – Hands on a Hardbody was a classic documentary, like you just saw people trying to – I don’t know if you know this one. It’s a contest to win a truck and you have to have your hand on it for days and days and days. This amazing human drama emerges from this very simple thing. And not to minimize the art of the filmmaker there, but then there are other documentaries that just do – I don’t know, just wildly creative things and sort of synthesize and pull out of something, some story you never could have ever have found, that like they’ve told that story. And I enjoy them both, but I like to sort of reflect on sort of what’s the hand of the filmmaker. I know we’re not – there is an art to what we do, but we’re not making art. So just to loop it all the way back around to what you’re talking about, it is powerful to bring those stories in and kind of impact people that way. Again, I guess I’m going to lead by offering my point of view. I would hate for someone to listen to this and feel like what the field researcher does is just like point the lens at people and sort of – that we’re just scooping. We’re not digging or we’re not synthesizing. So when you come back with those videos of the husband and wife arguing, like you didn’t just – you weren’t just out shooting video, you were doing something else. Liz: Obviously – no, we’re doing a lot more, we’re probing in a really meaningful way and I actually – I’ve given talks here at Comcast to groups of product managers. You know last year Charlie Herrin who is our – now our head of Customer Experience and a whole new group, had asked me to set up a training program for all the product managers so that they could come, ride along, or be empowered to do their own listening in the call centers, or ride along with agents, but to really understand like what is the art of what we do and how do you listen to customers. I talk about things like how to build rapport, how you want to go in. You don’t want to necessarily go in with an agenda. You want to ask open ended questions. Don’t be leading. But it’s so important to build rapport because once you build a comfort level with the people who you’re visiting – you’re visiting them in their homes – it’s amazing how they’ll start to open with you and it gets to the point where – I think I ripped this from your book, but I say this anyway, it does have a tipping point from where it’s question and answer to question to story to just you say something and they just start talking. And that’s the point that we really always strive to get to and that takes experience, that takes practice and I think that’s really important. It’s the art of – it’s the art of having a conversation because that’s really what we’re going in there to do. And sometimes you don’t know where a participant is going to take you and that’s okay sometimes. You know you can get some of the most richest or most interesting insights when the participant takes you down the path that’s most interesting to them and then you can bring it back to some of the things that you want to learn about, but I really think that it’s so important to be open minded and let the participant talk to you and show you what’s most important to them. Steve: There’s an interesting tension – I don’t know that either of us can resolve it, or at least not in this conversation – between that territory with no map – I don’t know, like the exploring to get to what you don’t know you’re going to get to. At the same time the result is things that you mostly knew or knew to some extent, but maybe not with this amount of emotion or this amount of richness. Liz: I’m thinking of a good example that I can share. When we first got into the home security arena, it was maybe 3 or 4 years ago, we went to go do a field study with some of our early customers and we were just hoping to learn. We were still mostly UX Product. We wanted to see how are they using the system, what are the different ways they’re using the system – you know to improve it or to be inspired for new features or ways to improve the UI of some of the apps or the touchscreen and things like that. And what we found actually was a direct correlation between you know how engaged people were with actually using their home security system and how well their installation and first time use of the system was. And that wasn’t something we were looking for at all. That was something that started to come out naturally as my lead researcher was in the field and she came back with this like amazing story and this amazing end to end process map. It was like well, it was like 6 steps and we went in the home with this expectation of studying what was step 6 and what we learned was X, Y and Z about steps 1 through 5 and how it affects step 6. You know and step 1 is just becoming aware of the system. Step 2 is calling in to order it. Step 3 is getting it installed. You know all these things that happened along the way and that was because the participants wanted to talk about that and we found it necessary to let them talk about that to understand why they were or were not using the system in a certain way. So that’s an example I think of this sort of non direct, or the power of being non directed. Steve: So that’s an example where using this non directed approach helps you find something that you didn’t know. But you’ve also talked about examples where there’s a great power in the way that you can bring back stories that are not necessarily surprising as you said, but they’re brought back in a way. I’m just wondering, does that approach of – now I’m messing up your words – but sort of a non directed, rapport building. Letting the participant talk about what they want to talk about. Does that also bring back things that you knew, but in that powerful way that’s helping you drive change? Liz: Yeah. So one of the things that we hear a lot about, we talk a lot about obviously in the world of cable and television is live television. And how attached are people to live TV? I guess in our own little rarefied world we tend to think that people are really moving towards cord cutting and consuming everything on Netflix and over the top HBO and that kind of stuff, yet we’re going in the homes and we’re seeing this incredible attachment still to live television and we’re still seeing that in a lot of the numbers too and a lot of the Nielsen surveys and things like that. So we know it’s true and it seems like there’s this dichotomy and people want to understand it better. So we can go in the homes and you find that people – when you let them speak in a non directed way all of a sudden they start talking about like you know why they’re attached to live TV and it has a lot to do with like a lean back experience vs. maybe a lean forward experience. You know we think of live TV very much as like oh it’s still that classic – you know Alan Wurtzel would call it television the way God intended you to watch it. That’s what he says. Like everyone sits down at 8 o’clock to watch the Cosby Show on Thursday nights. But really the attachment to live TV is so much into sports, into the low commitment, into the serendipity, into I just want to flip around and see what’s on for the next 20 minutes while I decompress, like that kind of thing. And I think we tend to overlook that. So to be able to bring back videos of people showing, like they all have different strategies for browsing the grid, the traditional grid, but how that’s happening, to really illustrate that, to explain like why there’s still an attachment to it, but letting people show us through the context of their own stories and letting us take us down whatever path they want is really interesting and really meaningful. Steve: I feel like you just hit on what the power is of those humanizing stories. They’re not necessarily surprises, but they can help people reprioritize. We knew that, we weren’t really paying attention to it. By humanizing it you’re helping people to make decisions and understand that something’s really important. That it doesn’t have to be a surprise that blows your hair back. It can be… Liz: Right. It’s almost like it is really important so we definitely want to still make sure we have a strong way to help people you know browse what’s on live. And it’s not to say we shouldn’t be looking at like alternative ways to do that in the future. We know that live TV is still important. We know this from numbers. We know this anecdotally even though anecdotally there’s also the sense of there’s still cord-cutting, but being able to go in the homes and ask people to just like hey, take me through your day in TV and having them illustrate that sort of brings home this notion that you know some of these behaviors are still happening and we knew they were still happening and here’s a way to illustrate them and help people understand why it’s still so important and so meaningful to the user. Steve: What are some of the challenges that – maybe it’s less about your team, but just the kind of work – I mean you know user research as a thing. You know in an organization like yours where you’re having these successes and you’re having this kind of impact, you know what’s – how do you characterize the types of challenge that you all are looking to address? Liz: I mean some of the challenges that we’ve been looking to address lately, at least with my team, is we’ve historically been a qualitative research team you know and qualitative research is intrinsically linked to traditional market research techniques. So there’s a lot of familiarity with many stakeholders outside of UX and especially outside of UX and product with those methods. So really trying to bring in that awareness of how important it is to – you know we’re talking a lot about talking to people, but really the heart and soul of what we do is we come in and we build rapport and then we get them showing us and that’s really what usability is too – you bring people in, but you watch them use your system. When we go in their homes and we try to get them to just recreate their lives and watch them. So really you know trying to illustrate like different approaches to qualitative research and trying to help explain what those different approaches were, that’s been one challenge that I think we’re starting to really successfully illustrate. And then the other challenge too is like I said we’ve always traditionally been a qualitative research group and you know everybody loves numbers, everybody loves analytic surveys and stuff so we’ve started to – I have a quantitative researcher on my team now and we’re starting to bring in some of the quant to compliment the qualitative research we’re doing to emphasize some of the points that we’re making, sort of establish them – some credibility to them at scale. Steve: And what’s the difference between what the quantitative research on your team does vs. other kinds of quantitative research that Comcast is doing? Liz: We’re trying to do it from a product development approach. So you know one of the techniques that we just started implementing, or at least we’ve started piloting is Kano analysis which is surveying in a specific way, asking people about features and their expectations, but not do you like it or are you satisfied with it, but do you expect it to be there? If it’s not there are you upset, are you indifferent? There’s a formula that sort of segments things I believe into like three outputs – the must haves, the indifferents and then the delighters. You know we’re always getting – it kind of came because we’re always being asked about what will delight our customers? What new features should we be building to delight people? It’s a great language that really speaks to the company and one of the interesting things we’ve found too is that delighters decay into must haves over time so things – you know for example about the television interface that we would hear were delighters 5 years ago. Anecdotally we kind of see how they’ve decayed into must haves as we started to do some surveying. Like these must haves are things that people just expect to be there as a baseline, almost like a neutral. So anyway, having the – I think having some numbers behind it, but not in a traditional way, but in a way that reframes it, but again still amplifies things that we’re hearing from other types of surveys and other groups in the company I think is really, really powerful. Steve: It makes me think about car features and how they seem – that industry seems to have sort of figured out – you know things come in at the ultra high end and then you now you can rent a Toyota Corolla at the airport, if they make Corollas or whatever, and it has a backup camera. Whether that’s driven by legislation, whatever, all these features seem to trickle down. Liz: That’s a great example. Steve: And is that the same – do you think that’s the same – I don’t know what’s going on in the car industry, if that’s about economics or is that about you know… Liz: No, but that’s a great example. Like right now backup cameras, it’s a delighter. People who have them love it. But it’s not standard and there are still a lot of people out there driving cars that don’t have them, but I would predict you know in 5 years maybe that’s just going to be a must have and an expectation, especially if they’re putting it in Corollas. So it seems like that’s where you see the decay from delighter to must have when it’s no longer necessarily a luxury or something high end. It’s something that’s expected in any make and model. Steve: It seems like it’s easy to see that framework in the car industry and probably harder in the things that you work on. Liz: Right. Steve: Because there’s just not… Liz: It’s probably more practical stuff in the things we work on, like the number of tuners that you can use at once to record shows. It used to be like so exciting, I have a DVR. Like oh my God I can watch one show and record the other at the same time. Now people have four tuners and they can record four shows and watch one live at the same – or five tuners. You need 5 tuners to do that. And so things like that have become more than must haves. Same thing with the Internet, like the must haves. The speeds keep going up and how much you’re able to download in a certain amount of time. Steve: Is the tuner thing about multiple viewers in a household? Liz: Sometimes, yes. Sometimes. Or sometimes it’s about like a TV passionate – you know there are people who are super attached to TV and like to record a lot of things. They like to stockpile stuff. So there’s a lot of different behaviors out there that drive that. Steve: Right. I’m doing that thing that people do where like 5 shows at once, that seems like a lot. What’s wrong with that person? But your answer is of course the appropriate, empathetic one. There’s use cases that you support and I’m just doing that thing that people do. Liz: Exactly. Steve: And that’s right. Maybe there’s a… Liz: How many people use five tuners at once? I actually – I’m sure we have analytics on that, but it’s even more the perception. Steve: Ah, right. Liz: I think. Steve: Right. I bet the – you don’t have to answer this, but I bet the incremental cost to add a third tuner and a fourth tuner is probably low and so right now you have more capability. I mean it goes back to what you’re saying about things that are delighters vs. must haves like – and I’ve seen this in studies that I’ve done unrelated to your business area where you know more capability then you think that you could possibly is like that’s very comforting for people. Like, I’m set. Liz: A great example – a great example within the DVR space maybe is the, like remote DVR. So I remember when I was working on the first app for mobile DVR – oh my gosh I want to say that was like 2009 or so. You know where it was like super exciting to like be able to be like oh my God I’m out and I forget to set the DVR and now I can log in and actually set my DVR. That’s just – you know you can do that now. It’s still a delighter. So I want to say that I don’t know that’s it decayed to a must have, but I would say for – it’s getting close. Steve: Interesting. Okay, let’s shift gears a little bit. You’ve talked about your team and some of the people on your team. Can you give kind of a 1000-foot view of the kinds of people, the kinds of skills? Like what is – what have you built and kind of who have you looked for and what’s the makeup I guess? Liz: So like any UX research team we come from a variety of different backgrounds. You know the 31 flavors or whatever you want to call it. So my background is in physics. I’ve got people with backgrounds in information systems, anthropology. You know we’ve got someone from CMU (Carnegie Mellon’s graduate program). I’ve got people from – a PhD at Temple. So it’s really definitely a definite mix. And I think everyone of the team sort of has an area of strength. You know I’ve got someone on the team who’s just this absolutely amazing, almost like anthropologist – go out in the field – you know some of the things we were talking about before, about just being non directed and building rapport. You know that’s a real strength. Someone else on the team who really, really loves doing more of like the usability type evaluations. As I said I have a quantitative researcher on the team so – I don’t want to say – you know everyone is a generalist and we don’t have the luxury of being specialized, but I think everyone on my team has an area of strength I think. And I think it’s really important to have that mix as well. You know we get an open head or we’re looking to hire – among my senior group I have two managers on my team and a principal. You know the four of us will sit down and talk about you know what do we really need? What kind of skill sets do we need? Where do we have gaps or things like that. We’ll look for resumes and people who will fill that gap. Steve: I have to call an open head as like the best piece of corporate jargon that any of us are going to hear this week. I have an open head. Liz: I have an open head. Steve: Well we’re going to send you to the doctor then. Liz: It just means I have a requisition to hire somebody for a full time job with benefits. One of the things we talk about in user research of course is like using natural language with our customers. Steve: And do we talk about making fun with our natural language because that’s the principal of interviewing that I just leveraged. So when you’re thinking about managing workload how do you think about – you know with people being generalists, how do you think about you know here’s an internal client with a need, here’s who’s going to work on that? How do you make those decisions? Liz: That’s a great question. Right now I do have, I mentioned I have like a senior team and I tend to have them embedded in one subject matter area. So I have someone who is really leading most of the television and entertainment research, someone high speed Internet and home and then another one on customer experience. Underneath that I have a pool of researchers. They tend to move around a bit more. I like to try to keep people – you know again it’s kind of like the strengths in terms of methodologies. I also have people who have affinity to certain subject matter. So you know I have someone on my team who’s just a DIY geek about home automation so I try to keep that person working in the home security/home automation space because they just have a lot of subject matter expertise there. So some of it’s like you know if I can and if the demand is there, or if the need is there I’ll try to keep people – you know keep that subject matter expertise like running so that we can leverage that and move faster and build upon you know insights from the past. So that’s kind of how we do it and usually there’s enough work balance between each of the different verticals that it isn’t too much of a – you know people aren’t switching around too much. Steve: Let’s go back to the open head bit. So now I’m using natural language – I did it straight and you giggled so I was like I’m back in natural language. Liz: Calling me out on my corporate lingo. Okay. Steve: So when you’re in an open head situation and you’re looking at resumes what do you look for? What do you see and what do you want to see I guess? Liz: What do you see and what do you want to see? You know it’s hard with a resume actually. I’m almost always more interested in if someone reaches out to me with a note, you know, or a personal connection. I mean I can look at a resume and sometimes people have, what looks like they check the boxes and then we’ll want to talk to them, but it’s really about evaluating someone over the phone then on that first pass. So the resume you know should have like you know some experience, unless we’re looking for someone really entry level, but even if they’re entry level you know some sort of interest in doing something UX and user researcher related while they were in school. But anyway, yeah, it’s really more about I would say how are you going to demonstrate to me your level of passion for user research and your soft skills. So like for example you know I hired someone a few years ago – I actually got a note from one of our engineers – one of our scrum masters I think she was. And she said I met this woman at a party, she’s getting ready to finish her master’s degree at Temple and she’s really, really interesting and she had a background in mass communications and anthropology. She loved technology and she was doing work in like presence. And she was just a really neat person and I was telling her about your – we have a coop or like internship program and I was telling her about that and she was interested. So and I said put her in touch with me. But to me that’s like the test right, put her in touch with me to see if she actually gets in touch with me. Not only did she get in touch with me, she got in touch with me right away with this incredibly thoughtful email about all the different things she was thinking about and she wasn’t like I want a job or I absolutely must have a career in UX research. She was almost like I’m really interested in technology. I’m really interested in people. Not really sure where this is all going to go and I would just love to learn more about your team and what you do. So I got on the phone with her and I just thought she had amazing aptitude, she had great maturity and so then I brought her in to meet the team and we actually felt like she was qualified to be an entry level researcher. I thought she was way beyond an internship position. But it was really about you know the impression she made with me in terms of the way she communicated with me and communicated beyond like I know how to do these skills. It was more like I’m curious, which I can’t teach. You know I’m personable. I can get along with people. I ask good questions. Those are things that are harder to teach, the aptitude. And she demonstrated that she had that. Steve: Some of those seem very general to try to reach out to someone inside a company and make them feel like they would want to work with you and some of them seem like there’s elements of like this is what makes for someone who I want to have work with me as a researcher specifically. Can you highlight what are the researcher ingredients and how you’re trying to suss those out? Liz: Yeah, sure. I mean a big one I just mentioned is curiosity. Like if you don’t have a natural curiosity then I can’t teach you to be a researcher ‘cuz that’s like, it has to be ingrained in you, right. I mean to be a researcher is to want to understand. You know like I was that kid that was 5 years old, like why, why, why, why, why? And you know I’m still like that as an adult. I need to know how things work. So I look for that. Another thing is like pattern recognition. I think a lot of being a researcher is you know when you’re out, especially when you’re doing qualitative work, like really being able to understand. You know you hear this one story over here and you hear this one story over there and how are they related and what’s the common thread between them? I’m trying to think of a good example off the top of my head and of course now I’m blanking. But being able to really recognize patterns I think is important. Being a good listener too. I talked about curiosity, but the other side of curiosity is I love it when I interview somebody and then at the end when I say do you have questions for me, not only do they have questions which demonstrates their curiosity, but they’re not canned questions. They’re questions based on things that I’ve talked about, about my team and about what we do. So demonstrating that you can actively listen and process what you’re hearing and then ask thoughtful questions back I think is really critical. You know I can teach someone you know to go read about heuristics or good read a book, you know how to right tasks, how to run Morae and all these other hard skills, but I can’t necessarily teach someone how to be curious or you know how to be able to recognize patterns. I’m trying to think of another good one that I look for. I mean maybe it’s the soft skills too ‘cuz to be a researcher you know you really need to be able to just interact with people and make them feel comfortable. So I look for people who are good conversationalists, who are just comfortable talking to people they haven’t met before, come in, you know have a certain amount of confidence to them, with still some humility. ‘Cuz that’s what you need when you’re going to be going in and interviewing. Customers – and that’s also what you need when you walk in the room to present to your stakeholders. You know you have to be sensitive to their needs as well and what they need to hear in order to take action on your findings. You know and you may get some – when you’re presenting too you know you get questions at the end and sometimes the questions can be challenging, right. You know, especially if like something that you’re presenting on isn’t you know finding sort of challenges. It’s like one of their core beliefs about what they thought about their product. So how do you handle those questions? How do you react in that kind of a situation where you might be challenged I think is really important to observe as well. Steve: What’s the soft skill – what’s the soft skill that describes that, like being able to handle these questions? Liz: Yeah, that’s great. I’m not sure what the word is for it. Steve: It must be categorized. Liz: It must be categorized right. You’re going to make me dig back into my IA world in my brain. I think it’s about not being you know – I mean maybe it is humility. You know it’s not being so attached to your idea and your agenda that you need to be able to compromise. Steve: That’s a great challenge in an interview situation to be able to demonstrate your confidence and your humility together. Liz: Yeah it is, isn’t it. Steve: Like that’s a bit of a magic trick. And I’m not saying that people can’t pull it off. It is sort of about who you are as a person. If I had to think about how would I do that I would definitely have to scratch – be scratching my head and want to practice that a little bit. I agree with you those are essentially, but you just – I don’t know it makes me have some empathy for people that are knocking on doors and trying to put all those best feet forward at the same time. Let’s switch gears again and maybe sort of build on this. You talked a little about – I feel like at the top of the conversation you mentioned some of your other experiences that you had had. You’ve had other careers before this one. You want to talk a little about your background and maybe some of the highlights or milestones for you as you have… Liz: Sure. How much time do you have? I’m just kidding. Steve: How much time do you have? Liz: How much time do we have left? No, I’m just kidding. Yeah, so my degree, my undergraduate degree is actually in physics which I always joked after I was done, the only reason we – my friend and I – would say we majored in physics is because we liked the way it kind of stopped the conversation at a party when someone would ask us what we did, what we majored in. You know I kind of grew up being just you know one of those women that was smart at math and science which is a little bit unusual, especially 20 years ago, and I just assumed I’d major in math and science and then when I was in school I was working in labs and I was like God this is so boring, there’s no way I can do this the rest of my life. I thought I would just get a PhD and go do research and get published in journals and I just thought God how boring is that. So I wasn’t sure what I wanted to do. I went to Penn and I worked in their admissions office and through that I actually just got a job right out of school doing, as an admissions counselor for a local college for a couple of years, which wasn’t as interesting as being an admissions officer for a school like Penn, you know. But I thought it would be cool, but really the lessons I took away from that – you know it was a lot about being independent, being on the road, learning how to sell, how to speak to things, how to present. And I knew that wasn’t going to be my career or anything so I had a friend working for a textbook company, Prentice-Hall, big name, and she was working up at their headquarters and said hey we have an opening for a marketing assistant, you’d be great and it’s in science and math textbooks. So ended up getting a job up there and again I think some of things I learned up there – first of all I worked for two really awesome women who were leaders in the marketing area and you know they taught me not to be afraid to sell myself and to make sure that people knew that I was like doing a good job. It wasn’t enough to get a compliment. It was to make sure you forwarded your compliments on so that people who could help your career knew that you were being acknowledged. You know and I knew – I remember one time they were talking about moving me up to manage a big sales territory in Ann Arbor, Michigan which is you know, in the textbook industry, you want to move up you often need to go manage a sales territory. You need to go out into the field and one of them said to me, she’s like you need to go knock on everyone’s door of these executive editors and stuff and just ask their advice and let them know you’re interested in this position and I like freaked out and I got like all nervous and stuff and they’re like no you have to go do that ‘cuz these are the people who are going to promote your career. So I got like really good advice that way, you know and they were from women which I thought was really important. So anyway I spent five years doing sales marketing and editorial acquisitions in the science and math space in textbooks which is the only job I ever had that was even tangentially related to my degree when I was working on physics textbooks and I was actually a little conversant with the authors. But that was where I sort of made the transition to UX. I always loved – I always loved graphic design and typography and people told me that like I was too smart to do that and I would never make any money and I finally just decided well I didn’t care, I really loved it and we were working on a brand new textbook for biology and the biology editor, I was talking to her about this a little bit and she pulled out this book, it was Richard Saul Wurman’s Information Architects book and it profiled people like Erik Spiekermann and a lot of really just interesting people who thought about how to design things. This was way back in the print age, or signage systems and stuff, but thought about how to design things based on how people were going to actually use them and I was like this is just awesome. And they were hiring somebody as a consultant who was featured in that book to help design like the program for how they were going to design this new textbook. I just thought that was so cool and so I finally just decided you know what I think I want to change careers. And I left Prentice-Hall. I kept doing some freelance work, but I started studying graphic design full time at School of Visual Arts. You talk about key moments. A key moment for me is I was taking a typography class and we were supposed to redesign a spa brochure. It was just one of those handmade things on like – I forget what they were called at the time, but just one of those things where anyone could put together a brochure on their desktop computer and it was pretty ugly. We were supposed to just redesign it with nice type and special attention to typography and I actually rewrote all the content before I redesigned it and I was the only person that did that and I came back in and I remember a couple of the other students were like why did you do so much work? And I was like I can’t design something I don’t understand and that was like almost like the epiphany for me and that’s when I realized I was actually probably more of an information architect then a designer and so that’s how I got into, made the transition into UX and that was – it wasn’t even called UX at the time obviously. This was like around 2000/2001. But it was the right time, right place to sort of jump into that arena and so I moved back to Philadelphia which is where I’m from and through some connections you know got a job at a local agency, basically doing jack of all trades what would be UX now – small agency that built websites for nonprofits and educational clients. So I did everything. I did stakeholder interviews, content strategy, information architecture, you know building site maps. I actually did the visual design. I even did some front end HTML. So that’s where I really learned all my chops and then from there I just got really excited about understanding how people would actually like use the stuff I was designing. Back to like thinking about Erik Spiekermann and the case study of how he thought about redoing the map for the Berlin subway I think it was. Anyway that’s when I went off to Institute of Design for grad school and learned about design research and then got really excited about you know watching users and learning what they do. So from there I just went to Steelcase for a couple of years. Worked for a really awesome woman there named Joyce Bromberg who again was just a great inspiration. She was the one – she’d give me a compliment and I’d be your typical woman like oh it was no big deal and she’d be like take it like a man, say thank you. So she taught me to always just say thank you when someone compliments me. You know you never downplay your successes. It was neat to be out of the digital arena for awhile and then I decided I wanted to move back to Philly and I’ve been here at Comcast ever since. So I think I asked you how much time you had and that was probably the longest answer ever. Steve: No I thought you like managed to do like a really clipped, swift kind of active version of the whole career arc. That’s great. I think we will keep paying attention to time here. What are some things about you as a leader or a manager that I think might be important for us to understand or maybe things that we wouldn’t see on your LinkedIn profile? Liz: I mean I don’t know if this would come through in my LinkedIn profile, but as I’ve moved up I never thought I would be a manager, let alone a director or a senior director. When I started here at Comcast it was like hey here’s this lab and we started up and I thought I’d be a team of one, probably forever. No actually, I didn’t. But I think as I’ve moved up and gotten more into the management side of things I’ve really found like a passion for passing it forward. You know as I was talking about in the past you know some of these great mentors I’ve had who you know taught me how to take a compliment or make sure other people knew my successes or you know taught me how to be patient and have a vision for something. I think that’s something I didn’t talk about before. You know sometimes people ask me like how did you grow this team so quickly, because I know that can be a struggle for some folks, you know, in the corporate world, trying to build up a fledgling UX or user research practices. How do you go from a team of one to ten in a short amount of time? You know and it’s really having the patience. It’s like you’re not going to go from 1 to 10 in one year so it’s like celebrate your small successes and have a vision for where you want to go but be patient for it. So I think it’s kind of passing that forward to the folks below me. You know I’m really interested, I read a lot of books on how to motivate, what people are motivated by and trying to understand like a little bit about the psychology of that aspect of our work, and really trying to do for some of the younger folks what those folks did for me and helping them build their confidence, move their own careers forward, you know feel like they can do it and to figure out what you want to do and not be afraid to take risks. Steve: So as we think about wrapping up I will fall back as I often do in my stock questions because I’m a creature of habit, or I’m lazy or something else. Because they’re great questions. Is there anything that we should have talked about, about you and your work at Comcast or your background that you’d like to make sure that we include in this? Liz: I mean again, just to sum up, I think one of the things that maybe, that isn’t emphasized enough – you know we talk a lot within UX about like the hard skills and the methods and stuff like that and I know you go around – I think you said you were just at World IA Day and you gave Soft Skills are Hard and I think that that, that’s just so important and I think that’s something I’d love to emphasize to folks that’s so important in your career. Don’t neglect your soft skills. And I think I got lucky in that I had this – you know that period for 5 or 6 years where I was you know working as basically like almost in a sales and marketing capacity where I learned to be comfortable interacting with others. People who – you know learning how to work a room. Learning how to make conversation with people I really didn’t know. Learning how to negotiate. Learning how to present. And then again something I think I talked about is you know never turn down an opportunity. So I’ve had people like present opportunities to me where like I gulp really hard and then I’m really glad I did it you know. And one example is the University of Pennsylvania actually, they contacted me a few years ago for – to come back and speak on a panel at parent’s weekend. If you know anything about Penn, Penn is a very pre-professional school. So you basically go there and if you don’t say I want to be a doctor, a lawyer or an i-banker, or a consultant – it’s like oh my God what’s my child going to do and how are they going to make any money. And so they had noticed that I had this unusual major and this unusual career yet I seemed to be pretty successful. So they invited me to come back and speak on this panel and I remember when the guy called me and I was thinking alright, you know I’ll do it. And then when he told me there would be like 400 people in the audience I almost choked because I had never spoken to a crowd that big before. I think the biggest crowd I ever spoke to is maybe 40 or 50 people and I just like looked out and I was like okay I’ll do it. And it was just so energizing and so much fun and I think you know that was the beginning of like so when Alan Wurtzel invited me to speak at his offsite and they said that would be like 200 people I was like sure. I got a little nervous, but now I’ve gotten to the point where, like I’m super comfortable speaking in front of large groups, to the point where like now I can almost do it on the fly if I already have the content ready and I’m not afraid to be animated or to tell stories, or even be a little self deprecating in front of a large group which helps like humanize yourself and build some rapport with your audience. So don’t ever be afraid to take an opportunity even if it scares you ‘cuz you’ll always learn something for it and the more you do something the more comfortable you become doing it. So I think that that’s just really important. Steve: But Liz, what if I fail? Liz: Well you can’t think that, right. I don’t know. You know someone actually asked me that once. I’m so glad you said that because someone asked me. I remember I was working at Prentice-Hall. I can’t remember the context now but they said well what if you fail and I was just like I’m not afraid I’m going to fail. I don’t know, maybe that’s something intrinsic, but I guess don’t be afraid of failure either. Steve: Well I like – I mean just your framework is revealed when that question sort of just brings you up short, like I don’t know. I’m not thinking about that. Liz: Right. And it sounds like cocky. We’ve been talking about humility and I’m like I’ve never failed . I mean sure I’ve failed here and there, but I mean it’s not really failure. Maybe it was like a little bit of setback. Oh you know this didn’t go – I didn’t get that job that I wanted to get, but I got the confidence to do even better on the next interview and got another job that turned out to be even better. Steve: I didn’t hear you say I’ve never failed. I heard you say like… Liz: I’m not afraid of it. Steve: …don’t think about it in terms of what if you fail. Liz: Right. Steve: You sort of were attacking the question. I didn’t hear you say I don’t fail. Liz: No. Steve: Okay. Well those are some lovely sort of parting thoughts. Just to wrap up, wrap up, do you have any questions? No. Liz: Have other people asked you questions? Steve: Is that a meta question or an actual question? Have other people asked me questions? Yes, sometimes. Liz: I don’t know. Steve, is there anything else that maybe, that I could add that would be unique perspective. I listened to some of your other podcasts and there’s always been a lot of talk about the methods and the structure of the team. I think we covered that a bit though. If there’s anything unique… Steve: I mean you’re going to ask me if there’s anything unique about you? You’re the person that can answer that. Liz: And I don’t know. Steve: Okay. It sounds like we’re done then. Alright, thanks very much Liz. Liz: Thanks, Steve.
Feb. 16, 2016
Today I chat with Kavita Appachu, the Senior Manager of User Experience Research at Kelley Blue Book. She describes the different roles she’s had in different organizations, moving from design to research, and explains the change effort underway at Kelley Blue Book. It’s more like “You are a researcher?” Yes, I am and I can also help with providing input towards design. As a society we like to simplify things and compartmentalize people and the way I think about it is design is a mindset. You can apply it towards culling insights or you can apply it towards creating or defining experiences. And I feel like I can do both, but I don’t know if it is just me that thinks that way, or maybe it’s just the way things work right now. That’s the way it is and at some time in the future those boundaries will merge and go away. – Kavita Appachu Show Links Kavita Appachu Kelley Blue Book The Cognitive Systems Engineering Laboratory at Ohio State University Visual Communication Design at Ohio State University Don Norman Design for the Real World: Human Ecology and Social Change by Victor Papanek Liz Sanders Qwest CompuServe Intuit Fitch (circa 2000) Cox Automotive Media AutoTrader Follow Dollars to Donuts on Twitter (and Stitcher) and take a moment to leave a review on iTunes. Transcript Steve Portigal: Well, thank you very much for being here today. Kavita Appachu: Thanks for having me. Steve: So what might be a nice way to start with you is to maybe have you do kind of a narrative or walk us through some of the major steps in your professional career. Kavita: Oh boy. So I’ve had a fairly long and eventful professional career. I started as a designer back in the days when all a designer did was visual design and then I evolved to being an interaction designer and I went back to school for that and then started designing. And then when I started designing I very quickly realized that I didn’t have the right input or insights to help me create compelling experiences and that’s when I went back to school yet again to really learn about how the human mind works and you know how we could do a better job of creating experiences that align with people’s behaviors and needs. Steve: What was the program you went back to at that point? Kavita: That was – I went back to get my masters in – you know cognitive systems engineering and visual communication at the Ohio State University. Steve: We’ll keep going through time but I want to check with this. It seems like at that point where you had that realization that you were lacking this input, you know around behavior, if you say that now in 2016 it’s very different I think then in the time when you had that realization in terms of what are standard design practices? You know how users are or information from users is involved? I don’t know. I’m leading you a little bit, but tell me where you were at at that point. Kavita: Right. Absolutely. This was back in the days where I think Don Norman was just sort of emerging as one of the advocates of designing for human behavior and consumer needs. And you know – I don’t know if you remember, there is an author called Victor Papanek and he wrote a book on designing for the real world and I remember that had a deep impact on me because the traditional – back in those days the traditional design school curriculum was really I am the designer, I know what you need and I’m going to design it and you’re going to learn to use it. Steve: So that was a very different way of talking about it. Kavita: Exactly. It was a very different way of talking. It was a different paradigm. And that was – and that’s what designers were taught and that’s what they did and the book really had a deep impact on me in terms of just empathy – you know people’s needs and how you want to think of those first before you even start thinking of what you want to design. Steve: I love that. I love that we can reflect back, those of us that were around, we can reflect back on times when this – it wasn’t – you know user research wasn’t on CNN or wasn’t just so ubiquitous and that many of us found our ways to it and interesting paths. Anyway I took you away from your story a little bit. You were talking about going into this program at Ohio State. Can we pick up from there? Kavita: Yeah, absolutely. That was I think a very fulfilling experience for me where I was really able to apply myself and learn about, you know, what – how the human mind works and it was completely fascinating – about short term memory, long term memory and all the – you know everything that cognitive psychology has to teach and that really laid the foundation for my approach to creating experiences and how we involve users in every step of the process about collaborating with everyone, all your stakeholders and collaborating with your users and it’s not just about people’s functional needs, but it’s also about their emotional needs. And how you can create these great experiences that not just meet their needs but delight them. Steve: It seems like there’s an element there that the cognitive science gave you peek into the – my term anyways is like the soft skills around research. Working with stakeholders, it is an empathy activity as I think is what you’re saying. Am I – do I have you properly? Kavita: Yes, yes you do and I think I was – so when I was at Ohio State University there’s a couple of things that happened. One was I got to work very, very closely with Dr. Liz Sanders who has been my coach and mentor to this day, someone I really admire. And I think she was almost – you can call her a trailblazer in terms of just using for participatory design techniques to involve users in the design process and also using emotion as a way to both understanding consumer needs, user needs and also creating experiences that help meet those emotional needs. And I know, especially the lit – cognitive psychology literature there’s two schools of thought where it’s all about cognition and emotion does not play a role and she was – even though she is a cognitive psychologist by training she was very – you know she was one of the first people that said emotional does play a role and they work in tandem and you cannot ignore one and design only for cognition because you’re not going to be creating great experiences for a successful product. Steve: And I may be banging this drum too much in this conversation, but you said trailblazer. I think that’s an important characterization because that was not how design was approached then. The notion that this was you know the world of human factors and man/machine interfaces kind of was the legacy that I think the field had inherited and that you didn’t think about this person with empathy or as someone that was an emotional, an emotional user and that was so important. So that – it sounds like this was – some changes in the default practice were starting to come out of the work that you were doing, that Liz was doing. Kavita: Yes, absolutely and I think to your point in those, and we’re going so far back, and I think I’ll get back to why I’m even sort revisiting those days in my journey, not just because you asked me a question, but I think given where I’m working today has some bearing on that. That said, I think to your point in those days there were two kind of areas or avenues for design. One was the human factors, very product-driven approach and then the other was agencies and advertising. And they were very different approaches and they had very different goals. But those were the two and I think where we are today is a good mix of both, right. Steve: Yeah. Kavita: And I think because those boundaries, you know, don’t exist anymore and given that the virtual world and the real world are so closely intertwined, what we – we don’t pay for everything we consume. So there’s just – everything’s changed and so those roles are really all one now. Steve: That’s a nice observation. So, at your request, let’s get back to sort of the stages of your own journey. Where did you head to after that experience? Kavita: Right. And I think the second thing that also was a great experience for me was those days Apple used to, you know, had something called Apple Design Projects. And these ident – these sort of identified about 8 or 10 schools from all over the world and they would give them design projects and they would work with you for six months on a project and the culmination of that was you went to Cupertino and you worked with everyone there and you made a presentation to the execs. And I think that was sort of – that really put everything I was learning at Ohio State into practice for me. And it was really great to be able to work – and during the sixth month period they actually designed a designer from Apple that was your coach and mentor and worked very closely with you. Steve: What was the project about? Kavita: The project was how do you use the virtual world to get people to come closer in the real world. That was the project. That was the project for that, for the year that I participated and it was a competition. It was interesting because you would have a few teams in each school and you competed and then one team, the team that won that competition got to go to Cupertino and then competed with the other teams from different colleges and schools. So that was a, that was a fun experience. I really enjoyed kind of working with everybody and getting everyone’s perspective. We had people from the Netherlands. We had people from India. We had people from Japan and all over the world actually coming with their projects and sharing and learning from each other. So it was, it was a very, very rich cultural experience that sort of defined the way I really think and work with people. Steve: Wow. That’s fabulous. Kavita: Yes. It’s been a long time but I still remember it so vividly. Steve: So did that lead you back to the working world? Kavita: Yes, that did lead me back to the working world and I actually started in telecom which was in sharp contrast to this dynamic world at Apple. And I was the first UX professional hired by Qwest in – and they were an up and coming telecom company at that point. So I was working in this telecom, hard core engineering organization, but what I loved about that place and I think now that I’ve worked in a few places I realize what I really enjoy is the challenge of going in and setting things up where the organization has a thirst for evolving their design expertise and design maturity, building their design capabilities. And I really enjoy going in and working with the people and the organization and helping make them make that happen. Steve: And how did it play out for Qwest? Kavita: I think it played out well. At least, you know, they had developers who were actually coding – you know not just coding, they were designing the experience which was called the user interface in those days. And while it worked for – you know it worked for – well it wasn’t ideal, but they were able to make do for their customer service applications and internal applications when they started getting into the consumer space which is what they were getting into at that point. They felt they were at a big disadvantage so they needed this expertise to help create products and experiences that consumers would be able to use, that were usable and useful. Otherwise they were losing consumers. So I think that’s what drove them to actually start thinking of having this capability in-house. Steve: And how much was – you know in that particular organization and the work that you were leading there, how much was research part of the design practice? Kavita: Research, so it was interesting. I think, you know, in traditionally technical organizations they do what’s called user acceptance testing. You know it’s a very technical term. It’s a very engineering focused term and they – otherwise they were using – they were doing – there was nothing formal, but they were informally doing that and using that, not just to test the performance of their applications, but also to some extent get feedback on the usability and the usefulness of their applications. But that was it. There was not a lot of formal research and in all honestly at that point I really was not able to do a lot more either. I mean I had to – you know I had to pick – I had to prioritize and I certainly prioritized with okay let’s at least start with getting them to understand and learn about how – you know what it takes to design a good experience and it is a very unique sort of skill set and it requires a certain approach and the research at that point was very informal. Steve: That’s an interesting point you made too that the priority is to kind of get a design process happening. Do you think you need to get that set up first before you can start to kind of expand and flush out the formality of research? Kavita: You know today if you asked me that I would say probably not, but I think in those days, because design was not – design was, at that point for the most part there – you know on that maturity continuum either it was a tactical player or did not exist at all. Right. So you sort of prioritized and said okay, you know what I’m going to start – it’s baby steps, so I’m going to start with what, you know the most important – what they’re more likely to accept and then work my way into incorporating research. But if you were to ask me today, because thanks to design thinking and just the awareness and the impact of design, where it – in a lot of work – I can’t say it’s in all organizations, but in a lot of organizations it’s moved from being a tactical player to a strategic partner. I think you would go in today and really say we need both. Right. They work in tandem. Steve: Yeah, that’s a good explanation of the evolution since then. Kavita: Absolutely. Steve: So Qwest, how did you know when to move on from Qwest? Kavita: I think at some point I just was looking for other challenges and that’s when I decided to move to something that, to work on something that was just more consumer facing and moved faster. A telecom industry doesn’t move at the pace at say a media company does, or a financial company. They’re even slower. You know, I mean finance and software is not known for its speed or agility, but telecom moves very slow. And so that – so it was more – I think I was ready for a new challenge and that’s when I sort of moved to work for AOL back in the days when they were doing well. Steve: And what was the role you had there? Kavita: I think that was where I switched from being a designer to a researcher. I started as a designer there and we would outsource research and what happened was I think one day the GM at – and I was working in Columbus, Ohio which had – which – where CompuServe was based which was an AOL acquisition and the GM in Columbus, I think saw – he got the expenditure spreadsheet if I recall and he saw the line item for the expenditure on research because we were outsourcing it and he realized oh, maybe we should bring it in-house and that’s when he asked me if I would be interested in heading that. And it’s been a long time, but I think at that point I was like absolutely, this sounds like a great opportunity. I really enjoy – I like designing, but I really enjoy providing insights and gathering insights because to your point – you know as I mentioned earlier there’s data and there’s insights and I would get data, but I wouldn’t get good insights from the research that was coming my way all the time. So that was why I sort of took on that challenge if you will. And I think since then I’ve stuck with research. I enjoy doing it. I love working with people. As a designer you do interact with people, but as a researcher it’s imperative. And I absolutely love that. It’s such a fulfilling experience. You learn from – you learn so much about everyone and yourself, every day. Steve: And yourself is – yes. I’m not saying anything except just agreeing emphatically. Both those pieces are true. And then you never stop learning about yourself from your research. Kavita: Yes, absolutely. Yes. It’s very humbling. Steve: Yeah. One of the things I enjoy about research is learning about how judgmental I am. I think I’m a good advocate for all the good principals of research and deferring judgment and so on. It’s interesting to find myself in those situations where I can just feel all this judgment come up. I don’t know what you think. I think maybe as researchers we kind of learn just to sit with and hold onto that judgment as opposed to taking it on ourselves fully. I guess for me it can be fun to – like even just in the same conversation with someone that I’m interviewing, kind of flip-flop or maybe sway back and forth between sort of accepting and being open minded and curious and then sort of flaring up a judgment and setting it aside and then going back to being open. You know just seeing what my own limitations are for all the values I might proclaim. You know I’m pretty flawed as a person, as we all are. I don’t know, is that something that you – am I making any sense at all with this? Kavita: Absolutely. I completely agree. I think I go through that a lot, both in my professional and personal life, right. And especially in your personal life because you know you are much more emotionally vested in your personal life. It’s very easy to jump, you know into drawing conclusions and being judgmental and to your point I have to tell myself hey I’m able to do a reasonably okay job as a researcher, why can’t I also do the same in my personal life? Steve: Yeah. Kavita: You know what I mean? Steve: Oh, I do know what you mean and I feel like anything that I discover as a principle or write about that makes for good research is clearly applicable to everyday life, but I also think it can be a big burden for those of us that have reflected on that and have tried to develop those skills. It’s hard enough to be good as a researcher and just sort of moderately good as a person. I always want to be cautious of raising the bar for myself too high, like oh I should always be listening to people and I shouldn’t be judging them and I should be patient and open minded. That’s hard to do in research and sort of coming out of that mode sometimes I want to just set that aside and just you know, just be a jerk I guess. Or just a regular person I guess is maybe a better way to put it. Kavita: I know. But you know, believe me, my 16 year old reminds me every day about how I jump to conclusions and how judgmental I am. So I don’t even need to do that myself. He’s there to do it for me. Steve: I want to go back to one thing you said. You sort of described this point at which you went from being a designer to a researcher. Was that an identity switch for you? Obviously it’s a title change, but – do either of those labels – were those the right – you know sometimes we sort of transcend our labels. I know you as a researcher, so to hear that you were a designer like I think you’ve told me that, but I forgot until this conversation. So I am trying to like put you in a box as we’re talking. Like well is she a researcher or a designer? You kind of described those transitional moments. I don’t know. Kavita: You know it just seemed like. So that’s a good question. At that time I just was so excited I didn’t even think about it, right. And I just took it on because it was so much fun, just kind of – like I said I like to go in and set things up, make things happen and this was the starting of something new that I enjoyed doing and I didn’t think much of it because I had a passion for it. But to your point I didn’t realize that it would – in my mind it wasn’t because – you know a change of identity because I’ve always thought the two are very closely related and at least the way I work, the way I’ve been trained, I kind of have thought of myself as both because of my formal training and experience, but then to your point yes, I was asked to pick an identity even though I didn’t want to. And I think I sort of struggle with that even now. Steve: So you want to talk about how you experience that now? Kavita: Right. I think it’s more like, of you are a researcher? Yes, I am and I can also help with providing input towards design. Right. I think, so we – as a society we like to simplify things and to your point compartmentalize people and the way I think about it is design is a mindset. You can apply it towards culling insights or you can apply it towards creating or defining experiences. And I feel like I can do both, but you know I don’t know if it is just me that thinks that way or – and so maybe I’m way off from everybody. Or maybe it’s just the way, you know things work right now. That’s the way it is and at some time in the future those boundaries will, you know we merge and go away. Steve: You know people are going to be listening to this and they’re just going to be nodding vigorously right now going no it’s not just you. I think you expressing it that way I think is going to be a nice moment for people that are listening. Can we talk about your work at Intuit which is where I first met you? Kavita: Yeah, absolutely. So I think at some point I was looking for other challenges and Intuit came along. When I – and I think Intuit was a great experience for me in terms of watching and growing with an organization, in terms – on that design maturity continuum. Right. When I started at Intuit they believed in design and they felt design played a role in defining experiences, but they were not where they are today which is a design driven company. And it was – it’s been – not that I’m not at Intuit anymore I look back and I go wow that was a great experience living through – going through that journey and living it, watch – you know working through – you know starting with hey here’s the product go design it and go design – they were always insight driven and very consumer focused so that was always good and that was what helped them move so fast so quickly. That said they were – it was not – design was – design happened after the product strategy was defined, but today, you know, design plays a role. And when I say design I mean design and research, again using the term design to cover all aspects of the user experience. You know, design plays a role in providing input towards their product strategy. How cool is that? How many organizations do that? Steve: And did you – I mean what things did you see happen to help drive that progression in the maturity? Kavita: I think that there’s a few things. So one is the leadership believed in it and was committed to it. Right. So that was one. And they enabled the right people to go ahead and create an environment and the operating mechanisms to make that happen. The leadership was like well yes we believe in this, we do believe that when everything else becomes a commodity, you know software is a commodity, what’s your differentiator? It’s going to be the consumer experience and we also believed in doing right by our stakeholders and that sort of drives the fact that they are very insight driven – talking to consumers, researching, talking to their other stakeholders and every consumer touchpoint needs to – both – you know take into consideration consumer behavior and needs and also consumer – you know – and also ensure that you meet the consumer’s emotional needs. So you have empathy every step of the way. So I think that was one. And then knowing that okay we are committed to it, but we don’t necessarily know what this will look like or what needs to happen and then you sort of get the right people to lead it. And I think that’s great because a lot of organizations don’t always do that. They declare it and then they don’t know how to make it happen or they don’t always enable the right people t make it happen. And I think that’s commendable, you know, and so they, you know this – people who were leading the design leaders set up the right programs. They had the right success criteria, enabling not just the user experience professionals, but also everybody else within the organization to understand what UX is, how to work with UX. They also made sure that design thinking was embraced by the entire organization. So it was almost like design is a mindset. It’s not a function. And what that allowed you – by opening your doors to let other people in allowed you to have more impact because they understood, they embraced and they practiced it. And they also realized that you bring a certain expertise that will help them make – you know create a better product or create better experiences for our consumers. Steve: So as you’re involved in this, over this time the organization is changing and kind of progressing, what’s, what – how do you think you were different – how did you progress? You know if you think about when you started versus when you left, what was your evolution in that experience? Kavita: Yes, that’s a good question. So I think given my background the fact that I had worked with Liz Sanders and at Fitch with her for awhile, I always had practiced that approach, but in a very limited way and very sort of informal, internal way. What this did was just allowed me to practice it in a more formal way, more purposeful way. And also not limited to just design experiences. You know the design thinking provides you with tools not just to create better designs and experiences. It provides you with tools on how to collaborate with everybody, how to create better processes, how to even work through your interpersonal differences. So it just broadened your horizon and your perspective and you know made you grow, not just as, not just in your profession, but also personally. And I think that’s a very empowering experience and thought. Steve: I like how we’ve kind of hit upon a couple of times the overlap, or the kind of intertwining of our professional practice and our personal developments. Kavita: Right. I think design is a very introspective profession, right. I mean I’m sure others are too. I’m not saying design is the only one, but given what we do, if we have to be successful it does require a lot – a fair amount of self reflection. Steve: It certainly seems to be a characteristic we like to celebrate in ourselves. Kavita: And that may well be. Maybe we are just more vocal about it, yeah. Steve: I mean, I guess I shouldn’t be critical. That’s what this podcast is, right. We’re sort of looking at individuals and surfacing stories to kind of look at ourselves as a field. Obviously – and researchers especially like introspection because that’s part of the toolkit. Kavita: Yeah and maybe that’s what – maybe that’s why I kind of enjoy doing research because you are much more introspective. By nature of what you do you have to make sense of things. You have all this information and you have to try to make sense of it and it just comes naturally then. Steve: Can we talk about your current role and your current organization and maybe you can give a bit of an introduction to where you’re at? Sometimes I start off the conversation with that and we just went all the back and kind of all the way forward, but maybe we should talk about where you are and what you’re doing. Kavita: Absolutely. Yeah. And so – I am currently – I work for a company called Kelley Blue Book. Actually we are known – that’s the brand, but we are known as Cox Automotive Media now and what Cox Automotive Media is, it’s Kelley Blue Book and AutoTrader. So we recently integrated. I manage – I’m the Senior Manager of UX Research for Cox Automotive Media and Kelley Blue Book. I’ve been here for about I think a year and a half almost now and I – this opportunity just happened to come along. The Director of UX at Kelley Blue Book asked me, reached out to me and was wondering if I was interested and I – we started talking and then I wasn’t really actively looking or anything, but it just sounded like such a great opportunity to again going back to what I enjoy doing. You know the challenge of coming in and setting things up and getting an organization to develop and evolve into a more mature design drive organization. So that was what attracted me to Kelley Blue Book. I love the people here and I love the passion they have for insights and consumer research and – in terms of again the maturity I think it’s interesting because things have come full cycle. They are really I think where Intuit was when I started at Intuit. So it’s sort of I’m starting – it’s almost like starting over. The good thing is I have so many lessons learned that I can use them hopefully to build on the successes from my past and really learn from the failures as well so that I don’t repeat my mistakes. Steve: Can you talk about some successes that you have had in the year as you’ve been setting these up and trying to help with the evolution of the organization? Kavita: Absolutely. So Kelley Blue Book is in the automotive industry and with that sort of comes – that’s the – the automotive industry is, how should I put it, it’s not – on the design maturity continuum is not as sort of – the capability is not as evolved as the other industries if you will. But the good – so that’s one. But the good thing is there is – at least within Kelley Blue Book there’s a thirst for learning and being design driven, embracing consumer insights, because they also realize that that is going to provide them with the competitive edge. So a couple of things that I feel I’ve been able to accomplish is, the first is you know, as with most organizations that have a research function, a value to research where you sort of come up with a design and you kind of validate your concepts. That was being practiced here. One of the things that the organization was not practicing was generative or exploratory research. And one of the things that I’ve been able to do is get them to really embrace exploratory generative research to one both get the teams all excited and have more input towards brainstorming and designing experiences going broad and just designing different concepts and experimenting and testing their way into the best experience to launch. And also we’ve started – you know I’m not saying we’ve been able to do it consistently and successfully but we’re trying to – we’ve started using those insights as input towards defining product strategy. Are we there yet? No. But it’s a start. So one of my goals when I came here was to kind of move UX research from – and even design, but primarily – you know I’m starting with UX research. Move that from being a tactical player to being a strategic partner and I think we are well on our way to that. So that’s one. Steve: Can I ask a follow up before you get on to the second one? Kavita: Absolutely. Steve: You know just imagine that someone is listening to this and they’re thinking that’s great, I want to do that, but how? You know what – can you advice – and then you talked about lessons learned and so on so how do you help the organization start to make use of exploratory research and think about it strategically? Kavita: Right. I think one of the things – you know as part of my meet and greets the consistent theme I heard was you know we’ve been trying this and we’ve trying that and we’ve heard that there’s a lot of opportunity based on industry data or market data we have, but we just haven’t been able to figure out how to successfully monetize things. And I think that was my opportunity to say okay let’s step back and see what questions you have and how I can help you. And once we brainstormed a list of questions and concerns people had it became very clear to everybody that they really needed to look at things differently. And I’ll give you an example. One of the things that I know is big in the media, and especially in automotive media, is video. There is I think – I don’t know and I don’t have the numbers, but I think AdAge or somebody has projected that revenue from video is going to be in the millions in 2016 and we were – you know the KBB revenue was very, very small last year compared to – you know – so we were trying to – we were saying okay the numbers are telling us that there is a lot of opportunity but how do we make it grow? So you were sort of looking at the business – you’re sort of talking to them from the business point of view, but also having them flip their thinking to understanding okay what are your questions and there is another part of this equation which is the consumer and also, in the case of the automotive industry, the automotive manufacturers, which they’re known as OEMs, and dealers. So when we started brainstorming the questions it became very clear that they didn’t have the answer. So it was like well how are people going to use video in the shopping process? At what point do they want to watch videos? What kinds of videos do they want to watch? So very basic – so there were some very high level questions which you probe deeper and you realize we really didn’t have specific information in terms of what point in the consumer journey do people want to watch videos? What point in the journey do they want to – you know what are the relevant video topics? So just having that conversation allowed me to make the team realize that you can’t build something and ask for consumer feedback or test it in the market and then grow your revenue. Does that make sense? Steve: Yeah. And – I mean I love that story – it echoes a theme that I’ve heard from folks before which is you know you’re not – you didn’t start with a conversation about methods. You started with a conversation about goals, outcomes, what are we trying to get to. And then you’re able to say well given the questions that we have that we haven’t answered, here’s how to go about answering those and here’s how it supports the business. So you were not – you’re not being dogmatic about I love this and this kind of research. You’re able to bring it up in context of what the problems that the business has that you’re there to help them solve. Kavita: Right. Yes. Yeah, and I think that – and that’s something that I’ve learned in my career over the years. It’s not about the methodology and it’s not – and it’s not about the data. It is about the business needs, unanswered questions and insights. You know I think everybody trusts you – for the most part people trust your expertise in terms of the methodology. They’re – you know they really want actionable information so I always sort of – when I have conversations with people I kind of try to use this phrase, it’s a cliché, but what’s in it for me. So what’s in it for me, Product Manager? Or what’s in it for me, VP of – you know VP of Advertising. You have to speak on their terms. Steve: Yup. Did you have another example that you were going to describe of some things that you’ve had impact on in the time there? Kavita: Yes. And so this is sort of – this is what I like to call, you know for want of any other way to describe it, almost an serendipitous. So when I sort of – when I started at Kelley Blue Book one of the things that – my goal was to make research more collaborative. Research, the way it was being practiced, was more of a service. The researcher or research as a function wasn’t embedded in the product development design – design and development process. And as part of that I started doing some collaborative design thinking exercises to get the team to you know not just hire – not just as a researcher do generative research, but get the team immersed in talking to consumers. Going out in the field, you know – and then come back with ideas to generate concepts and brainstorm. And as part – so I was doing that for the different projects where it made sense and around that time our VP of Product and Finance attended a design thinking workshop and they were sold. They felt – they believed in it and they felt that was a really good framework for us to use in the organization, not just for designing, but I think one of the things they wanted the organization to do was to be more collaborative, do a better job of problem framing and then just move quickly. Those were their goals and they felt that this was a good framework for the organization to use and we just happened to have this conversation and at that point they decided they wanted to formally introduce design thinking which we call human centered design here in the organization and this was a cross- this is a cross functional initiative where we are work – we work with a consultant that came in-house and trained – to start with 17 catalysts that actually helped facilitate these sessions and those have ranged from anything for the sales team to generate ideas to monetize, sales team to generate ideas to create play books and pitch books, to operations teams to increase efficiencies in their processes. And of course for product design and development brainstorming ideas which is the natural fit anyways for design thinking. So I think that’s been very, very exciting and it’s been both very fulfilling and humbling because I don’t – you know like I said there was a thirst in this organization for user research and insight driven development but I didn’t expect the team and the organization to embrace design thinking the way they did. At one point we were conducting – and this was in June – we launched this in June and we were conducting at least – forget about the informal sessions which happened if there was a team, embedded team, you know catalyst on any team, we were conducting about one formal session a day and the catalyst couldn’t keep up. Steve: Wow. Kavita: So that’s been amaze – it’s been both humbling and amazing and you expect product to embrace it anyways, but sales embraced it. Operations embraced it. So I think it’s just been so exciting and fulfilling and very, very humbling. Steve: And within Kelley Blue Book and AutoTrader and I guess I don’t – you talked about them becoming integrated, is the kinds of change that you’re describing, is that happening everywhere within Cox? Kavita: So we’ve started with Kelley Blue Book, but now my – you know now we’re trying to kind of, as we integrate with AutoTrader we’re trying to build those relationships and get them to also experience and then we’ll take it from there. But we are kind of, at least within Kelley Blue Book, we’ve been practicing it. And we’re working with AutoTrader on embrace – you know – you know just sort of getting them to practice it and then build on it. Steve: I also neglected to ask you this earlier. What are sort of the basic businesses that Kelley Blue Book and AutoTrader are in? Kavita: Yes. So like I mentioned they’re in the automotive industry and Kelley Blue Book, for those of you that have grown up with those literally blue books, will remember that your parents or grandparents used to look at those books to get the value of their current car when they were planning to sell it. So that’s where Kelley Blue Book started and today, even though they make those – they do make – they do print a few of those books – you know our core competence is really the valuations which is pricing of cars and that’s what we offer and we are the market leaders in that. And one of our – you know the thing that – Kelley Blue Book as a brand stands for trust. People trust our values and that’s why they come to our site to get the value of their vehicles when they are ready to sell them and also the value of new vehicles when they are looking for another vehicle. So that’s Kelley Blue Book. And AutoTrader is in the – they, like our CEO says, they really invented online classifieds. They were the market leader and the first company to bring classifieds online – automotive classifieds that is. So you know – and that’s what AutoTrader does. They are a sort of match maker that provides classifieds or allows dealers to host classifieds and then provides consumers access to them and connects the two. Steve: Does AutoTrader have a user experience or a research group that is I guess analogous to what you’re leading? Kavita: Yes they do. They do have UX group and they do have a research team. Steve: At Kelley Blue Book what’s the size of the research group? Kavita: We are a team of four. Steve: Has that changed? Kavita: That has not changed. You mean since the integration? Steve: I mean I’m just wondering. Like a lot of research groups are say – I mean research groups are growing in general is the trend that I observe. Sometimes where demand exceeds supply even for researchers. Kavita: Right. To your point yes, that’s absolutely right. In fact I have open head count and I can’t find people. So the change is that I have to grow my team and I am not able to find people right now. Steve: What are candidates not succeeding in? Kavita: I think the challenge we have for us here in – you know Kelley Blue Book is located in Orange County and the challenge we have is that there’s San Diego to the south and there’s L.A. to the north. And within – in Orange County there aren’t many organizations that have such a big UX group. We are about 32 people here, UX which includes researchers, designers and copy writers or editors. And it’s very – so we’re a big UX team as far as – you know compared to the other organizations in Irvine and Orange County. And they don’t have this – you know the way their roles are defined in the other organizations, they are not defined by designer, researcher and copywriter which – so I don’t get people with strong research experience or skill sets. I think that’s where kind of the challenge is, at least for us here. Steve: This is when you’re pulling kind of from the local community – there are people that are elsewhere? Kavita: Right and so most of the people are either in San Diego or L.A. and they don’t want to move. And while you’re happy to have people relocate not many want to move to Orange County. Steve: It’s a kind of a combination of factors that are challenging you there. Kavita: Yes, absolutely. And while it’s easy to get entry level people I think when you are looking for people with some experience that gets very challenging. Steve: What are you hoping to accomplish in the next X years. I don’t know what the horizon for your vision is, but what are you looking towards? Kavita: So a couple of things. I think my hope and wish is that we evolve on the design maturity continuum at Kelley Blue Book. And I think we are working towards that – is removed from being, you know, an organization that uses design, you know as a way to define and create experiences to really using design as an input towards strategy. I think that’s my hope and wish. And if we get to a point in the next year where – in the Cox Automotive Media Group which is AutoTrader and Kelley Blue Book we’ve embraced design thinking and we are at least starting to work towards leveraging UX design research, not just to execute on product, but also in some little way use insights as input towards defining that strategy, I think that’s – I would be really happy. I would consider that a success. Steve: That’s great. So I’m just thinking about our time here and looking maybe starting to move towards a wrapping up. Maybe I can ask a question that goes all the back to some of what where we started. So you talked about your trajectory and the different educational and professional steps that you took and kind of how that brought you to where you are now. And we also have talked about kind of the personal and the professional and how for people that do what we do, and maybe from any professions – as you said they start to overlap. You know and a question that I often ask in these interviews is what – and it’s different I think then – I mean you’ve covered sort of the right things for this conversation, but sometimes there are other parts of our background or other parts of our personalities or our passions that find their way into the work. Are there things that really make you excellent at what you do that are different than the sources we’ve talked about today? Kavita: So I don’t know about that, but one of the things that I kind of do make a very conscious effort to do is really ensure that everything and anything I do or my team does has relevance for the business. I think sometimes we as UX professionals do ourselves a disservice by only thinking about solving for the user or the customer or the consumer, whatever you want to call them – you know so there are different labels in every industry. And what happens is that one of two things happens there. Either people do not listen to you because it goes back to what’s in it for me? Or even if they do they’re not able to use it because the question at the back of their mind is but how will it make me money? Right. Steve: Right. Kavita: And I’m not saying that oh you need to talk about monetization or your business model or revenue model every time or right away. That said, I think as UX professionals for us to be impactful and have a seat at the table when strategy is being discussed we need to be very, very mindful of that. And I think that’s one of my challenges when I look for UX professionals – you know when you interview them. It’s a unique sort of mindset and I – it’s very rare. I don’t always find it in UX professionals. Steve: Part of what you’re describing to me sounds like empathy and you’ve mentioned this in a number of different points and a different words, thinking about what their concerns are. It’s not about the dollar signs in our eyeballs. It’s about thinking about – as you said what’s in it for them, as an empathy activity. Kavita: I think that’s a very good way to put it. I had not thought of it that way, but thank you. Steve: You can use that. Kavita: I will absolutely actually. It’s more going back to yay, pat on my back. I always struggle with myself. You know this is an internal struggle I have about maybe I am not empathetic enough, right. I don’t have empathy for people. So thank you. Steve: Cool. Is there anything else we should have talked about today? Kavita: You know, this was fun. I’m feeling very introspective going back to my Fitch days and talking about Liz and everything. So yeah, thank you for just making me relive my – both, you know the fun moments and the reflective ones. Thank you. Steve: I think it makes for a really nice story when you put it all together, especially as we bring it up to today and the challenges you are facing today. Do you have questions for me? Kavita: I don’t. And there’s two things. I realize these are the kinds of challenges I like, getting into – Intuit had challenges, but they’re different. You know they’re already mature as a design organization and they have different challenges. Some people enjoy those. I do too, but this just getting in and sort of being part of something that is just formative is just what I enjoy and so I kind of love it. Right. Do I have frustrating days? Absolutely. But then when you see the rewards in terms of, you know, what you’ve accomplished and how people feel you made a difference in their outcomes, it’s just – yeah, there’s – it just feels good. And it’s very validating. Steve: Well that’s just such a lovely positive note to end on and so let’s leave it there. thank you so much for sharing so deeply and personally and as you said reflectively, it’s been really wonderful. Kavita: Well, thank you.
Feb. 2, 2016
In today’s episode I speak with Aviva Rosenstein, the Senior Manager of User Experience Research at DocuSign. We explore how to make all types of research actionable, the benefit of doing your own recruiting, and the evolution from building a usability lab to having an in-house research capability. They weren’t going to limit themselves to telling us about what was going on with sending. And I’m not going to shut those people down. I’m going to listen to what they have to say because it might be something amazing or new or super important. And of course it’s important to the customer so you don’t want to shut them down. So sometimes the scope of the study can get very quickly past where you wanted it to be, but it’s still going to be valuable so you still listen. – Aviva Rosenstein Show Links Aviva Rosenstein Follow Aviva on Twitter DocuSign API Snowball sampling DocuSign University Sean McLeary Practicing What We Preach: designing usage centered deliverables UX Process Improved: Integrating User Insight Advanced Communication Technology Lab (ACT Lab) Allucquére Rosanne Stone Radio Television Film Program CSS RE:DESIGN/UXD 2015 Erin Malone Follow Dollars to Donuts on Twitter (and Stitcher) and take a moment to leave a review on iTunes. Transcript Steve Portigal: Well thank you. So let’s start broad and big as we always do and have you tell us about you, your role, your organization. Aviva Rosenstein: My name is Aviva Rosenstein. I am the head of user research at DocuSign and I’ve been here for two years. DocuSign is a rapidly growing pre-IPO startup that’s been around for over 10 years. So it’s different on a lot of ways from your usual startup. We are interested in making signing documents and digital transactions easier for everybody from consumers to large enterprises. We are very concerned with making sure that we’re trustworthy, that you can trust your digital documents with us, and so we care a lot about security. And we are more than just signing. We are looking at all kinds of ways of making that digital transaction easier. So what people think about us is oh you’re just an e-signature, but we’re actually a lot more than that. Steve: Was signing what the company started off as? Aviva: We started off as e-sign, that’s absolutely right. Electronic signature. We actually started off with a lot of real estate agents who, if you’ve ever bought a house, you know how many papers that you have to sign as part of that process and so that was a really easy set of use cases to start out with of a very motivated group of users who wanted to make that easier on their customers. But now we are doing e-sign in the whole gambit of industries and we’re moving into digital signature in other parts of the world that have different laws and policies related to electronic signatures. So we are expanding our technology to accommodate those other countries. Steve: Is e-sign and digital signature, are those different technologies or standards? Aviva: They are different standards, that’s correct. And different technologies. Steve: Are those terms that are created elsewhere? I’m trying to sort of unpack the language. Aviva: They are created by their standards bodies although DocuSign has been active in those standards bodies, at least with the development of e-signature and getting e-signature accepted in North America and other commonwealth countries. Digital certificates and digital signatures are something that came out of the European market and South American market and other markets. So we are accommodating to those standards as we move into those markets. Steve: In some ways it just seems so – I don’t know, Silicon Valley, 2016 – sort of hilarious, silly and awesome that there’s a business whose legacy is about signing stuff. Aviva: It’s less – I think it’s really awesome. If you think about how much paper had to move physically from place to place to place using things like FedEx or UPS and people had to sign – let’s say you got a job offer. You had to wait for the job offer to get mailed to you, or delivered to you on a truck, with a dead tree. And now none of that is necessary. And then there was – in between that there were faxes. So if a fax came to you, you still had to print it out, sign it and then fax it back or mail it back. With e-signature there’s none of that dead tree. There’s none of that carbon footprint. So we’re actually doing a lot for the environment here. Steve: I mean your example of the real estate one rings true to me. I mean every time I’ve signed or refinanced it’s easily – I mean I’m not being hyperbolic, it’s an inch thick of paper that you’re just signing and signing and signing and signing and signing. Aviva: So here you put your signature in once and you agree that that’s your signature and that’s going to represent you. So there’s that handshake that you have with the technology and then it’s a click. It says sign here and you click on that and your signature appears where you just clicked. So that’s our standard way of doing things in e-signature. It’s a little bit different than the other technologies. Steve: When I think about – yeah, just sort of reflecting on – it is starting to become a more ubiquitous kind of thing. I bought something I think with Square yesterday and I signed with my finger and I didn’t think about it. And I got an email. And I walked into your building today and did the normal sign-in with reception – fill out a form, it prints out a badge and there was a nice DocuSign interface that I go through to kind of sign in here and it makes a lot more sense. Aviva: And it’s completely auditable. So we know where you were physically and what time you signed that. So if that stuff – if there’s a court case, if somebody challenges and says that’s not my real signature and tries to defraud a company, the company who is our customer has recourse to look at the information about when that signing even happened and try to connect it up with the person who signed. So it’s a lot safer and more secure than a signature with a pen on paper because anyone can forge that. Steve: So as many things have moved in our recent lifetimes from analog versions to digital versions there’s all these things that you get and one of them is being able to go back and then trace them to audit them. Aviva: Right. Steve: That helps me to think about this category as sort of an old school technology becoming a modern technology and what that affords. Aviva: Like so many technologies they start out as being – you know just replacing a single medium like – oh it’s like signing something with a pen only digital. But then as you keep looking at it there’s more and more opportunities to build out on that from and to pass that information into other systems. And that’s where things get really interesting because we have a set of APIs that our customers can use to integrate our signing technology into their workflow. Steve: Can you say more about what an API is and how someone else might use that? Aviva: APIs are the building blocks for making an application. So it’s a way of determining, in our technology, what inputs somebody is looking for and what the system does in response to that and capturing it in such a way that somebody else can use it to build an application Steve: So the point is though that other people can use sort of a core part of the technology without having to have DocuSign write a piece of software. Aviva: Right. Steve: If you wanted to say have a point-of-sale that people were going to sign with their finger you could use all the technology that DocuSign has and get all that auditing and all the security and all the greatness of the technology and then put your piece around it. Aviva: Right. So our service you can use the way we ship it, but if you wanted to build your own system and include the e-signature component then you would plug in the API and it would take you through the DocuSign interface and allow you to sign, but it would be integrated with your technology. So there’s a lot of different ways that people are using our technology in their own products. Steve: And how is user research part of the effort that DocuSign is making to create stuff for people, for lack of a better way of putting it? Aviva: I think that we have an impact at every point in the product development lifecycle at this point, the two years now that I’ve been here. We started really with the basics of usability studies of making sure that what the UX team was building could be used by people before it got shipped. So we did a lot of early stage prototype usability studies just to validate things before we ever showed them to customers in a production environment. But we also have done other kinds of user research from just concept interviews, finding out what people are looking for, whether our concepts are going to work for them, what would need to change about our concepts in order to work for them in their environment. We’ve done more field, follow me to work, kinds of studies where we’ve gone in and watched people use our production technology in their environment and talk to them about what was and wasn’t working or find out – sometimes we would watch them and find out, hmm, they’re using things and not understanding something correctly and we need to make that clearer. Or they’re making a mistake and they could be doing this much more easily and so we need to make that more apparent. And then that goes back into the next round of features that are up for – possibly including the product. So we’re both informing the roadmap as well as making sure that things that are on the roadmap are getting built in a way that people can use them and are delighted to use them. Steve: I’m curious about sort of the types of organizations and the roles within those organizations that you guys are looking at, again within what you can say. Aviva: Well I think that’s pretty clear. The people that use our products kind of fall into specific roles. There are people who are mostly just signing stuff that other people send them. So if you were the person who is buying a house and your realtor sent you something and you were the signer – so we could talk about the signer and that experience. There’s also people who are sending documents and they may be sending a single document or they may be using a reusable template where they’re sending the same kind of document over and over and so they’re customizing it for each signer and that may be more or less automated depending on their systems and what they’re doing. So we talk about different kinds of senders. They may be working in a small business. They may be at a big call center. They may be somewhere in between. They may be out in the field as a salesperson getting somebody to sign something on an iPad. So that’s a varied experience, but we can talk about senders. And we also have admins – administrators – people who administer the product for their sets of users because like any other kind of enterprise software there’s different roles that people can play within the software and they have different kinds of access and different permission levels. So there’s an interface specifically for administrators. So we talk to them a lot because they spend a lot of time with the product and have a lot to say about how we can make it better. And then of course there’s the developers who are using our tool kits and our APIs to build their own stuff and they have their own specific needs. So those are four of the most maybe top of mind end users that I think about on a daily basis. There are other people in the whole realm of our customers that are making decisions about whether or not to purchase the product and I think about them too. But the day to day end users is really what it’s for. Steve: What have you learned about trying to get in front of those different user types? Aviva: So consumers are relatively easy. Signers are often consumers. And so sometimes we just use Craigslist to get those folks. Or we use other recruiting services to find people who have experienced e-signature or maybe specifically experienced our product. But we also maintain our own customer panel that people opt into. So they fill out a short survey that describes them and their working environment and we can sift through our panel to find people who meet a particular kind of target and then send them an email inviting them to take a screener for a specific study and if they’re interested then they’ll answer some more questions and we can use that to schedule. And that’s been super helpful. Growing that panel has been a tremendous asset in speeding up our research process. But sometimes we need people who aren’t in our panel yet and then we use other groups within the company to help us identify the right people to talk to and get their information so we can contact them. So for example when we recently did research where we went to some of our customer’s sites we worked very closely with our sales team to identify customers they thought would be interested in talking to us. So that was hugely helpful. Steve: So a few different approaches. Aviva: Whatever works to be honest. Steve: Yes, that’s how recruiting works is whatever you can do. Aviva: Sometimes it’s a snowball. Sometimes it’s friends and family but we try to get the best sample that we possibly can because then the research is that much more trustworthy and resonant. So we’re a little bit ADD about our sampling. Steve: How so? Aviva: Currently we do not have a full-time recruiter. Currently the researchers are responsible for recruiting their own studies. So that means they spend time on the phone with every participant before they arrive on-site, or remotely, to participate in a study, unless it’s a survey. But if it’s an interview or a usability test they’ve often been vetted on the phone first and that means that when the team shows up to observe a usability test, for example, or listen to an interview while it’s happening, their time isn’t being wasted because we got the wrong person in the door. And we’re not stretching the process out because we have to re-recruit. So getting the right sample is really critical. Steve: Is there a counter side to it, maybe just in the kind of work you’re doing there isn’t, where those people that shouldn’t have – you get those people that you don’t know how they screened in and once you get over being irked about that fact you – something different can happen which sounds – it’s a nice sort of hand wavy Steve thing to say, but in the real world you’re in, is that true? Aviva: That’s especially true when you’re doing qualitative work in the field. So for example when I went into the field last summer I was particularly interested in the sending experience and I was trying really hard just to recruit against that. But our customers had so much they wanted to tell us that they didn’t feel constrained by that. They wanted – we were on-site, they had a lot to say. They weren’t going to limit themselves to telling us about what was going on with sending. So we also heard a lot about the administrative experience and we also heard second hand a lot about the signing experience. And I’m not going to shut those people down. I’m going to listen to what they have to say because it might be something amazing or new or super important. And of course it’s important to the customer so you don’t want to shut them down. So sometimes the scope of the study can get very quickly past where you wanted it to be, but it’s still going to be valuable so you still listen. In usability testing we try to figure out in advance like do we want people who have experience and people who don’t have experience? Or what is it, depending on the complexity of what we’re testing because we want to be able to do a fairly lean, fairly tight script. So for those kinds of studies it’s maybe less important to get in the stray, but for actual behavioral observation or opinion interviews it’s a lot more interesting sometimes to let those people in the door. Steve: I think your point about participants that have a lot to tell you and I think given that you work for a company that has customers, a lot of your research is with those customers. They know you, they know who you are, they have a relationship with you. They use the product. They have needs out of that product. What they want out of those interviews is different than some other kind of contextual research where it’s more exploratory, it’s not branded, it’s not tied to the relationship. Aviva: Right. Steve: So when you’re doing your work you’re representing the company. Aviva: I’m part of – I’m representing the brand. That’s absolutely true. And I like to think that we’re part of the company’s competitive advantage because we care about listening to our customers and we’re out there doing that every day. So if they have something to tell us it doesn’t matter what channel it’s going to come in through – whether it comes in through a customer support channel or through user research conversations, but it’s going to get back to the people who need to hear that information. It’s going to get back to the product managers or whoever it is that is going to get value from that information. Steve: Can we do like a – it’s not quite a day in the life – can you kind of give a Platonic ideal project where – you know where it’s coming from, who’s talking to the people and then where that information goes? Aviva: There are so many different kinds of projects that that’s a little bit hard to do but I guess the Platonic ideal is probably the usability test. So let’s say we’re doing a rapid iterative study with a functional prototype of a product that’s not been released yet. So the people who are involved are primarily the design – the designers that working on it and the product owner, however that’s defined – it’s usually a product manager here who is responsible, ultimately, for that feature or for that product. So we will do a round of testing – it might be 5, 6, 7, 8, depending on a lot of different factors – a lot of different sessions with time in between each session to debrief and decide what did we see? Did we all agree on something that went wrong? Do we all agree on why it went wrong? Okay let’s make that change in the prototype right away if we can and then bring the next person in to validate that. So at the end of a day or two we’ve moved the functional prototype in a different design direction, frequently, and because the stakeholders were right there observing the tests we can make those decisions really, really quickly. And then at the end of that process there’s usually a report that may or may not have video clips, depending on how quickly we’re working, or what it was that was observed, but some report that gets shared with a larger group of stakeholders. That might include more people at the VP level for example who are over the product managers. Or it might include the marketing people who are working on that product, or the people who are supporting that product, or – depending on what group of people we were talking to. And that they would then hear the results of that report. And then that report would then get mailed out to everybody who is a stakeholder that we know of and that maybe they weren’t able to attend the report debrief in person but they’ll get an email with a link to the report as well as sort of here are the top things that we found in the email, just so you know about it, and there’s more detail in the report. If you have any questions come and talk to us directly. So we’re doing this broadening out the communications circle at every point. So the people who are making the decisions about what the design looks like is a much smaller group, but the people who are hearing about the information can be pretty broad. Steve: So that broader group, what are they going to – what are they going to do with that information? ‘Cuz your owners are right there and you’re making changes and making decisions as part of that process. Aviva: Right. Steve: But it ripples out, so why do they – I mean not to be so negative, but like why do they care about it? Aviva: If we’re in a usability test and we’re showing off a feature and some of the participants volunteer information about how much it will help them or what they see the value is of this, or what they like about it, then giving that to our marketing team, so they can use it to evangelize for that particular feature or product, is hugely valuable, right. ‘Cuz we’ve just said here’s what some people are saying about this that other people might resonate with. So it’s not just the usability of the product, but it’s also the entire customer experience of the product that we’re caring about. Steve: So there’s qualitative information in something like that? Aviva: Yes. And there’s also information about well we’re shipping this, but we know people are having some problems understanding this, one aspect of it. Or this particular concept’s a little bit difficult. So we’re going to do the best we can to explain it, but we’re also going to let support know in advance that hey you might get some calls about this. Or we’re going to let training – the DocuSign University folks – know we think that people are going to get a lot of value out of this but it’s kind of a complicated concept, maybe you should include it in your training. So there’s lots of different stakeholders involved. Steve: And are – now I’m just grasping at straws here a little bit, but I can also imagine as you describe these scenarios, there are things that you uncover in usability testing and you can make the best design decisions you can, but they may – if there’s an underlying mental model issue or something it needs to get shared more broadly because it’s a bigger decision that’s going to have to get made that’s going to happen elsewhere. Aviva: Absolutely. Right. Right. There’s a limit to – like if you’re on a deadline and you’ve got to ship something and you find out there’s a huge mental model conflict then the product team has to make a decision – do we keep going forward in this direction and try and spackle over it with help? Or do we step back and rethink this and take another look at it? Or do we ship this now and schedule that we’re going to take another look at this in some future release because people need this feature now one way or the other. So the product owner has a lot of decisions to make and you need to give them the best information that you can for them to make a good decision. Steve: So I want to talk a little about how you got to this point here in this organization. You know you described to me and this – it sounds very state of the art. Sort of getting everyone together, being efficient, iterating, making decisions, sharing it. And you know this as well as I do that people will hear this or people come up to you and meet you at events and so on, we’ll say well, you know, I don’t get to do that at my job because this and that. You know there’s a lot of causes. But you’re sitting in a situation where you came into this organization – we haven’t talked about your own history here. I guess it’s about the question which I’m winding my way towards is, you know, how did it get to this point where you’re able to work this way? Aviva: Well quite honestly it was starting in a particular focus and place which was the usability test. Proving the model, showing its success and its impact and just kind of driving that wedge in. But there’s a couple of things that are required in order to do that successfully and one is to realize that everybody is on the same team. They may have a different focus, but at the end of the day we all care about the quality of the product and we all care about the customer experience. If you don’t have that, if people in the company are not on that same page, it might be a little harder. I’m really fortunate to work with an incredible team of professionals here who really do care deeply about product quality and about innovation and about what can we do to make the customers’ lives better? So because I’m fortunate in that regard it’s easy to collaborate. It’s easy to trust. It’s easy to know that we all really want to do the right thing. We might not have the same opinion about how to do the right thing, or the timing of that right thing, or the cost of that right thing, but we can all agree that we want to do the right thing and so we have that shared goal, that shared mission. And so that has made it easier for the research team to – can I say be a victim of our own success? I mean we’re – there’s many more opportunities for us to have an impact than we have time. So we’re able to pick the teams that want to work with us and they’re going to commit to acting on the research insights that we deliver. If they just want us to do a study to prove that they’re right then we don’t have to take that project. But that really doesn’t happen here at DocuSign. It really doesn’t. It’s a great group of people. Steve: So can you talk a little more about what was research before you got here? What was your brief? How did you sort of move it towards this victimhood of success (which is a lovely phrase)? Give us a little background. Aviva: And I do have open positions available so we are trying to increase our size here and be able to serve more product teams. When I got here there had been a researcher sometime previously. I hadn’t met her. I wasn’t familiar with that person’s work. And we also had one person who was an intern who was doing the best he could. He was still working on his masters at San Jose State and he didn’t have an active mentor really. So it was a bit in a holding pattern. They had decided that they were going to build a lab but nobody was really looking at it really closely to see how it worked and so the lab that was delivered was not functional. So my brief was come in and build a research program, really from the ground up. Start with the basics. Start with enabling us to do user research. Start with building a lab on-site. Help us create personas because they didn’t have those. Basically I was given a bit of carte blanche to build the program and because I’ve been doing this for I guess 15 years now, 16 years now, that was great for me. I’ve gotten to build programs in other places and I’ve learned a lot along the way about what works and doesn’t work. So we just kind of got our foot in the door with some product teams and showed what kind of impact we could have and how quickly we could move. How rapidly we could do this work. How it didn’t have to take months and slow down development. How it could speed up development in some cases. And how it would just help make the product better. And fortunately everybody has been very responsive to that here. Steve: If you can go back to when you were having those first conversations. You know they have something in mind. You can – you start to assess what you think the need is and what you might be able to do, but you know you pointed out that this is a place where everyone cares about working together and cares about creating the right kind of experience for people. How do you assess, before you start living here, can you be successful here? Can you accomplish what you all want to accomplish? Aviva: That’s a tough one. I mean you do the best you can when you interview to ask questions about what the culture is like and what people’s goals are and how the different teams collaborate and work together. I’ve heard of companies where it’s a little bit more toxic, where design and engineering are sort of at each other’s throats and blaming each other a lot and not working collaboratively. I think that you can always intervene in those cultures and turn them around. I’ve seen that happen. But not here. That was other times in my career. I just had a sense when I first interviewed here over two years ago that this was a company that really cared about its customers and cared about making the experience for them as good as possible. And I’m pleased that that seems to have been the case. Like it’s held up. I interviewed with the Director of the UX team here, Sean McLeary. He’s a wonderful guy who is impassioned. You know he really wants to do the right thing and he’s willing to argue about it with his team, about, you know, tell me why you want to go in this direction? Have you considered these other things? Why have you made these choices? And if he disagrees he’s very direct about that at times. And that’s great because that means that people can be honest with each other and trust each other and not try to guess what they’re supposed to do, right. So it’s a fairly open environment. It’s an extremely collaborative environment. And the UX team has been very careful in hiring so that when we bring people on that they’ll fit into that open, trusting, collaborative culture. Steve: So you’re within User Experience? Aviva: I’m within User Experience and User Experience is within a larger customer experience group. Steve: And so what’s the size of the research team now? Aviva: Right now there are three of us, but I expect that to double within the next six months. Steve: As you said there’s more to do then you have… Aviva: There’s more to do then I have person power to do and there’s different kinds of projects that I would like to get into like doing more survey research. Right now survey research is extremely valuable at the place we are as a company. Just understanding user feedback and coding that. So that kind of part qualitative, part quantitative research takes a different kind of researcher sometimes then somebody who’s going to be doing a lot of moderating or a lot of interviewing. And so we’re opening up a contract role for that. We’re actually going to bring on a recruiter sometime soon just to give the researchers a few more cycles. And to be honest, they’re not thrilled about letting go of that time on the phone, but they’ll get used to it. But it has enabled them to deliver really good samples. We just need to be able to train the recruiter to be that good, to be as good as researchers and that’s certainly possible. Steve: Well they’re coming in when there’s already a set of values and a set of approaches that you guys have created. Aviva: Right, right. They’ll take over management of our customer panel and growing the customer panel and tracking participation and all the kinds of things that you need to do logistically in order to meet policy requirements. Steve: What do you look for when you talk to people about working here? When you’re interviewing candidates as opposed to participants what are some things that jump out for you? Aviva: Two things mainly. I want people who are passionate about research and nerdy about research. Like who love to discuss that you know – the value of doing this method over that method and can understand when you might choose a particular method over another one or know how to merge methods and create new things. But who also are incredibly good communicators and story tellers. Who can not just spend time collecting that data, but know how to then turn around and tell a compelling story and deliver compelling insights. So it’s actually more than two things because I’m also looking for people who understand not just research, but also design, well enough to be able to give good recommendations. So if somebody has a huge background in research but doesn’t know anything about design basics they’re not going to be able to translate what they’ve seen into any kind of recommendations that will be useful. So it’s helpful to have people who have some background in design as well as research. Steve: I’m so fascinated by the relationship between design and research. You know if you look at – and you can probably explain this better than I could, but the history of a lot of the stuff is not through design. You know I came up through industrial design, before there was user experience design. And that was kind of the – at least they were the industry vanguard in insights and contextual research was about that. And so you didn’t have to understand design because designers would sort of take care of that part of it. And now user experience is kind of – maybe it’s just how I see the world and where I live and everything – but, you know, I’m not doing – I’m not in consumer package goods as much as I am in software for example. I think there’s an interesting relationship. I agree with you, if you can’t talk about the implications of your thing in the language of the people that are going to act on your thing, which is usually design, then you’re limited. Aviva: Right. I really tend to hire generalists or try to create generalists because I’m a generalist. I came out of a background of doing demographic research and Nielsen audience research before I got into this whole web design world and most people aren’t generalists. Most people are either really good at qual or really good at quant but can’t necessarily pair them up. So that affects who I hire. But because the bread and butter of what we do always comes back to usability testing and user testing designs then understanding the context of web design or software design becomes really important. Otherwise you can say well they had problems doing this, but you can’t tell them why. And you can’t tell them what to do to fix it. So that’s why design is so important. Steve: So is it okay for a researcher to say here’s what you could do to fix it? Aviva: Yes, absolutely. It’s collaborative. Again, they are making a recommendation and it’s not a prescriptive recommendation, but it’s a suggestion. Like consider moving this label closer to the control. You know there’s all kinds of ways that the designer could choose to interpret that recommendation or act on it. They might change the control completely. They might – or they might just move the label closer to the control. Lots of things can happen, but it’s a place to start talking about it. Steve: So the observation might be that, you know, people consistently didn’t appreciate that this control performed this function. So one thing you might do is move the descriptor or the label closer to it. Aviva: Right. Steve: And then that’s – so that’s a way, and I think researchers – I think researchers struggle with the “A” word , actionable, right – being actionable. And so what you’re saying is just putting your observations or your synthesis, your interpretation, in the language of what you might do, moves the ball – I’m mixing metaphors here – moves the ball down the field significantly. Aviva: Well, yeah. I mean part of it is tone, right. If a researcher is writing recommendations with the word “should” a lot I’m going to have a conversation with them about their tone because should is a really strong word. Like telling a designer how their product should behave, or how their product should look is not what we’re trying to do here. We’re trying to say hey there’s a problem and here’s some ideas about ways of fixing it that you should consider, but you’ll probably come up with better ways anyways. Or they might say, huh, that is a really good idea, I’m going to go ahead and do that. Mostly designers – I think everybody wants to be given latitude to make their own decisions. Just like a designer doesn’t want a product owner to tell them what design or control to use – like put a dropdown there. Well there’s all these other ways we could do the same thing that doesn’t involve a dropdown so let me be free to consider that. Researchers are the same. They don’t want to be told do a usability test because that might not be the right solution for the problem. We all want to be told here’s the problem, here’s some ideas for how to address that problem, now you’re a professional – go ahead and find the best solution. So we take that into account. Steve: But I also I think you’re saying, if I’m not putting words in your mouth – I think you’re saying that even that dialogue about here’s the problem – I mean so a research request comes in and people sometimes say, and we all experience this, here’s the study we want you to do. And then you have to say well what’s the problem you’re trying to solve because people aren’t necessarily trained to say here’s the problem I’m trying to solve. So that’s also what’s happening – you’re trying to facilitate that happening between researcher and designer. Here’s what we’ve learned. Here’s the problem and here’s how you could solve it, not here’s how, as you said, here’s how you should solve it. Aviva: You should solve it, right. Well and also there’s a matter of – reflecting on what you said about people aren’t trained how to ask. There’s questions that we ask when somebody comes to us and has a request for something. It’s like okay what decision are you going to make based on this information? And when do you need to make that decision by? And what’s the risk if we don’t do this? So it’s a fairly limited set of questions. It’s fairly easy to ask in a meeting or an email, just to get enough information to know huh, okay you’re asking for a survey, but what you really need to do is a card sort. They might not know what a card sort is so then you have an opportunity to explain that and to demonstrate how it’s worked well for other people and usually they’ll go that sounds really cool. Let’s do that. As long as it fits within their timeframe and their budget. But if somebody comes to me and needs a decision in two weeks I might prefer that they do a long field study that’s going to take a month to get back anything meaningful, but they don’t have the budget and the time for it so we’re going to figure out something that can work within the timeframe even if it’s only directional. Steve: And there’s a managing expectations part of it that I’m sure you’ve kind of championed in your time. Aviva: Yeah, I haven’t thought about it that way, but that’s actually a good way of putting it. Steve: You know one of the things I enjoy about these conversations is hearing the comparing and contrasting what it’s like to be someone who works outside the company. Because you and I are both taking requests. We’re both having to kind of push back, clarify, go back and forth, but the context is just so different. You know the dynamic is so different. The money is different. Everything about it is different, but yet that’s – anyone that needs research done has some amount of familiarity and some amount they don’t have nor should they. That’s kind of what they’re coming to you for. Aviva: Sure, sure. And I’m just happy that they’re asking us. You know they might be asking for the wrong thing, but at least they’re having a conversation with us. So it’s great to be asked. And then it’s an opportunity for us to maybe teach a little bit. But at the end of the day the research isn’t the product right? The research is how you get to the product. So we don’t expect to be treated with the same kind of importance that somebody would be putting into the actual development process, or shipping code, right, ‘cuz we’re there to help enable that. But they could do it without us. The outcome might not be as good. It might, but it might now. Steve: And historically – I mean you know this ‘cuz you’ve been around, they did do it without it us. Aviva: Yes, that’s right. Steve: So – you know, just to sort of reflect on where we are at at this point in time and you’re in a leadership role in this function and you’re evangelizing it, advocating for it and now a victim of your own success. That this is – I think you represent a different state of the art of how this work is being done. I don’t know, it’s interesting to sort of see where have we all come from in this – where have we all come to I guess is the phrase. Aviva: Well you know I’ve dropped out of enterprise from time to time and gone and worked as a consultant and it’s been really fun and I’ve been successful in my own way at it. I came and joined this company because I thought something special was happening here and it was the right time to do it. And I got to choose. Like there wasn’t any urgency for me about oh I need a job, I’m going to take the first one that comes up. And I think knowing as a user researcher that your work is valuable and that there’s high demand for your skills and that you have the ability to choose the place that you want to work and can choose the kind of culture that you want to contribute to is really important. That you have power as a user researcher. I think we’re in a different time right now than we were 10 or 15 years ago. The model has been pretty well proven. I think a lot of people understand the value of UX and the role that research plays in that. The role that talking to the customer and understanding the customer and the user experience plays in a company’s success. And so they’re actively looking for people in those roles. Steve: You started off our conversation by describing how you guys are an unusual type of startup, but – does that – does the – does that business context, start up, what kind of start up, how old? Does that change some of the – you know how proven the model is among these different audiences? Aviva: Absolutely, absolutely. You need a fairly mature management organization and fairly experienced product managers to – who know the value of this stuff. I think any MBA who’s been reading the Harvard Business Review gets the importance of UX on some level, but that may not translate into really understanding what it takes to do it. So you want to look for the people who have shipped before and who have worked with design before. I’m not the kind of person who wants to be the design team of one. There are other people who are really, really good at that. I don’t think that’s where user research shines obviously because you need a design team of at least two in order to support the kind of work that I want to do which is research. But that’s just me. I don’t want to be a designer. Steve: Meaning a researcher and a designer. Is that what the team is? Aviva: There are a lot of designers who do research, right. It’s a really common thing and you hear about the UX team of one where there’s one designer. That designer is going to have to do some research. A company has to be large enough to support a user research team. They need to get to that point and otherwise there’s not going to be a role for somebody who’s a pure researcher, let alone a research manager like me. So I’m going to pick those companies that are past that inflection point where they actually have enough designers on staff to support a user research team so that I can do what I like to do which is train user researchers and build teams. Steve: So design precedes research in this example. Aviva: You know that’s been the case everywhere I’ve worked. I think there’s been some cases where some really farsighted start ups have actually started with the research before they bring in the designers, but I’ve never encountered any myself. Steve: I mean my theory is that just the research industry and the design industry, that we lag. There are people like you who are in-house leaders of research functions but that’s still relatively new, but not that long ago there was no one that was a leader in design. That was a very rare thing. Now it’s a very common thing. And then we’re kind of trailing them. You know what are people going to general assembly to learn? You know what is the market like out there? That those things are – that we’re always where they were a couple of years ago. Aviva: Yeah, yeah. I think user researchers are just, user experience researchers in general are sort of odd ducks in a lot of ways. But yeah it is definitely a trailing career path in some respects. The classic example of a start up is the business guy and the tech guy, right, and sometimes you get lucky and you get a tech guy who also knows design and maybe they know research, but not usually. That’s not usually how it plays out. Steve: I have to ask you to expand on the odd duck comment. Aviva: Well you know we’re – researchers tend to be really fascinated by how things work. More so than we are in creating stuff that’s new. So our perspective is at the level of pattern. Like what are the patterns that are happening here and how can we evolve those patterns or how can we take advantage of those patterns or whatever it is. We’re looking at different measures of success for ourselves then a designer might. And we’re doing different kinds of things every day and yet we’re still in lock step with them when we work, at least that’s how it’s been for my teams. My teams have always been part of design. They’ve always been part of UX. And without UX there’s no UX research. Steve: Well that tendency you’re describing seems like some of what you’re trying to mitigate with how you’re asking people to report. So don’t just say this thing is happening or this works this way, but lean in (if that’s not too overused a phrase) lean into the thing that’s not your default and say how it could be different. Aviva: And I think the reason for that is in my time in this field the biggest complaint I’ve seen from decision makers are you’re giving me this information but you’re not telling me what to do with it – I’m not sure what I should be paying attention to. I’m not sure whether I need to get involved or not. Help me understand what I need to do next. And so part of that process of designing a report, or even a memo, or an email about a study, or designing the study itself is helping – helping the team understand what’s important and what they need to pay attention to and how they can fix things is a part of that. Otherwise you’re just kind of saying here’s some information that’s undifferentiated and kind of sending that over the wall and you don’t know what they’re going to do with it ‘cuz it’s divorced from the product experience. But I want the research team to be part of the product team. I want them to care about the user experience and that means caring about what happens as a result of their research and how it gets implemented on and acted on and built into the product, if that makes sense? Steve: Yeah, it does. I think the whole sort of research analysis and synthesis process is – I mean here’s how I look at it I guess. It’s a series of iterations of oh wow, so what, oh I think it’s this, oh wow – you know you’re kind of picking things out, being excited about them, discarding them. It’s maybe like kneading dough or something. You pull some things out and push them back in. Aviva: Right. Steve: And constantly kind of refining and I think a certain amount of it is sort of for your own enjoyment and I don’t mean that in a self-indulgent way, but… Aviva: That’s true. Steve: …you’re just trying to find the edges of the space. You’re trying to figure out what it is. Play with it, work it. But at some point you have to start to speak to someone who doesn’t speak your language which I think is a theme from our conversation here that you’re trying to communicate with other people that have other needs that need to act on it. Aviva: Some years ago I gave a talk at a conference about walking the walk and what I meant by that is we talk about user experience and understanding the user’s language all the time, but we don’t always use that same set of skills for our own deliverables. We don’t always understand the mental model of the people that we work for and what their needs are and try and accommodate to that in how we deliver information, and that’s a huge fail. We have to not just talk the talk about understanding the user experience, but think of our own colleagues as the users of our insights and information and how can we communicate those in ways that are going to be useful and usable by them. That’s a big thing for me. Steve: Yeah. That’s great. I think walk the walk is a lovely label to put on that. I’m going to switch gears slightly. I think it will, as it always does, it will I think maybe pull in some of the things we’ve talked about. I’d just love to hear you talk a bit about – you’ve eluded to it, but talk a little more about your background. I don’t know… Aviva: My background is weird. My background is crazy. You know everybody gets into user research through these bizarre channels. It’s only recently that there’s programs where you can go and study user research in a master’s program and a design program and then apply it. So I came to the field because I was getting a doctorate in understanding how people communicated online which is a weird thing to get a doctorate in and I had to sort of make that up for myself. I started out as an educator and as part of that master’s program I was doing a lot of demographic work on different populations. So I learned how to do survey analysis. I learned how to use survey tools. And then at some point while I was being an educator I was a – I did a lot of work in group process situations. So it was information education and community education. So I learned a lot about how to get a group to move in a particular direction or how to run a meeting, things like that. I decided I didn’t want to work in non-profits anymore. This interesting thing was going on at the time with bulletin board systems and people creating community and talking to each other at a distance through these technologies that I thought was fascinating. I was a member of The Well at the time and some other bulletin board systems that were places where people could chat and there was this beginning of a academic area of cyber studies happening. So I decided I was going to change careers and go to school and study that. And I got into the University of Texas at Austin where they had this amazing group called the ACT Lab, or the Advanced Communication Technology Laboratory under the oversight of a really amazing woman named Allucquére Rosanne Stone who had done some of the early research in these cyber studies. And I started putting together a program for myself to understand how people communicated online. While I was doing that the Web sort of happened. This was in the early 90’s. And all of a sudden people were putting up these billboards of self-presentation on the information super highway. They were talking about themselves online. They were talking to other people online as a part of that. They were developing community through the Web. And I decided yeah I’m studying that. That’s interesting. So I started – a couple of things happened. Because my program was part of basically a film school, the Radio Television Film Program, production was really important and I wasn’t a film person so my production teaching was about the Web. So this new technology, the Web, was happening and I taught the first for credit class at the UT of Austin in designing web pages. And anybody could do it then. It was like this was in the days where image maps, being able to click on an image and have it do something was a big deal. So this was before CSS. So that’s how I got started on the Web and taught some classes in designing interface. That was part of what was going on in this lab. But when I decided what to study as a doctoral student I was studying what people were doing with the Web and how I decided to do that was to interview them in person and have them give me tours of their personal homepages. So it was kind of like doing digital ethnography. In fact it kind of was doing digital ethnography. We just didn’t really have words for it then. But sitting next to somebody over a computer and having them share their experience of it with you led very naturally into a user research role when I got out of school and decided I didn’t want to be an academic anymore. This was during the first bubble and there were a lot of opportunities so I ended up in a small e-commerce start up as a user researcher and just stuck with it and kind of learned as I went. We were all making this stuff up I think. When I got to Yahoo, which I think was my third job in the field, that was an amazing place to learn the trade. At the time there were – it was such a big, active, busy environment with so many great people at the time and a really solid group of user researchers that I could learn from. So that’s kind of where I came up and was trusted with building a team for Yahoo Media in Santa Monica. And so left the Bay Area and went down to Santa Monica. Was just a user researcher by herself in this team and built a team. Hired more people. I think there were six of us when I left and that included some market research as well as user research because we became a customer insights team at some point. But I’ve been doing this since the year 2000 and it’s been a great career. It’s been a great ride. I would not – I couldn’t have chosen a better time to change fields because I love the research aspect of it, but I also love the collaboration aspect of it and the fact that we’re building stuff and sending it out in the world and that people are actually using it and what we do in the lab affects what they’re doing every day. Steve: Do you think your other pursuits, and I’m thinking – I know that you play music and have been involved in music I guess since I’ve known you. Do things like that fit into the creative process that you harness for research? Aviva: That’s a really interesting question. I think any kind of creative process is going to kick – kick start other creative processes. I think sometimes your brain just needs a break. When you’re trying to solve a problem sometimes it’s nice to step away from banging your head against it and do something else creative until you can get back some perspective. Steve: So from head banging to head banging. Aviva: Yeah, pretty much. That’s funny. Yeah, at some point in the past 10 years I went from being mostly an acoustic player to increasingly being a loud electric player so head banging probably applies. But I really do think of the music as a hobby. I’m not sure that I can draw an active connection between what it’s doing for me in research. It’s just a different place for my brain to be. One of the things that it does for me is it forces me to live in the moment more because when you’re playing lead guitar you’re reacting kind of on the fly and jamming with people around you and so that’s giving me more confidence with trusting my judgment in the moment. So maybe it’s doing that for me a little bit. Steve: It reminds me of the workshop that you led that I participated in where you had people play instruments that they knew or didn’t know, kind of on the fly, and you pulled out all kinds of great lessons about being in the moment and collaborating and trusting each other and probably 10 other rich things that I’m omitting right now. Aviva: Yeah, that was fun. And that wasn’t a workshop that I would ever have come up with on my own. It was something I was invited to do and it was a lot of fun doing it, but I honestly – I’m still kind of amazed that I got to do that, that I got to go to a UX conference and get people to play with musical instruments. It was fun. Steve: I didn’t know it was like a – it just seemed to make sense in the audience. Aviva: No, I was invited to do that by Erin Malone who coordinated and curated that conference. Steve: If we were to sit down in five years and talk about some of these kinds of things – the field, DocuSign and the work that you’re doing – I don’t know, any of the things – we’ve been talking about the past up ‘til now, but if we flip it to look forward are there any things that leap out for you as either aspirations or expectations for the future? Aviva: I think a lot about scale. How to do user research at scale? How to communicate findings at scale? How to make sure that we’re not losing insights when they’re being communicated at other parts of the organization. So I’ll probably be thinking a lot more about tools between now and then. And I also think a lot about how to train the next generation of user researchers because I know there are opportunities for them and I don’t think we’re – we’re churning out more designers I think in the system then we are churning out user researchers. I often talk to new graduates and they want to do both, but they don’t want to just do design and I want to know how to identify and train the people who would be stellar user researchers and maybe aren’t born designers, but would be great in this role and bring them up. So I’ll be thinking about that a little bit. Steve: That’s great. Is there anything that we should have talked about, that I should have asked you? Aviva: I love that question. I use that question all the time. If you are a user researcher who is looking for a great opportunity and has some years of experience under your belt and want to work in a place where your work is appreciated then please get in touch. I’d love to talk to you. Steve: Do you have any questions for me? Aviva: Is this what you expected? Steve: Well yes and no. I mean yes this is the kind of conversation I was hoping to have. No, I didn’t expect exactly what I – the content was not expected. So yeah, this is… Aviva: Well good. I always want to be a little bit surprising. Steve: Yeah, and this was really good. Aviva: Well thank you so much. Steve: Well thank you very much. Aviva: This has been fun.
Jan. 19, 2016
We kick off the second season with Judd Antin, the Director of Experience Research at Airbnb. Judd and I speak about their model for embedding talented generalists with product teams, skill-sharing among researchers, and just what exactly makes research “sexy.” I don’t know of another way to do things better than to give and get feedback. It should flow like a river. And I think that can be hard, to be open, to focus on strengths rather than weaknesses. To be humble is a thing we’re always seeking to be better at, but that’s how I approach the task of building a team. – Judd Antin Show Links Judd Antin Follow Judd on Twitter Airbnb Trello Alex Schleifer Kathy Lee Janna Bray Steffen Kuhr Natalie Tulsiani Brian Chesky Joe Gebbia Katie Dill Adrian Cleave Improv and user research French Culinary Institute Your Data Are Wrong: The Hype and Reality of Big Data The Incredible Hulk Follow Dollars to Donuts on Twitter (and Stitcher) and take a moment to leave a review on iTunes. Transcript (French version) Steve Portigal: Welcome Judd. Judd Antin: Thanks, Steve. Steve: It’s great to speak with you. Maybe let’s just start as we always do with just kind of the broad strokes. Do you want to introduce yourself? Tell us a bit about you and what you do, what your role is. Judd: Sure. Sure, happy to. So I am the Director of Experience Research here. So experience research at Airbnb is a team that was formerly called Insights, but what we really are is a UX research team. We are embedded in our design organization so we have designers, content strategists, product managers, engineers, data scientists as our partners and we’re a team of 17 folks at the moment. We have a pretty diverse group of people on the team which is something we do on purpose. And everybody is sort of embedded in their teams, working directly on the day to day of products for guests and hosts all the time. So, yeah. Steve: That’s good. And of course those intros, as we know from research, like we could just follow up on all the things that you said. That would probably fill our whole interview. Let me go back to some things. Can you explain Airbnb? Judd: Sure I can. Let’s see if I can do this in a nutshell. So Airbnb is a marketplace that let’s hosts who have space come together with guests who need it. So…we provide an opportunity to travel in a much more local way. So I think you know, it’s – you know Airbnb has been one of the poster children for the sharing economy but I think for us the way that we think about it is that Airbnb is an opportunity to connect hosts and guests together, to have much more genuine local experiences. Local in the sense that you’re traveling to a place that is pro-, potentially just in a neighborhood. You may be staying in a spare room or a place where your host is either there or they’re on vacation and you’re sort of experiencing their city from their point of view and you know getting recommendations from your host about where to go in the neighborhood. So I think that is the thing that makes Airbnb really beautiful and unique is that it’s kind of this view of travel which is to say yeah you show up in San Francisco, the Golden Gate Bridge and Alcatraz and Coit Tower, those are important things that we know from research that everyone wants to see, but if you never left Union Square and Fisherman’s Wharf it would be hard to say that you got a real sense of what San Francisco is like. So Airbnb really lets that happen. Steve: And you started off at that great explanation by contrasting sort of a common or public perception around it – you said “poster child for sharing economy.” Is there something that – and obviously everyone in the company – everyone in any company understands the contrast between how they’re perceived and – I’m just thinking about you in the role that you have and what your team does. Do you deal with that – I don’t know if it’s a gap or delta – the world thinks of us as this. You know we go out and people to sort of understand. I don’t know… Judd: Oh yeah Steve: You’re nodding so I’ll let you answer. Judd: All the time. We spend all of our time talking to guests and hosts on Airbnb. We make a concerted effort through travel and remote research to get outside of the Bay Area bubble for example because, you know, Airbnb is very much present here. But what that allows us to do is see what the everyday experience of traveling or hosting on Airbnb is like for people and I think that means that we get a huge dose of real talk. You know we have a mission – you know sort of our tagline as a company is belong anywhere which is something that we all pretty deeply believe. We want people to be able to have a travel experience which lets them have a really genuine welcoming experience of a place that they can start to feel at home there and belong there. But we learn a lot about how that vision sort of translates to reality in the everyday life of a host, or the everyday life of a guest who is traveling. For host that might mean we expose the diversity of hosts. So for a lot of hosts we talk to the – the money is really an important reason that they do it, but they also feel a strong connection to hospitality. They like to experience people who are visiting them from all over the world and they talk about that as a motivation for doing it. For travelers, similarly, we find out that finding a cheap place to stay is often – I mean let’s – however much – that’s not going to be a part of our vision at Airbnb, but it is a reality of the way that people think about Airbnb when they travel. It is a part of their motivation for getting on the platform. And so I think one of the great things about the research team is that we can be that dose of real talk. We can say like okay this is important. This vision is important. The mission is important. But let’s talk about the real lives of guests and hosts and what matters to them. Steve: How do you characterize – I don’t know, I can only ask it through suggesting an appropriate metaphor that – and I know the synthesis or the ping ponging, if you’ve got – you know here’s public perception, here’s what real people involved – here’s what the real talk says. Here’s what our internal aspirations or you know Kool-Aid or whatever it is for any organization – I mean how – what’s the sort of – tell me about how those things come together. You have to deal with what’s believed and what the aspirations are, with what the world says and with what the real talk from real people is. Judd: I think we have a pretty happy balance between the Kool-aid, or mission driven aspect of what the company does, and the real thing, the real talk of people. What we do most consciously is probably to remind people in this building that we are nothing like the vast majority of people on Airbnb. And that’s where I think the real talk matters because if you make a bunch of assumptions on the basis of what seems obvious or intuitive to us here in this fantastic building in San Francisco in SOMA, like we’re going to get it wrong. We’re going to get it wrong for California, for the United States and certainly for the world, the vast – the majority of our business doesn’t happen in the United States. So I think that is one of the most important roles of real talk is just to say… It’s not that difficult. I don’t think we feel a real tension between the mission and the values and the everyday lived experience of guests and hosts. We just have to remind people that most people’s daily lives are nothing like ours and so we just have to keep that in mind when we make design and product decisions. Steve: I love that that’s not a tension. Those things can work well together. Not every organization is the same and maybe elsewhere you’ve observed that as more of a tension. Judd: Yeah Steve: But, why? How does that work well here? What’s going on? Judd: Here’s my theory. My theory is that it’s because we’re a travel company and here’s a list of things that we don’t have to worry about – advertisers, monetizing engagement. What that means is that to the degree we can get people to find the perfect place for them, where they want to travel, get them out of our digital products and out into the world to travel, that’s success. You know what I mean. And so I think we don’t have to make a lot of assumptions about what motivates sort of digital behavior. We have to get people to travel and I think it attracts a certain type of a person who – and you know that combined with our mission for belong anywhere, we think it’s easy to have that kind of empathy for the world because traveling is such a – as an experience is kind of full of empathy. It’s like understand the world from someone else’s point of view. And as a research team we kind of focus on that too. We’re like hey product team, let us help you understand the world from another point of view, the point of view of guests and hosts that are nothing like you. Steve: That’s wonderful. So the company that’s about getting out of – giving people the chance to experience something differently than what they’ve experienced, in other words travel, is one way that a culture gets created that there’s a hunger to understand – a willingness and a hunger to understand. Judd: Yeah. Steve: The world is different than what we’ve assumed. Judd: Yeah. And I think that’s a big selling point for researchers who want to work at Airbnb because you know they think – travel is such an evocative experience. You know if you look back at the – I think the average person who thinks back over their life to the 10 experiences that were most educational, most instructive, most inspirational for you, a good chunk of them are probably travel experiences. So people’s eyes are open. Like it’s emotional. It’s intellectual. And so as a topic for research, you know facilitating that kind of travel experience is really sexy for most researchers we talk to. It feels really good. It feels really real. You know I think sometimes working in the domain of digital products you can think what does this all add up to? You know what experience I’m really making. Like we don’t have that problem because travel, we’ve all had experiences of travel that are truly sort of transformation. Steve: So what are the experiences that researchers have? Can you give some context or situations where researchers are involved in – are researchers traveling to study travel for example? Judd: Yes. So we have done a fair amount, not as much as I want us to do, but we have started to do travel, primarily as a way of understanding – getting outside that Bay Area bubble. We spend a bunch of time in the homes of hosts. That’s one thing I think we do a lot of. So one thing we haven’t done yet is kind of like I think a travel along project. You know if you research at Uber you could like ride along with an Uber rider or driver. We haven’t done a travel along. I don’t know, that seems a little invasive and weird. But we spend a bunch of time in hosts homes and one of the reasons we do that is because that’s one of those things where it’s very difficult to get a full picture of the day to day of a host, the things that challenge them just in the flow of trying to provide hospitality and use our products without really getting into it with them. Find out what’s their routine of sort of messaging with potential guests – you know scheduling, cleaning, doing key exchange, all of that day to day stuff. It’s really hard to find out the mechanics of that without being there. And it can be very difficult to get the context and depth and empathy we need about motivations for hosting, let’s say, without a really in depth face to face conversation and where better to have that conversation than in the home. Steve: Right. It’s just funny sitting here that you refer to it as their homes. Of course it’s their homes, but I’ve only ever been a guest, never been a host. So the unit that you’re referring to, this environment, it’s a place that I stay. I forgot that it’s somebody else’s home. So to hear you say that, even the language, which I think is probably a key – that’s probably key for you guys to be talking about these as homes. Judd: Yeah. No, I think it’s true. And you know I’ve been thinking a lot about the language we use. You know I think in research – you know I don’t know many people in user experience who use the word subjects. We mostly talk about participants and we use – you know sometimes we talk about users. You may – you know we can talk about whether that’s a good thing or a bad thing, but I kind of always want to turn to guests and hosts – the things we do for guests and hosts because it’s a reminder of – it’s similar to the way you said oh right it’s someone’s home. It’s like right, this is a guest, this is a traveler and this is a traveler who is showing up in somebody else’s space, in their home where they live and sharing that space whether or not the host is physically present at that time – they’re sharing that space. And on the flip-side like this is a host. Not only is hospitality and hosting something that we care a lot about as a company, something we try to promote with our products and look into with our research, but it’s a home. You know it’s really personal. Again, like that is a sexy thing for researchers because it’s like one of the most important things in your life is where you rest your head. And the idea of letting somebody else into that home can seem – you know that’s where the issues of trust come in that are so interesting to look at because you have so many dimensions of that. You have the sense of trust that’s instilled in Airbnb as a brand, Airbnb as a platform. The way we facilitate relationships between guests and hosts. And then you have this face to face personal think where I am letting you into my house if I’m a host or I’m going into your house if I’m a guest. And you know I have a lot of people when I talk about Airbnb that ask about trust and safety and things like that. And the only thing I’ll say about that is it gives me a huge amount of faith in humanity how rarely there’s a serious issue. You know. The number – like millions and millions of nights are stayed in Airbnb’s and the number of serious issues that happen when people trust each other and guests go into homes is just minuscule. From our point of view it’s too many – one is too many. Steve: Right. Judd: But it’s really sort of restoring a faith in humanity and I appreciate that we are constantly on the lookout for doing research that can facilitate that. Like creating wonderful rich, trusting interactions between guests and hosts. Steve: And another reframe for me that’s helpful to think about this is these two groups of people that are being connected as opposed to a bed and a toilet for the night. Judd: And to the end of real talk right. There are some guests who do see it as a bed and a toilet. You know, it’s a cheaper bed and a toilet then they can get at a hotel. But not all guests, probably not most guests. You know the aspect of hospitality, the aspect of getting into a neighborhood, living like a local, it’s really powerful and I think it’s a real motivator for Airbnb. Steve: I want to dive into some other language that you used and maybe get you to unpack that a bit. You said a couple of times that certain kinds of problems or areas are sexy for researchers. So that suggests some insight into what appeals to researchers. Like what we like doing. So what does sexy mean for researchers? Judd: Yeah. So in my experience, when I say sexy – when I say sexy I mean two things. One, a topic for research which is really meaty, which you can really wrap your arms around and spend years at a time diving into. It has a lot of dimensions. It’s not easy. It’s probably a challenge. I think most researchers find the challenge sexy. For example it is something that requires us to deeply understand a problem and figure out how to translate it into design and product and communication. So that’s one aspect of sexy. And then the other thing that’s sexy I think for a researcher is the fact that they can be set up for a direct line to impact product with their work. I think when I talk – I talk to a lot of candidates and when I think about how research is structured in their organization what I hear is well they had to advocate for budget, for a study, and it took them several months to do that. And then they completed the study and then they had to present it up the chain three or four or five levels to get that VP to advocate another VP to get that PM to put it into the roadmap. And it hurts my heart. You know it’s like the worst way to do research. And so I think researchers find it sexy when they don’t have to do that which is – which is one of the beautiful things of Airbnb which is that we don’t have a perfect situation but we have researchers set up as partners, with designers, with PMs, content people, data scientists, engineers, at every stage of the product cycle. The way that they have impact is through being there every day for direct conversations about what we should build? How should we build it and why? Are we doing it right? Iterative development, launch and learn, repeat, you know. And that is sexy to a researcher. Steve: Can we talk more about that staging or how people are set up? So there’s a number of individuals with different roles that are partnered together. Judd: Sure. So, yeah – so we have an embedded model for research where – which I would contrast to the more service-based model where there’s like a central organization that sort of fields requests and sends out a researcher into the field. Our model is an embedded model where a researcher gets situated into a product. Let’s say that product is search. That researcher sits physically next to, hopefully between the PM and the designer, the engineer, the data scientist. And I make a big deal of that physical presence because I think that’s where a lot of the action is. There are a lot of – I’ve had conversations with researchers before where they say I need to get into that meeting. I go, what meeting? And they go that meeting where all the decisions happened. And usually I look into it and find out that meeting doesn’t exist. That meeting happened like in between her desk and his desk at 1:45 and it happened because – like I’m elbowing a shoulder right now. Steve: Um-hmm. Judd: Like that’s why it’s so important to physically have that person there, physically present for everything that’s going on in the product cycle. And that person finds partners in all the other functions. So certainly with design – we have very close relationships with design, product managers. We have a strong and wonderful data science team here. Engineers of course. Content strategists. Everybody is kind of at the table, one very collaborative team trying to make great product. Steve: So I have a dumb question and we can say there are no dumb questions. Judd: There are dumb questions. Steve: Thank you. Thank you for affirming my dumbness. You know product cycles, development process focuses on different tasks are hot and heavy at different points. So I’ll just ask this in the dumbest way possible, what’s the researcher doing at different stages of that? What does it look like for them? Judd: Sure. Yeah. So I guess it would start with an exploratory or formative stage where we’re trying to figure out what we should build. What are the problems? What do they look like? Why are they problems? So very often that’s a place for in-depth qualitative research. We love to have a model where we do in-depth qualitative research and then we follow it up with really rigorous larger scale survey work to check prevalence. So that kind of one-two punch is really powerful for us so we can say here’s a deep, contextual rich understanding of these problems and we know a lot about how prevalent they are and we can segment them a little bit in ways that matter to us. So cool, we did that. Okay now maybe we have a product road-map. Or we’ve decided to build a thing in a certain way and the designers were hopefully participants in that research from the very beginning. Very often they are in the observation room and they’re coming on home visits. They’re participating in feedback on interview guides and surveys, all that stuff. But then I think they really kick into gear at that point where the implications for design kicks in from the study and we are starting to do sketches and ideation and visioning work and I’m a huge advocate of low fidelity prototype testing as a way of sort of pointing the ship. So at that point the researcher might be doing sort of a rolling study where at first mocks maybe Framer prototype, sort of golden path, Wizard of Oz-y things that are a little bit more rich and interactive. Doing repeated user studies with those. And then at a certain point in our cycle we’re sort of iterating and we end up passing off to engineering. Again, like the engineers are often there through the whole process, but then the sort of hot and heavy engineering kicks in. At that point there might be some more design refinement that can go on, or if there’s a concurrent project that the researcher is working on. But we get towards a stage where we have a functional prototype we can test we tweak that before launch. We launch. Now we can talk to real users in the wild and we get all that great ecological validity and then we’re back to square one in a way which is okay. Maybe we worked on a thing, we improved it, but a whole new set of problems reveal themselves. Like no design solution is perfect and so the researcher’s job is never done. Steve: That’s awesome. I feel like applauding that narrative. So everybody’s busy doing this stuff. Again, I guess that’s the embedded model kind of brought to fruition. In a service model you probably wouldn’t either have the resources or wouldn’t choose to do it that way. Judd: Yeah, I mean – so here’s – like the way that I think about that trade-off is that – the greatest thing about a service model that I can see is that you can really provide the person with the perfect expertise for a product – for a research project that’s being requested. We have to have a model that’s a little bit more like a generalist. So people on my team are generally expert at something. So maybe they’re just more passionate and experienced with diary studies, or surveys. They do little stats. You know we’re a very multi-method team. But they kind of have to be a generalist because whatever – if you’re embedded, whatever research need is kind of thrown at you, you need to handle it. You need to handle it. You need to collaborate with another researcher who’s an expert. If it’s sort of at the edge of your skill set you need to work with the data scientist. But it demands this generalist model. And then the other thing that I think is important is that in the service model – like I – we don’t spend a lot – like the amount of time we spend on reporting varies a lot, but it is generally not a huge proportion of their time. I think of a huge amount of time spent on beautiful Keynote or PowerPoint decks as the price you have to pay for a service organization rather than an embedded organization because when a researcher is embedded presumably that PM, that designer, the engineer, they were along for the whole thing. They’ve seen it at every stage. They were in the back room and you debriefed after every session or at the end of every day. They participated in the affinity diagramming and so we may still do a video reel, cut together some clips. We may put together a Keynote, but that’s not the primary deliverable. They’ve been getting it the whole time. It was a constant back and forth. I just think that’s so much more efficient and it lets a researcher develop like real product expertise in that area rather than parachuting in and out all the time. Steve: Are there ways that – as you said they have to work as generalists. But you have diverse staffing, people with different backgrounds and different experiences. Is there any sense of kind of community practice that you get very specifically in that centralized model? Like the researchers sit together and they share stuff, but what’s the gestalt you can create in this model? Judd: I think we have to work just a little bit harder for it because the peripheral awareness isn’t necessarily there, the way that it would be if you had one team and sort of everybody knew who was allocated to which project at which time. And honestly like we don’t have a perfect solution to that yet. I’ve never been a part of a research team where we didn’t have a problem with that type of information sharing and it’s really important and we’re trying to do better at it. Actually we’re talking about it right now because number one we want that community of practice. We want feedback to flow. We want people to find beautiful, like synergy of research questions. We don’t want to have people doing the same project in parallel or reinventing the wheel. And all those things require that kind of peripheral awareness of what everybody is working on, or the ability to get that information quickly. On the other hand you have this sort of classic, collective action problem where if it’s too onerous for any one person to provide that information, or consume it for that matter, then the whole model falls apart because it’s sort of only as strong as its weakest link. So we have to figure out how to create a lightweight model. And to be honest the way we’re doing that right now is with Trello. I don’t know if you ever use Trello? It’s a pretty low tech tool in the sense that it’s like the closest analogy – Alex Schleifer, our VP of Design and my boss, he calls it like a digital whiteboard. It’s got columns and cards and columns. So a product team has a column and a card is an active project. And it might have a link to a research brief or an interview guide, but it’s just that simple. And that’s it. And people can check that to find out what’s going on. They can update it. They can move a card into the done pile. I don’t think it’s a perfect solution, but you call that a real problem that we work hard at. Steve: So everyone can see what’s being researched? Or see what has been? You can look in the done pile and see what’s been done? Judd: Yeah. So we’re still working this out, but the model is that things that are recently completed stay in the done pile. Eventually they kind of get archived and they’ll live in kind of a research repository elsewhere. But things that are active or recently completed, it’s sort of a one stop shop to view them all. We’re going to try it out and see how it works. Like I don’t think there’s a perfect solution for this, but it’s something that I hear from the team that we need to work hard at. Partly because we find those collaborations and cross-pollinations and partly because it helps us feel like a team and like we have that community of practice and that’s something that we work on actively promoting. Steve: That sounds like another sort of – that’s the tension of the distributed – I’m sorry, the embedded vs. the centralized model. Not that the centralized model has it figured out because you know creating deliverables – I don’t know how many times I’ve encountered the like – I’ve had – I’ve been the vendor outside an organization and had people in that organization come into a new role and not know of the project and have to reach out outside the organizational walls to get that collateral or those deliverables from a vendor from a year ago, me. So it’s not like if you – it’s not like the centralized model always works for that. The deliverables just have a shelf life and they disappear. Judd: For sure. Steve: So is this knowledge management we’re kind of talking about here? Judd: Yeah. You have somebody with institutional memory and they’re always there. Although of course moving from team to team to sort of cross-pollinate is a good thing, but we probably do that on the order of like 18 months to 2 years. I think everything is a trade-off. I choose to optimize for perfectly positioning a researcher for impact and to be a voice, a constant voice in the product process. The trade-off of that decision is that we have to work harder at the community of practice bit. Thankfully we have a really passionate and collaborative group of researchers at Airbnb who are devoted to that, actively seeking that and we’re small enough that we can have – you know small enough physically in that we can all sort of see each other in the space of 30 seconds and that we can meet in a reasonable sized room and share ideas on a regular basis. Steve: Are there – do you have regular meet-ups or certain types of activities that are meant to make that happen? Judd: Sure. You know we are in the midst of a concerted effort for information sharing. So we meet all the time. We meet every Monday morning for a standup which is sort of go around the table and everybody say the two things they’re working on that week. We meet every Tuesday for a kind of what I could call more like a team meeting. Like we get into it. We talk about recruiting and hiring. We often have special topics. We talk about growth. We have – growth in the sense of team growth and individual growth. We talk about sort of big picture product and design context stuff that’s going on. We also have something that Kathy Lee (one of the researchers on the team) put together which is Shop Talk. So it’s like a Friday event in which people can have a beer and give a presentation they would have given to their product team to the other researchers. And we’re also instituting skillshares. I sort of demand that everybody on the team – so my model for this is sort of, in my memory as somebody who got a PhD – so when we get a PhD you become the world’s foremost expert at some tiny little thing that nobody cares about and then you write a dissertation which if you’re lucky your advisor will read and then you’re kind of done with it. But, so clearly that’s why I’m not in academia, but one of the great things about that model I think is that you carve out this niche for yourself which is your calling card and that’s a thing that I ask every researcher on my team to do. Yes there’s this baseline that we must be rigorous, beautiful generalists, but what is the thing that you’re uniquely good at as a researcher? What is your calling card? What is the thing that you want to at first sort of really consume all the materials you can find about, go to seminars, take workshops and read books on and then before you know it you’re writing the books and you’re teaching the seminars and you’re giving brown bags to the rest of the team. I just think if you have a team where everybody has that unique team, an aspect of methodology or practice, communication or whatever it is, that they are passionate about developing into a unique skill and they’re all committed to sharing it with each other, that’s my model of a team that grows individually and grows as a team. Like that sounds awesome to me and we’re work- you know I don’t think that we are perfect in that regard, but we have that as an aspiration. Steve: What are some types of things on your team – topics for different researchers, that’s there thing? Judd: Sure. There are methodological topics – just as simple as survey best practices. So we have somebody on the team, Janna Bray, who has made it sort of a thing she’s really vocal and generous about to raise everybody’s game at surveys. We have another researcher, Steffenn Kuhr who is really passionate about rigorous evaluative research in which you are trying to bring as much objectivity to the sort of usability and user testing process as you can and he has some really interesting ideas about how to do that that he’s honed over his career. There are also folks who want to bring aspects that are a little further from sort of core research methodology as their special sauce. So for example, Natalie Tulsiani is a researcher on the team who recently gave a workshop on moderating the observer room. So the topic is as a researcher moderating the group of people who are observing a study. That just kills me. I’m like that is so great. You know what I mean. I worked with a researcher at Facebook who her thing was improv comedy. You know she thought improv could make you a better qualitative researcher and I believe she’s right. So that was her thing. She was passionate about that. I think it could be all over the map, but I just love the idea that everybody on the team has their thing and they’re just chasing it and bringing everyone along with them. Steve: Those are great. Those are good examples. So the idea of the individual passion is a driver here. Judd: Yeah, I mean look, the research shows just in general that when somebody stops growing in their job and they feel like they’re in a rut they’re going to leave. So just from a purely practical point of view, it’s like in my interest as somebody trying to build a team to keep everybody growing around things they’re passionate about. But I also think that’s the shortest path to everybody building everybody else’s game to be stronger. Everybody raising everybody else’s game. We will – that is the path to becoming a world class research team and I think it’s both demanding of an individual to always be raising their game and demanding of that person that they be generous and collaborative with the researchers around them to do the same. Steve: Right. So as a member of that team I am getting energy and reward from being able to follow that passion, but also I’m receiving from others. Judd: Yeah. Steve: From them, so I’m always growing and developing as part of that team. Judd: And if you do that I think developing that community of practice is easier. You know it’s easy to see where it comes from and how it grows. Steve: So you’ve talked about growing the team. Can you go back in time and describe a bit of the history, the evolution? Where you’ve come from? Where you’re looking to go? Judd: Sure. So my history at Airbnb is admittedly a little bit short. I’ve been here since the middle of May, so what amounts to about 6 months. I was the 10th researcher. As of this week we have 17. So we’re growing quickly, but we’re not growing for growth sake we’re growing because – I think because the embedded model we have means that awareness of research is high, the team is producing great work, and it’s just I’m getting more demands than I can reasonably service right now. So I need more researchers and that’s a beautiful problem to have. Steve: So there are teams that don’t have an embedded researcher that… Judd: That’s right. Steve: That team needs somebody. Judd: That’s right and we’re in this moment of accelerated hiring which is necessary in my mind because there are enough teams that have no point of contact, that getting – I mean going from zero to one is sort of technically an infinite improvement, right. And in the research that we do it is really dramatic at that moment. And I think this is – so we’re trying to take advantage of that inflection point. A lot of research teams in industry go on a ratio, right. So a ratio of designers to engineers, researchers to designers, for example. So they might say like 3 to 1 is the right ratio. One researcher to three designers, or one researcher to three designers and PMs together. And what we’re trying to do is just get ahead of that and then back off. And the reason that we’re trying to get ahead of it right now is because we’ve – I think we’re at that moment where having a research voice in all of these teams that have sprung up and don’t have any research representation is going to be disproportionately valuable. So we’re trying to concentrate growth right at this moment and then we’re going to slow down. Steve: So there’s a context here of sort of shift in product development – let me see if I can ask this properly. You’re describing the teams are springing up so that says that what the product is is evolving and growing – do you call the thing a product – what you as a company provide and the different pieces you’re creating to do that… Judd: Yeah. Steve: …is evolving and growing so there’s a thing that’s being made that maybe wasn’t being made at some point and that’s a new team? Judd: Right. Or you know – and I don’t – I’m not an expert in the growth of companies or product teams of course, but in this one, you know I think as the product organization matures there are two things that happen. One, new products spring up. The other, existing products, the pie grows, right, so that the amount of work is – becomes specialized enough to carve out a new team. It’s just too much. So where there was just one team before that for example looked after all the things we do around host dashboards and host tools, for example, well now we’ve a reached a scale and a complexity of the product such that there are several important work streams. And then they break them out and PMs and designers and researchers and data scientists specialize in those areas. And so I think – that’s a thing that’s kind of been happening, like spawning teams at Airbnb, and in that moment having a researcher at the table is super crucial. We don’t want to lose that momentum and there’s enough important work there – a lot of those people have come from other teams where they had a researcher and they’re like hey I’m working on this new thing, where’s my researcher? I love that. I love getting those requests. Why isn’t there a researcher in the room for this. I means I have to say no more than I want to, but I’ll take that problem any day. Steve: Are you trying to hire with specific team roles in mind? Judd: Mostly no. Mostly what I want to hire for is talented generalists and that’s because of what I think of as the virtue of that embedded model and because I don’t want to create – well I don’t want to create an environment in which there’s a bunch of people who are really only either happy or well suited for one type of work because stuff changes. The product moves. The demands of research shift such that okay I need to move some researchers around to tackle this new product area and I don’t in general want to have researchers who I think I’m going to make dissatisfied or their skills aren’t going to be a good fit if they need to slot into a new areas in 6 months or a year. Steve: So you’re using the demand to figure out how big your team needs to be, but it’s not sort of slot filling, it’s capacity building. Judd: Yeah. Overall capacity building. I think if we chase anything at all as a team we chase diversity. Like I want a team full of people with different methods, different perspectives, different backgrounds. We just hired this talented researcher named Andrew Sweeney who is going to be our first researcher in Portland. Portland is where a lot of our Customer Experience team is and so Customer Experience, those are the people who answer the phones. If you have a problem before or during your trip that’s what they’re for and they have a whole suite of internal tools that they use to handle these problems, to track information about cases that are open. Those tools are not as good as they could be and we have never done systematic research on those tools. So in comes Andy Sweeney who has – Andrew has a background working on complex, information rich tools, right. So his approach to the problem is different than the one that I would take. He goes to system usability scale for example which is not the first place that I go, but that’s brilliant for this situation because we’re talking about understanding information flows in a complex information environment. Efficiency is important because we want to enable Customer Experience agents to be going as fast as they can and not blocked by the fact that they have the information tool, like a CRM, and three spreadsheets and a notepad and their workflow is just broken. So chasing that diversity as a team is something that I think goes to the earlier point about raising everybody’s game where everybody is helping each other out. I want to learn about system usability from Andrew. Steve: And so Andrew’s – he’s tackling a new problem space. That’s not hosts or guests? That’s the new organization and being able to deliver a good experience. Judd: That’s right and I think he – so in a way we’re taking on a new sort of constituency which – as a research team – which is our internal CX agents who we call crewbies. So we have hosts and guests and crewbies. Steve: Say the word again? Judd: Yeah, crewbie. People who work in CX, manning the phones and doing Chat and email. They call themselves crewbies. But I mean they’re the most talented, empathetic people on the planet. Like we – many people who work here do these shadowing sessions where we’ll sit with crewbies and sort of just observe their day to day and we do that to gain empathy for the problems that guests and hosts have during and before trips and after trips. But these are some of the most patient, amazing people on the planet and we’re, you know, in the process of working towards better tools for them. Steve: Okay. So that sounds like a marked evolution in what – in how the work that your team is doing is impacting the company. Judd: Yes, it’s a more internal focus than we’ve had before. I think it’s fundamentally the same sorts of things which is like do rigorous research that is deeply understanding the problems and needs that people have and understanding potential solutions, but it’s in the context of making our business more efficient and providing better service. Steve: So will Andy have access to the designers and the engineers that are creating those tools for the crewbies? Judd: Oh yeah, there’s an entire products team that’s spun up in Portland and in San Francisco just to work on this. Steve: So once you have a product team then that’s – in your embedded model – that’s when the request for the researcher comes? Judd: That’s exactly it. And I think I look at a lot of things that way. So I could have hired Andrew 6 months ago when there was no product team, but the reason I didn’t is because he would have had nowhere to land. I think that’s setting a researcher up for failure. You know I want to know – you know a researcher can do the best work in the world, but if there’s no landing zone for that research then they’re not going to be able to have a lot of impact most of the time. Steve: So at the risk of re-asking a question I brought up earlier, we were talking about slot filling or not, is Andrew someone where you saw his particular – what made him a diverse candidate? Can an individual be diverse? I don’t know. What was unique about him that contributed to diversity? It sounds like the thing that he was brought in to do is perfect. For the company, perfect for him. Six months ago were you saying that the right project for him wasn’t there? We talked before about capacity building versus slot filling, but I’m wondering is there sort of a nuance to that that goes with the diversity aspect? Judd: Yeah, I mean – I guess I would be lying if I said that there was never an appropriate time to look for a specific skill set and that it was always just general capacity building, looking for great researchers. This is a case where absolutely I thought I’m sure we can find an incredibly talented researcher who has experience working with information rich internal tools and making them work better and understanding complex workflows and efficiency and all this stuff. I’m sure we can find that person and low and behold we did. At the same time I think, you know Andrew has done a bunch of things in his career, not to emb- I’m sure he’s going to be embarrassed about this and – and he just started on Monday by the way, but – he’s done a lot of things in his career and he would be perfectly capable of filling almost any other slot. He just happens to be a great fit for this role. Steve: Let’s go back in time. As you said you’ve been here “x” amount of time, but you must have some sense of the history. You were #10 you said. Judd: Yeah. Steve: Do you know the story of #1? Like how did research become a thing that was hired for and done? Judd: Well research is in the lore of Airbnb. One of the stories that gets repeated internally is about one of Airbnb’s early investors who at a very early point in the company turned to Joe and Brian (Brian Chesky and Joe Gebbia, two of our founders) and he famously said go to your users. And so they spent several weeks in New York City with some of the first hosts on the platform, deeply understanding what they were going through and what their experiences was like. And Joe and Brian are designers by training. They were at RISD and I think that having designer founders – having you know two of our founders be designers has sort of infused this kind of design and research sensibility into the whole thing . And that’s been beautiful for us as a team because I don’t – let’s say to the degree anyone cares about buy-in from the top, which I think realistically every team needs to care about, relatively speaking we do not have to worry about that. We have people like – we have a whole bunch of leaders saying where’s the research. Let’s look at the research. Go to your users. So it’s really embedded into the lore of the company. I wish I could be more specific about the history of this team in particular, but honestly I can’t. Steve: So with that caveat I’m going to ask another specific question which you can then throw back and say you can’t answer. Especially for this podcast, I think an implicit topic – I think it’s implicit – is this idea of research leadership. And the trajectory often in organizations is a researcher is brought in. Sometimes they’re a junior person and they kind of work for a UXer design person, depending on the industry. And then some organizations there’s a point at which it’s acknowledged that this is a specific thing and it needs to be led, not by someone who’s a designer, but by someone who that’s what they do. That’s my sort of general sense of the pattern and I wonder, do you know when did that happen at Airbnb? When was research a leadership – when was there leadership in research? Judd: Yes I do. Steve: See! You said you couldn’t answer it. Judd: I can answer a few questions. So in the beginning of the team it was just one or two researchers and obviously the design team was much smaller and it was just one big team. Everybody reported to Joe Gebbia. And over time what happened is the design team grew. I think the research team, which was at that time called Insights, was not growing at the same rate, and partly because I think of a – the fact that it’s hard for somebody who’s not a researcher to know how really best to leverage and grow that organization. But the partnership was there. They were doing great work and then at a certain point Alex Schleifer showed up. Alex is our VP of Design. He’s been at Airbnb for about a year and a half and he took over leadership of the organization and almost immediately recognized the need for research leadership. And it took him awhile – it took me awhile to meet him and I think to his credit he gives me a lot of freedom and says like Judd I want you to build a world class organization and I want to support you in that, what do you need? And he gives me a huge amount of context and feedback, but he’s set me up with his team such that my peer now is Katie Dill who runs sort of the organization of designers. Adrian Cleave who runs our design operations organization – but we are Alex’s first team and he treats us like peers and so there is no sense that even though I’m in the design organization I don’t have the sense of – that some – of research being – the sole value of research being to feed design for example. I feel genuinely like we are equal partners in a product process and I give Joe a lot of credit and Alex a lot of credit for recognizing that that was a useful and valuable thing to do with research from a pretty early time. Steve: So the position that you came in to take was a new position? Judd: It was a new position which was to lead the research organization. Yeah. And I was and have been given like a fair amount of leeway to say how should we do this. And that feels great. I don’t have the answers. You know when he said that I was like great I don’t know how to do that, but I’d love to work with you and figure it out. Steve: That seems really key and I don’t know if that’s just part of the overall culture or something that is unique to you, but – and I’ve had this conversation with people a number of times about knowing the solution versus knowing how to get to the solution and the comfort with the ambiguity. You’ve mentioned a number of times in this conversation that we don’t have this figured out. We don’t have the best solution but we’re trying this. Judd: Yeah. I think anybody – you know it’s interesting because we were recently having some conversations about the structure of product teams in general and how hard that is and going why hasn’t this been figured out by now? Shouldn’t there be one canonical way to set up a product team? No. There is no one canonical way because every business is different, every product is different. Values are different and that implies different things for structure. Well I think it’s the same thing for research, for design, for product overall. I feel everything that I do is to make good principal decisions in response to the realities of the situation at Airbnb. To try to build a research team which is uniquely responsive to Airbnb and at the same time embody some qualities that we really care about – world class rigorous research, perfectly positioned researchers every time. Those are kind of our two mantras. Center of excellence, perfectly positioned. And those are sort of – there’s no playbook for that. I think we apply those principals. We apply them systematically and we communicate a lot. And I don’t – it sounds stupid when I say it out loud, but I don’t know of another way. I think I’ve been really, really hard with my team and they are embracing of the idea that feedback is the most important thing we do. Feedback for each other, feedback for me, feedback for me with them – because I say to them I don’t know of another way to do things better than to give and get feedback. It should flow like a river. And I think that can be hard, to be open, to focus on strengths rather than weaknesses. To be humble is a thing we’re always seeking to be better at, but that’s how I approach the task of building a team. I welcome any and all feedback. I don’t pretend to have all the answers. I just think I have a principled way to get there. Steve: I’m going to ask sort of a clichéd research question but I think it follows from what you’re saying. In a purely speculative way, like if we were to be talking in five years let’s say, what do you think – what could you imagine the state of research at Airbnb being? Judd: Five years is a long time. Well, hopefully we will have scaled to the point where we have what we would consider full research coverage. To me that means being sort of lean and agile, but having a researcher represented kind of at every level, from the ground level up to the leadership level and we’re definitely not there yet. It takes time to build that capacity. You want to build it from within too so we’re growing that capacity at the same time. I also think in 5 years we’re going to be a company that has already hopefully long since already embraced the idea of like a really global/local product so that we have product teams that are staffed all over the world and I have researchers embedded in those teams and they are connected back to San Francisco and they are providing this very difficult thing which is how do you create a product which is sort of you know 80 or 90% core, but that 5% or 10% on either edge is the real local bit where the rubber meets the road. So how do we create a uniquely Singapore product or a Germany or Brazil product where we represent belonging in a way that resonates culturally where we have tweaked our onboarding process to highlight different value propositions. Where we do special things that facilitate trust between guests and hosts that are unique to an Indian market for example. And I think that’s going to take an organization which is global. Like I don’t think that sending researchers from here out to those places is going to cover that. So that I hope sooner than 5 years will be true. And then the last thing I’d say is that I hope that by that time we will have created not just a product organization, but an entire company which is totally driven by research and insights in which – you know I don’t think that we need to do all the work, but whether we’re talking about marketing or legal or policy or local operations or customer experience there are research questions in all of those places and I want us to be able to have an organization where somebody on my team is the glue to every one of those teams. Steve: So taking embedded beyond product? Judd: For sure. Because I mean – for example name a – it’s a little bit stupid to me that most organizations have a product research and a market research organization. Name a product question which doesn’t have an influence on marketing and advertising and vice versa. I’m sure somebody could snark and think of an example, but that would be the exception that proves the rule, you know. I’m not saying – there’s no need to have one big kumbaya team, but there is a huge amount of value to saying like look fundamentally we’re just researchers applying our expertise in different disciplines and in different ways, let’s be more together than not. And in my experience most organizations are not like that. So I don’t know. That is a big aspirational goal for this company. Steve: I love it. Just so well put. So that’s kind of the looking ahead. It’s a lovely vision. I want to go back. You mentioned a few things about yourself – PhD and some other places that you’ve worked. Maybe you could just talk about – I’m just curious about sort of parts of your background, whether it’s professional, personal. Judd: Yeah. Steve: Go back and talk about some of the things – what are some of the things that are in your background, experiences or education or whatever, that are really present for you now in the way you’re looking at the world and the kinds of things you’re trying to make happen? Judd: Yeah, I mean, I guess if I – that’s a really – you’re forcing me to be introspective. Well I began – as an undergraduate my major was cultural anthropology and so I began my career as an anthropologist. You know very focused on meaning and understanding and writing culture and the idea of culture and what it meant. And I feel like that’s really important to me now, especially at a company that’s focused on travel. It gives me a lot of empathy. You know I feel like I studied a lot of kind of epistemology that gave me like a fundamentally subjectivist approach to research which I think makes it valuable – makes me a better multi-method researcher and leader in the sense that I think objectivity is a myth. That I think everything has a cultural and social lens and all we’re doing is like seeking confidence and that confidence is built through multi-method research, through looking at the same problems from multiple directions and perspectives. So cool, if you do that there’s no territoriality. We just need all the methods. They’re all flawed and they’re all powerful. So I think that was influential for me. My PhD was in social psychology and information systems so I took a real jag into experimental social psychology and data science. That I think informs me because I, you know, have had now very deep experience with both the most ethnographic qualitative work there is, you know in which I spent six months at an after school arts and media program in the Bayview learning about informal learning outside the classroom. And I’ve done a huge amount of experimental social psychology and data science and I can appreciate – you know I can hold my own with basic statistics and write R and Python and PHP and talk to an engineer in code and stuff like that. So I think having had that experience is deeply influential for me. And the last thing I would say is that I think – so after I graduated from undergraduate I went to culinary school. So I spent six months getting a degree at the French Culinary Institute in Manhattan and then I worked as a chef for awhile. I actually – I was not destined to work as a chef. That was an extremely difficult life and I found that out and was grateful to have kind of something to go back to which was anthropology, and graduate school. But having been a chef, having been somebody who worked with my hands and really embraced that kind of creativity and appreciated flavor and the craft of perfect taillage, like knife skills. I think that helps me have like a little bit of a window into a design world. I don’t think that I have a lot of a window compared to the amazing people I work with on the design team, but I think it gives me an appreciation for craft, for art, for the details, for the menial but beautiful handwork that makes great food for example. I think I can apply that to research and to design. Steve: That’s awesome. You know we talk about researchers as kind of translators and even this – the diagram you drew of somebody elbowing somebody they’re sitting next to. It’s kind of like a translation thing and it seems like the experiences that you’ve had have given you a lot of vocabularies to translate between and maybe even the translation was sort of an ingredient in those. Judd: I agree with that. I think – you know recently I gave a talk – I’m not sure I should say this, but I’ll say it anyway. Recently I gave a talk at a learning lunch here at Airbnb, mostly to a data science crowd, and the talk – this is also a talk that I gave at a design conference in Philadelphia. And it was basically a social psychology talk. It was about the ways that we are all biased. It was about confirmation bias and minimal group bias. It was about post hoc bias. And like the point I was making is research is human and research requires human – humans. Every type of research does and humans are flawed. We cannot avoid our biases. The best we can hope to do is counteract them. By hanging out with each other. By trying multiple things. And that talk got me into a little bit of trouble only because I think it seemed like what I was doing was being unduly negative about different methods, and I was. It was kind of one of those talks where you crap on everything and that was my point. It was like okay everybody spends time exulting the virtues of their particular method and that’s great. They’re all strong, but they are all weak and we are weak. And so I think part of the translation for me is to be able to speak to everybody and go okay I get the strengths of your method and I get its weaknesses and this one too and this one too. So let’s just get past that and work together. Steve: And that’s being human. Judd: How can you avoid being human? Steve: Yeah. Is there anything we didn’t talk about in this conversation we should cover? Judd: I guess one of the things that is on my mind a lot is, because I’m growing a research team, is the idea of responsible growth for a research team. And what does that mean? I’ve been thinking about it because we are growing and I want to make sure that we do it in the right way. I’ve experienced both types of growth, responsible and irresponsible. And the way I think growth gets irresponsible is not even really in a headcount way or a budget way. It’s probably more like a communication and culture way. So the things that I want to promote in the content of responsibly growing this team at Airbnb is radical transparency and communication. The worst outcome in the world is if new researchers look at old researchers look at managers look at me and go I have no idea what these people are thinking. Or I don’t know how to plug into this decision making process and I don’t feel like I have a say. How did this get to be this way? I don’t think it’s right but I don’t have a voice. Okay, terrible, terrible. Need to avoid that at all costs. So the way we do that is by being fundamentally transparent and collaborative. Like everybody from me on down, we talk about what we’re thinking and why we make the decisions we make. We open them to feedback and maybe we form a committee in which everybody has a voice. Okay, committees, that sounds kind of bureaucratic, but what I mean is like involve everybody from the intern to the senior manager in doing something like how should we build a skillsharing system? Cool, we can all be involved in that. And then the other thing I think is responsible growth is making sure that everybody has a path forward in their career because other than feeling like you’re stuck in a rut, the other thing that I think is a recipe for a researcher being dissatisfied is feeling like they have nowhere to grow. And so making sure that everybody feels like they have a path is really important to me as the team grows. That path is building out your unique niche, working on your core skills and expanding them, taking on more stakeholders, taking on more senior stakeholders. And you do not have to become a people manager to be a lead. That is another thing I feel like is really important because people management is a set of skills that is unique and crazy difficult and learned. And you choose to focus on them. Not everybody wants to do that. It is absolutely not the case in this organization that the only path forward toward seniority as a researcher is through people management. It can be through research leadership, through product leadership, not just people leadership. And so as we grow I think the responsible bit is making sure that all of those paths for growth are open to everybody and everybody knows about them. I spend a lot of time thinking about that because it’s scary to think that we were 10 when I started, we’re 17 now. I don’t know, we’ll be 25 or 30 in the next year or year and a half. That’s a lot of growth and everybody thinks the next person in the door is going to be the one that changes the culture, but I don’t think that. But I think we should be deliberate. It should happen on purpose not by accident. That kind of growth, the planning around it. So that’s what I seek by doing it this way. Steve: That’s fabulous. Do you have any questions for me? Judd: Why are you doing this podcast? Steve: Why am I doing this podcast? You know it’s self-serving I think. I’ve been around long enough that the best work was being done by vendors. Let’s just say that. And I still am one. I hate the word vendor… Judd: The “v” word. Steve: …we all know what that means. You know people who work outside organizations. Teams like – companies like this didn’t exist. Teams like this didn’t exist. Leadership roles like yours didn’t exist. It’s a big change that’s happened in the last few years where now – I mean the kinds of vision for what research can be and how it can impact. You know it has to be done inside. It’s not to say that – I’m not saying that my work doesn’t have value, it’s just different. If you work with organizations the context has shifted. There are people inside organizations that have roles and titles and responsibilities that didn’t exist before. So my professional life has changed. So it’s self-serving because this is a chance for me to learn about this shift. It’s fun to be able to do that. It gives me an excuse to have conversations with you and learn things I wouldn’t otherwise learn. And you know I think much of the best work – whatever the percentage is – amazing things are happening inside organizations. So that’s the place to kind of look and learn. I’m not – I like working outside organizations so when you’re a consultant or a vendor you journey from place to place, like in The Incredible Hulk. You know you have these adventures and then you have to leave at the end which is an obscure reference for people that didn’t watch the T.V. show in the 80s or 70s or whenever that was. Judd: I didn’t get it. Steve: Okay. Judd: I liked it though. It’s a good reminder of how young this field really is. Steve: Especially in the form that it’s in. The conversations we’re having about change in design, insourcing and acquisitions and so on. I feel like research follows design and kind of we’re trailing by a couple of years. But we’ve seen research firms get bought too, not just design firms, you know in our recent history. We’ve seen that happen. So I’m curious. Judd: As much there is no one canonical way to build a research organization there’s also no information out there about how to do that. And so to have a resource where you can learn from people who are trying to build those organizations is really valuable to the UX community so thanks for doing it. Steve: Alright. Well it was great to speak with you. Thanks so much for being a guest and for being my host here today. I’m throwing those words out in a really confusing way now. Judd: Nicely done. Thank you, Steve. It was a pleasure.
Feb. 18, 2015
Today’s guest is Carol Rossi. She’s the Senior Director of UX Research at In our conversation, we discuss her small-but-mighty team,’s collaborative workplace culture, and the personal driver of “doing good.” We take this business question that’s been given to us…everybody goes into the field, everybody is involved in understanding what the objectives are and laying those out, everybody is involved in regular reports to the executive board…everybody is involved in running the study. – Carol Rossi Show Links Carol Rossi Follow Carol on Twitter Edmunds War Stories Carol’s War Story: Driving Force Interaction Design at Santa Monica College Follow Dollars to Donuts on Twitter (and Stitcher) and take a moment to leave a review on iTunes. Transcript Steve Portigal: Well Carol, thanks for being on Dollars to Donuts. Carol Rossi: Thanks Steve, thanks for including me. Steve: Maybe I’ll ask you the first question, which is always my favorite first question just to go broad and have you tell us about where you work and what you do. Carol: Sure. I work at and I manage our user experience research team. We focus mostly on qualitative research and we work really closely with the quantitative researchers and marketing, analytics, other people in the organization that do research. Steve: Is analytics different than quantitative research? Carol: Yeah, I guess I’m making a distinction between site analytics and quantitative research, which includes large scale surveys–like we have ForeSee running on the site, which is like the voice of the customer tool, and other surveys that we send out to our customers. Steve: And that’s a separate team from the analytics team? Carol: Yeah. That team is run by my colleague, Michelle Shotts, and we’re kind of joined at the hip. We sit next to each other, we work together all the time, and we both report to the VP of user experience, Jackie Remus. The analytics team is in a separate area of the company. We work really closely together but they don’t organizationally report up the same way we do. Steve: Okay, that makes sense. What’s your history with Carol: Oh my gosh… I’ve been here for about four and a half years and I was brought in actually to establish some research function here that didn’t exist in any formal way before that. There had been people, maybe more junior researchers here, maybe one or two at a time, and they would do usability studies or a survey occasionally, but there hadn’t been any formal attention to collecting and understanding customer needs and keeping a pulse on the experience on the site and measuring it on a regular basis. Steve: Do you remember what those conversations were like when you were trying to get the lay of the land and talk to them about what you would do in that role? Carol: Yeah. I mean, it was actually really awesome because the first month that I was here, I literally spent every day just conducting stakeholder interviews and would sit down for like an hour with like twenty-five people at the company–some of them, a lot of them, are still here and are the people I still work with every day in product, in design, in marketing, customer service, all these different areas– sat down to understand what they needed. “What do you need to know about our customers? What do you know now? What’s helpful?” All those things. What I heard resoundingly was “We don’t really know very much.” From there it was almost obvious that we needed to just start with a combination of big scale personas, brand perception research, all the way down to regular usability studies and interviews with customers. I just laid out this eighteen-month road map to get us started. The company had just decided to move in a direction of design thinking and had hired some people to come in and train a lot of the company on really basic design thinking skills. So, that was happening and then I realized “Okay, the first thing I think we need to do is create personas” because I think it’s a really good tool for giving people a model the lay of the land and at least have some idea of what broad segments exist. And because I think it’s really important to get people involved in the process in order to get their buy-in, I sat down with my boss and the VP of product and said “I think we need to just take people on their own and get them out interviewing customers.” We had like thirty-five people running around the country interviewing all these people, came back into the room and did our clustering and kind of proudly, together, presented the first set of personas that gave everybody a way to at least start thinking about customers as people and in terms of the things that are important to them instead of the things we might think are important to them. Steve: I’ve neglected to ask you at this point what kind of business is. Carol: We are a third-party auto website. The company is almost fifty-years-old and it’s been a publisher–first, we published a book with car prices and then we had the first third-party auto website in I’m going to say 1995. We were really focused as a company on being experts and we still have this amazing editorial team of professional writers and people that keep cars for a long time and review them over months and years, so we can really write intelligently about what it’s like to own the car. But anyway, it was traditionally a publisher and so there was this very strong kind of “We are the experts” mindset, and I think was what started to shift in 2009-ish to 2010, when I was brought in, to think more about what customers need and how we can give them more of what they need instead of thinking that we understand that already. Steve: In that personas project where you had thirty-five people out and about doing research, were you looking at car buyers? Carol: Yeah, because we didn’t know exactly where we wanted to target, so we literally did a panel recruit in various cities and got people who either said that they were in the market to buy a car or had recently bought a car. I can’t remember the exact parameters, but it was probably something like “Had bought a car with six months and intended to buy one within three,” something like that. So, we got people that were pretty close to the actual purchase and collected their stories. Steve: And that’s the focus for much of the research that you’re doing, is people having some relationship with the purchase? Carol: Yeah. Obviously as we’ve learned more over the last few years, it’s become more and more specific and tailored and focused on certain types of buyers and all of that. As we’ve matured, our understanding of our customers–that original set of personas–has been replaced, so we have a much more sophisticated understanding of shoppers and who’s on Edmunds. Of course, we’re always learning but we’re in much better shape to target better. But we’re still generally focused around people actually making a purchase, and part of that is that the business of the company has shifted over the last couple of years and we’re much more focused on connecting car shoppers with dealers now than we were four years ago. Steve: That’s a really specific use case or a situation that you’re looking to talk to people in. Have you found ways to deal with that? Carol: Yeah, it is very specific. I think we’ve learned a few things and we’ve kind of evolved our thinking in terms of how we can find the best people. I think one thing for sure that we’ve done is that towards the end of last year, Michelle, who’s my research partner and who does the quantitative stuff, and some people from the analytics team, did a really fabulous analysis of our site users to understand who gives us the most benefit. So, now we’re actually focusing on them and we’re actually recruiting people that have used Edmunds for whom we have contact info as our primary market. Right now, our focus has been a bit more on them. We’ve also realized that we can’t go out and say “Do you intend to buy a car within a month” because they’ll say “Yeah,” because they may intend to but then something changes, somebody loses a job or the parent gets sick and they have to take time away to do that. So, there are these things that prevent that from happening or, for example, we ran a study last quarter and this guy said he intended to buy within a couple of months, he was looking at a couple of cars, and went out for a test drive and ended up buying a car. It can be challenging to pinpoint the exact moment of purchase, but we’ve got some ways to either be realistic about what we can get or target people that are on our site because we know a little bit more about their potential behavior. Steve: That sounds like it’s the product of years of working on this and refining it. Carol: Yes, it is. [Laughs] Steve: I guess that’s how you learn most things–trial and error. Carol: Yeah. Steve: Often the kind of qualitative research that we do gets split into generative versus evaluative. I don’t know if that’s the right framework for you, but I wonder how you characterize the kinds of ways that the research you’re doing is supporting other parts of the organization. Carol: Yeah, I think that is a way that we might characterize it. We’ve tried to be very responsive to the needs of the organization and we’ve been asked to be on both sides of that. For example, a year ago this time, we were asked to do pretty regular usability benchmarking and focus on the more evaluative piece. At this point, in the last six months, we’ve been working really hard to get ahead of the product team and use an integrated research approach so all of us are starting to come together in a more formal way–the marketing team that’s doing research, the analytics team, the quant./qual. on our team, customer service, because they’re getting a lot of feedback, and try to drive product changes and drive new business initiatives through research by answering some very specific questions. It’s actually a really exciting moment because we’re getting more and more ahead of the product team. It’s pretty cool. Steve: Can you point to anything as influencing that shift? Carol: That’s a good question. I think there was executive recognition that we could be doing things a little bit differently in a way that could help us get ahead of the competition. I’m not sure if there’s a specific root cause. Steve: When you describe the shift in where the idea comes from, what the product opportunities are, is there a risk of something like that being a political issue? By that I mean is something being taken away from someone in favor of this more integrated process? Carol: You mean being taken away from somebody else who was potentially driving that? Steve: Yeah. From the outside, I have the meanest, smallest narrative in my head that “Product people who reported to so-and-so used to make decisions about things and they would dictate what was getting made, but now they’ve lost that power.” Again, it’s a very negative characterization, but I wonder, how does that manifest itself inside your teams? Carol: I neglected to say that as part of this integrated research team, we have product people and designers involved. In the past two quarters, we’ve spent one quarter each doing an ethnography project–we’re in the middle of one right now–where I’m paired with a product person or two and a designer, and we’re running the whole deal together. We take this business question that’s been given to us, we figure out some of the logistics–my team or some other people helping will work through the recruiting and that kind of business; but the planning of the study, the executing–everybody goes into the field–everybody is involved in understanding what the objectives are and laying those out, everybody is involved in regular reports to the executive board that check in on this process as we’re going along, everybody is involved in running the study, so everybody goes into the field. Usually we have me or another researcher leading it, but one of the product people or designers is there as the co-researcher. We all get together and do the whole affinity diagramming and understanding what we’ve learned, extracting the trends and insights. The other thing that we’ve added now is that we all work through in ideation to come up with some product concepts directly resulting from those insights. Everybody is going through the process together, so at the end everybody feels ownership of it. As I was saying when we did the first set of personas, we had some of these same people out in the field and we’ve always gone into the field with a partner from product design, other areas of the company–marketing, etc. But this time we’re saying “No, the partnership starts at the beginning and it goes all the way through instead of just involved in the planning or running the study, or one or two phases along the way.” Steve: That’s a much happier narrative than the dramatic one that I was constructing. Carol: [Laughs] Yeah, it is. It’s a pretty happy narrative so far. Steve: I love that. Carol: I just don’t see the point in going out, especially for field research–even when we’re doing on-site visibility testing, everybody is in the room. We have a researcher in the room with a participant and also, again, one of these co-researchers who’s from productive or design, etc., but we’ve got all these other people in the observation room watching and we’re building the affinity diagram during the study. So, by the end of a day or two, everybody is together, everybody is on board. It sounds more “Kumbaya” but let me put it this way: it works pretty well because this is a very collaborative environment–like seriously collaborative and fairly non-competitive in a really good way that I haven’t seen at every company I’ve worked in, that’s for sure. Steve: It makes me think of how far the field has come, and I don’t just mean the research field but all of the fields, because more than a decade ago, the dominant narrative was–and maybe this is still true for a lot of places–but I used to hear a lot about “Well, how do we get the engineers to come to the usability test? How do we get other people interested in what the researchers are doing?” But I feel like what you’re describing, and it’s certainly consistent with what I see out there talking to people, is that there is a real interest and a real hunger across the disciplines for understanding what’s going on with people so that we can all do a better job together. So, I’m doing a little Kumbaya but– Carol: [Laughs] Steve: –But I think maybe it’s one of the consequences of just my age, is that I remember some of the “bad ‘ol days.” The world has moved on and a lot of those problems where we dug our heels in what felt like the early days just aren’t there. It’s not trivial to do what you guys have done and establish that kind of collaborative culture and then put a program in place, but it seems like the conditions are there in workplaces in general where that could actually happen, and there was a time when you would be winning an Academy Award for what you just said, it was so in a new category that I just created. Carol: [Laughs] Steve: I think you and I have similar times in the profession. Do you see that we’re in another era from when we all started? Carol: Oh my God, yes. It’s funny, because I remember I worked for one company, which shall remain nameless, years ago and that was when we were still called human interface people, or human factors or whatever, and we couldn’t even get into the field. It was like a battle getting into the field, and it was an enterprise product, so it wasn’t like you could go to a panel company and get participants–we needed these business people and it was really hard to get out. But the other thing that’s funny, I’ve been reflecting on this lately–my career began in probably a really odd way because my first job in college, I was a senior, I wasn’t even in graduate school yet, and I worked for Honeywell, and it sounds so ancient [laughs] but we were making training simulators for F16s or something. The reason I’m telling this story is that we had all these interns, me and like six others who were undergraduate or graduate students in psychology, and we were working with a subject matter expert from the air force and a programmer from Honeywell, and there was somebody else, there were like four people on these teams. We each had a section of the airplane that we were responsible for and we worked together–all of us sat in a room. These air force guys would come from Arizona or something for like six weeks at a time and we’d work on this thing and then they’d go away and we’d develop something and then they’d come back. I just thought that was normal. It’s like when you’re a little kid and your family is a certain way and you think that’s normal, then you go off to other people’s houses and you realize that other houses operate differently. My career started with this totally collaborative team thing happening, so I thought that was normal, and then I got other jobs where that wasn’t the case. It wasn’t until the internet like fifteen years ago, so a few years later then, that collaboration became the way to just do things. So yes, I’m seeing exactly what you’re seeing and maybe more. Steve: I wonder–if this was a podcast about some other part of a business operation, like HR or accounting–I wonder if what we’re describing is just a shift in the western workforce and professional cultures or if it’s specific to our field, which is still very young. Carol: It’s hard for me to answer that because Edmunds is this freakishly collaborative place. I mean, you’ve been here. For example, we have this experience team that I’ve never seen anywhere, with these really awesome people that can just make anything happen. In fact, one of them has become our chief recruiter; she’s amazing, she gets these participants just by talking on the phone to these people. I’m probably in an extreme kind of environment for that kind of thing. Steve: One of my favorite stories about meeting people of Edmunds was someone from the workplace experience team had someone who was traveling outside of your area, Los Angeles, to go to a conference. It wasn’t travel agency stuff like “Here’s your hotel,” but they were really helping them with a new life experience, which was to go on a business trip. I thought that was just an interesting sign of the times and I felt my age. I was like “Wow, no one helped me do that kind of stuff.” Carol: [Laughs] Steve: But how great that there’s just so much thoughtfulness put into that. I took that as an indicating characteristic of your workplace culture. Carol: Yeah, that’s great. Steve: Since you mentioned recruiting, maybe I’ll use that as a pivot here. Can you talk a little bit about who else does research, who’s on the team that you manage? Carol: We are in the process of backfilling. We had someone on the team for three years, so we’re looking to replace him now. There was a period when Michelle, who’s my counterpart, was on the team and she shifted to be her own team. As far as the people that report to me, it’s me plus a researcher. We have a coordinator that we’re also hiring, and that’s somebody that’s going to do recruiting and work on another area of the company and that person is going to report to our VP of user experience. Then the other people doing research are in other organizations–like our brand strategist is doing a lot of brand research, and marketing is doing ad studies. So, the other people are the ones we coordinate with. Steve: What do you look for then when you talk to people? I’m thinking about the backfill example that you mentioned. So, you’re looking to bring someone in that role–what do you look at when you talk to candidates? Carol: I’m looking for somebody who, because we’re what I call the “small but mighty team,” that can really take the lead. The title is senior user experience researcher, so we need somebody who can move forward, who can work really comfortably, somebody that plays nice with others and also has enough attention to all of the logistical details that you have to pay attention to to be his/her own project manager in terms of making things happen. We need somebody who can wear a number of hats–so, field research, usability studies, some light surveying, some attention to quant., although you don’t need to be able to do big analyses. Because we’re a pretty compact team, we need somebody that can do a lot of things but really focus mostly on qual. and being able to lead a project. Steve: I was speaking to somebody today at lunch and they made one of those declarations that they acknowledged they couldn’t back up, and it was an interesting hypothesis. They said “I think you can get away with being a crappy researcher longer than you can get away with being a crappy designer.” It led me to think, in the situation we’re talking about, you meet a candidate and you’re thinking about not only all these hats but that this field research piece is important. How do you assess somebody in terms of that? Carol: I ask them to tell me very specific examples or stories about experiences in the field and how that’s led to specific products that have been implemented. How do they tell the story? Someone told us a story that was really compelling the other day in an interview, and it helped put me in the room with him and the participant, it helped me see a little empathy, but it also helped me see how he turned that into the product that he described that they made as a result of that experience. I’m looking for, first of all, the ability to tell the story because that’s a big piece of it. I’ve seen a couple résumés lately where I’m like “I know this person’s got some skills here, but the way that it’s being conveyed, I’m not engaged.” We have to be able to advocate for the stories we’re hearing in the field. Actually, I think I was talking mostly about shoppers but we’re also doing research with car dealers. So, it’s B2B and B2C. You need to be able to tell those stories and represent them and also role that up into insight and be able to take action. I’m kind of looking for how they articulate how they do that. I’m not there with them, so what’s the quality of their listening skills? I don’t really know. So, I don’t know if that’s what your friend meant by “You can be a crappy researcher for a longer time.” I’ve been asking people this question: “Tell me how you deal with feedback that you get,” like in a 360. We all get this positive feedback and then we get occasionally something that we need to work on. What I’m finding is that everybody says “I always want to work on it,” but the way they talk about it a little bit more, it tells me something about the character of the person. Steve: It reminds me a little of your point about “When we realize that not every family is like the one that we grow up with.” It just feels like those realizations about how you’ve defined the world around yourself continue to happen. I think, to me, feedback is something like that. You reach a point of expertise and confidence and there’s something you didn’t know that you didn’t know about doing that, and those feedback opportunities are tremendous but also terrifying, because there’s things that we need to improve. But sometimes that feedback, no matter how compassionately it’s done, still kind of pulls the rug out from under you. I think if you hear it well, it can be really like “Oh my goodness, I understand the world differently now and I don’t know what to do.” Your story reminds me of getting stories from people about their fieldwork. I had somebody that dug themselves a hole; they were on a bad project and that was the story they had chosen to tell me. I think they were doing a good job, but they were basically hamstrung in a subcontracting gig that was wrong and shouldn’t have been sold. But to your point about how our ability to tell a story says something about us as researchers and as collaborators, I could see the person just following the story’s trail and it was just taking them down, down, down. I tried to lift them all the way out of it–”What’s another story you could talk about? Another project where things didn’t happen this way and you were given the resources to do your thing?” The poor person just could never get out of it. I guess that’s a good thing for me to have experienced in the interview because ultimately I didn’t end up working with them because you don’t want to see that happen, you don’t want that to happen in front of one of your colleagues, or your boss, or your client, or anything. I want to go back to something that you said when you were describing this point of evolution, which I think is really remarkable. You talked about how this integrated team is first handed a business question and comes together to figure that out. Do you think that this integrated team–I don’t know what to call it–but all of you who are kind of doing this design thinking basically to be innovative, is there a role in that team to be identifying other business questions? Carol: I just had this conversation with someone earlier today; not literally in that way, but just that we want to be open. You do research and you wind up with more questions as well as insights, so we want to be open at the end of this quarter to be able to say “Here’s what we learned and what we want to study as a result because we think it could have an impact on the business” and have that feed into the conversation about what the next quarter’s business questions is going to be. Steve: That’s nice. You answered that much better than I asked it. Carol: [Laughs] Steve: I knew there was a reason I would talk to you. So, maybe then there’s a bigger question that I’ve been thinking about, and I guess this is not speaking to you about specifically, but Edmunds and the other places that you’ve worked and how you’ve seen the industry evolve. Maybe it’s the platonic ideal, but what do you think research does for a company? What is its benefit? Carol: It’s so variable from one place to another. What I want to think is that what research does is both create some way of thinking about the people that use our, in this case, website, service, app, whatever, as humans, as people that are trying to solve a real person problem. So, in terms of giving us empathy for the people that are our customers, and again I mean car shoppers or car dealers. I hope it does that. I hope it also expands the business and gives us some sense of pain points that these people are experience that can be business opportunities. Steve: And making that connection. Carol: Making that connection so people aren’t just statistics and the business isn’t just operating based purely on intuition without people data. The fact that there’s a big emphasis on ethnography now–this is a very quantitative company and they’ve been looking at big numbers for a long time; Edmund’s website has 18 million visitors a month, so they’re used to big numbers. I think what research does for us right now, focusing on field work, is to make it really human through very specific stories that can balance the numbers. So, I feel like, yeah, both creating empathy and getting a human experience to the people on the other end of whatever we’re offering, and also to impact the business in a positive way. Steve: Part of what you’re describing the benefit of research is is it humanizes people, it takes them from data to being real people that we can empathize with. Why is that important to a business? Carol: I think it’s important to a business because we’re not just putting something out there. It’s not “If we build it, they will come.” We need to create something that somebody wants, whether we’re in a completely generative mode, like this thing doesn’t exist and we’re trying to figure it out but it fits a need somewhere along the way–there’s no reason to create something unless it’s going to fit a need because we want people to engage with it. So to me, unless you understand the story, “What’s going on in that person’s house? How does this iPhone app fit into their lives? Where are they using it? What are they trying to do there? What are they trying to do on the website?” I just feel like you have a very shallow picture of what’s possible. I think also we can get a lot of the “what” through the numbers but we don’t know the “why.” So, we know what people are doing, we could tell you who’s clicking where, but why are they doing that? That’s the transition that we’ve made in the last few months; very specifically–”Okay, who are the people on the site that are the most valued to us. They’re the ones that are making us the most money. Let’s go to their living rooms and study them, let’s find out why they’re doing these behaviors that we like because they’re good for the business. Why? What value are they getting and what can we offer them that maybe we haven’t thought of?” So, I don’t know that you can know that if you’re just looking at the numbers. Steve: That’s a great way of explaining it. Maybe I’ll switch topics a little bit again and ask you to talk about you. You gave us one formative story, or a story that may be formative for you, with this air force project. I’m wondering, if you look back at your own trajectory and what’s core to you as a person, what is it that makes you great at what you do now? Caro: I think what makes me engaged in this is that I feel like I’m kind of a do-gooder and I want to feel like I’m creating a really fun and engaging experience for people with technology. I love technology–I guess that’s part of it, too. Which sounds kind of odd because we all use it every day but… You know, there was no internet when I was a little kid. I talk to my fifteen-year-old nephew and he’s like “What do you mean there was no internet?” So, I don’t know, I’m still kind of intrigued by the gadgets and stuff, I think it’s really cool. Otherwise maybe I’d be doing something else. I left the field for awhile, I thought I wanted to go to graduate school and get a degree in anthropology and become an academic, and then ended up coming back into it because I realized I like technology and I think it should be fun and useful for people and I want to be a part of that. Steve: I like that you frame that as being a do-gooder, because I feel like socially we have a very immature and un-nuanced view of that word, that a do-gooder picks up trash or rescues kittens… Carol: [Laughs] Steve: …Or pays for babies in Africa to have fresh milk or something–and those are all important things that need to happen, but– Carol: I’ve probably done all of those, yeah. [Laughs] Steve: That’s good. You’re “good-er” than I. Carol: [Laughs] Steve: –There are many, many opportunities to do good and to approach one’s work to say “I’m going to do good at this and I’m going to improve things and enable other people to have something better in their lives,” it doesn’t mean that you’re getting rid of fracking or something. It doesn’t have to be the thing that keeps all awake, the worst things in the world. There’s lots of ways to shape the world around you that fall under the category of doing good. Carol: I think it took me awhile to get to that, but yeah, I agree. I think I had that pretty narrow view of it even though I consider myself to be that type. Steve: That sounds like a little bit of self-actualization, to be able to say “What I’m doing is putting some good back into the world in some way.” Someone asked me–I don’t know, it was about a year ago–about my service. “What do you do?” I had to think about that because where I live there’s a thing where you can and clean up the beach, there’s a thing where you can pick up trash, like trail cleanup day, and I never do those things. My answer on the face of the question was “Well, nothing,” because I don’t volunteer out of this or do that. But I started to think about other things that I do and I think sometimes we have to kind of search for that, personally. But I would suspect in our field people are drawn to it because that’s a thing that has meaning to them, that they are improving things. Carol: It’s funny, there are these personality systems, like there’s the Myers-Briggs, there’s one called DiSC, and there’s one called the Enneagram in which the type that I am is called “the reformer.” When you read about who that type is… this work draws a certain personality type for sure. Steve: “The reformer,” yeah. Can I ask you about your war story that you wrote for our series? Carol: Yes. [Laughs] Steve: For someone listening to this, what would you say to them to get them to go read your war story, which is linked right here. Carol: [Laughs] Oh, what’s the most compelling thing… So, yeah, in the spirit of what we were talking about earlier, like running an interview and taking somebody who’s not an interviewer with you, I was taking with me the guy who’s VP of editorial, so he’s responsible for all these people that write these really awesome articles on the site about cars and reviews them and whatever, and he hadn’t been out in the field like this before. He said “Well, I’ll drive and I’ll take one of the cars from the editorial fleet.” So, we meet and we go downstairs and we’ve got this $100,000 red convertible BMW or something and we’re driving through this really terrible neighborhood, this section of Hollywood that’s really not-so-great. We get to this guy’s apartment and it’s really dingy and it was quite an enlightening experience for Scott, my research partner. It was kind of a fun story. Steve: Good, and there will be a link to that for people to go and read more. This is totally me sucking up or setting you up to say what I want you to, but hey, I’m the host so I get to do that. Carol: [Laughs] Steve: What do you think the war stories project brings to our field or to people that are learning about it? Carol: Reality. It’s a total reality check. And some of them are not funny; some of them are quite… what’s the word… you really become quite pensive with them. But yeah, I think it’s reality. It gives somebody a flavor for what really happens and kind of everything else that happens–it has absolutely nothing to do with exactly what you’re trying to capture but that’s the person, that’s the whole picture. It’s kind of what I was saying earlier about making people human for your colleagues to get the stories. Yeah, sure it’s about how they bought the car and how they got the price and what they thought of the dealer and all that stuff, but what’s this guy’s living room like? He and his wife lived in this tiny apartment with a baby, and he looks forty-five-years-old and she looks like Gisele [laughs] and you’re trying to figure out the backstory. It’s the reality of the real human experience that goes with the business question and the product that you’re trying to develop and all of those things that are important. You can’t make this up. We interviewed somebody for this coordinator role who was a comedy writer. I said “If you work here, you’re going to have more material.” [Laughs] Steve: It’s interesting, those stories taken in the aggregate, all of them have extracted away any actual research content, but they sort of have distilled that element of the humanness of field research. Carol: Yeah. Steve: What didn’t we talk about yet that we should talk about? Carol: This is nothing to do specifically with Edmunds but you were talking specifically about the evolution of the field–last week it was announced that fifteen community colleges in California are going to be able to offer BAs for the first time, because usually they only offer AA degrees but not BAs. The BAs were chosen based on the ability to train people in a field that’s important and relevant in the neighborhood of the college, and Santa Monica College was just awarded their first BA in interaction design. Steve: Wow. Carol: Yeah–wow. It’s kind of crazy; crazy good. It really points to–and we’re in “Silicon Beach” and companies are bringing themselves to L.A.–but it was pretty great for the field. I didn’t read in detail all the other schools and what they’re doing, but a lot of them are like, you know, dental hygiene–you need that, you need people to do that. But yeah, SMC got a program in interaction design, so that’s pretty awesome and a pretty good sign of the times, I think. Steve: Do they need instructors? Carol: Maybe. [Laughs] We will find out. Steve: Since you mentioned you like to teach. Carol: Some of us have teaching experience and that would be cool. Steve: We’re putting it out there now, so. Carol: [Laughs] We’re putting it out there–”One can do that and this job.” Steve: Do you have any questions for me? Carol: Um… So, how’s it going? How are you feeling about your new series? This is fun. Steve: Yeah, well you’re someone I’m able to talk to after the series exists because the first few interviews were ones that I conducted before it was posted. So, there we’d be talking about the series in the abstract and now, as I’m talking to people, they’ve heard it. I’m enjoying it. I’ve been getting good feedback from people. Someone said something really nice to me the other day, they said “I wish you’d been doing this for five years so that I could have had all of those to go back and learn from,” which is sweet but also sort of an exhausting thought because I don’t know that I would continue to do it for five years. I mean, this project seems to make a lot of sense now, these are conversations that aren’t being had. We hear a lot about design, about user experience and design thinking, but we don’t hear a lot about research and how it walks and talks in the corporate environments. For me, it’s exciting because what’s changed over the last few years is that many, if not most, of the leaders in our field are in roles like yours and I think there was a time when those people were in roles like mine. But you and your ilk, if I can use that word, you guys are doing amazing stuff so I’m loving getting to hear these stories and hear from people. It’s one of these things where we’ll just see where it goes and what values it has for people. Someone said to me that they thought these would be interesting to people besides researchers, and I’m getting a little bit of that, other people listening to them. You gave this lovely articulation of what researchers can bring and I think we would like more people to hear that. So, if this draws a broader audience than just us–kind of “insider baseball stuff,”–then I think that’s good for everyone. Carol: Well, I think it’s evolving, it’s brand new. To me, it seems like the theme is that it’s evolving. Our understanding of the process and how we need to be and how we need to engage with the rest of the organization is evolving. It seems like nobody really has a set in stone process, at least of the people that I’ve heard you speak with so far. Steve: I would agree with that. But I think the evolution is tremendously based on where we all started. This is so much recursion: we’re now in an episode of a podcast talking about the podcast. I thought only Howard Stern could do that. Carol: [Laughs] Oh, geez. No profanity here though. Steve: Yeah, I think we have a clean rating in the iTunes store, we don’t want to lose that. Carol: Clean, no beeping, no beeping. Steve: Anything else or does that take us to where we want to get to? Carol: Unless you have any other questions. Steve: No, I’m really happy–I love this conversation. So, thanks for taking the time and for sharing all these stories and giving us a lot to think about. Carol: No, thank you. Thanks for doing this. It’s really great to be a part of it, I appreciate it. Steve: Very good. Carol: Alright, talk to you later.
Feb. 11, 2015
Today’s guest is Kerry McAleer-Forte, the Director of User Experience Research for Sears Holdings. We discuss how researchers need to think like storytellers, getting at the underlying need behind a research request, and the risk of using research to make recommendations. Tools are important. It’s important to choose the right one, but at the end of the day, they also won’t replace that thing inside your skull, which is your most important tool, which is being able to observe and listen and pull the important story out for your user. – Kerry McAleer-Forte Show Links Kerry McAleer-Forte Don Norman Michael Kronthal Images from Michael’s talk Sears Kmart Sears Home Services Sears Parts Direct Follow Dollars to Donuts on Twitter (and Stitcher) and take a moment to leave a review on iTunes. Transcript Steve Portigal: Well, thanks for joining us today Kerry. Kerry McAleer-Forte: Absolutely. Glad to be here. Steve: So, let’s start, as we do, with a bit of an introduction. Maybe you could describe the organization that you’re at and the role that you have there. Kerry: Sure. I am the Director of User Experience Research for Sears Holdings, and I have a team of seven full-time researchers and we specifically focus on experience research. By many accounts, it’s a pretty straightforward organization where UCD and UX people are concerned. We sit directly within a customer experience organization that has user experience architects, and designers, and copywriters, and product managers, and product and UX work hand in hand also with analytics, and we do a really wide range of methodologies to support answering the questions that they have around design and around the experience in general. Steve: You mentioned Sears Holdings–how do we know Sears? What do we know Sears Holdings as, as consumers, or as consumers in North America at least? Kerry: The two main brands that everyone would be familiar with would be Sears and Kmart since we are under one company. So, the websites, the mobile applications and devices, and then obviously the physical stores are all things that we spend time researching. Steve: So, the websites for both brands, the stores for both brands, and then the mobile apps or other kinds of tools for both brands as well? Kerry: Mobile and tablet. Right, because of where our team sits, the UX and the product management does have a particular digital focus. So, most of our time is spent focusing on the website, mobile phones, and tablets. However, Sears has been in the integrated retail arena for a long time now, so we definitely take an approach wherever we can. We’re looking at the holistic end-to-end experience, so somebody might be at home on their tablet, they’re on the iPad watching television, they see something that’s interesting, and then the next day they’re at work and they pull it up and they’re like “Mm, that’s pretty interesting.” Then, on the weekend, they visit a store to go check out the refrigerator, open the door, and then pulling it up on their phone to check out reviews. So, again, most of our work is done with a digital focus, but it’s definitely running across all of the different channels. Steve: So, what’s the geographic emphasis for your team? Where are you looking at consumers? Kerry: The entire US–from coast to coast there are stores, and online is obviously omnipresent, and the demographic of who our customers are goes back for 125 years. So, we’re interested in understanding all of them. Steve: And we quickly went to talking about consumers, which makes sense since your business is retail, but is that the key focus for you? Looking at–I don’t know what your term is–shoppers, consumers? Kerry: I think we interchangeably use shopper/user, but specifically member. Sears is a very member-focused organization, so we’re looking not just at a terminal transaction of coming and buying a product one time, but people have had multi-generational relationships with the brand, so we’re focused on understanding how that relationship thrives over time and what its different needs are over time. Steve: Does your team think about, or is tasked with, looking at, say, the work of someone at a Sears or a Kmart store? Kerry: So, say, from the associate side? Yeah. Supporting integrated retail requires a lot of digital support in and of itself. Some of the work we do is focused on apps and sites that associates in the stores or in other parts of the company are using to help run the company. But by far the most common type of research that we would do would be around understanding what it’s like to shop at Sears. Steve: That makes sense in retrospect. I think I have the experience sometimes of talking to people where I know their product or service as a user of it, only to find out that there’s a million things going on behind the scenes that they work on. So, I guess that’s behind my question, kind of where your emphasis is. But really you’re looking at the members and their experiences. Kerry: Yeah. Steve: Also at the beginning, you described a little bit about your team. I think you said there was seven of you? Can you say a little more about kind of what the skill sets or roles that you have in that team? Kerry: The skill sets are really wide-ranging. Like a lot of people, UX people have come to UX research through many different paths, whether it be psychology or rhetoric or something completely unrelated. I’d say that the vast majority of what we do is evaluation of a specific design. So, that might be a current website, it might be a current phone app. Then we also do some sort of generative work, we do some field work. Again, when we get into the integrative retail point of view, we really need to understand what it’s like to transition from one device to the other, and how those needs change, and where are the gaps between them. So, the team is poised to change very quickly from different types of tools to meet the different needs of every project that we’re doing. Steve: I like how you characterize the world of UX and UX research in general. You didn’t use the word “generalist,” but there’s something there about how we all do a lot of different kinds of things. Kerry: Yeah, if you tried to write the job description–and I have–it gets very, very long and it seems very intimidating because, frankly, to be a great experience researcher, you need to understand experience design and you also have to have an understanding of how this design is relating to a business that needs to support itself and needs to be successful. So, there are a lot of multi-faceted demands put on a researcher, in addition to their immediate job. Steve: And I like how you’re using the words “experience researcher”–the terms that we use are so fraught, and I don’t mean to pick on you in any way, I’m not asking you to defend that term, but I wonder if you can explain why that’s the term you’re using. Kerry: Well, to use a technical term, this is a ginormous company and an enormous organization within the company, and there are so many different types of research out in the universe, that most people are much more familiar with other types, so as a marketing research or analytics, and these things are always very much focused on numbers. So, I do like to stay away from the terms “quantitative” and “qualitative.” Everybody likes a number. Everybody would like to, from a business perspective, have that great confidence you get with being able to say “See, this is 5.2 and not 7.8.” When you’re talking about experience, you start to throw the psychosocial layers into it and it gets a little bit more difficult to attach a number to. So, by referring to it as experience research, I think that people can relate to having an experience and they relate to it as a three-dimensional thing and it’s involving personalities, it’s moving through time and space, and it’s not just attached to an analysis of ticking numbers at the end. Does that make sense? Steve: Yeah. And is that a term that you’ve advocated for internally? Kerry: It’s one that we often use. I think that within our business, the term “usability” has become a little confining, frankly. I think that it has almost some very limited definitions. I mean, our focus at the end of the day is “What’s your question?” and our job as researchers is to figure out what’s the best methodology and the best tool to pair together to answer that question. So frankly, whether it’s quantitative/qualitative, or whether it’s a usability or HCI focus or storytelling, whatever my particular flavor is, I don’t care so much, as long as I do a great job answering your question within all of the constraints I have. If I’ve got three days as opposed to three weeks, my choices are going to be really different, so that’s where I like to keep our focus. A lot of times people will come to us with requests for assistance and they say “I want to do this type of a study on this to show this,” as opposed to starting from “Tell me what’s the question you’re trying to answer? What is it that’s gone wrong? What is it that you’re noticing that makes you feel like you need some more insight?” You don’t have to worry about what techniques, what tools are out there because people–it’s not their job to be a researcher and know our whole tool box. So, we’re going to put together the best constellation of method and tool to get you what you need. Sometimes they’re really knowledgeable and they know exactly where we’re going to go with that, and sometimes we use a little methodology jazz and we put things together in different ways and hook things up in a way that can get at some of the nuances better than things they were familiar with to begin with. Steve: Methodology jazz is a great phrase. I love what you’re saying and I’ve been talking a lot about my own terminology, I guess–the business question, the research question, and the research method, and that they’re all related. The business question is “What does the business want to do?” and the research question is “What do we need to learn in order to help inform or guide that business action?” and the research method is, of course, “How are we going to go about it?” and that there’s a relationship between the three. But I find, like you, that people come in with one of those three and that there’s some work to be done to kind of develop all three and connect them. Do you have a theory or any thoughts about why do people approach and say “I want to do these activities” as opposed to “I want to answer this question”? Why is that a starting point, do you think, for some people? Kerry: That’s a great question. I think that in the grand scheme, there are so many different types of research because there are so many different types of questions and it would be unusual for there to be one team that did them all, that did formal marketing research, and analytics, and experience research, and whatever else we’ll all come up with next. So, I think that people–you go with what you know. “I’ve been around, I’ve heard some marketing studies before, so I guess that’s what I need. I need to find out some marketing information,” or they’re familiar with web analytics, so they want to know click rates and things like that. So, I think that where experience research comes in, it’s often very much a matter of educating them how to work with us, letting them know what our capabilities are and then really acting as their consultant to guide them through–”Focus on your business aspects,” or “Focus on the design questions that you have and we’re going to bring our tools to the table.” Steve: That’s a very wise approach and I wonder if our field has kind of done ourselves a disservice, especially in our relationships with the media. There’s stories that we think we can get attention for that we like to tell and the media likes to repeat, that “We went and we watched people sniff perfume bottles,” whatever sort of weird activity. That seems to make for a good business article, even though we’ve been reading those articles for a couple of decades now. People are coming to us and saying “Hey, we want to do a blank,” and I wonder if we’re the ones that are putting those stories out there. It’s easy for me to criticize other people that don’t know and then I have to educate them, and I can feel myself being a little judgmental sometimes of people that don’t know how to make a proper request for me. But as I hear you talk, I think “Well, maybe that’s collectively our fault for promoting those stories that seem to have a little bit of sizzle to them.” Kerry: I don’t know, I’m going to let you off the hook. I think that they didn’t come to you because they heard about a tool. They came to you because they heard something that sounded promising to deliver them some results. “So, whatever it is you call that thingamajig over there, yeah, let’s use one of those.” I mean, I’m sure people have come to you and said “My business is down, so let’s do some eye tracking,” and you take a deep breath and you’re like “All right, well let’s talk about what’s going on with your business and then we’ll get to the tools last.” But eye tracking is the most fantastic sales tool ever invented and its actual applications, frankly, I think are really limited. What it does, it does well. So whatever it was that got them in the door, that’s fine. “I’m glad you’re here, I’m glad you’re interested.” The most important thing is they’re wanting some feedback, they’re realizing that “I can’t and I shouldn’t make decisions in a vacuum about how to run my business, about what I think will please and delight my customers.” Steve: That’s a very empathic way of thinking about providing help. A lot of what we do is a form of service to people that have some other type of question or business need, so I think to approach it the way you are is good–and I’m happy to be off the hook. Kerry: We spend a lot of time on our team thinking about our own company as our users and that really helps us make better choices about the type of research that we do. Sure, you would love weeks to put together a shining report that would get an A+ if you handed it in some place. Sometimes I say if we take that long, we have, by definition, made ourselves useless to the people that have to be running as fast as they can think. So, you have to figure out what the most important headlines are, get it to them, be okay with a B-. You know, we’re all overachievers on the team and that’s hard because you know you can get it a little clearer, you know you can get it a little shinier, smoother, nicer. You might have to leave out a couple of things, but that’s the tradeoff. “What are the needs of our users?” Our users need it fast, they need to get to the heart of the issues and we’re going to have to bank on the fact that we have a long-term relationship with them, that even if we don’t get it this first time, this first report, we’re going to keep engaging with you, and we have that relationship and we build the rapport between the engagements where my team isn’t embedded within every single one, but we assign domains to people, so they develop subject matter expertise, they develop relationships over time, and I think that at the end of an experience study, what do you know? We’ve made some observations, we have derived some feedback for them from it. Our approach is we do not make recommendations, and that’s where we differ from a lot of researchers and research organizations. In my early days with Sears, I had somebody come to me and they showed me a report that had been done during a previous time, and somebody had dutifully executed a pretty straightforward usability test, they had observed users doing things, and then they said “Here’s your problem as evidenced by these three observations, therefore here’s the recommendation,” and the person was struggling because there was clearly no causal link between the observations and the recommendations that were being made. I could kind of see where they were coming from, but because it was in a report in a document and it has a great sense of authority once it’s wrapped up in that document, somebody was very dutifully saying “Well, research says this is how we’re going to solve our problem,” and you start marching down that road. So, we come at it from the point of view of “We need to let you know what we’re seeing, we need to let you know what we think the implications of it are, why these are important things, how they relate to other parts of the experience that we’re also observing, but the actual solutioning is something that we’re going to do with you after this report.” Steve: So, what I hear you talking about is taking kind of a facilitating role as opposed to an instructive role. So, rather than saying “You should do X,” you’re saying “Hey, we’ve learned these things and we want to work with you to help determine how to act on it.” Kerry: Exactly, right. When we’re working with newer teams, an analogy they’ll use a lot is “Think of us as your radiologist. A patient comes to us, we’re going to study them, and then we’re going to come away, we’re going to tell you where it’s broken, the nature of the break, how bad it is and then we’re going to work with the surgeons–the plastic surgeons, the orthopedists, those are the specialists that are going to come in that are going to be much more familiar with maybe things the patient has already experienced or maybe things that are more realistic within the whole context of running the hospital.” So, we’re coming in to help identify and then we’re going to then consult when it comes time for the solution, but not actually be the ones to deliver it. Steve: So, in this analogy, do your users, your internal clients, do they have access to surgeons? Kerry: Yes, absolutely. So, many times the surgeons are the ones that have come to us requesting that the patient be studied. We’re working with user experience teams made up of the user experience architects, and the designers, and the front-end developers, and the copywriters, and then that team is working closely with a product management team that’s very focused on understanding the business analytics and the business goals behind that. Steve: Can you describe, in as generic a form as is appropriate of course, some kind of narrative about a project that your team was involved in? Kerry: Sure. Our typical choreography is that for our major teams and initiatives, we’ll have a researcher assigned to that domain. So, one domain, for instance, might be top of funnel, which for us is from the home page to the cart, or it could be bottom of funnel from cart to checkout. Our structure is set up that way because it mirrors the way other teams are set up. So, again, we’re not embedded per se–I wish we had those numbers–but you get a general domain. So, you meet with your domain teams and you start to formulate. I’d like to call it a road map, but being in retail, you have to be very, very flexible, so we look at a forecast of what’s coming up. There’s a two way communication between our team and the other teams. Teams come to us and they say “Here’s a project we’re working on and we know that we would like input beforehand, or input during, or doing some sort of prototyping.” The other form of communication is as the researcher develops experience within the domain, they have the advantage of not being as in the weeds as the other people are about the details and the requirements and the system operation, and it allows them to have those fresh eyes to say “Hey guys, you know what? Those are important things but we’re also going to study this,” and so they will propose additional studies many times that are needed to really get a great understanding of whether or not something is successful or the domain is working well. And then as we get closer to, say, a particular project, we’ll work with the team on starting, first and foremost, again, going back to your question, “What are the research questions that we’re going for?” and that’s really, really important because, by definition, we have to move extremely fast because they do. So, identifying and articulating “Here’s what we’re answering with this study,” it therefore means that we are not answering all those other great questions, they’re not listed right here. Then we do the algorithm from “Here’s the question. What measures can we gather to answer this question? Okay, what tool is the best tool to carry out those particular measures?” and then the researcher comes up with a test plan, reviews it–this is all very tidy assuming we have a tidy project–and they run the study. I think that one of the things that has surprised me over the past five years is how much our focus has shifted from using our in-person usability lab to using remote tools. We still use our lab, it’s still a great asset to have to be able to do these in-person studies, but remote tools are where it’s at for our team. They allow us to do an incredibly fast turnaround in terms of getting out to finding the right kinds of users, getting the stimulus in front of somebody, whether that’s a live site or a prototype–it could be a low fidelity thing that we get out there very quickly. Then we’re focused on getting that report back to the teams, again, extremely fast. So, from the day of the launch of a remote test, we are able to, after we’ve set up the test, to get it out there, get all of our tests completed and the videos back–the longest within a day, usually much faster than that–and we’re going to have the report back to the team, fully annotated and with video clips as well. We’re going to have that to them anywhere from two to four days after that. I think our land speed record was four hours from beginning to end. We just needed to study the interaction of one particular toggle, so we formulated the test, ran the test, reported it and got it back to the team within four hours. Obviously there are going to be studies that take longer. Field studies, for instance, or diary studies–something that unfolds over a number of weeks. Then once it’s reported–we always, always, always have a report–we meet with the immediate team and we present the report and then that’s when the discussion starts. “All right, here’s what we know. Now what are we doing about it?” Then we become consultants and cohorts with the design and the project management teams to talk through the options. In addition to presenting directly to the teams and working with them, we maintain an online wiki where we publish our research. So, while teams are focused very much on the things that they’re doing, we know that they can also learn from things that have been done before, and being aware of–again, things happen in other points of the experience, so we have an online wiki complete with all of the video that people can go in and search themselves and pull up things, and it’s really important for scalability because, even with a team of seven that’s great, you can’t begin to meet the research needs for every team for every project that’s out there. So, we get a lot of support leveraging that. Steve: What are the attributes of less tidiness? Kerry: Less tidiness? I think a good experience study is predicated on being able to get the participant to play along in your land of make believe with skewing them as little as possible. We know just by participating and research, they will, of course, be somewhat altered in their behavior, but I think that some things are easier to get them to fake than others. So, if we say “Pretend you feel like shopping today,” that’s not as big of a stretch as saying “Pretend like you really want to figure out how to use a coupon today,” because the first just presupposes that you’re somebody who shops, which is pretty common behavior and something that you can actually screen for rather easily, and therefore write a scenario to, write a task setup fairly easily. The more specific or the more narrow the topic when you get to things like, say, a coupon, you’re really telegraphing a lot of information to the user, first of all, about what’s going to be going on, what you want them to look for, and these are just the types of situations that it’s hard to set up a study so that you can be pretty confident that “I’m pretty sure if I have them do this, they’re going to end up doing what I hope they’re going to do,” because you can’t just say “What do you think of this?” But sometimes it’s tricky, and sometimes at the end of the day what the team wants to know is “Which is better, A or B?” and a small sample test isn’t a great way to assess that. Like I say, “We don’t pick winners. We’re going to let you know where the strengths and weaknesses are in each of these, we’re going to let you know what people did and how people reacted to them, but I know what you want to know, is you want me to point to one and say ‘Do that one, go,’” and so sometimes the untidiness, the messiness comes from having to squirm a little bit and say “Sorry, love you, mean it, but we’re not doing that. We’re going to answer this for you instead.” Steve: I love to hear that given the timeline that you are working on, which is quite amazing, there’s a lot of investment in your limited timeframe in aligning on the problem and the approach, and I think you said something about “We’re going to answer this, we’re not going to be able to answer that right now, and in order to answer this, we’re going to do this other thing as well.” So, you’re really focusing on having that conversation to figure out together “This is what we’re going to do in order to address your question,” and that seems like that’s a hard space for some teams, that they’re trying to get away from sort of taking dictated requests. But you’re really bringing the expertise and working together with them to figure out, like you said before, “What are your questions and how are we going to get there?” Kerry: Yeah, and we can do that because we have an ongoing relationship as an internal in-house team. If you have an organizational model where, say for instance, you know you’re only going to get one study, then you’re going to be really, really starting to answer all of your questions and want to cram as much in there as you can. Whereas if you have an ongoing relationship and hopefully, ideally, you’re not going to get just one study, you’re at the very least going to have sort of an ad hoc relationship with the research team subsequent to your study, there’s not as much pressure on that, and I think that people are a little more open to some mentoring about how to approach breaking down what their question needs are because teams know what they need to know. They’re usually right on. They usually just have one hundred questions and we need to narrow you down to ten per study and then go from there. Steve: Anything else about this relationship that you have with these teams? Because this seems to be really key to how you’re being successful and what you’re trying to accomplish. Are there other things that are going on in that relationship maybe outside–you’ve mentioned the wiki and you’ve mentioned sort of what the choreography of what the project is. Anything else? Kerry: I think that some of the greatest success relies on having a great communication flow with our teams, but I’ve been really, really passionate that we remain a separate organization, that we remain a separate team, because as much as we want the company to succeed, we’re Switzerland and you look at our flag and you see the face of the member on it. So, we really need to have kind of this dual relationship of having great open communication and yet I need to be just as excited to deliver terrible news to you about your study as I am to deliver good news. I also want our teams to succeed in being able to explore their ideas, but when somebody comes to me and says “Hey, I want to run a study because I need some ammunition for fill in the blank,” a flag goes up and “Well, okay, I’m not going to worry about what you’re doing with this. What I want to know is what is your question?” Everybody has heard that Switzerland reference a million times around here because they can get the fact that to have research “in your pocket” does nobody any good. We have no interest and it’s very dangerous. We run the risk of being seen as biased if we’re only attending to the needs of one group or one particular kind of initiative. Steve: What you’re describing, the Switzerland example, reminds me of something that Michael Kronthal, who up until recently was in a similar role to yours at Yahoo, is not anymore, but I saw him give a talk and he had this really cool model which I’m not going to be able to replicate properly and I haven’t been able to find online, so maybe as a result of saying this someone will dig it up for me, but he described three or four different models as to what the relationship is between the researcher–the, in your parlance, member and the user. One of them was sherpa, where the researcher brings the internal team kind of to that world of the member. He had these great graphics that kind of showed different organizations that can work in different ways, and it wasn’t one was better than the other. I think that Switzerland aspect is in there as well. We’ll have to find that. I haven’t been able to. I’ve been looking for it for awhile because it was so concise, and when you saw it, it was like “Yeah, we do all those things and I’ve worked with groups that are one more than the other.” Kerry: Yeah, that’d be interesting. Steve: But I like the Switzerland thing because I think that speaks to not only sort of what the process is but much more what the mindset is, that you are there to kind of bridge and represent and enable and connect these different constituencies. Kerry: A good test doesn’t always mean good news, but you’re definitely going to benefit from getting your bad news because there’s no escaping it if you let it out without changing it. Steve: Can you talk a little about the history of experience research within your organization? Kerry: I think it’s a really exciting time to be a researcher within retail in particular because there’s so much change going on. I need a word that’s bigger than “change.” Five years ago when I first joined, there was a team of one and I became the team of one, and what you had were a lot of people that, like a lot of other organizations, were used to being hybrids, so I’m going to be a UXA until I need something tested and then I’m going to put on a researcher hat and then I’m going to do that. There’s definitely a benefit of being able to knock off a test on your own and know how to do that really well. But there was a definite craving in the organization for more feedback. We tend to use the word “feedback” rather than “data” or “research” because I think that it allows you to be more flexible in what you actually mean by that. Feedback can be something you’re getting from ad hoc research or it could be something from a more formalized study. But what was clear–obviously, how much can one person do? So, it all comes down to scale and the appetite to invest. So, no big news to anybody listening to this podcast, it can be difficult to convince people to invest in research or invest in experience research in particular. They can feel a little more comfy investing in marketing, say, or investing in analytics because it’s very literal and it feels very knowable in many ways. In order to be able to feed the starving kids at the door, we needed to find a way to start to prove the value and to prove some sense of scalability, that it’s not something that’s going to take this massive investment, it’s not going to slow down timetables, that there’s a way that you can have a really practical, savvy, lightweight research practice while still delivering great results and still be thinking about great design experiences. In the earlier years it was very focused on “How light can we go? How short can we go? How fast can we go?” because I think you can go too fast if you skip especially those early important steps of focusing on “Okay, let’s clarify, what are we doing? What are we answering? Okay, go.” You do tend to get your favorite methods and your favorite routine, and sometimes at the end if you cut the wrong corners, there’s a “gotcha” at the end of “Oh, that’s not what you actually wanted to answer. You wanted to answer something else.” So, a lot of my focus on remote tools and lightweight methodologies really came from needing to attend to the needs of our users who are our internal teams. This type of information and feedback is such an easy sell. Once people get it, once they experience it once, typically people are like “Well, I would never want to do a project without that again” because it’s so enlightening, and it’s enlightening in a way that’s, I think, even more approachable and even more knowable than a number at the end of the day. If I watch somebody doing something, if I’m watching a twenty-second clip, cognitively I’m getting so much more information out of that than looking at a chart with a line. Charts with lines are really important but they’re only part of the story, so we’re really able to deliver the other side of that. As demand grew and as confidence in the leaders grew, some other very important leaders came into the organization and they were pivotal and coming in from organizations that didn’t have the long, long legacy of any company with a longstanding legacy. It’s a double-edged sword–you’ve got great culture and traditions, and sometimes there are routines that are very difficult or attitudes that are very difficult to change. So, leadership coming in from other more new-world organizations really brought the point of view of, “Well, of course, it’s just table stakes. Of course you have to understand the experience,” and so the team started to grow–the core team, meaning the team that’s attending to the universal websites, the universal applications, things like that. Then we started to develop such great relationships with some of our internal businesses, and to us, a business, an internal business for instance, would be hard-lines businesses or software, or soft-lines, clothing. If we’re researching what’s a great shopping cart, it’s not necessarily speaking to all the most important needs of somebody who’s, say, shopping for clothing, so we started establishing relationships with some of our other business units to focus on other types of things. That allowed our team to grow even more–people who could specialize not just in an area of the funnel but in a particular point of view or type of shopping. The other part that’s been a really interesting space for us to grow into has been the non-shopping sites. So, Sears Home Services, Sears Parts Direct. If you’re thinking about it as a lifelong relationship with the brand, where I’m getting to know a brand, I’m making purchases, and then I’m maintaining and taking care of not just these things but my entire home over a lifespan, there’s some really great opportunities that come up there. Once we started to connect and establish relationships with those teams, we were allowed to grow even more and look forward to continuing to grow because I think that the spaces where we need to answer questions about experience are just ballooning with all of the different social and technological advances. I think that we’re on this really exciting doorway right now into the land of the internet of everything, where what happens when everything you look at is connected and can speak to each other. I think that’s, for researchers, just fascinating and I think that it’s also going to be really interesting to find out what we have to let go of and what’s not going to apply anymore, and what new lessons and paradigms and principles are going to come from the new ways of being. Steve: Your team has been growing and you’re looking forward to this imminent growth as the topics explode–who and what makes for a good addition to your team? Kerry: A type of researcher? Steve: A type of person or a type of researcher. What is it you’re looking at? Kerry: That’s a great question. I’ve been really lucky to find some really special researchers on my team, and I think that what they all excel at is regardless of their beginnings, regardless of what their undergrad or their graduate degree is, they really understand–people are going to click off this podcast–they understand why the research question is important. You can do methodology jazz, but you have to understand your scales. You have to be able to do the basics. So, as long as you have a really clear understanding of the academic principles behind qualitative research, research of any kind, scientific theory, the scientific method, then I think that then frees you to explore and to play around. “Oh, there’s a new tool out, there’s a new remote that’s going to solve the world and run your business-kind-of-tool out. Does it work, or doesn’t it?” I think that you have the foundation of the classical theory, the foundation of the academic, and then the curiosity to really play around. Not that we don’t take it seriously, but I think that if you want a template–a template should only ever be your springboard–so, to think that we would do the same method, the same tool forever or for most, to me, is missing out on one of the most important parts of what we do, is to pay attention to the nuances and pay attention to the direction, to the trend. Because if you’re expecting everything to fit into this box and you’re not open to spotting “Oh, that seems to be heading over there, I didn’t expect that. We need to follow that, that’s important.” So, you have to be okay with that uncertainty and chase things down as they emerge. Steve: Is that what you mean by “play”? Is that uncertainty and looking for those moments and following them up? Kerry: Yeah. I think that there are many different personality types that can make a great researcher, but for me, somebody that needs things very checklist-oriented, “I’m going to do this, and this, and this, and this, and therefore I will have a great study”–checklists are great, but they’re very two-dimensional. It’s not just thinking like a computer, it’s not just thinking like somebody who’s going to tick off boxes. I mean, you’re really thinking as a storyteller. You’re looking at the entire stage, you’re looking at the entire scene and understanding “I’m watching this thing, but this thing is happening in a much bigger context.” That possibly only makes sense to me. A lot of what we do is about overlay structure and limits and very definable things on experience, and experience–it’s messy, it’s emerging, it’s fluid, it’s dynamic, so I think that you need to be comfortable of both worlds, of knowing the importance of the constraints, knowing the importance of articulating the structure and not suffocating it in the process. Steve: I almost feel like Gollum, kind of holding onto this idea that instead of Precious, it’s really about–it’s creative. I think “Oh, researchers, creative, researchers, creative,” and because research informs people that make things and that think divergently and often our work is to kind of facilitate that and sometimes I feel like “Oh, we don’t get credit for being creative,” but I haven’t necessarily articulated that. Certainly you’ve given a really just lovely kind of call to creative arms for what that play means and what the thinking is that researchers have to have in the work that we’re trying to do now. Kerry: Yeah, I think you have to be–if you aren’t creative, if you’re somebody who really does need that unchanging rigid structure, then my guess is that you’re going to have some pretty limited success in what you do just as a business person would. It would be “If business were that easy and that predictable, everybody would do it well, right?” But it’s not. It’s like they say in medicine, “You’re still practicing medicine.” You don’t quite ever know exactly what you’re doing but you work through it. Steve: What was your background? How did you get to be great in the way that you are great? Kerry: I, like many of the researchers on the team, have kind of a wind-y background. I started off in undergrad in theater and I was very interested in theater history, and performance, and design, and ended up working in film production for quite a few years after that and had a blast. It’s like going off with this very gentle non-violent army and going to war, and you go to war for a couple of weeks and then you have something in the can and then you’re done and you move on. That was very interesting, but most of the work I did was in advertising and I think that there was a part of me that felt like I wanted to branch out a little bit more because advertising is, by design, pretty predictable a lot of the time. I ended up connecting with an arts organization here in Chicago that I ran the programming for and it was really interesting because it was arts education for kids but it had a jobs training aspect to it. So, it was paying kids minimum wage to be mentored by a professional artist and in all of the different kinds of arts. So, what a blast, but some really, really interesting things caught my attention as I was there over the years. We had an inclusion program where kids with physical and cognitive disabilities were mainstreamed with all of the other kids and we would start most of our programming–it first took place in the summer, and in the fall we would start to get calls from teachers saying “What did you do with this kid? Johnny used to be limited and had trouble relating and would be very antisocial and now he’s flirting.” So, we would hear these great anecdotal stories about kids that, after having gone through the two-fold arts experience and also this social experience, just would have enormous personality and brain development, cognitive development, and that’s kind of what drew me back to grad school. My graduate program was an interdisciplinary mix of cognitive science and instructional design and social context, which are my three favorite flavors of soup. I got to grad school never having heard of Don Norman and then read Don Norman and said “That’s it.” So, I became hooked on research and thinking about experience since then and have worked in a number of different sectors of digital experience research since then. Steve: What kind of soup are you serving in ten years from now? Kerry: Oh, ten years… You know, that’s funny–I think my brain is wired to really live very much in the now. I’m often not the most accurate future-thinker. I can spot immediate things but I do remember somebody first described email to me, I kind of scrunched up my face and said “What would you do with that?” So, maybe not the most visionary. All I can think is that the pace of technology becoming invisible to us is just astonishing. Whether it’s physically invisible to us, we just can’t see it, it’s embedded, it’s the internet of everything, it’s my refrigerator talking to the grocery store and things just show up, or whether it’s the cognitive invisibility where you’re just “Of course it’s not unusual to send an email now, of course it’s not unusual to think that I can show somebody my computer screen on the other side of the globe now.” I think for me in ten years, I’ll just continue to be fascinated in watching how people’s perceptions and behaviors change according to how technology is really transforming the world around us. Steve: I love that, that was great, and you didn’t mention soup, so that was my bad in trying to belabor your lovely metaphor. Kerry: That’s okay. I love cooking, we’ll talk about soup after this. Steve: What didn’t we talk about in this conversation that you think we should cover? Kerry: Yeah, ask a researcher to ask you a question and you’ll never stop. “Do you have any questions?” That’s a dangerous question. First of all, I’m really excited by this series. I think that very often researchers tend to be off in a room watching video and analyzing things, so it’s exciting to hear more about the practices and just the people that are in this space around the country, so thanks to you for that. So, after you’ve been listening to us yammer on for awhile, I’m interested right away–what are the things that are surprising you of either trends or topics or points of view you didn’t expect? Steve: Wow, turning it around. Yeah, you’re right, researchers are dangerous with each other. Yeah, I mean, I’m in this interesting position, as I’ve talked to a number of people now and I think this is what happens any time you do research, is you can feel that there’s something to synthesize. There’s sort of a “spider sense” that I think that I get when I have parallel conversations with people in related roles, and sometimes I try to keep myself in the living in the data. “Don’t go there yet,”–of course, when you have the luxury of that time. So, I get these little signals here and there, like “Oh, that’s a thing that someone else said,” “Oh, that’s very kind of different,” and it’s funny too because I’m trying to be present in our conversation, but yet I keep having these little moments where my eyebrows are like “Oh, wow, that’s the pull quote, that’s this,” and now you’re asking me kind of at the end to go back to that, and I’m like “I’m right in this part of the conversation thinking about soup and the future and your phrase ‘the internet of everything.’” I’m just going to defer. I don’t know if I can answer that question right now, because that’s the question I was going to ask you too, is like “Where do you see some of your unique things?” You said early on that the balance is really on the evaluative versus the generative, but I feel like–and I’ll just expose a bias–I feel like you talk about evaluative work with the soul of a generative researcher. Kerry: Oh yeah, for sure. Steve: Because I feel like evaluative research can get a bad rap, that it is kind of convergent, it’s not nuance, it’s not rich. It may be actionable but it’s sort of a subset of the problem. That’s a bias that I have in general and I’m sort of hearing my bias come out by the way that I hear you talk because, again, the deep focus on building rapport and framing the problem, I love that and that’s sort of the research working at its best with teams. So, that is a really important theme that I’ve heard you articulate. And I’m not saying that that’s a gap with anybody else, but you just said it kind of in an interesting and, I think, unique way. There’s probably others but I think I’m kind of rambling here, so I might stop. Kerry: Well, I think it’s interesting for you to say you’re calling yourself out on your biases and turning your nose up at certain types of things. I think earlier you heard me throw some shade at eye tracking and the difficult relationship I have with that, and I think that marketing people roll their eyes when you talk about being able to understand anything with five people in an experience study, and analytics people. So, I think that it is really easy to be dismissive of other genres really, and I think that while I definitely am the worst possible candidate to be a data analyst, my best scenario is to link pinkys with them and say “Thank God there are people that like to do that, because we need that too.” You can’t just watch five people and know everything. You can’t just look at numbers coming off of web analytics and know everything. I say “I know a lot, but I don’t know everything,” and there are other people that know things, and so I think that while it’s good to have, a healthy scrutiny of what something’s good at and what it’s not good at, I think it is important to think about, to use our favorite geeky term, triangulating all of them, because there’s, out of every question you could answer, no one technique or genre of research is going to answer all of them. So, we all have our piece of the puzzle to take on. Steve: You said that really well earlier on too when you talked about figuring out the right methodology, and I totally agree. That’s an activity that I’m involved in every time I plan to do something. But also, as you talk, it sort of raises one of my fears for myself and my own career arc I guess, is the tendency to be the person with the hammer that sees everything as a nail. Obviously there’s many hammers of different sizes and so on, but I think the tools and the approaches are continuing to evolve, and I think the way that you and your team–you have some diversity there in the way that you can “link pinkys,” as you say, to bring many different kinds of approaches to bear to different kinds of problems. It makes me feel a little anxious but I have great respect for how you’re describing it. It sounds great, like that’s just the way to go. Kerry: Yeah, and I’m laughing because every time I find a new gadget, and there are scads of them coming out every day, it’s the same thing–suddenly the world is a bunch of nails and we have a new hammer. We have a lot of different tools at our disposal but I also very purposefully keep the toy-box kind of small because they’re often variations on a theme and there’s never been a tool invented in the history of mankind that does exactly what it set out to do or is as automated as they say it is. So, tools are important. It’s important to choose the right one, but at the end of the day, they also won’t replace that thing inside your skull, which is your most important tool, which is being able to observe and listen and pull the important story out for your user. Steve: I think that’s the great high point to leave it on. So, let’s say thanks and goodbye. I appreciate your time, and really, this is a great conversation and really insightful stuff, and I thank you for sharing it with me and with everybody. Kerry: This was a blast. Thanks Steve. I can’t wait to listen all of them coming up after this. Steve: Very good.

Podcasts like "Dollars to Donuts"

By Per Axbom & James Royal-Lawson
By Wesley Noble
By Jared Spool and many others
By ShopTalk
By - Todd Henry
By Fresh Tilled Soil
By Edwards, Clifford, Ritchie
By Boxes and Arrows
Disclaimer: The podcast and artwork embedded on this page are from Steve Portigal, which is the property of its owner and not affiliated with or endorsed by Listen Notes, Inc.