The Answer Was Never Us — with Leilani Battle

WEBVTT
00:00:02.640 --> 00:00:04.080
Welcome to The Test Set.
00:00:04.080 --> 00:00:10.720
Here we talk with some of the brightest thinkers and tinkerers in statistical analysis, scientific computing, and machine learning.
00:00:10.720 --> 00:00:16.725
Digging into what makes them tick, plus the insights, experiments, and OMG moments that shape the field.
00:00:17.484 --> 00:00:23.085
In this episode, we talk with Leilani Battle, who likes things she can repeat, study, and refine.
00:00:23.085 --> 00:00:30.649
Whether that means finding her disc golf skills, tweaking a recipe, or deriving the perfect vegan buttermilk sourdough pancakes.
00:00:30.649 --> 00:00:34.969
This same experimental instinct runs through a research at the University of Washington.
00:00:34.969 --> 00:00:48.354
Leilani built her career straddling two very different research cultures, the computation behind databases and the human centered experimental world of visualization, an experience she describes as essentially getting a PhD in two disciplines.
00:00:48.354 --> 00:01:01.020
We talk about the magic that happens when those fields meet, how what loads first in a visualization can shape what people find in their data, and how our work now applies that same technical and human perspective to responsible data science.
00:01:01.020 --> 00:01:04.579
So I'm so excited for you to listen to Leilani.
00:01:04.780 --> 00:01:09.094
Leilani, welcome to The Test Set. So glad to have you.
00:01:09.094 --> 00:01:24.099
So Leilani Battle is the Robert E. Denning Endowed Associate Professor of Computer Science at University of Washington and co director of the UW Interactive Data Lab and also a member of the database group.
00:01:24.099 --> 00:01:33.939
And you do a lot of really interesting work, it seems like on the computational aspects of visualization, but also the cognitive human factor side of it.
00:01:33.939 --> 00:01:39.665
Why would we say speed up visualization and in and in what situations for people?
00:01:39.665 --> 00:01:46.224
And also more recently, it seems working a lot on responsible data science as well.
00:01:46.224 --> 00:01:48.584
So thanks thanks so much for coming on.
00:01:49.200 --> 00:01:50.799
Thank you for the invitation.
00:01:50.799 --> 00:01:55.959
So often I feel like I'm like, at the very end, we get to people's hobbies.
00:01:56.799 --> 00:02:02.840
But I I hope it's okay that we flip it around because I think you have so many incredible, interesting hobbies.
00:02:02.954 --> 00:02:15.500
I I thought maybe we could go hobby forward, which was, first and foremost, I I do wanna know you mentioned you do a lot of pickling and sourdough, and I think it's incredibly important we get to that.
00:02:15.500 --> 00:02:20.819
But also, you said your most surprising hobby is probably disc golf.
00:02:21.979 --> 00:02:25.900
How did you could you just unpack for us your history with disc golf?
00:02:25.900 --> 00:02:29.259
Because I know you're in the Pacific Northwest, so I don't know if that's related.
00:02:29.259 --> 00:02:32.944
Or how did you get into disc golf?
00:02:32.985 --> 00:02:34.025
It's a good question.
00:02:34.025 --> 00:02:43.264
So long story short, my husband got me into disc golf, and he's been playing for a very long time, since high school, actually.
00:02:43.460 --> 00:02:52.180
But I didn't really have any interest in disc golf whatsoever until the pandemic when we were all stuck inside.
00:02:52.180 --> 00:03:02.764
We couldn't necessarily go and do our more social hobbies, But I still wanted to kinda go out, hang out with my husband, just do something outside of my house at the time.
00:03:03.525 --> 00:03:07.044
And he had been suggesting disc golf to me for years.
00:03:07.044 --> 00:03:10.805
I think I tried it once in high school or college or something and was like, nope.
00:03:10.805 --> 00:03:11.460
No thanks.
00:03:11.460 --> 00:03:19.300
And then he managed to get me to do it again during COVID, because he's like, we could go outside and be in nature and throw Frisbees.
00:03:19.300 --> 00:03:21.139
And I was like, okay, cool.
00:03:21.139 --> 00:03:22.020
I'll do it again.
00:03:22.020 --> 00:03:26.099
And then that time, something stuck, and I really fell in love with it.
00:03:26.099 --> 00:03:28.314
And I just kept doing it.
00:03:28.314 --> 00:03:41.715
And something I think that helped at the time, so I was an assistant professor at the University of Maryland College Park, and we had bought a house that was like a fifteen, twenty minute walk from a very beginner friendly disc golf course.
00:03:42.090 --> 00:03:48.930
So I developed this habit of every weekend going with my husband to this course and playing disc golf.
00:03:50.250 --> 00:04:03.344
And I just kept doing it, and then when we moved back to Washington, Washington has so many beautiful disc disc golf courses, most of them free to play, actually, too.
00:04:04.185 --> 00:04:15.159
And there's a so I I never checked out the disc golf scene, like the community in Maryland when I was there, but I started checking it out here because I wanted to be able to play with more women.
00:04:15.159 --> 00:04:17.480
Most of the people I'd see on the course are men.
00:04:17.480 --> 00:04:21.105
And I love playing with my husband, but I just wanted to kinda get into it more.
00:04:21.105 --> 00:04:31.064
And then I started playing tournaments in our local area, and that's really where I kinda shifted and really got obsessed with the sport.
00:04:31.185 --> 00:04:32.545
Wait. I love that.
00:04:32.545 --> 00:04:35.425
I didn't realize you're you're a competitor.
00:04:35.425 --> 00:04:38.399
You're entering the tournaments, disc golf tournaments.
00:04:38.399 --> 00:04:43.360
Amateur amateur division term or I could go on forever about disc golf.
00:04:43.360 --> 00:04:52.355
So there are mixed divisions, meaning you could be, like, any gender and play in those divisions, then there are some gender gender and age protected divisions.
00:04:52.355 --> 00:05:02.899
So there are female specific amateur divisions and professional divisions, and then age protected divisions that start about every decade and eventually go about every five years as you get older.
00:05:02.899 --> 00:05:06.019
Wes, have you disc golfed at all?
00:05:06.019 --> 00:05:10.660
I played a little bit of ultimate Frisbee in in high school, but it's been it's been a long time.
00:05:10.660 --> 00:05:14.899
I'd have to I'd have to really think hard, like, what's the last time I I threw a Frisbee?
00:05:14.899 --> 00:05:26.404
It's been it's been a while, but, I mean, I think, like, it it it's interesting, like, all the the changes to our hobbies and our personalities that that were caused by caused by COVID.
00:05:26.404 --> 00:05:30.245
Like, I I think I understand, like, the people that got into pickling and sourdough.
00:05:30.245 --> 00:05:39.990
And I didn't do sourdough, but I did pickling because all of a sudden I wasn't traveling like I I used to before COVID and I was like, am I I'm stuck at home, like, what can I do that I didn't used to be able to do?
00:05:39.990 --> 00:05:44.910
Like, oh, sit and watch a thing, you know, do science like inside my my kitchen.
00:05:45.634 --> 00:05:55.634
So but, you know, now I enjoy you know, I I haven't pickled late lately, but it it is it is, you know, now, like, a new hobby that I indulge in from time to time.
00:05:55.634 --> 00:06:01.019
And the the pickling I thought this is interesting in a profile that was written on you.
00:06:01.019 --> 00:06:05.420
They talk a lot about a bit about you doing some life logging.
00:06:05.420 --> 00:06:13.540
I feel like you mentioned, like, cooking in batches and enjoying sort of doing, like, big pickling batches.
00:06:14.214 --> 00:06:17.735
I'm I'm curious if you could say a bit more about some of your yeah.
00:06:17.735 --> 00:06:20.574
Why you pickle and what you like about it.
00:06:21.495 --> 00:06:25.654
I thought that was an interesting note, some of the batch batch cooking.
00:06:25.654 --> 00:06:27.774
Yeah. Are these things connected?
00:06:28.029 --> 00:06:45.425
So what I'll say is probably the most satisfying feeling for me when it comes to cooking and pick, like, any kind of food preparation is when I'm able to come up with recipe or a combination that to me feels optimal.
00:06:45.425 --> 00:06:50.785
Like, it's really easy to make, it tastes great, and it's repeatable.
00:06:50.785 --> 00:06:53.904
Maybe it's a scientist to me.
00:06:50.785 --> 00:06:53.904
I love it when it's repeatable.
00:06:53.904 --> 00:07:00.480
If it's not repeatable, to me it's a terrible thing to cook because then it implies you can never do it again that well.
00:07:00.480 --> 00:07:11.160
So I think to me these kinds of hobbies where even disc golf kind of falls into this, where I can just kind of do it over and over and over and hone the recipe, hone the skills.
00:07:11.714 --> 00:07:15.795
I I know I really like how it feels.
00:07:15.795 --> 00:07:25.160
And pickling and cooking, even meal prep where I cook in bulk, like, they're the kinds of things where I can do them over and over and over and get better and better and better over time.
00:07:25.160 --> 00:07:35.560
And also where documenting and just recording my experiences or experiments, if you will, pays dividends.
00:07:35.560 --> 00:07:42.975
It sounds like you really like you're solving the reproducibility food recipe crisis, but you like the fluency.
00:07:42.975 --> 00:07:44.735
Like, by yeah.
00:07:44.735 --> 00:07:51.629
By repeating an action, you can kinda, like, see yourself getting more fluent at it from sort of Yeah.
00:07:51.629 --> 00:07:54.670
I can experiment with different techniques.
00:07:54.670 --> 00:08:05.415
I also like to do our call making base recipes where you can take kind of the core of that recipe and then modify it in different ways to get a a different, like, a fresh experience.
00:08:05.415 --> 00:08:09.095
Because you can imagine if you ate the same food over and over, you might get bored.
00:08:09.095 --> 00:08:14.454
So having these kind of paste recipes where you can mix things up, it's still repeatable, but gives you variety.
00:08:14.454 --> 00:08:16.894
Like, I love things like that.
00:08:17.149 --> 00:08:17.949
Yeah. Yeah.
00:08:17.949 --> 00:08:24.310
It's it's sweet of you to think about the human factors too that you're like, if I eat the same thing, I'll be bored with it.
00:08:25.389 --> 00:08:28.189
That's just practical. I like to save money too.
00:08:28.189 --> 00:08:31.564
I can't can't risk throwing all my food away.
00:08:31.564 --> 00:08:46.820
I I really love the the Ratio cookbook, I think by Michael Ruhlman, which is like, okay, actually across all of these different types of things, like mostly baked goods, there's just like the same underlying ratio, which I think is really cool.
00:08:46.820 --> 00:08:50.740
Like, I loved, like, I loved learning, like, a pound cake.
00:08:50.740 --> 00:08:56.179
It's called a pound cake because it's like a pound of butter, a pound of sugar, a pound of flour, a pound of eggs.
00:08:56.179 --> 00:09:08.475
And that, like, one one one one ratio is like the basic ratio between basically every type of cake or if there's kind of a like, I think that's just so, like, empowering and, like, cool to cool to know.
00:09:08.475 --> 00:09:11.475
That's exactly the kind of thing I love.
00:09:12.235 --> 00:09:22.190
An example of recipe that I worked on for a really long time and really enjoy doing is, like, my buttermilk pancake recipe.
00:09:22.230 --> 00:09:26.955
So I started making buttermilk pancakes in graduate school, and I got really good at this.
00:09:26.955 --> 00:09:33.995
But then I became a vegan, which kind of destroyed my buttermilk pancake recipe, at least the old version, and I had to rebuild it.
00:09:33.995 --> 00:09:40.674
But because I had all those experiences and what makes a good pancake, I figured out what makes a great vegan pancake as well.
00:09:41.019 --> 00:09:45.019
Do do you kid, do you think we can get that recipe to share with our listeners, Leilani?
00:09:45.019 --> 00:09:47.019
Like I don't know if it's secret.
00:09:47.019 --> 00:09:52.139
It's more like I'm probably shy about sharing a recipe because I'm like, oh, what if other people don't like it?
00:09:52.139 --> 00:09:54.019
Or I don't know.
00:09:54.034 --> 00:10:03.235
So like a side of my life I'm not used to sharing widely, but I am happy to write up the vegan buttermilk pancake recipe.
00:10:03.235 --> 00:10:03.634
Yeah.
00:10:03.634 --> 00:10:09.590
Or just or at the very least Hadley gets it and then it just shares the secret with you.
00:10:09.590 --> 00:10:12.230
Yeah. I'll make it and then I'll I'll rate it.
00:10:12.230 --> 00:10:12.549
Yeah.
00:10:12.549 --> 00:10:19.870
I should also mention it's a vegan buttermilk sourdough pancake recipe, so maybe not all your listeners can make it anyway.
00:10:20.595 --> 00:10:27.794
I I feel I I just feel like there was there'll be like, there's enough of our listeners who will be able to make it.
00:10:27.794 --> 00:10:33.409
Who are vegans who have sourdough, I just feel like this this they're probably overrepresented.
00:10:33.409 --> 00:10:33.730
Yeah.
00:10:33.730 --> 00:10:35.889
If you have a sourdougher, it's actually pretty easy.
00:10:35.889 --> 00:10:37.250
So happy to share.
00:10:37.250 --> 00:10:48.524
I do feel like a lot of this does hit on what seems so interesting about your work is a lot of that balance of, like, there's a lot of computation in it, but also experimentation.
00:10:48.524 --> 00:10:57.205
Like, how do we for example, in your studies of visualization, like, well, how can we technically speed up visualization?
00:10:57.245 --> 00:11:02.769
But also, like, in what situations would a person benefit from this?
00:11:02.769 --> 00:11:11.889
Both, like, the computational problem of how, but also, like, how do we stay tapped into the sort of human centered part?
00:11:11.889 --> 00:11:16.250
Like, what part of a person's task does this actually improve?
00:11:17.625 --> 00:11:20.424
Yeah. I'm so curious to get into it.
00:11:20.424 --> 00:11:29.065
I wonder almost if just leading up to it, you could tell us a bit of how you got sort of to where you are studying these things.
00:11:29.065 --> 00:11:33.490
Tell us a little bit about your journey to where you are today.
00:11:33.490 --> 00:11:40.250
I'll try to maybe sprint through kinda my path, career path.
00:11:40.769 --> 00:11:48.955
Initially, when I was in high school, I was really into drawing and art, and I loved playing video games.
00:11:48.955 --> 00:11:58.595
So I had assumed that because I was passionate about those things, like drawing and video games or art and video games, that I would end up there somewhere.
00:11:59.995 --> 00:12:11.409
That being said, I think the influences of interacting with that kind of technology, you know, like game consoles, my family always had a computer at home growing up, things like that.
00:12:11.409 --> 00:12:17.049
I think I was more open to kind of being in tech in general.
00:12:17.955 --> 00:12:31.715
And thankfully, I realized right before applying to college, like, universities that, you know, there's more to life than being a game developer or an artist or designer for games.
00:12:31.715 --> 00:12:43.950
And I, like, based on my dad's interests and experiences around electrical engineering, I figured, oh, that seems like a viable route if my dad was interested in it.
00:12:44.950 --> 00:12:54.504
And this is kind of the first of some coincidences, I guess, but when I applied to the University of Washington, because I was for me, Washington is home.
00:12:54.504 --> 00:13:02.059
I grew up in the state of Washington primarily, and the University of Washington would be considered kind of a home university for me.
00:13:02.059 --> 00:13:09.700
I knew they had a strong engineering program, including electrical engineering and computer science and engineering.
00:13:09.820 --> 00:13:17.595
I was directly admitted to the computer science and engineering program at UW Because originally I wasn't even going to do computer science.
00:13:17.595 --> 00:13:19.754
I was gonna do electrical engineering.
00:13:19.754 --> 00:13:24.634
And with that direct admission, I was like, oh, well this is a pretty competitive program.
00:13:24.634 --> 00:13:26.235
It's hard to get into.
00:13:24.634 --> 00:13:26.235
I can give it a try.
00:13:26.235 --> 00:13:28.480
And if I hate it, I could always do something else.
00:13:28.480 --> 00:13:31.360
And so that's kind of where I ended up in computer science specifically.
00:13:31.360 --> 00:13:37.600
So not every part of my life was, you know, carefully engineered to end up exactly where I am.
00:13:37.600 --> 00:13:40.159
There have been some kind of happy coincidences here and there.
00:13:40.159 --> 00:13:41.960
I think that is one of them.
00:13:42.615 --> 00:13:47.735
So I started going through the program, started taking the intro computing courses, ended up loving them.
00:13:47.735 --> 00:13:52.855
Did not see that happening, but I thought they were incredibly fun.
00:13:52.855 --> 00:13:57.254
You know, kudos to, back then, the instructors teaching the intros courses.
00:13:57.254 --> 00:14:10.129
Because I'm at UW now as faculty member, so kinda seeing the other side of it, it's been really fun to see everything that goes into designing a really good CS course, especially at the introductory level and how that had such a huge impact on my life.
00:14:11.965 --> 00:14:14.565
So I I was enjoying it.
00:14:14.684 --> 00:14:26.164
My early years as an undergraduate student, I actually struggled to get good, like, competitive internship offers.
00:14:26.169 --> 00:14:33.129
And my first summer after my freshman year, I didn't have any opportunities to do a tech internship.
00:14:33.129 --> 00:14:40.845
So I I decided to apply to research internship programs, and I did an on campus research internship program that summer.
00:14:40.845 --> 00:14:44.524
And that was my first introduction to computer science research.
00:14:44.524 --> 00:14:48.605
It was actually in professor Rajesh Rao's lab here in the Allen School.
00:14:48.605 --> 00:14:54.320
And at the time, his he does a lot more, like, kinda neuroscience and engineering related things now.
00:14:54.320 --> 00:14:58.279
Back then, his lab was a bit more of like a traditional robotics lab.
00:14:58.399 --> 00:15:01.519
And I I've mentioned this to him too.
00:15:01.519 --> 00:15:06.120
I'm really grateful for that experience, but I knew I was not a roboticist.
00:15:09.024 --> 00:15:25.279
So after that, my next summer research experience was actually in databases, in the database group here with Magda Balazinska, who is our school's director.
00:15:25.480 --> 00:15:30.759
So she's she's been in my life and in my career for a long time, and I really look up to her.
00:15:30.759 --> 00:15:42.565
But kind of seeing the database group, how it operates, the kinds of research projects they were working on, I started to develop an appreciation for databases.
00:15:42.565 --> 00:15:50.970
And the work that I contributed to in particular was a little bit more oriented in the human computer interaction direction.
00:15:50.970 --> 00:15:57.409
So thinking about how we can make people's lives easier when they're trying to use database systems.
00:15:57.690 --> 00:16:01.049
And I think that kinda set me up for graduate school.
00:16:01.049 --> 00:16:05.024
So as I was finishing my bachelor's degree, I was looking at what opportunities I had.
00:16:05.024 --> 00:16:13.065
By then, I was able to get, you know, more competitive internship opportunities and and experience a little bit of what industry is like.
00:16:13.105 --> 00:16:24.629
And I just felt like I didn't have enough time to explore what I wanted to learn in academia before leaving and joining the workforce.
00:16:24.990 --> 00:16:28.990
I felt like I got a decent sense of what to expect in industry.
00:16:28.990 --> 00:16:34.384
I did not get a decent sense of what, you know, what potential I had in academia.
00:16:34.384 --> 00:16:35.985
So I wanted to keep going.
00:16:35.985 --> 00:16:44.784
And so Were you pretty set on joining academia by academia by that point?
00:16:44.784 --> 00:17:00.019
Or I was somewhat conflicted because part of it was my parents, not so I'm the first person in my whole family to get a doctoral degree, and including my extended family.
00:17:00.225 --> 00:17:02.144
Oh, that's incredible.
00:17:02.144 --> 00:17:02.945
Yeah.
00:17:02.945 --> 00:17:20.980
So for them, this is really a foreign concept, and it's even when I had made the decision, it took some convincing of my parents that I wasn't throwing my life away by not automatically taking six figures from Microsoft or Google or whoever, Amazon.
00:17:22.140 --> 00:17:24.539
And they were really worried, like, we gonna have to pay for this?
00:17:24.539 --> 00:17:26.859
We already paid to help you go to college.
00:17:26.859 --> 00:17:28.275
Like, are we gonna have to pay again?
00:17:28.275 --> 00:17:37.674
Like, just a lot of kind of educating my whole family on what it means to go to grad school in computer science and what the benefits are and what the value could be long term, stuff like that.
00:17:39.730 --> 00:17:44.289
But, yeah, it's a little conflicted because it's a hard decision to make for a lot of people.
00:17:44.289 --> 00:17:46.289
It doesn't make sense to go to academia.
00:17:46.289 --> 00:17:47.730
It doesn't make sense to get a PhD.
00:17:47.730 --> 00:17:51.690
You you can do awesome in industry without it.
00:17:52.035 --> 00:17:54.914
So, you know, it's a it's a tough decision to make.
00:17:54.914 --> 00:17:58.835
But I I landed on the side of I wanna do this.
00:17:58.835 --> 00:18:02.674
I wanna try. I wanna see if possibilities are out there.
00:18:02.674 --> 00:18:13.180
The the database group was incredibly helpful and supportive in in helping me apply to graduate programs, and I ended up getting into MIT, which to me was really exciting.
00:18:13.180 --> 00:18:17.059
That also changed my parents' minds because they've heard of MIT.
00:18:17.099 --> 00:18:18.585
Yeah. That's incredible.
00:18:18.585 --> 00:18:20.184
Yeah. They're like, oh, MIT.
00:18:20.184 --> 00:18:22.505
Well, I guess this grad school thing would be okay.
00:18:22.505 --> 00:18:26.025
They're like, we could we're we're into it.
00:18:26.025 --> 00:18:27.944
We're willing to, like, see where this goes.
00:18:27.944 --> 00:18:29.664
Yeah. Exactly.
00:18:29.944 --> 00:18:32.704
It was at that point where they were like, oh, okay.
00:18:33.420 --> 00:18:36.339
Some name brand school, I guess I get it now.
00:18:36.859 --> 00:18:39.660
I don't no shade to my parents. I love them.
00:18:39.660 --> 00:18:48.059
It's just they're kind of funny because they don't really they're not academics, so the way they see my job in academia is very different from the way I see my job in academia.
00:18:48.059 --> 00:18:50.865
But, so then I went to grad school.
00:18:50.865 --> 00:18:52.625
I started working with Mike Stonebraker.
00:18:52.625 --> 00:18:57.065
He was my PhD advisor. I was in the MIT database group.
00:18:57.744 --> 00:19:06.880
Thankfully, MIT also, at the time, and still does, it's even stronger now, but had really great HCI researchers as well.
00:19:07.000 --> 00:19:21.815
And in fact, it was Rob Miller who doesn't do any viz work, who connected me with another professor at Tufts University, Remco Chang, who is a visualization researcher.
00:19:22.174 --> 00:19:35.654
And it was with Remco and Mike that I started charting out what I actually wanted to do for my dissertation and learning what it takes to be a researcher, not only in databases but also in visualization.
00:19:35.670 --> 00:19:50.684
I did get a very small taste in conversations that people in the database group were having at the time at UW around kind of more HCI and visualization oriented work because there were starting to be some interesting intersections.
00:19:50.684 --> 00:20:04.849
Like, I think at the time, people were talking about, like, Many Eyes, which was, like, a pretty influential platform around back then when I was an undergrad and graduating and moving to MIT.
00:20:06.170 --> 00:20:16.329
But I didn't actually do any visualization research as an undergrad, and it was really in grad school when I started getting exposure to that and and learning about it.
00:20:16.329 --> 00:20:19.585
It was a little tough to do it in a database group.
00:20:19.585 --> 00:20:27.904
I loved being in the database group and I feel like it was a strength as far as my technical ability and my strengths as a researcher.
00:20:27.904 --> 00:20:29.984
I I feel like I really needed that.
00:20:29.984 --> 00:20:38.819
But it's hard to say publish papers at visualization conferences when you're used to reading and learning about database papers in conferences.
00:20:38.819 --> 00:20:42.460
So I really had to find my way.
00:20:42.740 --> 00:20:49.285
Like, the data the database research community is, like, not by and large, like, not that interested in kinda human factors.
00:20:49.285 --> 00:20:49.525
Right?
00:20:49.525 --> 00:20:55.684
It's more about making stuff fast and efficient and correct, like, not, like, easy to use for humans.
00:20:55.684 --> 00:20:57.204
Is that is that right?
00:20:57.204 --> 00:21:05.609
Or is it, like was that kind of, like, the bit of database is like a fringe community, or is that kind of fairly central?
00:21:05.730 --> 00:21:10.450
I love that question actually because it there's some nuance to it.
00:21:10.450 --> 00:21:18.744
I would say a lot of people in the database community had thought that way when I started my PhD.
00:21:19.184 --> 00:21:32.619
And I think today, there's much more of an appreciation for user centered database work, And I see more of it getting published at SIGMOD now than before.
00:21:32.619 --> 00:21:38.220
But on the whole, I would say database researchers do not focus on it.
00:21:38.220 --> 00:21:46.875
I think they can if you spend time talking to them about it, they can understand the value, but they they just don't prioritize it.
00:21:46.914 --> 00:21:52.355
And, of course, there's people there's gonna be people out there who are like, oh, this is a waste of time, or this is not my job.
00:21:52.355 --> 00:21:53.875
I don't see why I should focus on this.
00:21:53.875 --> 00:22:05.769
And there'll be other people, know, like my PhD advisor who really see the value and opportunity there and don't see it as a sacrifice as far intellectually speaking for database research.
00:22:05.769 --> 00:22:19.775
I mean, I think what's been really interesting in the last, you know, in the last twenty years has been, like, the that all the work that's happened at the inter intersection of of database systems and data visualization and human computer interaction.
00:22:19.775 --> 00:23:00.509
And and I think, like, the HCI community has embraced work happening in databases and data processing because without, like, that fast slicing and dicing, like, the cross filtering, you know, being able to, like I think when you see a really, like, a really compelling dashboard with, like, the instantaneous interactions, low you know, having low latency interactivity, like, being able to, like, drag a slider and see, like, sort of start to experience the data through interacting with an interactive data visualization and then seeing, like, oh, the thing that made this possible was, you know, DuckDB, or the thing that made this possible was, like, this analytic database.
00:23:00.509 --> 00:23:06.325
And so, like, without the database technology, you wouldn't be able to create, like, these compelling, you know, data experiences.
00:23:06.325 --> 00:23:10.724
And so I feel like there's this virtuous cycle between, like, between the worlds.
00:23:10.724 --> 00:23:32.970
And so I'm you know, as somebody who's, like, deep inside that world, I I mean, it's just it must be incredibly, you know, incredibly interesting to be able to, like, have influence, like, on both sides of, like, you know, sort of more of the HCI data visualization research as well as, like, the computational side of, like, how do like, what can the computations make possible that wouldn't be possible without it?
00:23:33.384 --> 00:23:34.345
Absolutely.
00:23:34.345 --> 00:23:39.865
I think this is one of the most fun parts of my job because I love straddling that boundary.
00:23:39.865 --> 00:23:59.279
It was really difficult in the beginning just kinda finding my way because you have to I'm I'm fond of saying I basically had to get a PhD in two disciplines to be able to talk to them, but now that I'm there, I really love talking about those research problems and understanding the strengths and weaknesses of both communities, because when you put them together they make magic happen.
00:23:59.279 --> 00:24:14.150
And one classic example that I love to point to that is kind of very easy the very easy win for that intersection is, you know, Tableau and originally the Stanford project Polaris.
00:24:14.150 --> 00:24:22.950
That was an example of how putting those two things together can make magic happen in a way that no one community could have possibly done that on their own.
00:24:22.950 --> 00:24:41.984
With Tableau, just to kind of unpack it a a bit, are you saying so for context, for people to see maybe, like, Tableau is a, in a way, a BI tool where a person can use, like, a graphical interface to click through their data and, like, chart it or present it as a dashboard.
00:24:41.984 --> 00:24:51.750
What what what are you saying Tableau brought specifically kinda to that to the world?
00:24:51.750 --> 00:24:52.549
Okay. Cool.
00:24:52.549 --> 00:25:01.865
So what I love about this because the Tableau paper I think the original Tableau paper came out in, like, two two thousand two.
00:25:01.865 --> 00:25:06.144
And at the time, it was this product this research prototype called Polaris.
00:25:07.704 --> 00:25:27.150
And it was kinda funny too when you read the paper because the paper is when you read the paper, the introduction is basically making the case for, hey, viz community, this is why we should care about what databases do and how we can map our visualization primitives onto database primitives to automatically generate SQL queries to process our data.
00:25:27.150 --> 00:25:31.525
That's basically what that paper was bringing to the viz community.
00:25:31.525 --> 00:25:42.849
It was saying, a, we should care about this, and b, our theory and database theory could actually be integrated to produce this really nice automated stack going top to bottom and bottom to top.
00:25:42.849 --> 00:25:47.970
And the grammar that kind of tied it all together is this this grammar called VizQL.
00:25:47.970 --> 00:25:50.210
Basically, this mapping across the stacks.
00:25:50.210 --> 00:25:51.569
Because you can think of it as a stack. Right?
00:25:51.569 --> 00:25:57.125
You have your data processing and storage layers, which database management community typically owns.
00:25:57.125 --> 00:26:05.125
And then you have the visualization and interaction and is there interface design layers which the visualization community typically owns.
00:26:05.125 --> 00:26:14.180
And that paper was saying, why bisect the problem in that way when you can actually do this more integrated approach?
00:26:14.180 --> 00:26:21.299
And nowadays people would look at that, and I even had PhD students after reading the paper read that and be like how is this groundbreaking?
00:26:21.299 --> 00:26:22.579
Everybody just does this.
00:26:22.579 --> 00:26:25.494
And I'm thinking that's the point. That's the point.
00:26:25.494 --> 00:26:29.974
That paper was basically saying everybody should be doing this.
00:26:29.974 --> 00:26:34.134
And in twenty twenty six, it's very easy to say, oh, but everyone's doing this, whatever.
00:26:34.134 --> 00:26:49.940
So just the idea that those communities should be working together and integrating what they're doing on a foundational level, I think was a tectonic shift, intellectually speaking, for the viz community and the database community.
00:26:49.940 --> 00:27:05.394
And I think that's also why Tableau as a company had this, like, wild commercial success soon after that because those communities just weren't connecting in a way that that company was able to kinda put them together at the time.
00:27:05.394 --> 00:27:11.950
So to be fair, there were definitely companies before and after who've done great things that are kind of us in a similar spirit.
00:27:11.950 --> 00:27:12.589
Right?
00:27:12.589 --> 00:27:17.309
I didn't realize though there was a research paper associated with Tableau and a lot of it.
00:27:17.309 --> 00:27:19.190
That's pretty crazy to hear.
00:27:19.204 --> 00:27:30.404
I I do find it so so fascinating, like, how we kind of build these, like, boundaries around, like, this is the area in which I work, or this is the area in which we work.
00:27:30.404 --> 00:27:37.029
And then it becomes almost, like, heretical to kinda, like, step outside of it or combine it with something else.
00:27:37.029 --> 00:27:39.430
Like, it's very difficult to like, you yeah.
00:27:39.430 --> 00:27:47.204
Each community builds up its own kind of standards, its own, like, vocabulary, and that's how these you you generate these, like, shibboleths.
00:27:47.204 --> 00:27:56.164
Like, this is how I advertise myself to other people so they know I'm a serious database researcher or a serious viz researcher.
00:27:56.164 --> 00:28:03.039
And then when someone comes out from the outside and they don't know how to, like, signal all of these, like, oh, I'm a serious whatever.
00:28:03.039 --> 00:28:06.960
And just so can be, like, so hard to, like yeah.
00:28:06.960 --> 00:28:14.444
Like, that I guess I never really understood why, like, interdisciplinary or cross disciplinary work was so hard, but it's not really about the work.
00:28:14.444 --> 00:28:18.765
It's about the people and the way they define themselves and, like, their communities.
00:28:18.765 --> 00:28:19.404
Yeah.
00:28:19.404 --> 00:28:21.964
Absolutely. Yeah. I think so.
00:28:21.964 --> 00:28:28.724
And lots of times, a community defines itself by the problems it chooses to focus on, that community chooses to focus on.
00:28:28.730 --> 00:28:54.704
And that work, and then also just my, I like to think the work that I've focused on in my career is trying to break down some of those walls to reclaim some of what we lose by those, you know, those boundaries that we those artificial boundaries that we put up to kind of just define and protect our communities in some ways.
00:28:54.704 --> 00:29:13.414
Do you feel like today you mostly are the are the conferences you go to today mostly split still, like like, bucketed in one of the two areas, or are there many sort of, like, cross disciplinary conferences for sort of the HCI, viz, and computation stuff?
00:29:13.414 --> 00:29:14.855
That's a really great question.
00:29:14.855 --> 00:29:23.775
So as far as conferences I publish in, I will publish wherever I believe the work is a good fit.
00:29:23.815 --> 00:29:25.494
That could be an HCI conference.
00:29:25.494 --> 00:29:27.380
That could be a visualization conference.
00:29:27.380 --> 00:29:29.779
That could be an AI ethics conference.
00:29:29.779 --> 00:29:36.819
That could be a, you know, per more programming languages oriented conference or a database conference.
00:29:36.819 --> 00:29:40.779
I think to me, the most important thing is that community cares about the work that we did.
00:29:41.065 --> 00:30:01.390
So this would be different if I was working really hard to get tenure because I would want one community to know me well, so I'd probably want my home base to be in one or maybe two communities so that when, you know, letters go out recommending my tenure case, they're saying, I know Leilani. She does great work in x.
00:30:01.670 --> 00:30:08.309
If you're too diffuse in your portfolio, then people can't write those kinds of letters because they'll be like, who's that?
00:30:08.309 --> 00:30:12.154
Leilani, who's that? Or I don't really know her work.
00:30:12.355 --> 00:30:25.035
So there's that danger, but if you can avoid that danger, I think it's better to focus more on where the work will be best received and have the most impact rather than worry about, you know, oh, I published all my papers at this.
00:30:25.400 --> 00:30:30.119
That being said, practically speaking, I can't jet set around the world all year.
00:30:30.119 --> 00:30:33.559
I have to be practical about what conferences I actually go to.
00:30:33.559 --> 00:30:42.375
I try to make sure, when possible, try to make sure I go to at least one, like, viz or HCI conference and one database conference a year.
00:30:42.375 --> 00:30:47.654
The past couple years have been weird, and I've been prioritizing sending my students to conferences instead.
00:30:47.654 --> 00:30:52.654
But, yeah, typically when I attend a conference, I'll try to go to one of each.
00:30:53.490 --> 00:31:01.490
And and are you sort of conscious of, like, code switching when you're writing, like, a database paper versus, like, a HCI paper?
00:31:01.490 --> 00:31:02.609
Yes.
00:31:02.609 --> 00:31:12.884
I am deliberate, and I do code switch between communities because the things that they talk about, the terminology they use, the things they emphasize are different.
00:31:13.164 --> 00:31:16.684
That being said, with AI, things are blending a little more.
00:31:16.684 --> 00:31:22.839
We're sort of ending up in similar places more so than I've seen in the past because we have kind of similar goals we wanna achieve.
00:31:22.839 --> 00:31:30.039
So I I think things are blending more now, but I do have to code switch when I talk to different communities.
00:31:30.039 --> 00:31:33.720
And and when you say AI because of AI, you mean, like, the problems it faces.
00:31:33.720 --> 00:31:40.065
Not that people are using AI to write all their papers now, and hence, all of the papers seem the same regardless of discipline.
00:31:40.065 --> 00:31:41.065
Yeah.
00:31:41.184 --> 00:31:51.009
I mean more that, say you want to use AI to help people achieve data science tasks, like complete data science tasks, analyze their data, things like that.
00:31:51.009 --> 00:32:04.009
Database researchers looking at that problem and visualization, and HCI researchers looking at their problem will end up in similar spaces because there's things like how exactly do we prompt models or design infrastructure around the models to facilitate this.
00:32:04.265 --> 00:32:13.785
When a model manipulates the data, how do we communicate that to users in a way that they can feel comfortable that they understand what's going on and they can trust it?
00:32:13.785 --> 00:32:29.974
And then also, if we want to empower users specifically to analyze the data, how can we help them do that in tandem with the model or maybe co plan and coordinate with the model such that, you know, they achieve the goal that they wanna achieve in with hopefully, you know, minimal effort?
00:32:29.974 --> 00:32:45.839
So with those things in mind, right, database people are starting to have to worry about interfaces a bit more and transparency in the behavior of the system because users are a little less trusting of black boxes and wanna know what's going on.
00:32:45.839 --> 00:32:56.640
And also, when a mistake happens, it's not super obvious why the mistake happens, so users are gonna need more transparency as well on, oh, why did this AI powered or LLM powered operation do this thing?
00:32:56.640 --> 00:32:57.680
I didn't understand.
00:32:57.680 --> 00:33:01.160
And then also on the flip side with these people. Right?
00:33:02.704 --> 00:33:10.944
Being able to provide enough context to the model can very easily and quickly become a data management problem as well.
00:33:10.944 --> 00:33:19.160
So there it's interesting how those two sides are blending with this shared interest of maximum utilization of LLMs.
00:33:19.160 --> 00:33:38.275
I mean, thing I'm curious about is, I mean, is there is there a sense kind of in your in your community that your research community that that, you know, your field is going to experience, like, the same type of you know, or different you know, certain type of disruption compared with more general computer science or or software engineering.
00:33:38.275 --> 00:33:43.829
Like, I think, you know, software engineering is clearly, like, in a state of upheaval and and disruption.
00:33:43.829 --> 00:33:54.869
But I imagine, like, in in data visualization and interactive data that that there is disruption, but perhaps we don't know, like, what things are going to look like on the other side.
00:33:54.869 --> 00:34:13.710
I know, at least in the business world, there's a lot of discussion about, like, headless BI and some and, you know, semantic layers becoming more important, like trying to give agents, like, a simpler language to talk about, you know, relationships and data models and describing data visualizations and and and things like that.
00:34:13.710 --> 00:34:20.590
And like, I feel like databases have become even more, you know, even more important because they can't be they can't be vibe coded quite so easily.
00:34:20.590 --> 00:34:46.010
Whereas I I feel like we're gonna see a lot more like just in time, you know, dashboards and like just in time data interfaces where you aren't necessarily looking at, you know, something that was developed over a long period of time, but maybe something that was stood up on top of, you know, stood up on top of DuckDB or on top of one of the interactive, you know, data visualization frameworks, you know, and and like a as like a, you know, throwaway prototype.
00:34:46.010 --> 00:34:54.250
And, you know, in the past, you you can't you couldn't imagine, you know, building some interactive data application only to throw it away and then rebuild it, you know, the next day.
00:34:54.250 --> 00:34:59.425
But now that's something that, you know, you could build these, like, just in time interactive data interfaces.
00:34:59.425 --> 00:35:11.989
And so I feel like that is going to lead to, like, new interesting avenues of research, like how to incorporate, like, how to incorporate LLMs into the, like, you know, essentially into the process.
00:35:11.989 --> 00:35:14.070
So Absolutely.
00:35:14.070 --> 00:35:22.670
I think we're already experiencing a massive shift in the way people do data science work with the introduction of large language models.
00:35:23.030 --> 00:35:30.614
One easy example I can point to is in chart generation and visualization recommendation.
00:35:30.614 --> 00:35:43.219
A lot of a lot of my prior work has been in visualization recommendation and automated methods to support that, And now we always have to answer this question of, oh, well, how would LLM solve this problem?
00:35:43.219 --> 00:35:51.099
Or how would someone use an LLM to, you know, generate a chart or generate code to generate charts or even generate dashboards?
00:35:51.795 --> 00:35:58.275
So when we try to introduce new ideas in that space, we always have to be thinking about it.
00:35:58.275 --> 00:36:12.860
And also, practically speaking, a lot of people I know, researchers, folks in industry, they'll just very quickly go to a model and ask it to make charts for them.
00:36:12.860 --> 00:36:20.655
So just on a practical level, people are using LLMs in a way where they would traditionally use maybe viz tools and stuff like that.
00:36:20.655 --> 00:36:24.815
And also even tools like data science tools themselves are doing it as well.
00:36:24.815 --> 00:36:28.414
Right? Just automatically using providing AI recommendations.
00:36:28.414 --> 00:36:31.820
So I think huge, huge change.
00:36:31.820 --> 00:36:40.739
I can spotlight one ongoing project in our lab around this because we are curious, you know, what the shift looks like.
00:36:40.780 --> 00:36:57.445
So Will Wong, a PhD student in our lab who's co advised by myself and Jeff Heer, He's investigating this problem by interviewing statistical and and data analysis consultants at universities.
00:36:57.965 --> 00:37:32.239
Because traditionally, pre LLM days, if you're a scientist and you weren't totally sure how to, you know, analyze and statistically test your data that you collected for an experiment or something, your university, there's a very good chance, like, you could talk to the statisticians there, or there's even a consulting office you could go to, and then a consultant could work with you to figure out what questions you're trying to answer from your data, whether you collected the right data, what statistical models and tests make the most sense, how to process your data properly for it and so on.
00:37:32.280 --> 00:37:44.644
But you can imagine with LLMs, like, statistical consulting must be seeing a humongous tectonic shift, like, experiencing a huge shift in their, like, client base and how they are utilized, how consultants are utilized.
00:37:44.644 --> 00:37:57.364
So we are wrapping up an interview study with consultants to see how, like, their perception of language models and how it's affected their jobs.
00:37:57.364 --> 00:37:59.539
And it's been it's been really interesting.
00:37:59.539 --> 00:37:59.780
Yeah.
00:37:59.780 --> 00:38:11.139
It's so interesting to hear that, yeah, that that avenue opening of, like, you're mentioning, like, just being able to ask an LLM to generate a visualization.
00:38:11.139 --> 00:38:27.974
And in the context of your really thoughtful, careful work on, like, visualization recommendation, still thinking critically about what's the quality of that recommendation and, like, who's asking for it.
00:38:28.110 --> 00:38:36.670
And it sounds like you're also saying, like, what happens to the people who were who they were going to before for that advice, all those kinds of elements.
00:38:36.670 --> 00:38:38.829
So from different angles, we've been tackling this problem.
00:38:38.829 --> 00:38:57.284
So, Will, he the PhD student I mentioned, he also had a a previous paper on evaluating the kinds of recommendations at the time that models were giving when you asked them to generate charts or to complete a partial visualization design.
00:38:57.284 --> 00:39:01.579
So if you had a partial design, you wanted the model to finish it, what would it recommend?
00:39:01.579 --> 00:39:07.460
Or if you wanted the model to just recommend charts, you know, what what do you get from it?
00:39:07.820 --> 00:39:10.860
At the time so to be fair, this is a paper from twenty twenty four.
00:39:10.860 --> 00:39:15.494
It's twenty twenty six now. Models move fast, update quickly.
00:39:15.614 --> 00:39:25.295
At the time, his work found that what the models were recommending was mostly fine, but there were some cases where they would recommend weird stuff.
00:39:25.295 --> 00:39:30.670
Like, would recommend visual encodings that didn't really make sense for the context.
00:39:31.510 --> 00:39:43.635
And I think kind of the big takeaway for me was that sometimes the models were recommending designs that didn't seem to align with the data that we have in our community.
00:39:43.635 --> 00:39:47.474
So as visualization researchers, as scientists, we've collected data.
00:39:47.474 --> 00:39:55.514
We've done tons of experiments seeing what kinds of design decisions make sense, which ones don't make sense.
00:39:56.239 --> 00:40:00.159
And it just didn't seem like models like, what they were learning on was not that.
00:40:00.159 --> 00:40:01.599
It didn't seem like it was that.
00:40:01.599 --> 00:40:03.519
At least not most of the time.
00:40:03.519 --> 00:40:09.514
And for proprietary models, who knows what exactly they're learning their viz recommendations from?
00:40:09.514 --> 00:40:12.755
But it seemed like the answer was not us.
00:40:12.795 --> 00:40:14.355
Sometimes, anyway.
00:40:14.474 --> 00:40:18.715
Do you think this is changing how the viz community, like, types of tools it makes?
00:40:18.715 --> 00:40:28.159
Like, my recollection, which is now very out of date, is that, you know, people would tend to produce kind of like, you know, polished UI, point and click kind of interfaces.
00:40:28.159 --> 00:40:32.639
But obviously, like, an LLM is gonna have a hard time driving that.
00:40:32.639 --> 00:40:43.744
Do you so do you think this is kinda pushing the viz community to produce tools that are like plain text or JSON or code based that then the LLM can generate the visual specification?
00:40:44.744 --> 00:40:46.344
It's a good question.
00:40:46.344 --> 00:41:04.079
So kind of before, during, and now, like this before and during the AI wave, I think we were already in some ways transitioning away from these kind of standalone viz tools, at least in the specific areas I work in.
00:41:04.184 --> 00:41:13.025
We're sort of transitioning away from these standalone viz tools and more towards, like, modules and things that can integrate into stuff like computational notebooks.
00:41:13.144 --> 00:41:16.744
And I I personally feel like that's the future, having a standalone.
00:41:16.744 --> 00:41:40.284
Unless you're providing an environment that other tools would integrate into, I I don't think it's really to your advantage to have a standalone tool that you create as a viz researcher because then you have to argue to everybody why they should leave whatever environment they already use and use your tool instead and then export whatever your tool makes back into their usual workflow.
00:41:40.284 --> 00:41:41.885
So, yeah, that I mean, that's fascinating.
00:41:41.885 --> 00:41:49.409
Because, I mean, I I haven't and not that I was ever particularly active in the viz community, but when I was active, that was, like, fifteen years ago.
00:41:49.409 --> 00:41:51.170
And that was the thing that struck me.
00:41:51.170 --> 00:41:57.889
Like, why would someone use your cool new visualization when they've gotta do it in a completely different computational environment?
00:41:57.889 --> 00:42:05.045
Like and so so it just felt like the viz community didn't actually care about people using their visualizations, which is, you know, like, fine.
00:42:05.045 --> 00:42:11.925
Not all research has to be, like, end user focused, but it did feel, like, weird to me.
00:42:11.925 --> 00:42:18.289
So that's that's neat that people are more focused and, like, let's integrate this into where people are already working.
00:42:18.289 --> 00:42:27.994
You still see a lot of stand alone tools just practically speaking because it's easy for a grad student to make a research prototype that way than to build a, like, Jupyter plugin or something.
00:42:27.994 --> 00:42:40.860
That being said, with large language models, Jupyter plugins are way easier to make now than these to So I think there's less of an excuse now and that integration, because I don't know.
00:42:40.860 --> 00:42:46.619
I think there is more of a question around computational notebooks years ago, like, earlier in my career.
00:42:46.619 --> 00:42:49.980
Nowadays, it's like a no brainer computational notebook environment.
00:42:49.980 --> 00:42:51.500
Like, why wouldn't we use it?
00:42:51.500 --> 00:42:55.179
So I think the viz community has been forced to adopt it more.
00:42:55.179 --> 00:42:55.844
Yeah.
00:42:55.844 --> 00:43:03.684
It's funny because, yeah, because computational notebooks are, like, such a like, in some ways, and, like, I can you know, that's such a terrible environment.
00:43:03.684 --> 00:43:06.644
But, like, they're they're clearly, like, the least worst environment.
00:43:06.644 --> 00:43:08.005
Like, everyone knows them.
00:43:08.005 --> 00:43:12.969
They're just this great, like, you know, ground floor that everyone can rest on.
00:43:12.969 --> 00:43:18.329
But I'm sure like, I I can just imagine, like, the viz community looking at them and being like this.
00:43:18.329 --> 00:43:19.130
These are awful.
00:43:19.130 --> 00:43:25.010
Like, there's so many problems with them that Kinda like MapReduce in the database community.
00:43:26.574 --> 00:43:31.175
You can't beat them, join them, or come up with a better version.
00:43:31.695 --> 00:43:34.974
That reminds me of this advice I heard years ago.
00:43:34.974 --> 00:43:47.320
Like, how like, if you wanna if you wanna pick some, like, machine learning algorithm to use in your actual applied work, like, how do you how do you do what how do you figure out which algorithm should should use?
00:43:47.320 --> 00:43:57.465
And the advice was, like, go read, like, three of the top recent papers, and then look at what they benchmark their method against, and then use that.
00:43:57.465 --> 00:44:00.144
Because, like, that's the gold standard.
00:44:00.264 --> 00:44:02.784
It's so true, actually.
00:44:03.465 --> 00:44:07.340
That's a smart way of picking a good commercial tool to use.
00:44:07.340 --> 00:44:15.260
I do I think it'd be nice too to talk about the responsible data science stuff, which I know is a lot more of your recent work.
00:44:15.260 --> 00:44:21.514
But I figured maybe before we get to that, I'm curious if you have, like, a favorite viz study you've done.
00:44:21.514 --> 00:44:27.835
Like, do you have a study of yours that's, like, a personal favorite that you've done on viz?
00:44:27.835 --> 00:44:33.510
I would say there have been, like, takeaways from different studies that I've done that I've really enjoyed.
00:44:33.510 --> 00:44:40.750
I don't know if I would say, oh, that whole everything about that study, I just love it about everything else and all the rest.
00:44:41.349 --> 00:44:43.429
You have some favorite takeaways, though?
00:44:43.429 --> 00:44:45.710
Yeah. I have some favorite takeaways.
00:44:46.344 --> 00:44:52.344
So one funny takeaway from a study we did in twenty twenty two, I believe.
00:44:52.344 --> 00:44:54.025
That was the twenty twenty two CHI paper.
00:44:54.025 --> 00:44:58.905
So this work was in collaboration with Michael Correll.
00:44:58.905 --> 00:45:03.369
He at Tableau at the time, but now he I believe he's at Northeastern.
00:45:03.369 --> 00:45:16.525
But the it was around visualization recommendation, and our our idea was to see, okay, how would people like automated recommendations versus, like, human curated recommendations?
00:45:16.525 --> 00:45:21.405
And what arguments are people making in terms of why they like one versus the other?
00:45:21.405 --> 00:45:33.519
So what we did was we picked up some picked up some commercial tools like Tableau, Power BI, and such, had them generate recommendations for visualizations of public health data.
00:45:33.559 --> 00:45:37.000
And this data was specifically a subset of data from NHANES.
00:45:37.000 --> 00:45:41.760
So the it's basically a national health survey.
00:45:42.405 --> 00:45:55.980
I'm not gonna make myself regurgitate that acronym right now off the top of my head, but it's basically a national health survey data from the from the, I think from the CDC or or the National Institutes of Health, one or the other.
00:45:55.980 --> 00:45:57.579
Maybe the National Institutes of Health.
00:45:57.579 --> 00:45:59.820
But we had the self survey data.
00:45:59.820 --> 00:46:07.539
We showed it to students, professionals, and researchers in public health.
00:46:08.625 --> 00:46:19.784
And we showed them recommendations from, you know, automated tools and also from just people crafting recommendations.
00:46:19.824 --> 00:46:22.440
And the professional like, not the professional.
00:46:22.440 --> 00:46:32.199
The the participants of our study generally tended to prefer the human created recommendations because they felt that the human created recommendations told more of a story.
00:46:32.199 --> 00:46:36.239
There's more of a story behind why that recommendation was generated.
00:46:37.184 --> 00:46:43.344
And at the time, the automated tools, they just they generated pretty overly simplistic recommendations.
00:46:43.344 --> 00:46:47.025
You could tell they were just putting variables together and then generating charts.
00:46:47.025 --> 00:46:52.230
And so that, like, is a a fun, like, favorite takeaway of mine.
00:46:52.230 --> 00:46:57.349
Of course, this is, like, pre LLM, so maybe the automated tools would do better now.
00:46:57.349 --> 00:46:59.429
So we should redo that experiment.
00:46:59.429 --> 00:46:59.670
Yeah.
00:46:59.670 --> 00:47:05.835
It's interesting to hear that it's people they prefer, yeah, recommendations from people.
00:47:05.835 --> 00:47:08.474
And it sounds like maybe was that surprising to you?
00:47:08.474 --> 00:47:17.675
What I would say was surprising wasn't even necessarily at the running the study stage, but at the experiment design stage when we were looking at the recommendations.
00:47:17.675 --> 00:47:20.570
Because we could also see it before we ran the study.
00:47:20.570 --> 00:47:24.730
We're like, how are these tools generating these recommendations?
00:47:24.730 --> 00:47:27.170
This seems really generic.
00:47:27.289 --> 00:47:28.730
Why would anyone want this?
00:47:28.730 --> 00:47:33.414
Maybe we should've given that feedback to the tool creators, but yeah.
00:47:33.414 --> 00:47:35.695
That to me was always surprising.
00:47:36.135 --> 00:47:42.695
But maybe it's because I'm a I'm a researcher who focuses on visualization recommendations, so I could see it when I was trying to use these tools.
00:47:42.695 --> 00:47:45.974
Oh, you know, these recommendations are not like, I don't understand who they're helping.
00:47:45.974 --> 00:47:47.989
It doesn't seem that they're helping anybody.
00:47:47.989 --> 00:47:56.750
It sounds like the thing that really struck you was the as you were running it, you were really struck by the kinda difference between the automated and the humans.
00:47:57.269 --> 00:47:57.750
Yeah.
00:47:57.750 --> 00:48:03.985
I was just struck by, like, how some overly simplistic the automated recommendation methods were at the time.
00:48:03.985 --> 00:48:13.344
Like, I guess they wanted to from a commercial standpoint, I can understand wanting to generate recommendations that will be, you know, consistent.
00:48:13.344 --> 00:48:14.789
They'll always work.
00:48:14.789 --> 00:48:19.909
They aren't going to make any customers unhappy, that kind of thing.
00:48:19.909 --> 00:48:36.844
So I can understand strategically why you would do that to but as a researcher who, you know, likes to do push the boundaries of what computationally we can do, I was thinking there's no way that someone would pick these recommendations over the human curated ones.
00:48:36.844 --> 00:48:39.724
And I I hope it was not a self fulfilling prophecy.
00:48:39.724 --> 00:48:44.565
I like to think I designed good experiments, but we we pretty much saw that in the study results.
00:48:44.869 --> 00:48:46.949
Yeah. It is interesting to hear.
00:48:46.949 --> 00:48:51.190
It's like you kinda see it coming, but you have to run run it through.
00:48:51.190 --> 00:48:51.909
Yeah.
00:48:51.909 --> 00:48:53.630
Yeah. Basically.
00:48:53.670 --> 00:49:00.269
And then one more study I'll highlight that was really fun, actually, was from the tail end of my dissertation work.
00:49:00.724 --> 00:49:02.244
Basically, you wanted to see, okay.
00:49:02.244 --> 00:49:10.164
Well, with recommendations, there could be this issue of performance where, let's say, you're analyzing a large dataset.
00:49:10.164 --> 00:49:14.164
What if the thing people look at is the thing you can recommend the fastest?
00:49:14.164 --> 00:49:15.949
Basically, the thing they see first.
00:49:15.949 --> 00:49:22.829
How could that potentially introduce manipulation of people?
00:49:22.829 --> 00:49:27.309
Like, what what conclusions they might draw or or what findings they might find in their data?
00:49:27.309 --> 00:49:31.670
So what we wanted to do is design an abstract experiment to try to test this.
00:49:31.704 --> 00:49:41.105
Could you, in a sense, manipulate what people found in a visualization purely through latency?
00:49:41.224 --> 00:49:46.864
Like, how quickly or how slowly people might see the data or different parts of the data.
00:49:47.599 --> 00:49:51.519
So I designed this, like, really abstracted version of this experiment.
00:49:51.519 --> 00:49:58.639
I created this image collage and asked people to explore the collage of images, and then they were looking for certain a certain target image.
00:49:58.639 --> 00:50:07.635
Actually, put two targets, like two copies of that target in the collage, but the goal of the task is to only find one.
00:50:07.835 --> 00:50:18.900
And you can imagine kinda like Google Maps, hey, you sometimes say you're on your phone and your data speed isn't that great and the map takes a while to load, maybe loads in kinda patchy.
00:50:18.900 --> 00:50:25.579
Imagine something kinda like that in this experiment, but it's an image collage, so some images appear faster, slower than others.
00:50:25.780 --> 00:50:33.474
So I designed the latency for the images such that it would look kinda random, but in fact, it was definitely not random.
00:50:33.474 --> 00:50:49.659
I was manipulating how quickly certain images would appear, and I was trying to make it such that I only use latency to try to incentivize people to find one target over another in the collage.
00:50:49.659 --> 00:50:57.659
And the the experiments showed that if the latency was pretty low overall, like relatively low, it didn't affect people as much.
00:50:57.659 --> 00:51:10.525
And then but gradually, as you ratcheted up the latency, like, you made it the tile, the images load in slow enough, you could definitely manipulate people to find a certain target in the collage.
00:51:10.525 --> 00:51:19.889
So it was just showing how latency matters, how you design a system matters, but also how you recommend things matters.
00:51:19.889 --> 00:51:39.260
If you do it purely based on performance and don't pay attention to what, you know, what you're prioritizing Like, if you're if you have options to show a or b and you always make a show up faster, depending on how much faster a shows up, you might be incentivizing the user to just always look at a and never look at b.
00:51:39.260 --> 00:51:46.619
That seems like super relevant in the era of LLMs again where, like, latency is becoming so important again.
00:51:46.619 --> 00:51:47.739
Absolutely.
00:51:47.739 --> 00:51:51.900
So that to me was like a fun experiment that I enjoyed.
00:51:51.900 --> 00:51:53.579
I really enjoyed the design of the experiment.
00:51:53.579 --> 00:51:56.755
The takeaway was pretty straightforward in my opinion. In my opinion.
00:51:56.755 --> 00:52:06.034
Like, I guess one thing that was surprising about it was people kind of thought of latency as a sort of binary thing, like, oh, there's either too much or too little, but that experiment was showing.
00:52:06.034 --> 00:52:14.789
It's actually kind of a gradual effect, And the more latency you have, the worse the effect becomes for people that get become kinda biased.
00:52:14.789 --> 00:52:21.630
And the, I guess, the latency between two things being relative as well, I guess, where two things are competing?
00:52:22.545 --> 00:52:22.945
Yeah.
00:52:22.945 --> 00:52:30.545
Like, in an image collage, there are all these different images you could be looking at, but your eye and your mind is sort of drawn to the stuff that shows up first.
00:52:30.545 --> 00:52:36.949
And if it shows up way earlier than everything else, you're just gonna kinda predicate your exploration on that.
00:52:36.949 --> 00:52:57.144
I feel like it's so helpful to hear the, like, surprising pieces of people's research, like, stood out and kind of whether it was the I think it's interesting to think about too, like, does the thing that could stand out to someone or that you might like might be the designing of it or the, like, what you saw early on and and the takeaways.
00:52:57.144 --> 00:52:59.905
So, yeah, I really appreciate you sharing.
00:53:00.105 --> 00:53:06.820
I know I know we're getting long, but I definitely wanna get to the responsible data science because I know it's what you're working on now.
00:53:06.820 --> 00:53:11.420
Could you could you tell us a little bit about that work and how you got into it?
00:53:11.539 --> 00:53:14.820
Yeah. So responsible data science is this giant umbrella.
00:53:14.820 --> 00:53:17.300
Lots of people work in this space.
00:53:17.300 --> 00:53:27.724
At a high level, respond the the whole point of responsible data science is the idea that we want the outcomes of data science work to be responsible.
00:53:27.724 --> 00:53:31.965
We want that people are not discriminated against.
00:53:31.965 --> 00:53:39.909
We want that people are not harmed by the outputs of, say, models, large language models and otherwise.
00:53:39.909 --> 00:53:43.070
We want people to be treated fairly.
00:53:43.429 --> 00:53:44.389
Okay.
00:53:44.389 --> 00:53:49.744
So everybody who wants that and is working towards that, they fall under this responsible data science umbrella.
00:53:49.744 --> 00:53:57.184
What I think is really cool about the work that I'm doing, and this is in collaboration with a few professors who I wanna shout out.
00:53:57.184 --> 00:54:14.670
So I wanna shout out Emily Wall, kind of my longest standing collaborator on this work, professor Amy Zhang, who Amy and I co advised an awesome PhD student working in this space, Teanna Barrett, and then also professor Yuichi Shoda.
00:54:14.670 --> 00:54:18.349
So Yuichi is a faculty member in psychology at UW.
00:54:18.349 --> 00:54:26.664
And without Yuichi, we wouldn't be I wouldn't be where I am now with these projects, so I think his perspective has been really helpful too.
00:54:26.664 --> 00:54:50.679
But the thing that always struck me about responsible data science work, where I think it's changing a bit more now, but years ago when Emily and I started talking about this problem, what struck us about it and and what struck me about it especially is how it was always sort of treated as an algorithmic problem, a model training problem, or a data cleaning problem.
00:54:51.585 --> 00:54:54.784
The problem was always someone or something else.
00:54:54.784 --> 00:54:57.144
The problem was never us.
00:54:57.664 --> 00:54:59.664
And I was just thinking, wait a minute.
00:54:59.664 --> 00:55:02.224
So who does data science work?
00:55:02.224 --> 00:55:03.985
Is the data science worker the model?
00:55:03.985 --> 00:55:07.579
Is the data science worker the data? Or is it me and you?
00:55:07.579 --> 00:55:15.900
So if we're going to, you know, try to make data science work more responsible, should we be emphasizing the artifacts?
00:55:15.900 --> 00:55:18.739
Or should we emphasizing the people doing the work?
00:55:19.284 --> 00:55:26.485
So what came out of that is this idea of behavior change interventions for responsible data science.
00:55:26.485 --> 00:55:33.619
So the idea that we should not just focus on whether or not a data set is responsible, a model's responsible.
00:55:33.619 --> 00:55:46.344
We should also be focusing on, you know, are the practices of the data scientists responsible, and how does that contribute to ethical or unethical outcomes in data science?
00:55:46.344 --> 00:55:52.144
So that's the perspective that we brought to the community.
00:55:52.344 --> 00:55:55.784
Because initially, people weren't really thinking about the problem in this way.
00:55:55.784 --> 00:55:58.224
They weren't talking about it in this way.
00:55:58.264 --> 00:56:00.344
People were actually kinda skeptical too about it.
00:56:00.344 --> 00:56:25.375
It took us a bit to actually publish our very first paper in that space, where it was kind of a mixture of a literature review and vision paper describing what behavior change would mean in the context of responsible data science and what kinds of tools in psychology and in computer science that we could use to actually achieve that goal of trying to help people shift their behaviors.
00:56:25.494 --> 00:56:53.054
And then since then, we've had a couple a few more papers on that line in that line of work where the most recent one is actually a paper at CHI, so top HCI conference, where we actually evaluated two existing behavior change intervention strategies in the context of a controlled data modeling task.
00:56:53.414 --> 00:57:10.650
So we asked people to train a machine learning model with some provided data that we gave them during the experiment, And then we they they experienced I think they experienced two different intervention strategies.
00:57:10.650 --> 00:57:13.610
So one was we primed them.
00:57:13.610 --> 00:57:22.625
So we had them read a little prompt that hopefully motivated them to be more responsible in their actions and training the model.
00:57:22.625 --> 00:57:31.579
And then a separate intervention where we actually just loaded, like, a a modular plug in into the notebook they were using to train the model.
00:57:31.579 --> 00:57:38.500
And they could use that tool, that specific tool, to check the fairness of their model and improve it.
00:57:38.699 --> 00:57:40.619
Yeah. Yeah. So sorry.
00:57:38.699 --> 00:57:40.619
I have it here.
00:57:40.619 --> 00:57:47.715
I was trying to grab the but this is evaluating behavior change interventions for responsible data science.
00:57:47.715 --> 00:57:48.755
Is that Yes.
00:57:48.755 --> 00:57:49.394
Yeah. Yeah.
00:57:49.394 --> 00:57:52.675
Evaluating behavior change interventions for responsible data science.
00:57:52.675 --> 00:57:53.235
Yeah.
00:57:53.235 --> 00:58:02.110
So a cool outcome of that work was that priming people to think responsibly did shift their behaviors.
00:58:02.110 --> 00:58:18.085
They they did have an increase in the range of responsible activities they did during the session, But the fairness of their models did not actually get better from a significance perspective.
00:58:18.085 --> 00:58:22.324
We did not notice a significant improvement in the fairness of the models.
00:58:22.324 --> 00:58:30.679
So just motivating people to do better, basically, just telling them to do better and them feeling like they wanna do better doesn't actually lead to anything any better.
00:58:30.679 --> 00:58:35.079
So also, some things change, but not necessarily a measurably better way.
00:58:35.079 --> 00:58:45.614
So giving them tools to actually show them, like, this is what better looks like in terms of outcomes and pairing that with the motivation, that actually works.
00:58:45.614 --> 00:58:51.775
I think that behavior change framing is, like, so interesting and, like, powerful.
00:58:51.775 --> 00:58:54.659
Like, it's not something I don't know.
00:58:54.659 --> 00:59:13.454
Like, I think in, you know, how a computer science and statistics, it's so easy to get locked in on, like, the the algorithms and the mathematics and forget, like, oh, actually, it's humans doing all this, and we have these levers to pull on how the humans use our tools, and that's just as valuable.
00:59:13.454 --> 00:59:14.414
Powerful.
00:59:14.414 --> 00:59:15.934
Yeah. But not more. Yeah.
00:59:15.934 --> 00:59:16.335
Yeah.
00:59:16.335 --> 00:59:17.775
What what was the tool called?
00:59:17.775 --> 00:59:19.375
Is it Aequitas?
00:59:19.375 --> 00:59:22.454
Oh, Aequitas.
00:59:19.375 --> 00:59:22.454
Aequitas. Yeah. Okay.
00:59:22.699 --> 00:59:25.179
Yeah. What does that look like?
00:59:22.699 --> 00:59:25.179
It runs inside the notebook.
00:59:25.179 --> 00:59:27.579
Is that It does run inside the notebook.
00:59:27.579 --> 00:59:34.619
It shows you some bias metrics, and you can pick specific subgroups in your dataset that you wanna compare against.
00:59:34.619 --> 00:59:45.485
For example, a dataset that we asked participants to train models with was sort of a classic, like, loan application dataset.
00:59:45.485 --> 00:59:51.525
So a you could a demographic group you could compare would be like men versus women and their loan applications.
00:59:51.900 --> 01:00:00.420
So you could see if your model would be, say, biased against women and be more likely to reject their loan applications.
01:00:01.260 --> 01:00:05.335
So it's like versus the prime, which is more of a nudge.
01:00:05.335 --> 01:00:12.335
It gives them some, like, tools to kinda implement some, like, checks for them.
01:00:12.614 --> 01:00:28.599
So the the prime is really just a prompt that encourages you to think, like you could think about saying, hey, with the loan the credit loan dataset, it's it's possible that women are more likely discriminated against than men and more likely to have their loans declined.
01:00:28.599 --> 01:00:35.715
So you could say something like that and like, oh, please keep this in mind when you're training your models.
01:00:35.715 --> 01:00:43.315
And as I'm understanding, it's like people will the prime actually does nudge people in that they kind of they've both, like, self report.
01:00:43.315 --> 01:00:48.474
They're like, oh, I did follow these kinds of, like, principles more.
01:00:49.380 --> 01:00:50.739
Yeah. People would say, oh, yeah.
01:00:50.739 --> 01:01:00.980
I was definitely more motivated, like, being reminded of that or, oh, as a woman, I can resonate that resonates with me, and I would not want people to be discriminated against in that way.
01:01:00.980 --> 01:01:05.215
It definitely motivated people, and we did notice a behavioral shift.
01:01:05.215 --> 01:01:12.135
But I think the important thing is just inducing a change in behavior does not guarantee the outcomes change.
01:01:12.175 --> 01:01:16.375
You have to induce the behavior in a deliberate direction.
01:01:16.894 --> 01:01:17.855
Yeah. Gotcha.
01:01:17.855 --> 01:01:27.110
Like, I guess, like, provide the skill or capability to kinda actually produce a fair model.
01:01:27.789 --> 01:01:28.110
Yeah.
01:01:28.110 --> 01:01:39.275
Giving them the the capabilities to measure the fairness of their model and then also to to fix a model when it's it's not at the level that they want or if they want it to be more fair.
01:01:39.275 --> 01:01:41.434
Yeah. Gotcha. Yeah.
01:01:39.275 --> 01:01:41.434
It's super interesting.
01:01:41.434 --> 01:01:49.389
And as I understand it too, the like, the evaluation of like, there's two parts as I understand it.
01:01:49.389 --> 01:01:52.750
Like, the the responsible data science, you have the framework for it.
01:01:52.750 --> 01:01:59.630
But this paper is really interesting that it's also attempting to evaluate it and provide, like, a behavioral experiment.
01:01:59.630 --> 01:02:11.664
And I think I recall, I think the paper says it's, like, one of the first controlled experiments in kinda, like, evaluating yeah.
01:02:11.664 --> 01:02:14.264
I'd be curious to hear a little bit more about that.
01:02:15.110 --> 01:02:15.590
Yeah.
01:02:15.590 --> 01:02:21.989
So behavior change behavior change for responsible data science is just, in my opinion, understudied.
01:02:21.989 --> 01:02:28.309
There should be way more people than the me and the faculty I mentioned studying this problem.
01:02:28.309 --> 01:02:37.014
And there are some people studying this problem, so I'll acknowledge this is not an empty bucket, but there should be way more people studying it.
01:02:37.014 --> 01:02:50.380
And I think part of the challenge in studying this problem is that we are not necessarily well equipped with the tools that we need to tackle these kinds of behavioral problems, but there are other communities that have studied it for a long time, like psychology.
01:02:50.659 --> 01:03:17.230
Behavior change is a really important aspect of psychology research, And so what we are trying to introduce through these papers are strategies and tools that we could adopt where we maybe we could even collaborate with researchers in psychology, like Yuichi's collaboration with us is an example of that, where we don't have to come up with all the answers as computer scientists.
01:03:17.230 --> 01:03:23.150
And in fact, we shouldn't because a lot of us are not behavioral scientists, a lot of us are not psycho psychology researchers.
01:03:23.150 --> 01:03:28.235
We should be collaborating with other people, and we should be learning from them to develop solutions.
01:03:28.235 --> 01:03:45.449
So all of this work is really trying to build infrastructure intellectually and, you know, mechanically speaking, like, was designing experiments to measure success to see, you know, what could behavior change interventions look like in this context if we were to build more of them.
01:03:45.449 --> 01:03:50.809
And, you know, what are the right things to do, and what are the kind of the pitfalls we should avoid in those designs?
01:03:50.809 --> 01:03:55.610
I saw a lot of interesting measures like cognitive load and things like that.
01:03:55.610 --> 01:04:03.605
So it's cool to see kind of the full array of, like, instrumentation, you know, in this experiment.
01:04:03.605 --> 01:04:04.005
Yeah.
01:04:04.005 --> 01:04:28.215
I'm glad you mentioned cognitive load too because an important thing to acknowledge in that work is that we're we're asking people to literally do more work than they would normally do when you're asking them to do data science work responsibly, which also means introducing a cognitive burden on top of what they already have to deal with as a user of data science tools.
01:04:28.335 --> 01:04:34.534
And the the load measures from that study show that, reflect that.
01:04:34.750 --> 01:04:35.869
It's not shocking. Right?
01:04:35.869 --> 01:04:45.110
It makes sense that if you're asking people to not just worry about model performance, but also whether it's discriminating against people, that's more work that they had to do.
01:04:45.630 --> 01:04:48.550
It was cool that those measures in the paper show that.
01:04:49.235 --> 01:04:51.514
Yeah. It's a really cool approach.
01:04:52.914 --> 01:04:59.474
And then I know the other one I saw that you put out recently was wait.
01:04:59.474 --> 01:05:00.954
What's this?
01:05:01.474 --> 01:05:07.539
A Discursive Framework for Black Decolonial AI.
01:05:08.300 --> 01:05:08.539
Yeah.
01:05:08.539 --> 01:05:12.659
I'm I'm really curious to hear more about that paper as well.
01:05:12.860 --> 01:05:13.260
Yeah.
01:05:13.260 --> 01:05:18.114
So the I I definitely wanna acknowledge the intellectual leader of that work was Teanna Barrett.
01:05:18.114 --> 01:05:23.474
So she's a PhD student here at UW, co advised by me and Amy Zhang.
01:05:23.474 --> 01:05:32.594
That was kinda her her first major project as a PhD student here at UW, and I felt like she did a fantastic job with it.
01:05:32.594 --> 01:05:45.679
And So what's interesting to me about that work is this idea that, so, know, in responsible data science, we care a lot about what it means to do something right and what it means to do something wrong, ethically speaking.
01:05:45.684 --> 01:05:53.485
And a lot of how various communities try to write that down are through frameworks, ethics frameworks.
01:05:54.244 --> 01:06:11.480
What we noticed is that in these ethics frameworks, you have them for a lot of different communities, but it seemed like African countries were being left out of that conversation of what, like, what is right and what is wrong in the context of interacting with, engaging with African peoples.
01:06:11.480 --> 01:06:24.864
And it seemed like we could develop a framework to reflect the values and the ethics of different African countries to fill that gap.
01:06:24.864 --> 01:06:27.780
So that was the idea behind that work.
01:06:27.780 --> 01:06:52.894
And Teanna worked really hard to take into account African like, sub Saharan African philosophy, ethics work, kind of thought leaders in terms of how AI should be incorporated into their cultures, things like that, and kind of putting it all together to review.
01:06:52.894 --> 01:06:53.735
Okay.
01:06:53.929 --> 01:07:03.849
If we were to create one framework to exemplify these principles that should be upheld, what would that look like?
01:07:03.849 --> 01:07:08.369
And what gaps are we filling compared to other ethics frameworks?
01:07:08.434 --> 01:07:12.594
Like, which of these principles might be covered in current frameworks and which are not?
01:07:12.594 --> 01:07:16.355
And I would say anyone who's really excited about it, please reach out to Teanna.
01:07:16.355 --> 01:07:26.820
She loves talking about her work and and that paper, And a lot of people have have been really excited about it, so I I just encourage folks to reach out.
01:07:26.820 --> 01:07:33.340
Do you know what y'all might do next as next steps building on the paper?
01:07:33.375 --> 01:07:34.815
Yeah. That's a great question.
01:07:34.815 --> 01:07:45.135
So one thing that really bothers me about ethics frameworks is that they're so hard to use in a practical sense.
01:07:45.135 --> 01:07:55.440
I could could go out, read a bunch of books, learn all there is to know about these frameworks, but then how do I actually apply it to my data science work?
01:07:55.800 --> 01:08:09.574
And then also a a more recent project that Teanna has been working on is this idea of, well, how exactly do data scientists operationalize their ethics and their values in their own work?
01:08:09.574 --> 01:08:17.869
And the answer is it's really dependent on the environment that they're in and the opportunities that they're afforded to express those values through their work.
01:08:17.869 --> 01:08:34.904
And some some people haven't thought as deeply about it as others, but even for people who think deeply about it, depending on where you work, you might need, like, outside projects to be able to do that, to really closely align your values with the practices in data science that you follow.
01:08:35.024 --> 01:08:37.585
Because your your company might be hostile towards it.
01:08:37.585 --> 01:08:40.145
They might not care.
01:08:40.145 --> 01:08:57.909
Like, you better meet your your key performance metrics, and I'm not gonna give you any extra resources for this other stuff, or it might be that resources get cut back at work, so then the time and the energy and resources you had to do some more of the responsible aspects might be reduced.
01:08:57.909 --> 01:09:02.824
So it's very situational, and there are a lot of factors involved.
01:09:02.824 --> 01:09:17.140
So what we're interested in is, well, for the question of if it's hard to figure out what your ethics are, what values you wanna uphold through your data science work, could frameworks help you and could we make frameworks easier to use in that direction?
01:09:17.659 --> 01:09:32.965
Another question is if there are these barriers, these environmental barriers to helping you, you know, practice data science more responsibly in the way that you want, From a technology perspective, what can we and what can we not do to support that?
01:09:32.965 --> 01:09:34.324
You're saying outside projects.
01:09:34.324 --> 01:09:38.244
They might need outside projects because they just don't have the resources.
01:09:38.244 --> 01:09:38.645
Yeah.
01:09:38.645 --> 01:09:47.079
Or the bandwidth or the company is just not interested or doesn't want them to do that work within the context of the company.
01:09:47.079 --> 01:09:58.335
So then maybe they may have a passion project outside where they're sort of I don't wanna say atoning, but, like, allowing themselves to to focus on where their heart is.
01:09:58.335 --> 01:09:58.654
Yeah.
01:09:58.654 --> 01:10:08.414
It's interesting to think about to the in terms of responsibility that I guess if you're a data scientist within a company, the question of, like, what is my responsibility?
01:10:08.414 --> 01:10:25.010
And if I don't have resources in the company to kinda, like, meet that responsibility, Should I maybe do do I need to put in work to kinda, like, build the skills to to behave responsibly?
01:10:25.635 --> 01:10:38.180
Or or even just how hard should I fight to try to make these resources available so that people can perform their work responsibly in the way that aligns with my values?
01:10:38.180 --> 01:10:43.779
Because the organization has values and then the individual has values, and there's this question of how they intersect.
01:10:43.779 --> 01:10:45.739
Yeah. That's a great point.
01:10:46.260 --> 01:10:53.805
Have you run into much discussion around or thought about responsible data science?
01:10:53.805 --> 01:10:54.204
Yeah.
01:10:54.204 --> 01:11:04.604
I I think we we think a lot about like, don't like, we don't use, like, responsible, but we think about a lot of these similar things.
01:11:04.604 --> 01:11:25.265
And particularly, like, you know, for for me, I think a lot about the, like, the pit of success and, like, how do you kinda guide people into doing, like, the right thing or the bit like, making the right thing or the better thing, like, easier than the hard thing easier than the the bad thing or the the worst thing.
01:11:25.545 --> 01:11:38.920
And I think it's fascinating if, like, we now have a lot more levers to pull in terms of, like, prompting LLMs and giving them tools and having, you know, different agents review your work afterwards with different with different lenses.
01:11:38.920 --> 01:11:53.614
So I I think it's like an it's a exciting time to be thinking about how can we make these frameworks more actionable for for, you know, real data scientists who, of course, wanna do their best but have so many other pressures on their time.
01:11:53.614 --> 01:11:55.335
Yeah. Absolutely.
01:11:55.375 --> 01:12:05.640
I was gonna say one direction I'm really interested in moving in the future is this idea of, yeah, it might not necessarily be right versus wrong in this kind of more nuanced space.
01:12:05.640 --> 01:12:14.199
Something I think we're running into and that we see is that inaction might be the kind of the biggest problem, inaction along different dimensions.
01:12:14.199 --> 01:12:42.994
Because it's pretty clear, right, that if data science data scientists and engineers and so on do what they believe is rational or what they believe is, like, reasonable and are not actively doing actions where they are like, you know, I should be doing additional things to make this as responsible as possible versus, oh, I don't think I'm doing harm, so I think I'm fine, we still end up in all sorts of bad situations.
01:12:42.994 --> 01:12:46.835
So it's not that people aren't rational or people aren't reasonable.
01:12:46.835 --> 01:12:56.795
It's that the rational and reasonable like, rational behavior is not the in the current context, the definition of rationality might not be ethical.
01:12:57.970 --> 01:13:09.890
So if you define rational as responsible, you end up in a different frame, which might lead a totally separate line of decision making that we come very different from what we do right now.
01:13:09.890 --> 01:13:24.734
So something I'm Emily Wall and I are interested in following is this idea of rationality in data science and where it might get us into trouble and how other frames, like other definitions of rationality, might help us kind of break out of that cycle.
01:13:24.734 --> 01:13:31.109
Wanting to make money is rational, for example, but can lead us all sorts of directions.
01:13:31.109 --> 01:13:33.069
Sticky situations?
01:13:33.189 --> 01:13:34.390
Yeah.
01:13:34.390 --> 01:13:35.989
And it's not necessarily bad. Right?
01:13:35.989 --> 01:13:37.590
But sometimes the outcomes are bad.
01:13:37.590 --> 01:13:43.034
So the question is, like, how does that frame that definition of rational get us in trouble?
01:13:43.034 --> 01:13:49.914
Do you have advice for people maybe starting out in data science today or recommendations for them?
01:13:49.914 --> 01:13:52.835
You mean for, like, responsible data science?
01:13:53.354 --> 01:13:55.739
Yeah. For responsible data science.
01:13:55.739 --> 01:14:02.859
What I would what I would say is you don't have to, like, read a bajillion books on it or anything.
01:14:02.859 --> 01:14:06.300
Maybe pick one book, like Data Feminism, and check it out.
01:14:06.300 --> 01:14:18.314
And you don't have to necessarily commit yourself to doing everything in that book, but just kinda I think it's useful to know what's out there, and then you yourself can decide how you wanna incorporate it.
01:14:18.314 --> 01:14:34.470
And then I would say for each tool or feature that you're gonna use to automate your work, see if there is another tool or feature that you can use to evaluate it from a responsible perspective.
01:14:34.989 --> 01:14:40.805
Like, you're gonna train models, see if there's already tools out there that could help you check how fair they are.
01:14:40.805 --> 01:14:48.765
If you're data cleaning and you're gonna do some automation, see if there's anything that can help check for, like, bias.
01:14:48.965 --> 01:14:51.204
It seems like such a helpful rule of thumb.
01:14:51.204 --> 01:15:01.010
Like, for each thing you use, find a thing to evaluate it as, like, such a practical piece of advice to kinda, like, try to counterbalance.
01:15:02.729 --> 01:15:15.324
Like, an an easy thing to kinda, like, nudge a person, I feel like that that strikes me as really helpful for being able to kinda like put this into operation.
01:15:15.685 --> 01:15:15.925
Yeah.
01:15:15.925 --> 01:15:18.244
If I was starting on the very beginning, that's what I would do.
01:15:18.244 --> 01:15:22.390
I would think about, you know, almost like unit testing.
01:15:22.390 --> 01:15:32.949
If I were to do the equivalent of that for any tool I wanna add to my toolbox, you know, what checks and balances can I just automatically put in place as I add to the toolbox?
01:15:32.949 --> 01:15:35.789
Hadley and Wes, any other questions?
01:15:35.944 --> 01:15:45.425
Just this I just I mean, I just really appreciate your, like, thinking about, you know, responsible data science, how do we kinda, like, steer people towards it?
01:15:45.545 --> 01:15:51.430
You know, not about being perfect, but how do we help everyone just get a little bit better?
01:15:51.430 --> 01:15:58.109
I think that's such a the the framing around behavior interventions to just feel really like it.
01:15:58.310 --> 01:15:59.109
Absolutely.
01:15:59.109 --> 01:16:00.109
Agreed.
01:16:00.524 --> 01:16:04.725
If you're fans of, like, The Good Place, I think of it kinda like that.
01:16:05.085 --> 01:16:06.364
The Good Place?
01:16:06.364 --> 01:16:07.685
Yeah.
01:16:08.444 --> 01:16:12.204
There are no perfect choices in life.
01:16:12.204 --> 01:16:14.909
You can only, you know, do better.
01:16:14.909 --> 01:16:18.750
And that's just such, yeah, such an antidote antidote to, like, paralysis.
01:16:18.750 --> 01:16:21.710
Like, you know, you can't you were never gonna be perfect.
01:16:21.710 --> 01:16:28.630
You're never gonna do the the best thing you possibly could, but you can do a little bit better than what you're doing right now, and everyone can do that.
01:16:28.715 --> 01:16:31.515
Exactly. It's so it's relative.
01:16:31.515 --> 01:16:34.755
I I think of responsibility in relative terms.
01:16:34.795 --> 01:16:37.034
There aren't good people and bad people.
01:16:37.034 --> 01:16:39.114
Well, Leilani, thank you so much for coming on.
01:16:39.114 --> 01:16:50.159
Sorry I've been so mean, the thing is I think you have so so many interesting studies that span, like, computation, cognition, and ethics.
01:16:50.159 --> 01:17:15.970
Like, I I think that I got too excited in the sauce, but, honestly, it's been so nice having you on and hearing from your work at the intersection of so many interesting areas and really focusing on or being sure to bring in also the human component into so much of it and and to think about the ethical side of it has been so inspiring to hear about.
01:17:15.970 --> 01:17:22.489
So, yeah, really appreciate you coming on The Test Set, and thanks so much for talking.
01:17:23.045 --> 01:17:25.364
Yeah. Thank you for having me. It's been a blast.
01:17:25.364 --> 01:17:29.524
I've loved talking about all the topics we covered today, especially responsible data science.
01:17:29.524 --> 01:17:32.244
I actually don't get that many opportunities to talk about it.
01:17:32.244 --> 01:17:36.685
It's not that popular of a topic in industry right now.
01:17:36.770 --> 01:17:46.729
What's super exciting, I think, if people it's cool that people both have the framework and can read some of the experiments you've done, you know, evaluating it too.
01:17:47.170 --> 01:17:50.170
Yeah. Awesome. Yeah.
01:17:47.170 --> 01:17:50.170
Thanks so much.
01:17:50.369 --> 01:17:52.090
Yeah. Thanks for being here.
01:17:53.215 --> 01:17:59.774
The Test Set is a production of Posit PBC, an open source and enterprise tooling data science software company.
01:17:59.774 --> 01:18:04.034
This episode was produced in collaboration with creative studio, Adjy.
01:18:04.034 --> 01:18:10.234
For more episodes, visit TheTestSet.co or find us on your favorite podcast platform.