#47 Human-centered Design in Data Science

WEBVTT

1
00:00:00.160 --> 00:00:04.360
In this episode of data framed,
a data camp podcast, I'm speaking with

2
00:00:04.559 --> 00:00:09.990
Peter Bull about the importance of human
centered design in data science. Peter is

3
00:00:10.029 --> 00:00:14.349
a data scientist for social good and
Co founder of driven data, a company

4
00:00:14.429 --> 00:00:19.670
that brings cutting edge practices in data
science and crowdsourcing to some of the world's

5
00:00:19.710 --> 00:00:25.539
biggest social challenges and the organization's taking
them on, including machine learning competitions for

6
00:00:25.660 --> 00:00:30.420
social good. We'll speak about the
practice of considering how humans interact with data

7
00:00:30.620 --> 00:00:35.619
and data products and how important it
is to consider them while designing your data

8
00:00:35.700 --> 00:00:41.810
projects. Will see how human centered
design provides a robust and reproducible framework for

9
00:00:41.929 --> 00:00:46.890
involving the end user all through the
data work, illuminated by examples such as

10
00:00:47.049 --> 00:00:52.880
driven data's work in financial services and
mobile money in Tanzania. Along the way,

11
00:00:52.920 --> 00:00:57.600
will discuss the role of empathy and
data science, the increasingly important conversation

12
00:00:57.679 --> 00:01:03.159
around data ethics and much, much
more. I'm Hugo bound Anderson, a

13
00:01:03.240 --> 00:01:14.670
data signed as the data camp,
and this is data frame. Welcome to

14
00:01:14.709 --> 00:01:18.739
data framed, a weekly data camp
podcast exploring what data science looks like on

15
00:01:18.819 --> 00:01:23.180
the ground for working data scientists and
what problems it consol. I'm your host,

16
00:01:23.340 --> 00:01:26.780
Hugo bound Anderson. You can follow
me on twitter that Hugo bound and

17
00:01:26.859 --> 00:01:30.019
data camp. At data camp you
can find allow episodes and show notes a

18
00:01:30.099 --> 00:01:38.730
data Campcom community podcast. Hi,
that Paida and welcome to data framed.

19
00:01:38.930 --> 00:01:42.409
Thanks you do. I'm happy to
be here. I'm, how about to

20
00:01:42.409 --> 00:01:45.769
have you here and I'm really excited
to be talking about human scenter design and

21
00:01:45.849 --> 00:01:49.200
data science, the role of design
and diatascience and what diatasigns can tell us

22
00:01:49.239 --> 00:01:51.959
about human scent to design as well. But before we get into this,

23
00:01:52.200 --> 00:01:53.280
I want to find out a bit
about you. What are you known for

24
00:01:53.439 --> 00:01:57.719
in the data community? Primarily,
I'm known for my work at driven data.

25
00:01:57.719 --> 00:02:04.829
Driven data is an organization that runs
machine learning competitions that have a social

26
00:02:04.909 --> 00:02:12.469
impact. So we work with nonprofits
and Geo's government groups to help them figure

27
00:02:12.469 --> 00:02:15.340
out a way that machine learning can
help them to be more effective. Then

28
00:02:15.460 --> 00:02:21.060
we put that online so that our
global community of data scientists can come up

29
00:02:21.259 --> 00:02:24.900
with the best solution to this particular
problem and then, after the competition,

30
00:02:25.419 --> 00:02:30.409
we help the organization to use that
coming forward. So that's probably one of

31
00:02:30.490 --> 00:02:34.169
the things. Is just that work
of driven data that we've been doing for

32
00:02:34.250 --> 00:02:38.250
the last five years. But outside
of that, there are two particular areas

33
00:02:38.289 --> 00:02:43.289
of data science that I often talk
about and I'm very interested in, and

34
00:02:43.569 --> 00:02:49.080
the first one is engineering best practices
for data science. Some one of the

35
00:02:49.159 --> 00:02:53.919
many maintainers of the Cookie Cutter Data
Science Project, which is project template that

36
00:02:54.000 --> 00:02:58.000
I hope we get some time to
talk about because it's one of my pet

37
00:02:58.080 --> 00:03:00.590
project and I think it makes a
big difference in our own work and I

38
00:03:00.669 --> 00:03:04.389
help it makes a difference for other
people. And then the other one is

39
00:03:04.469 --> 00:03:08.349
thinking about the social impact the data
science can have and it's relationship to the

40
00:03:08.750 --> 00:03:15.340
larger data ethics conversation that's happening.
So we just released a package called DM

41
00:03:15.740 --> 00:03:22.860
Deon that's for building ethics checklists that
you can use as part of your data

42
00:03:22.900 --> 00:03:25.139
science project. So if we can
chance, I'd love to talk about both

43
00:03:25.180 --> 00:03:30.050
of those as well, because there's
things that I care deeply about. I

44
00:03:30.129 --> 00:03:32.250
would love to jump into all of
these things. So to recap, we

45
00:03:32.409 --> 00:03:38.210
have machine learning competitions with social impact
engineering best practices, which I think is

46
00:03:38.449 --> 00:03:40.330
incredibly important, particularly because, you
know, there is an argument that in

47
00:03:40.370 --> 00:03:45.479
data science there is the idea of
best practices in general is in a woeful

48
00:03:45.520 --> 00:03:47.360
state and something that's trying to be
a lot of people are working on correction

49
00:03:47.719 --> 00:03:53.479
for and bringing engineering best principles into
that will be essential. And then,

50
00:03:53.520 --> 00:03:58.270
of course, the data ethics aspect
of your work. Very recently, I

51
00:03:58.389 --> 00:04:01.550
mean Michael Keaty's Hilary, Mason Dj
Battell have started writing the series on data

52
00:04:01.550 --> 00:04:04.389
ethics for o'riley, where they've actually
got a post, as you know,

53
00:04:04.590 --> 00:04:09.189
on checklists versus odes, versus codes
of conducts. So I think all of

54
00:04:09.229 --> 00:04:12.819
these things are incredibly topical. So
let's just spend a bit of time to

55
00:04:12.860 --> 00:04:15.860
go through each of these. In
terms of your machine learning competitions with social

56
00:04:15.860 --> 00:04:18.939
impact, could you just give us
a couple of examples? Yeah, sure,

57
00:04:19.259 --> 00:04:25.089
so start with one of my favorites
and that is the first competition we

58
00:04:25.170 --> 00:04:31.850
ever ran. So there's a a
nonprofit organization called education resource strategies, and

59
00:04:32.449 --> 00:04:38.569
really what they want to do is
they want to help school districts to spend

60
00:04:38.600 --> 00:04:44.639
their money more effectively. So schools
are spending money on things like student transportation,

61
00:04:45.240 --> 00:04:50.920
teacher salaries, textbooks and they have
a wide range of operational costs and

62
00:04:51.439 --> 00:04:57.509
right now it's very difficult for a
school district to say am I paying a

63
00:04:57.629 --> 00:05:02.269
lot more for textbooks than neighboring districts
and getting the same outcomes? Are My

64
00:05:02.430 --> 00:05:11.620
investments being effective? And the biggest
barrier to doing that kind of benchmarking or

65
00:05:11.699 --> 00:05:19.019
comparison is that school budgets come in
wildly different formats and there's no standard for

66
00:05:19.220 --> 00:05:26.089
reporting the categories of a budget expenditure
so that I can say we're spending more

67
00:05:26.129 --> 00:05:29.970
on textbooks than a neighboring school and
we need to look at this. So

68
00:05:30.689 --> 00:05:35.920
education resource strategies gathers all of this
budget information from the districts they work with

69
00:05:36.639 --> 00:05:44.920
and ultimately what their output is a
recommendation for how to think about the school

70
00:05:44.959 --> 00:05:50.029
districts budgeting after they go through this
process of benchmarking that district against other districts.

71
00:05:50.589 --> 00:05:58.069
So the big problem is that they
spend hundreds of person hours a year

72
00:05:58.790 --> 00:06:02.350
looking at excel spreadsheets, reading a
budget line item and trying to assign it

73
00:06:02.389 --> 00:06:08.300
a standard set of categories. So, as I'm sure your audience will have

74
00:06:08.420 --> 00:06:13.220
picked up on on that description,
they have a lot of labeled data that

75
00:06:13.300 --> 00:06:18.889
they've generated through the course of their
operations and that labeled data is a budget

76
00:06:18.930 --> 00:06:24.129
line out of the description of it, the cost of that budget item and

77
00:06:24.370 --> 00:06:30.810
then what category belongs to, whether
that is transportation, textbooks, extracurricular activities,

78
00:06:30.850 --> 00:06:38.160
administrative salaries, all of those things
they've captured over their history of working

79
00:06:38.279 --> 00:06:43.519
with school districts. So our first
competition was how do we help this organization

80
00:06:43.680 --> 00:06:48.430
that really cares about the output report
and not about this taxonomy process? How

81
00:06:48.470 --> 00:06:54.189
do we help them to automate that? So we ran a competition where people

82
00:06:54.949 --> 00:06:59.509
build an algorithms to Parse the natural
language in these budgets, to look at

83
00:06:59.550 --> 00:07:05.060
these budget costs and to automatically assign
categories for those school budgets to those line

84
00:07:05.100 --> 00:07:11.300
items, and we took the output
of that competition we turned it into a

85
00:07:11.420 --> 00:07:16.610
tool that fits into the workflow that
this organization already had. So it's saving

86
00:07:16.649 --> 00:07:23.730
their analysts tons of time in terms
of just reading through sal spreadsheets so that

87
00:07:23.850 --> 00:07:27.730
they can focus that time where they
can really add value, which is about

88
00:07:27.769 --> 00:07:31.279
making recommendations for how those pledgets can
be changed. That's great and it seems

89
00:07:31.279 --> 00:07:35.759
like it would reduce so much of
the manual labor involved. Yeah, really

90
00:07:35.839 --> 00:07:40.160
it's the last time we checked in, it was saving them about three hundred

91
00:07:40.240 --> 00:07:45.709
percent hours a year to automate that
process and for a relatively small nonprofit organization,

92
00:07:46.189 --> 00:07:50.509
that's actually a huge amount of labor
savings and really their goal is to

93
00:07:51.230 --> 00:07:58.110
employ those savings for effectively where their
employees actually add value rather than in the

94
00:07:58.149 --> 00:08:03.139
label lagua spreadsheets where it's just this
task that had to happen anyway. Yeah,

95
00:08:03.180 --> 00:08:05.500
absolutely, and we should mention that
if people find this type of stuff

96
00:08:05.540 --> 00:08:09.220
pretty exciting and interesting, they can
check out a lot of the competitions on

97
00:08:09.379 --> 00:08:11.819
at driven data, but if they
find this particular competition interesting, they can

98
00:08:11.930 --> 00:08:16.490
even take the data camp course that
you've built and that I that I collaborated

99
00:08:16.610 --> 00:08:20.889
on, which is learning from the
experts and you get to build the winning

100
00:08:20.930 --> 00:08:24.529
solution in the end. Yeah,
that's right. So that course will walk

101
00:08:24.689 --> 00:08:28.879
through not only what a baseline solution
to a problem like this is, but

102
00:08:28.079 --> 00:08:33.960
also how the person who won the
competition combined a number of interesting techniques to

103
00:08:35.120 --> 00:08:39.240
get to that best performing solution.
And I'm not going to spoil the punch

104
00:08:39.320 --> 00:08:41.950
line. I don't want you to
either, but I will say that it's

105
00:08:41.990 --> 00:08:48.429
not an lstm or any crazy deep
learning architecture that wins the competition. So

106
00:08:48.549 --> 00:08:50.389
now we've talked about the types of
machine learning competitions. You do it driven

107
00:08:50.389 --> 00:08:54.230
data. Maybe you can tell us
a bit about your thoughts on a engineering

108
00:08:54.269 --> 00:08:56.899
best practices for data. So it's
in in particular you'll cookie caught of Dita

109
00:08:56.899 --> 00:09:03.179
science project that you maintain great.
Yeah, so my background is in software

110
00:09:03.179 --> 00:09:07.779
engineering and so one of the things
that I think about while I'm working on

111
00:09:07.899 --> 00:09:13.889
data science projects is how software and
data science go together. I think there

112
00:09:13.929 --> 00:09:20.409
are some important differences between the processes
in that data science tends to be more

113
00:09:20.769 --> 00:09:24.919
open ended, tends to be more
of a research and development effort, but

114
00:09:24.039 --> 00:09:28.840
it's still the fact that a lot
of what we do in data science is,

115
00:09:30.159 --> 00:09:35.159
at its core, building software.
Even if that software exists in a

116
00:09:35.159 --> 00:09:39.669
Jupiter notebook or at our markdown file, it's still a piece of software and

117
00:09:39.789 --> 00:09:43.110
a lot of the best practices that
come out of software engineering can be employed

118
00:09:43.269 --> 00:09:50.990
to improve those products. And so
the cookie cutter data science project is what

119
00:09:50.190 --> 00:09:58.539
we think of as the first pass
at standardizing our process to make ourselves more

120
00:09:58.580 --> 00:10:03.259
effective, and that's to have a
consistent layout of what a data science project

121
00:10:03.379 --> 00:10:07.129
looks like so if you were to
look at a web application in Django,

122
00:10:07.210 --> 00:10:11.649
which is the Python web framework,
or in Ruby on rails or in not

123
00:10:11.850 --> 00:10:18.210
dot JS, all of these are
different programming languages, but anytime you build

124
00:10:18.210 --> 00:10:22.159
a web application in one of those
frameworks you have more or less the same

125
00:10:22.320 --> 00:10:28.200
structure. What that means is is
that anyone who's a web developer can go

126
00:10:28.360 --> 00:10:33.879
into a project like that and have
some expectation about where they would find certain

127
00:10:33.960 --> 00:10:37.870
kinds of code. The code that
talks to the database usually lives in one

128
00:10:37.950 --> 00:10:41.830
place, the code that talks to
the front end usually lives in another place,

129
00:10:41.429 --> 00:10:48.549
and those expectations make it very easy
to work together and collaborate on projects.

130
00:10:48.990 --> 00:10:52.259
And so the cookie cutter data science
project idea is to bring that kind

131
00:10:52.259 --> 00:10:58.500
of standardization to our own data science
work. So we have a defined folder

132
00:10:58.580 --> 00:11:03.580
structure where our data looms and we
have a set of folders for raw data,

133
00:11:03.019 --> 00:11:07.009
data that has some processing but is
in an interim state, and then

134
00:11:07.129 --> 00:11:13.610
processed data. We have a particular
folder structure for where we keep our Jupiter

135
00:11:13.730 --> 00:11:18.169
notebooks or are marked out files that
are built in the literate programming style,

136
00:11:18.570 --> 00:11:24.720
and we have another set of folder
structures for data processing pipelines that may exist

137
00:11:24.799 --> 00:11:28.840
as scripts and then ultimately make get
refactored in something like python package. And

138
00:11:30.000 --> 00:11:33.870
because of this consistency, it's very
easy for us to move from project to

139
00:11:33.990 --> 00:11:39.509
project and pick up something and remember
sort of where we are how to reproduce

140
00:11:39.669 --> 00:11:43.509
something. And because we work with
lots of different clients, are lots of

141
00:11:43.549 --> 00:11:48.980
different projects, this means that anyone
who works on the team can jump into

142
00:11:48.980 --> 00:11:54.659
any project without having to spend a
lot of time figuring out what happens where.

143
00:11:54.100 --> 00:11:56.299
That's great and I find all the
details very interesting. But, as

144
00:11:56.340 --> 00:12:01.139
you have had at home, the
idea of actually having an overarching, overall,

145
00:12:01.179 --> 00:12:05.730
consistent layout to data science projects and
a system of best practices, I

146
00:12:05.850 --> 00:12:09.970
think is incredibly important. I presume
this actually plays a role in your approach

147
00:12:09.090 --> 00:12:15.009
to building the package for data ethics
and ethical checklist, of having something that

148
00:12:15.090 --> 00:12:20.759
you can carry across projects and if
there are biases involved or challenges their systematic

149
00:12:20.840 --> 00:12:24.200
in the sense that they won't be
induced by a human working on the project,

150
00:12:24.320 --> 00:12:26.759
they'll be in this structure as a
whole. So you'll be able to

151
00:12:26.000 --> 00:12:28.240
you know that, will become aware, you'll become a parent of them.

152
00:12:28.480 --> 00:12:35.149
Yeah, so I think the to
a related in that we really spend a

153
00:12:35.230 --> 00:12:39.230
lot of our time building tools for
people who are data scientists. A lot

154
00:12:39.269 --> 00:12:43.419
of times they start out as tools
that we're using ourselves and then we open

155
00:12:43.539 --> 00:12:48.700
source those tools so that other people
who are working in data science can use

156
00:12:48.779 --> 00:12:52.460
them. So that's how the cookie
cutter data science projects started and that's really

157
00:12:52.580 --> 00:12:58.009
how this ethics checklist package dion started
as well. So the idea there was

158
00:12:58.409 --> 00:13:05.289
there are a lot of conversations around
data ethics that we found very compelling from

159
00:13:05.529 --> 00:13:11.570
a standpoint of really seeing where things
had gone wrong in the process and feeling

160
00:13:11.690 --> 00:13:16.559
like ourselves we were vulnerable to some
of these things that could go wrong.

161
00:13:16.840 --> 00:13:22.639
It's very, very hard to have
perfect foresight and to understand exactly what can

162
00:13:22.720 --> 00:13:28.350
go wrong in any given circumstance and
because we kept seeing these examples and feeling

163
00:13:28.350 --> 00:13:31.870
like would we necessarily have caught this
in our own kind of work, we

164
00:13:31.990 --> 00:13:37.470
wanted to have a more actionable way
of engaging with that data ethics conversation to

165
00:13:39.070 --> 00:13:43.620
make sure that we didn't fall into
some of these traps that just exist in

166
00:13:43.700 --> 00:13:46.580
the work that, if you're focused
on methods. If you're thinking about data

167
00:13:46.860 --> 00:13:52.299
you can get into these really technical
aspects and not have a chance to pull

168
00:13:52.299 --> 00:13:56.169
up and look at the ethical implications, and so we wanted to have a

169
00:13:56.889 --> 00:14:03.570
really process driven way of engaging that
conversation for our own projects. So what

170
00:14:03.809 --> 00:14:09.480
this package we built does is it
generates a data ethics checklist that's really framed

171
00:14:09.480 --> 00:14:16.440
around the data science process. So
it starts with the data collection phase and

172
00:14:16.879 --> 00:14:22.080
there are a set of checklist items
that ask you questions like do the participants

173
00:14:22.360 --> 00:14:28.950
in your data collection give informed consent
for the data that is collected? That's

174
00:14:28.029 --> 00:14:31.350
one example of an item on the
checklist. And what we've done is we've

175
00:14:31.389 --> 00:14:37.389
taken each of these checklist items at
different parts in the data science process and

176
00:14:37.509 --> 00:14:43.059
we've mapped them to real world examples
where something has gone wrong. So we've

177
00:14:43.100 --> 00:14:50.580
got this collection of news articles and
academic papers that explain when data science projects

178
00:14:50.620 --> 00:14:58.009
have hit and ethical implications and the
problems that have arisen. And there was

179
00:14:58.049 --> 00:15:07.129
actually just a really great article about
Amazon really trying to build a resume filtering

180
00:15:07.360 --> 00:15:13.440
algorithm and they get an unbelievable volume
of resumes for any position that they open

181
00:15:13.480 --> 00:15:18.120
up and they had the belief that
they could use all of their historical data

182
00:15:18.159 --> 00:15:24.470
to train an algorithm to identify the
top candidates that were applying for a good

183
00:15:24.669 --> 00:15:28.429
position. Now, as a sort
of framing, that may seem, from

184
00:15:28.429 --> 00:15:31.990
a data science perspective, like it
makes a lot of sense, and that

185
00:15:31.190 --> 00:15:35.269
we've got a long set of trading
data and we want to be able to

186
00:15:35.350 --> 00:15:39.220
replicate this with an algorithm. But
they just actually shut down the team that

187
00:15:39.340 --> 00:15:45.539
have been working on this project for
years because they found that the algorithm was

188
00:15:45.700 --> 00:15:50.889
biased against women and in particular,
it wasn't saying, Oh, is this

189
00:15:52.090 --> 00:15:56.210
person a woman and they've indicated that
on their application, now I want to

190
00:15:56.289 --> 00:16:00.090
use that and discount their application,
but it was using things that were a

191
00:16:00.210 --> 00:16:06.679
little more implicit than that. So, in particular, if the applicant had

192
00:16:06.759 --> 00:16:14.879
attended a women's College, then their
score went down. And they discovered these

193
00:16:14.960 --> 00:16:19.590
problems with their algorithm and disbanded the
team that was working on this project entirely

194
00:16:21.190 --> 00:16:25.629
because they couldn't get it over this
this pump in the process, and so

195
00:16:25.750 --> 00:16:29.669
they're still having humans review all of
these resumes that come in, and so

196
00:16:29.789 --> 00:16:33.220
this is just sort of a classic
example of something that seems like a great

197
00:16:33.220 --> 00:16:37.620
set up for a machine learning problem
in the abstract, but if you don't

198
00:16:37.659 --> 00:16:42.259
think about how it's affecting your outputs
that aren't just some measure of accuracy,

199
00:16:42.779 --> 00:16:47.450
it can go really, really wrong, and these are the types of things

200
00:16:47.490 --> 00:16:49.649
that they on would ask you to
check for. That's exactly right, YEP,

201
00:16:49.850 --> 00:16:55.450
and so it goes through the data
science process, from data collection to

202
00:16:55.769 --> 00:17:00.759
data cleaning to exploratory data analysis to
actually building models, and that's where this

203
00:17:00.919 --> 00:17:04.960
would come in. And then actually
it's got a section on deploying model.

204
00:17:06.039 --> 00:17:08.279
So when the models deployed, what
are the questions that you should ask?

205
00:17:08.559 --> 00:17:14.960
And really the goal is not to
have all of the answers about what's right

206
00:17:15.279 --> 00:17:18.829
and what's wrong the checklist. Given
all of the different domains people work in,

207
00:17:19.309 --> 00:17:23.109
that's really an impossible task. But
the goal is to take people who

208
00:17:23.109 --> 00:17:30.099
are already conscientious actors that want to
be doing the right thing and make sure

209
00:17:30.140 --> 00:17:33.940
they're having the conversations about the way
that things can be misused. So really

210
00:17:33.980 --> 00:17:38.660
the workflow we see for the tool
is you generate a checklist using the tool

211
00:17:40.180 --> 00:17:41.900
and then you submit a PR to
your project that says, Hey, here's

212
00:17:41.940 --> 00:17:45.890
our ethics checklist. Let's make sure
we talk about each of these items as

213
00:17:45.970 --> 00:17:51.250
a team, and it's really about
spurring that team discussion to make sure you've

214
00:17:51.329 --> 00:17:56.650
considered the particular implications of your project
and you've made the right decision about and

215
00:17:56.849 --> 00:18:00.559
so will include links to the cookie
Cutt of project and the Dayon package in

216
00:18:00.599 --> 00:18:03.119
the show notes. I think this, you know, you discussing data ethics

217
00:18:03.160 --> 00:18:07.319
in terms of thinking about the variety
of stake holders really dovetails nicely into our

218
00:18:07.359 --> 00:18:14.430
conversation about human scenter design in data
science and why it's important as a predude

219
00:18:14.509 --> 00:18:18.309
to humans scenter design, though,
I just like to ask you the quick

220
00:18:18.309 --> 00:18:21.869
question about the role of empathy in
data science as a whole. So I'm

221
00:18:21.869 --> 00:18:25.109
wondering what is the role of empathy
and data science for you? That's a

222
00:18:25.190 --> 00:18:30.700
great question. So I actually feel
like empathy is a term that has started

223
00:18:30.779 --> 00:18:37.140
to pop up in the data science
conversation as a core skill of a data

224
00:18:37.259 --> 00:18:45.089
scientist, and in my mind,
empathy is just one way to get at

225
00:18:45.170 --> 00:18:52.529
a particular kind of approach, and
that approach is to be problem focused rather

226
00:18:52.569 --> 00:18:59.000
than method focused, and what I
meant by that is we should start as

227
00:18:59.079 --> 00:19:04.000
data scientists that are really in a
professional service role. We're providing a service

228
00:19:04.160 --> 00:19:08.240
two different parts of the business or
different parts of an organization. We should

229
00:19:08.319 --> 00:19:14.789
start with what the problem is that
we're solving and understand the context for that

230
00:19:14.950 --> 00:19:18.670
problem, rather than saying, Hey, who's got an NLP problem that I

231
00:19:18.710 --> 00:19:23.029
can solve, a a an Lystam
or who's got a computer vision problem where

232
00:19:23.190 --> 00:19:27.740
I can try out the latest neural
network methods and get a really cool result?

233
00:19:29.059 --> 00:19:32.819
Because if you start methods first,
a lot of times you end up

234
00:19:32.819 --> 00:19:37.019
with a solution that's not going to
be really useful in the context in which

235
00:19:37.019 --> 00:19:41.450
it operates. So when we talk
about data science and empathy, what we're

236
00:19:41.490 --> 00:19:47.690
really saying is that you should empathize
with how your data science output will be

237
00:19:47.890 --> 00:19:52.490
used. You should empathize with what's
the problem we're solving. So when we

238
00:19:52.569 --> 00:19:56.079
talk about empathy, I think that's
one way of getting to a perspective that's

239
00:19:56.279 --> 00:20:03.640
problem first rather than method first.
And any concrete examples spring to mind?

240
00:20:03.839 --> 00:20:07.670
Yes, I think that. For
me, a good example is we worked

241
00:20:07.910 --> 00:20:17.990
on a project that is trying to
automatically identify species in videos species of animals

242
00:20:18.029 --> 00:20:22.539
that is. So there's a research
organization that has these motion detecting cameras that

243
00:20:22.619 --> 00:20:26.099
they set up in the jungle and
they try to record videos of chimpanzees,

244
00:20:26.460 --> 00:20:30.500
but they get a lot of videos
of other animals as well, and instead

245
00:20:30.500 --> 00:20:33.259
of sitting there, are watching all
of these videos and saying which one has

246
00:20:33.299 --> 00:20:37.650
a chimpancity which one doesn't. We're
helping them to build an algorithm to automatically

247
00:20:37.690 --> 00:20:42.930
identify the animals that appear. We
actually ran a competition around this last year.

248
00:20:44.210 --> 00:20:47.009
If you're curious, you can look
at driven data dot org and see

249
00:20:47.009 --> 00:20:51.000
the results of that competition. And
what was the name of the competition?

250
00:20:51.160 --> 00:20:57.880
Competition was primatrix factorization stomach. Thanks
for asking, Gud no, that's right.

251
00:20:59.000 --> 00:21:00.839
You really care about our data.
Sides Puns, and that's one of

252
00:21:00.880 --> 00:21:06.230
my absolute favorites. We're talking about
the dietosigns puns. We have to mention

253
00:21:06.390 --> 00:21:08.309
naive bays as well. Yeah,
and to be honest, you go with

254
00:21:08.390 --> 00:21:14.029
your acts and I wasn't even sure
if he were saying bees for bases.

255
00:21:14.509 --> 00:21:18.579
So it works even better than Australia, and a nive base, of course,

256
00:21:18.859 --> 00:21:22.259
is currently being toned into a series
of Diata camp projects as well.

257
00:21:22.500 --> 00:21:26.700
That's right. Yes, so the
naive bees competition was to build an algorithm

258
00:21:26.779 --> 00:21:32.700
to identify honey bees from bumble bees
and we're working on a set of data

259
00:21:32.740 --> 00:21:36.970
camp projects to help people work through
that problem, to give them that first

260
00:21:37.009 --> 00:21:42.650
exposure to a computer vision task that
fits into a classification framework and look at

261
00:21:42.930 --> 00:21:48.799
the more traditional methods and then move
on to deep learning, neural networks,

262
00:21:48.960 --> 00:21:52.720
in convolutional neural networks. Okay,
so, after that Pun testic interlude,

263
00:21:52.759 --> 00:21:56.319
back to empathy, the role of
empathy in identifying primacts to the role of

264
00:21:56.359 --> 00:22:03.150
empathy, and really this is about
going back to the context and understanding the

265
00:22:03.230 --> 00:22:07.470
context in which you operate. So
we working with this team of ecologists and

266
00:22:07.589 --> 00:22:11.390
Biologists. They spend a lot of
time in the field setting up these cameras,

267
00:22:11.430 --> 00:22:15.029
capturing data, watching videos and then
writing papers about what they see in

268
00:22:15.109 --> 00:22:19.420
the videos. And the output that
we ended up working on after the competition

269
00:22:21.220 --> 00:22:26.900
was an open source package that let
you run from the command line predict,

270
00:22:26.460 --> 00:22:32.329
here are my new videos, and
it would output a CSV with each of

271
00:22:32.410 --> 00:22:37.210
the videos and a probability that a
certain kind of animal appeared in the videos.

272
00:22:37.609 --> 00:22:41.410
And we were pretty pleased with this
output. It'd be super useful for

273
00:22:41.490 --> 00:22:45.200
us, and the first thing we
heard from the team we were working with

274
00:22:45.720 --> 00:22:49.680
is we can't even get this tool
installed. I can't get extra boost to

275
00:22:49.759 --> 00:22:53.519
install on my sheep my machine.
I'm having trouble getting the version of tens

276
00:22:53.519 --> 00:22:59.869
or flow installed, I'm having trouble
getting GPU drivers installed, and so all

277
00:22:59.910 --> 00:23:03.430
of this stuff that fields like second
nature to us as data scientists sort of

278
00:23:03.549 --> 00:23:07.789
blinded us to the context in which
this tool was actually going to be used.

279
00:23:08.150 --> 00:23:15.099
And it's by ecologists that aren't used
to all of this complex machinery around

280
00:23:15.779 --> 00:23:19.220
the packaging of data science tools that
can make it really challenging to use the

281
00:23:19.259 --> 00:23:23.740
latest methods. And so that's just
a really concrete example of a place where

282
00:23:25.339 --> 00:23:29.329
we weren't doing the right thing up
front to really understand that context and make

283
00:23:29.369 --> 00:23:32.970
sure we built something useful. We
were building something that we knew would be

284
00:23:32.970 --> 00:23:37.490
useful for us. A kind of
like clear that I'm sure a non trivial

285
00:23:37.609 --> 00:23:41.160
proportion of like working expert data scientists
have a lot of problems getting exgt boost

286
00:23:41.200 --> 00:23:45.960
installed occasionally as well. Yeah,
if someone wants to take on the initiative

287
00:23:47.359 --> 00:23:52.920
to improve the extra boost installation experience. That's a really invaluable project that someone

288
00:23:52.960 --> 00:24:00.789
could do for the open source community. Will jump right back into our interview

289
00:24:00.829 --> 00:24:06.309
with Peter Bull after a short segment, we're back with our favorite storyteller.

290
00:24:06.630 --> 00:24:10.740
Justin. Boyce has been telling us
about probability distributions and their stories. Hey,

291
00:24:10.859 --> 00:24:12.740
Justin, hey, here go.
Not Really. I'd ask you to

292
00:24:12.779 --> 00:24:15.420
tell us another story, but I
have some questions for you, so I'm

293
00:24:15.460 --> 00:24:19.779
just going to ask great ask the
way last time. You're talking about the

294
00:24:19.859 --> 00:24:25.329
exponential distribution. It's the first continuous
distribution you've talked about and you gave the

295
00:24:25.450 --> 00:24:29.930
time between the rare events of no
hitters in Major League Baseball, which can

296
00:24:29.970 --> 00:24:33.450
be described as a poss on process, as an example of an exponentially distributed

297
00:24:33.490 --> 00:24:37.599
random variable. As a reminder,
a No hitter is when a team allows

298
00:24:37.680 --> 00:24:41.160
no hits by the opponent in a
baseball game. But then you told us

299
00:24:41.319 --> 00:24:45.440
the time between no hitters in units
of games played. If I remember correctly,

300
00:24:45.839 --> 00:24:49.960
you told us the longest time between
no hitters was six thousand some odd

301
00:24:51.000 --> 00:24:56.549
games. But isn't the number of
games discreete and the exponential distribution describes a

302
00:24:56.630 --> 00:25:00.190
continuous time. Very good observation here, though I could almost hear the air

303
00:25:00.230 --> 00:25:04.990
quotes when you said time between Games. That's right. The time between Games

304
00:25:06.029 --> 00:25:11.140
is continuous, but number of games
is discrete. So strictly speaking, we

305
00:25:11.220 --> 00:25:15.380
should use a discreet distribution to describe
the number of games between no hitters.

306
00:25:15.660 --> 00:25:19.460
So we can't use the exponential distribution
then? No, we can, and

307
00:25:19.700 --> 00:25:23.769
as I explain why, I'm going
to introduce another distribution with a new story.

308
00:25:25.809 --> 00:25:30.369
Here goes. The number of successive
failures of a Bernoulli trial before success

309
00:25:30.970 --> 00:25:36.680
is geometrically distributed. So you can
think of a Bernoulli trial as a game.

310
00:25:37.279 --> 00:25:41.200
A failure is not getting a no
hitter and getting a no hitter as

311
00:25:41.240 --> 00:25:45.960
a success. So the number of
games between no hitters is geometrically distributed.

312
00:25:47.680 --> 00:25:52.789
So not exponential but related to the
exponential. Right, strictly speaking, not

313
00:25:52.910 --> 00:25:57.069
exponential, but the geometric distribution is
the discrete analog of the exponential distribution.

314
00:25:57.710 --> 00:26:02.589
It has the same shape as the
exponential. It's just defined at discrete values.

315
00:26:03.349 --> 00:26:07.579
So we can approximately use an exponential
distribution to describe the number of games

316
00:26:07.619 --> 00:26:11.740
between no hitters. It's called a
continuous approximation, and in this case it

317
00:26:11.900 --> 00:26:15.019
is a very, very good approximation. Ah, so now we have connected

318
00:26:15.059 --> 00:26:19.450
another couple of distributions, the geometry, which is discreet and it's analog the

319
00:26:19.529 --> 00:26:26.009
exponential, which is continuous exactly.
If you think about it, any digital

320
00:26:26.049 --> 00:26:30.529
measurement of time results in discrete data. For example, if we have timing

321
00:26:30.569 --> 00:26:34.200
data from a marathon and units of
seconds, we strictly speaking have discrete data,

322
00:26:34.839 --> 00:26:38.720
but the error and treating the timings
as continuous is not large. So

323
00:26:38.839 --> 00:26:45.559
how about giving us another distribution connected
to the exponential? Sure here's a new

324
00:26:45.640 --> 00:26:48.470
story. The amount of time I
have to wait for the arrival of to

325
00:26:48.710 --> 00:26:53.829
pass on processes is gamma distributed,
or, more generally, the amount of

326
00:26:53.869 --> 00:26:59.589
time I have to wait for n
pass on processes to arrive is gamma distributed.

327
00:26:59.789 --> 00:27:03.099
So the exponential is a special case
of the gamma distribution, where in

328
00:27:03.420 --> 00:27:11.380
equals one exactly. or You could
say that the gamma distribution generalizes the exponential.

329
00:27:11.220 --> 00:27:15.210
And the gamma distribution gets more interesting
as end grows. It has a

330
00:27:15.329 --> 00:27:19.930
peak, unlike the exponential distribution,
which just the case away. I should

331
00:27:19.970 --> 00:27:23.730
note also that n need not be
integer. And so where do we see

332
00:27:23.809 --> 00:27:30.160
gamma distributions like all distributions, they
pop up all over. Now gamma distributions

333
00:27:30.240 --> 00:27:33.799
come up a lot in my field
of biology. A common example is the

334
00:27:33.880 --> 00:27:38.519
time between spiking or firing of a
neuron. After its spikes, there's a

335
00:27:38.559 --> 00:27:42.599
refractory period and a few processes have
to happen to have it fire again.

336
00:27:44.240 --> 00:27:48.269
Now they're complicated objects, so not
all neurons have interspike intervals that are gamma

337
00:27:48.309 --> 00:27:52.269
distributed, but many of them do. Thanks Justin your stories have really been

338
00:27:52.390 --> 00:27:56.230
making my neurons file and thanks for
that intro into the geometric and gamma distributions

339
00:27:56.269 --> 00:28:00.339
in this stories. What distribution the
story you're going to tell us about next

340
00:28:00.339 --> 00:28:07.500
time? You know, I think
we're overdue for the granddaddy. Next time

341
00:28:07.859 --> 00:28:11.420
we'll talk about the normal or Gaussian
distribution. I can't white. Let's have

342
00:28:11.500 --> 00:28:21.569
story time again soon. After that
interlude, it's time to jump back into

343
00:28:21.609 --> 00:28:25.410
our chat with Peter, but first
I wanted to let you know that driven

344
00:28:25.450 --> 00:28:29.599
data is currently hiring. If you're
enjoying this interview, you might be interested

345
00:28:29.640 --> 00:28:33.519
in working with driven data. Currently
the team is looking for a software engineer

346
00:28:33.680 --> 00:28:37.880
who loves the idea of building python
applications for social impact and we'll put a

347
00:28:37.920 --> 00:28:41.509
link in the show notes. We're
here at data camp. Have collaborated with

348
00:28:41.589 --> 00:28:45.309
driven data and Peter on several insues
and they've always been a joy to work

349
00:28:45.430 --> 00:28:52.269
with. So let's jump into human
centered design and why it's important data science,

350
00:28:52.309 --> 00:28:56.259
because I think probably a lot of
our listeners wouldn't necessarily think of design

351
00:28:56.380 --> 00:29:02.420
principles as being something which would play
a huge role in the data science process.

352
00:29:02.420 --> 00:29:04.460
So maybe you can tell us about
human center design and why it's important

353
00:29:04.460 --> 00:29:11.410
in data science. Great. Yeah, so human centered design is a way

354
00:29:11.490 --> 00:29:17.410
of framing the design process and it's
really related to other terms that are in

355
00:29:17.529 --> 00:29:21.609
this field that you may have heard
of, like design thinking, the double

356
00:29:21.769 --> 00:29:26.039
diamond method and design sprints. Those
are all sort of popular terms that people

357
00:29:26.119 --> 00:29:30.640
may talk about and it's really referring
to the same set of ideas, which

358
00:29:30.640 --> 00:29:37.400
is about having a design process,
and human centered design in particular is the

359
00:29:37.519 --> 00:29:44.470
one that we are most familiar with
through our work with an organization called Ideo,

360
00:29:44.630 --> 00:29:48.430
cat or work. So Ideo is
one of the leading human centered design

361
00:29:48.589 --> 00:29:53.259
firms. They helped design the first
apple mouse. I have a long history

362
00:29:53.500 --> 00:30:00.420
of being designers both from an industrial
design perspective but also from a digital design

363
00:30:00.420 --> 00:30:04.740
and then eventually service design perspective.
And so they have a really long history

364
00:30:04.980 --> 00:30:12.930
and track record of working and using
these design tools to spur innovation and they

365
00:30:14.009 --> 00:30:18.809
spun out a nonprofit, Arm Ideo
Dot Org, that works with NGO's to

366
00:30:19.009 --> 00:30:26.640
have the same sort of results.
So we partnered with that organization to look

367
00:30:26.640 --> 00:30:33.279
at digital financial services in Tanzania and
just to take a step back, that's

368
00:30:33.279 --> 00:30:37.269
sort of the context that we're working
in, is with this team of human

369
00:30:37.349 --> 00:30:41.630
centered designers and what that human centered
design process looks like, and I'll give

370
00:30:41.670 --> 00:30:45.589
you the over of your first and
after that we can dig into the details

371
00:30:45.630 --> 00:30:49.940
of that particular project, which I
think are pretty enlightening for how data science

372
00:30:49.980 --> 00:30:56.660
and design work together. So the
big picture is that human center design is

373
00:30:56.740 --> 00:31:03.420
about starting with what's desirable. So
there's a perspective that the best solutions to

374
00:31:03.500 --> 00:31:08.769
a problem are desirable and that someone
wants to use them, they're feasible from

375
00:31:08.769 --> 00:31:15.569
a technological perspective and their viable from
a business perspective, and the best solutions

376
00:31:15.769 --> 00:31:21.200
sit at the intersection of that Venn
Diagram. You know you're not having a

377
00:31:21.279 --> 00:31:25.359
good data science conversation until you're talking
about a VN diagram. Right, I

378
00:31:25.519 --> 00:31:30.759
was waiting for the VENDA. So
this particular ven diagram is desirable, feasible

379
00:31:32.119 --> 00:31:37.549
viable. We want the intersection of
all three of those things and of course

380
00:31:37.589 --> 00:31:41.829
a lot of ditosions work. Maybe
we'll start with fasible, like the newest

381
00:31:41.829 --> 00:31:45.630
cutting edge technologies in the the COOLEST, most efficient algorithms of that type of

382
00:31:45.670 --> 00:31:48.700
stuff. Right. That's exactly right, and I think that that is one

383
00:31:48.740 --> 00:31:53.339
of our tendencies as data scientists that
I see in myself all the time,

384
00:31:53.619 --> 00:31:57.500
where I'm getting excited about lots of
these new technologies and I want to find

385
00:31:57.539 --> 00:32:01.809
ways to use them. And so
the trick is just to find the balance

386
00:32:01.970 --> 00:32:07.609
for really solving a problem where using
that is appropriate. And the human center

387
00:32:07.730 --> 00:32:14.049
design process, serve starts from this
perspective of what's desirable to a user and

388
00:32:14.170 --> 00:32:17.960
it gets there by moving through these
three phases of the design process. The

389
00:32:19.119 --> 00:32:29.960
first is inspiration, and inspiration is
about going and observing the people who will

390
00:32:30.000 --> 00:32:36.029
be users of your end product.
So in the case where you're a data

391
00:32:36.109 --> 00:32:40.109
scientist, let's say that your job
is to create this report. That's emailed

392
00:32:40.150 --> 00:32:45.500
out to your executive team on some
so what you would actually do is you

393
00:32:45.539 --> 00:32:49.859
would go and talk to the people
who get that email and you would say,

394
00:32:49.859 --> 00:32:52.059
Hey, when you get this email, what do you use it for?

395
00:32:52.539 --> 00:32:54.460
What is it going to do you? Do you say, okay,

396
00:32:54.539 --> 00:32:58.650
I need some top pine numbers from
here that I put into slides, or

397
00:32:58.890 --> 00:33:00.970
is it something you read to get
context that? Then you say to the

398
00:33:01.049 --> 00:33:05.089
people that you manage, we need
to change things x, Y and C.

399
00:33:05.730 --> 00:33:07.609
You would go and you would actually
talk to the consumers of your data

400
00:33:07.650 --> 00:33:14.279
science process to see how does it
fit in to the bigger picture? And

401
00:33:14.519 --> 00:33:20.759
so the inspiration part of the phase
is really about going broad brainstorming and trying

402
00:33:20.759 --> 00:33:23.799
to get inspired by everything you might
see around you. It's not about let

403
00:33:23.880 --> 00:33:28.509
me see the data and get inspired
by what's in my data. It's let's

404
00:33:28.509 --> 00:33:31.430
get inspired by everything before we even
think about the data. So that's the

405
00:33:31.549 --> 00:33:37.430
first phase. The second phase is
ideation, and what this means is trying

406
00:33:37.509 --> 00:33:45.220
to come up with particular solutions to
a problem and then testing those really quickly.

407
00:33:45.099 --> 00:33:52.059
So one of the core concepts here
is having the lowest fidelity prototypes possible

408
00:33:52.500 --> 00:33:57.890
and getting real user feedback on them. So it might be the case that

409
00:33:58.529 --> 00:34:02.529
you're working on a model to do
some classification and ultimately it's going to be

410
00:34:02.809 --> 00:34:08.289
like this big complex machine learning system
that's deployed in the cloud. But what

411
00:34:08.489 --> 00:34:13.840
you might do first is, let's
say we're working on this honey bee bumbling

412
00:34:13.920 --> 00:34:19.239
problem. You might just say,
okay, here's a spreadsheet of probabilities for

413
00:34:19.559 --> 00:34:22.559
each of these species. What would
you do with this? That's sort of

414
00:34:22.599 --> 00:34:27.349
my lowest fidelity one. Take the
most basic method, take the most basic

415
00:34:27.469 --> 00:34:30.349
output and say, is this useful, and then you take that and you

416
00:34:30.469 --> 00:34:36.909
learn from that. So the ideation
phases about these iterative cycles of learning from

417
00:34:36.989 --> 00:34:43.139
low fidelity prototypes that slowly and slowly
build fidelity around them. But it sort

418
00:34:43.179 --> 00:34:47.860
of helps to keep your project going
in a direction that ensures that the output

419
00:34:49.179 --> 00:34:52.500
is going to be targeted at a
real problem, that it's actually going to

420
00:34:52.579 --> 00:34:57.289
be useful and that as you come
up with new ideas throughout that process you

421
00:34:57.409 --> 00:35:00.449
can see which of those are good
ideas and which ones are and it keeps

422
00:35:00.449 --> 00:35:05.329
you want to stride in the sense
that you're not going to end up building

423
00:35:05.449 --> 00:35:08.880
something which is useless or going down
the entirely wrong. Bath that's exactly right.

424
00:35:08.920 --> 00:35:15.039
Yeah, so, I mean I've
seen so many times that, even

425
00:35:15.239 --> 00:35:17.639
work that we've done, where you
build a dashboard that no one ever looks

426
00:35:17.679 --> 00:35:22.750
at and everyone thought the dashboard was
what they wanted. But that's not the

427
00:35:22.789 --> 00:35:27.230
right tool for the job. People
really cared about answering one specific question and

428
00:35:27.949 --> 00:35:30.829
just if they would to a webpage
where the thumbs up for yes, the

429
00:35:30.869 --> 00:35:32.789
thumbs down for it. Now that
would have been even better than the dashboard

430
00:35:32.789 --> 00:35:37.099
you created. So in this dashboard
case, instead of building the dashboard,

431
00:35:37.139 --> 00:35:40.659
perhaps the inspirational audiation phase would involve
drawing the type of figure on a white

432
00:35:40.699 --> 00:35:44.619
board or on a piece of paper
that the dashboard would show and say to

433
00:35:44.699 --> 00:35:46.980
the stakehold is, Hey, is
this actually what you want? Yeah,

434
00:35:47.059 --> 00:35:53.809
that's exactly right, or even mocking
it up in powerpoint or using Microsoft paint

435
00:35:53.929 --> 00:35:59.570
to make a little prototype and say, Hey, is this graph something that

436
00:35:59.929 --> 00:36:02.090
you would need? How would you
actually use this in your process? So

437
00:36:02.570 --> 00:36:07.320
trying to get at not just saying
hey, do you like this? Is

438
00:36:07.360 --> 00:36:10.480
this something you want to see,
and more how would you use it then,

439
00:36:10.840 --> 00:36:15.239
because that question of how you actually
use something will change people's answers.

440
00:36:15.599 --> 00:36:19.510
I think in a lot of data
science work it's very clear that if you

441
00:36:19.630 --> 00:36:21.789
ask Hey, do you want to
see this? Hey, do you want

442
00:36:21.789 --> 00:36:23.150
to see that? Hey, do
you want to see something else, people

443
00:36:23.230 --> 00:36:27.309
say yes to all of those questions. How could you not want to have

444
00:36:27.469 --> 00:36:30.789
all of the information that you could
have? But really the question of how

445
00:36:30.869 --> 00:36:36.659
would you use it helps you to
narrow down on the things that are going

446
00:36:36.699 --> 00:36:39.820
to be valuable and not create this
information overload. So tell us about the

447
00:36:39.820 --> 00:36:45.260
third fights. We've got inspiration,
audiation and the third eye is the third

448
00:36:45.300 --> 00:36:51.530
eye is implementation. So implementation here
is not just okay, you've finished it,

449
00:36:51.730 --> 00:36:55.289
now go build it. Implementation is
actually a continuation of this process,

450
00:36:55.769 --> 00:37:02.199
so to go from prototyping to actually
piloting a solution. So we think of

451
00:37:02.280 --> 00:37:07.079
prototypes is being small scale, low
fidelity, something we do with a couple

452
00:37:07.159 --> 00:37:12.920
people to get some feedback. We
think of implementation and as okay, how

453
00:37:13.000 --> 00:37:16.030
do we become more data driven about
this decision that we're making? Now,

454
00:37:16.429 --> 00:37:22.869
implementation is about picking a pilot cohort, a set of users that will actually

455
00:37:22.909 --> 00:37:27.590
consume this and then saying, okay, here's the version we're working with.

456
00:37:27.829 --> 00:37:32.099
Now here's a higher fidelity prototype that
we have. Let's put it out there

457
00:37:32.380 --> 00:37:37.139
with a particular user group and let's
do some real testing of if this solution

458
00:37:37.500 --> 00:37:42.380
is working and solving the problem that
we want to solve. And so implementation

459
00:37:42.780 --> 00:37:46.889
is is piloting phase to get to
the point where not only do you have

460
00:37:47.090 --> 00:37:52.090
a lot of great anecdotical and qualitative
evidence that you've built up from these discussions,

461
00:37:52.409 --> 00:37:57.610
but you're also starting to get this
quantitative evidence for how what you built

462
00:37:58.130 --> 00:38:06.480
is changing the metrics that you care
about. We'll jump right back into our

463
00:38:06.519 --> 00:38:10.679
interview with pay to bull after a
short segment. Let's now dive into a

464
00:38:10.760 --> 00:38:16.110
segment called studies in interpretability. I'm
here with power coil, machine learning engineer

465
00:38:16.190 --> 00:38:21.989
and one of the core developers of
the open source statistical modeling platform pimcthree.

466
00:38:22.510 --> 00:38:24.510
Great to have you on the show. Pattern. Thanks for having me here.

467
00:38:24.550 --> 00:38:30.380
Go so potter, you've got very
strong opinions on interpretability and fantasy in

468
00:38:30.420 --> 00:38:34.099
machine learning. That's one of the
reasons I like you. Can you start

469
00:38:34.139 --> 00:38:38.699
off by telling me what interpretability and
fans are? interpretability is telling you why

470
00:38:38.739 --> 00:38:45.530
I model makes certain decisions. Often
we don't just care what a model predicts,

471
00:38:45.050 --> 00:38:51.130
we care why I predict something.
Fairness is making sure the protections are

472
00:38:51.210 --> 00:38:58.800
unbiased and not discriminating against protected groups, implicit or explicit. An interpreter model

473
00:38:59.159 --> 00:39:02.400
can tell you why I decided a
certain person is not worthy of a credit

474
00:39:02.440 --> 00:39:07.559
card, and for a human it
becomes easier to judge as a decision was

475
00:39:07.599 --> 00:39:12.710
based on a learned demographic or racial
bias. Thanks, podder. So why

476
00:39:12.750 --> 00:39:15.989
do you think this is becoming a
significant concern for data scientists? I think

477
00:39:16.030 --> 00:39:21.670
if you look at the trends such
as general data protection regulation, or GDP

478
00:39:21.789 --> 00:39:27.380
are, and changing consumer attitudes we
use data, people need to invest time

479
00:39:27.420 --> 00:39:32.820
in understanding interpretability and fairness. The
key terminology in GDP are that concerns us

480
00:39:32.940 --> 00:39:37.570
is the right to explainability, and
that means that in specific cases a black

481
00:39:37.690 --> 00:39:44.130
box model might not be sufficient,
and by black box model I mean one

482
00:39:44.170 --> 00:39:47.610
does not interpretable. For example,
you might want to know why a model

483
00:39:47.690 --> 00:39:52.519
predicted cervical cancer and a patient and
particular, explain that to a subject matter

484
00:39:52.639 --> 00:39:57.400
expert. Now those are really big
topics. Can you talk a bit more

485
00:39:57.440 --> 00:40:01.679
about fairness specifically? Well, it's
worth talking about what's happening in the research

486
00:40:01.679 --> 00:40:07.550
community. So specifically, in the
past few years we've seen tutorials at nips

487
00:40:07.630 --> 00:40:10.070
on fairness and machine learning. They
these will be in the show notes.

488
00:40:12.190 --> 00:40:15.230
I think it's very important to point
out this is a sad fact about humanity.

489
00:40:15.789 --> 00:40:20.070
The humans are prejudiced. You just
need to look at home. Few

490
00:40:20.110 --> 00:40:23.059
Minorities and women are represented at the
sea level of Forbes five hundred companies,

491
00:40:23.539 --> 00:40:29.099
for example, to see that.
In fact, Morse Heart, researcher into

492
00:40:29.139 --> 00:40:34.619
fairness and machine learning, points out
that in the financial space, automated underwriting

493
00:40:34.739 --> 00:40:39.769
increase approval rates for minority and low
income applicants by thirty percent while improving the

494
00:40:39.849 --> 00:40:45.929
overall acuracy of default predictions. So
this possible, depending on the rules and

495
00:40:46.130 --> 00:40:52.679
the process implemented and what data is
collected, to make less prejudiced decisions with

496
00:40:52.920 --> 00:40:55.760
algorithms. One of the concerns that
we see in the media and society at

497
00:40:55.840 --> 00:41:00.320
lodges the idea of trust in models. What does this mean for data scientists

498
00:41:00.480 --> 00:41:06.150
practically, especially in terms of fairness? I think for us to have impactice

499
00:41:06.190 --> 00:41:10.710
data scientists, we need to have
empathy, an attitude of move fast and

500
00:41:10.829 --> 00:41:15.469
break things, doesn't work in the
healthcare space and especially doesn't work in cases

501
00:41:15.869 --> 00:41:22.539
where you might be impacting human beings
lives, like models for recidivism. So

502
00:41:22.619 --> 00:41:25.420
we shouldn't dismiss these as concerns.
I think we should invest some time in

503
00:41:25.460 --> 00:41:32.250
learning interpret models. In fact,
my field of expertise is probabilistic programming and

504
00:41:32.369 --> 00:41:37.369
I've got a course of this online
if listeners are interested, called probabilistic programming

505
00:41:37.369 --> 00:41:45.010
primer. These models allow you to
both interpret the results you get uncertainly quantification

506
00:41:45.769 --> 00:41:49.559
of what you're modeling, and they
can also incorporate to be a knowledge.

507
00:41:50.400 --> 00:41:54.880
Fundamentally, also, the generative way
that models are built in probabilistic programming is

508
00:41:54.880 --> 00:42:00.599
less black box that machine learning methods
such as random forests. So what does

509
00:42:00.719 --> 00:42:05.590
this mean practically for data scientists?
Practically, I think it's very important to

510
00:42:05.670 --> 00:42:09.070
talk to humans. I think of
the regulatory aspects of what you're doing with

511
00:42:09.230 --> 00:42:14.590
your data. For example, when
I worked with credit risk models, we

512
00:42:14.710 --> 00:42:19.539
weren't allowed by law to use certain
features such as rice. I think that

513
00:42:19.659 --> 00:42:23.619
part of maturing as data scientists is
accepting the legal or ethical framework that you

514
00:42:23.699 --> 00:42:29.340
act within, and also think if
this model could leak to the press.

515
00:42:29.860 --> 00:42:34.010
Would I be embarrassed about it?
I think that's a good sniff test pattern.

516
00:42:34.409 --> 00:42:37.690
Thanks for that introduction to studies in
interpretability. Anything I can do to

517
00:42:37.809 --> 00:42:52.719
help listeners. Thanks for having me. Time to get straight back can do

518
00:42:52.800 --> 00:42:58.829
our chat with paid up. So
you and I've discussed a really interesting project

519
00:42:58.829 --> 00:43:00.550
you worked on the previously, which
I think illuminates all of these steps really

520
00:43:00.789 --> 00:43:05.349
wonderfully. It's a project where you're
looking at it a digital financial services in

521
00:43:05.469 --> 00:43:08.150
Tanzania. So maybe you could tell
us a bit about this project and how

522
00:43:08.349 --> 00:43:13.500
humans scented design actually played a pivotal
role for you. Great. Yeah,

523
00:43:13.780 --> 00:43:16.059
so this project, I'm going to
set back and sort of give you the

524
00:43:16.420 --> 00:43:22.340
the context that this project is done
in. So, for a lot of

525
00:43:22.500 --> 00:43:25.179
people, your money has always been
digital, and what I mean by that

526
00:43:25.380 --> 00:43:29.650
is that when I opened my first
bank account, my parents brought me to

527
00:43:29.730 --> 00:43:31.210
a bank in middle school and said
you're going to have a bank account,

528
00:43:31.449 --> 00:43:36.050
you have to be responsible for your
finances. Now. I gave that bag

529
00:43:36.210 --> 00:43:39.409
of money and they wrote down on
a piece of paper how much money I

530
00:43:39.570 --> 00:43:45.840
had, and that amount of money
wasn't physical cash that I was holding.

531
00:43:45.280 --> 00:43:51.559
That was actually in a database that
the bag had that was purely digital,

532
00:43:52.079 --> 00:43:55.829
and that's my native experience of what
it's like to interact with money, is

533
00:43:55.909 --> 00:44:05.309
to have what is really a digital
representation of that currency, whereas in lots

534
00:44:05.349 --> 00:44:09.219
of places bank infrastructure is not very
good, it's expensive to build banks,

535
00:44:09.340 --> 00:44:15.980
there are security risks of moving physical
money around and in many countries it's the

536
00:44:16.099 --> 00:44:23.059
case that people do most of their
interaction with money in pure cash, and

537
00:44:23.139 --> 00:44:27.050
that means when they want to save
money, they have cash and they want

538
00:44:27.050 --> 00:44:29.929
to spend money, they have to
get enough cash to buy something, and

539
00:44:30.050 --> 00:44:37.210
that can be very limiting in terms
of your ability to save money for time,

540
00:44:37.719 --> 00:44:43.360
your ability to get loans that you
might need to buy particular things.

541
00:44:43.800 --> 00:44:50.360
And so there's a belief in the
international development space that one of the ways

542
00:44:50.519 --> 00:44:54.349
in which we can help lift people
out of poverty is to provide them access

543
00:44:54.389 --> 00:45:01.789
to digital financial services. So this
same digital experience that people have had growing

544
00:45:01.829 --> 00:45:05.989
up, where they put their money
in a bank. How can we provide

545
00:45:05.989 --> 00:45:10.460
that without having to build all of
that physical banking infrastructure? And one of

546
00:45:10.539 --> 00:45:17.059
the approaches to this is to have
a digital wallet on your phone that's associated

547
00:45:17.139 --> 00:45:22.130
with your mobile phone account. Mobile
phones have been one of these lead program

548
00:45:22.210 --> 00:45:25.409
technologies where people who didn't even have
a landline now have access to a mobile

549
00:45:25.449 --> 00:45:32.369
phone. So the mobile phone providers
are now starting to offer services where you

550
00:45:32.489 --> 00:45:37.480
can actually save cash on your phone. You can say, okay, I've

551
00:45:37.480 --> 00:45:42.159
got a wallet with ten dollars on
it and that's just associated with my phone,

552
00:45:42.199 --> 00:45:45.239
and so there's this transition from a
world in which you just work with

553
00:45:45.400 --> 00:45:51.429
cash to a world in which you
have some digital representation and for people who

554
00:45:51.429 --> 00:45:54.989
haven't had the experience of always having
a digital representation, you have to build

555
00:45:55.070 --> 00:45:59.750
trust in that system. You have
to make sure that the digital financial services

556
00:46:00.309 --> 00:46:04.340
actually fit the needs that this community
has. So that's the big sort of

557
00:46:04.699 --> 00:46:09.980
project. Background in context is how
do we increase uptake of these digital financial

558
00:46:09.980 --> 00:46:20.489
services that are providing for people in
these environments that have very volatile incomes some

559
00:46:20.849 --> 00:46:27.210
stent sense of stability for their money, some access to loan infrastructures, some

560
00:46:27.329 --> 00:46:32.480
access to savings infrastruction. So that's
the context. This is a project that's

561
00:46:32.480 --> 00:46:38.400
funded by the gates foundation to try
to understand what levers we can pull to

562
00:46:38.559 --> 00:46:45.519
help engage people and give them access
to these tools so one of the particular

563
00:46:45.719 --> 00:46:51.389
things about this digital financial services system
is that you need a way to exchange

564
00:46:51.429 --> 00:46:57.030
your digital currency for cash. There's
always going to be some sort of transaction.

565
00:46:57.429 --> 00:47:01.860
That's like working with an ATM.
However, ATMs are relatively expensive and

566
00:47:01.940 --> 00:47:07.460
there's a lot of physical infrastructure you
need for an ATM to work. So

567
00:47:07.820 --> 00:47:13.219
what happens in places where they have
digital financial services, and this also goes

568
00:47:13.300 --> 00:47:16.170
by the name mobile money, so
I use those interchangeably. You have what

569
00:47:16.250 --> 00:47:21.449
are called mobile money agents. These
are usually people that have a small shop

570
00:47:21.610 --> 00:47:28.289
somewhere that are selling home gooods,
snacks drinks, and they also become mobile

571
00:47:28.369 --> 00:47:30.880
money agents. But that means is
you can go to them and say I

572
00:47:31.039 --> 00:47:36.719
want to trade five digital dollars for
five dollars in cash, or I want

573
00:47:36.719 --> 00:47:39.760
to take five dollars in cash and
put that in my digital wallet. So

574
00:47:39.840 --> 00:47:45.630
there the interface between physical cash and
digital cash. So the focus of our

575
00:47:45.670 --> 00:47:51.829
project was how do we make that
interaction between agents and customers one that can

576
00:47:51.909 --> 00:47:58.389
help build trust in this mobile money
system, and that direction was driven by

577
00:47:58.780 --> 00:48:04.260
this human centered design process. So
when we went out, the first thing

578
00:48:04.340 --> 00:48:07.579
we did was talk to people who
use mobile money and people who didn't.

579
00:48:07.619 --> 00:48:12.780
We went and sat in a number
of markets. We watch people buying things

580
00:48:12.820 --> 00:48:16.210
and we asked them why did you
buy that with cash? Why did you

581
00:48:16.289 --> 00:48:21.050
buy that with mobile money? Do
you have a mobile money account? What's

582
00:48:21.090 --> 00:48:24.489
your experience been like? And it
sounds like in this process as well,

583
00:48:24.530 --> 00:48:28.920
then you're getting qualitative dita as well
as quantituive dital. That's right, and

584
00:48:29.320 --> 00:48:35.519
really a big part of what you
do is try to gather that qualitative data

585
00:48:35.559 --> 00:48:39.519
as well. So we've been gathering
this quality of data through these interviews,

586
00:48:39.880 --> 00:48:46.030
but we also got mobile money transactions
from one of the mobile network operators for

587
00:48:46.190 --> 00:48:51.670
a full nine months and it was
tens of gigabytes of data, all of

588
00:48:51.710 --> 00:48:54.510
the transactions, hundreds of millions of
mobile money transactions in the country, and

589
00:48:55.099 --> 00:49:00.780
so our goal was to combine that
very rich data source about real behaviors,

590
00:49:00.219 --> 00:49:06.179
with behaviors that we heard about,
with these qualitative experiences that people actually had.

591
00:49:06.940 --> 00:49:12.530
And so one really great example that
I think just highlights the value of

592
00:49:12.570 --> 00:49:17.849
human centered design is the fact that
we kept talking to agents and saying,

593
00:49:19.570 --> 00:49:23.480
what are your biggest struggles with mobile
money? How do you see it fitting

594
00:49:23.519 --> 00:49:29.599
into the larger context of Your Business, where mobile money is one of your

595
00:49:29.639 --> 00:49:35.480
revenue dreams but you also sell other
goods, and what we kept hearing from

596
00:49:35.519 --> 00:49:38.750
these mobile money agents has well,
it can be really tricky to predict how

597
00:49:38.789 --> 00:49:45.750
much money I'm going to make on
mobile money transactions because I earned some commission

598
00:49:45.949 --> 00:49:51.550
on each transaction, but it's really
opaque what those commissions are and it's very

599
00:49:51.630 --> 00:49:55.019
hard for me to predict on any
aggregated time frame, for some given week

600
00:49:55.059 --> 00:50:00.179
or some given month, how much
money I'm going to get in commissions.

601
00:50:00.340 --> 00:50:07.369
And this was particularly interesting to us
because we, as data scientists, had

602
00:50:07.409 --> 00:50:10.610
been digging into this huge treasure trophy
data, thinking how this has all the

603
00:50:10.730 --> 00:50:14.730
answers in it. This is going
to be amazing. We can find so

604
00:50:14.809 --> 00:50:19.769
many insights in this huge range of
transactions that we have. One of the

605
00:50:19.809 --> 00:50:23.960
things that we realized after doing these
interviews is we didn't know what the commissions

606
00:50:24.000 --> 00:50:29.000
for an agent were. That was
not data that was in the data set

607
00:50:29.039 --> 00:50:31.559
that we had. We had the
amount of the transaction, but the portion

608
00:50:31.760 --> 00:50:37.670
that then an agent got as a
commission and was calculated totally separately by some

609
00:50:37.829 --> 00:50:43.710
business logic that existed in another application. And even if you did know how

610
00:50:43.750 --> 00:50:45.429
much they got, you might not
know whether that was a lot or little

611
00:50:45.510 --> 00:50:49.750
or how that affected them on the
ground. Oh, absolutely, yeah.

612
00:50:50.059 --> 00:50:53.099
And so just the thought that we
could have known what was valuable the agents

613
00:50:53.340 --> 00:50:58.659
by looking at the data set and
figuring out these patterns. That data set

614
00:50:58.699 --> 00:51:01.699
didn't even have the most important variable
to the agents that we were working with.

615
00:51:02.019 --> 00:51:05.969
We wouldn't have learned that if we
didn't talk to them and just tried

616
00:51:06.010 --> 00:51:08.690
to learn things from the data alone. So one of the big things that

617
00:51:09.170 --> 00:51:12.889
we try to do it all of
our work and I really encourage all data

618
00:51:12.929 --> 00:51:19.119
scientists to do, is to go
out and observe how your data gets collected.

619
00:51:19.639 --> 00:51:22.440
So if you're working with Iot data
and a factory, actually go to

620
00:51:22.519 --> 00:51:25.480
the factory floor watch how things happen. If you're, like us, working

621
00:51:25.559 --> 00:51:31.599
with digital financial services, go watch
people make transactions, see what actions in

622
00:51:31.679 --> 00:51:38.110
the real world correspond with something in
your data, because that perspective changes how

623
00:51:38.150 --> 00:51:43.429
you think about the data itself.
It changes where you trust the data and

624
00:51:43.510 --> 00:51:46.030
where you don't trust the data,
and so I really encourage people to get

625
00:51:46.030 --> 00:51:51.059
away from their screen, step away
from their desk and go watch the data

626
00:51:51.179 --> 00:51:55.500
collection in action and I think in
nearly every case you can go do this

627
00:51:55.940 --> 00:52:00.900
and it will have a transformative effect
on how you think about that data.

628
00:52:00.769 --> 00:52:05.130
I think that's really important, particularly
as we live in an age when people

629
00:52:05.530 --> 00:52:10.210
maybe first get associated have experience with
Diata signs, online competitions, for examples,

630
00:52:10.369 --> 00:52:14.929
such as yours, platforms like Diata
camp, getting tech tital or getting

631
00:52:14.969 --> 00:52:19.880
Dita online, and not actually thinking
a lot about the data generating process.

632
00:52:20.239 --> 00:52:23.719
Yeah, it's very easy to just
start with the data and say, okay,

633
00:52:23.880 --> 00:52:30.000
what's in here, but until you
really understand that data generating process,

634
00:52:30.550 --> 00:52:35.150
you won't know to ask what's not
in this data that I might care about

635
00:52:35.590 --> 00:52:40.190
or what in this data is not
reliable. For example, we saw a

636
00:52:40.469 --> 00:52:46.780
lot of these mobile money transactions fail
because of network connectivity, for example,

637
00:52:47.300 --> 00:52:52.500
and for some of those transactions we
wouldn't have seen that in the data.

638
00:52:52.539 --> 00:52:54.619
If the network failed, the transaction
every went through. It doesn't get recorded

639
00:52:54.699 --> 00:53:01.530
in the database. So understanding the
limitations of the connectivity and how that affects

640
00:53:01.570 --> 00:53:07.289
the experience is something that we can
only even start asking about. How do

641
00:53:07.409 --> 00:53:12.369
we measure those transactions when we've actually
observed them? So we're going to need

642
00:53:12.369 --> 00:53:14.280
to wrap up in a few minutes. Lay to all. Just wondering all

643
00:53:14.320 --> 00:53:16.639
that. Any other aspects of the
project, intents, any that you wanted

644
00:53:16.679 --> 00:53:22.400
to discuss? Yeah, so I
just just to share one other example of

645
00:53:22.920 --> 00:53:25.760
where the human centered design approach,
I think, really made a difference.

646
00:53:27.190 --> 00:53:34.429
We were looking at the Times of
day which a mobile money agent was busiest,

647
00:53:34.710 --> 00:53:40.150
so when we're people coming to trade
cash for digital currency or do the

648
00:53:40.309 --> 00:53:46.019
opposite, and we took the data
and we looked at it board days of

649
00:53:46.099 --> 00:53:52.980
the week and Times of day and
we built this really beautiful heat map visualization

650
00:53:52.219 --> 00:53:57.650
that you can think of as a
checkerboard, for each of the squares is

651
00:53:57.730 --> 00:54:01.369
either lighter or darker based on how
many transactions you have. We made it

652
00:54:01.489 --> 00:54:07.010
interactive so that you can hover and
you can see how busy for a different

653
00:54:07.050 --> 00:54:09.719
region is, a different time of
day is a different day of week.

654
00:54:10.000 --> 00:54:14.760
It really get a great sense for
the patterns and mobile money used that happen.

655
00:54:15.079 --> 00:54:20.039
And it's also color blind human friendly. It absolutely is. We did

656
00:54:20.159 --> 00:54:23.909
build it using forritis, which is
color blind friendly, and if you're not

657
00:54:24.070 --> 00:54:28.829
thinking about that for your visualizations,
you should be, because that's a little

658
00:54:28.829 --> 00:54:32.670
bit of human centered design and,
speaking as a color blond individual on red

659
00:54:32.710 --> 00:54:36.630
grain, color blind, as you
know, eight percent of Human Miles Ah,

660
00:54:36.989 --> 00:54:38.900
so you appreciate for it as even
more than the rest of us that

661
00:54:39.019 --> 00:54:44.300
think it's a beautiful color map.
I can't get enough of us. Well,

662
00:54:44.420 --> 00:54:46.579
I think it's a really compelling and
beautiful color map, which is one

663
00:54:46.579 --> 00:54:51.059
of the reasons that we loved this
visualization, which is one of the reasons

664
00:54:51.059 --> 00:54:55.289
that people we work would love this
visualization and how interactive it was. But

665
00:54:57.090 --> 00:55:01.730
none of the agents that we were
working with had access to a comp cut,

666
00:55:02.489 --> 00:55:06.650
they weren't sitting at a laptop and
they wanted to look at a dashboard

667
00:55:06.650 --> 00:55:09.840
that had this beautiful visualization that that
wasn't going to be useful to them.

668
00:55:10.719 --> 00:55:19.199
So what we ended up actually building
was a text based visualization that was essentially

669
00:55:19.320 --> 00:55:22.630
just a bar chart where would say
m for Monday, and then it would

670
00:55:22.630 --> 00:55:27.510
have three capital eyes in a row, and then it would have tea for

671
00:55:27.590 --> 00:55:30.670
Tuesday, it would have eight capital
eyes in a row. And by building

672
00:55:30.710 --> 00:55:37.219
these text based visualizations, essentially bar
charts built out of characters, we could

673
00:55:37.219 --> 00:55:43.900
actually give a data visualization experience to
these agents that were working on feature Phonas,

674
00:55:44.179 --> 00:55:49.539
and that process of taking something that
we think of as an amazing data

675
00:55:49.699 --> 00:55:54.849
science output, this really compelling interactive
visualization, and putting that visualization into a

676
00:55:54.969 --> 00:56:00.250
context where it can actually get used, is one of the transformative experiences of

677
00:56:00.369 --> 00:56:05.519
that project for us, where we
started to think about, okay, what's

678
00:56:05.559 --> 00:56:08.320
the context for all of our output, not how do we make the most

679
00:56:08.360 --> 00:56:13.920
amazing data visualization? Yeah, and
it's so rely on on the knowledge of

680
00:56:14.039 --> 00:56:17.199
what actual technology, what phones humans
have on the ground. Right. Yet

681
00:56:17.320 --> 00:56:22.710
that's absolutely right. So Paida.
As a final question, I'm wondering if

682
00:56:22.750 --> 00:56:24.550
you have a final cult to action
to well all our listeners out there.

683
00:56:25.110 --> 00:56:30.630
Yeah, so, I think there
are, in my mind, for core

684
00:56:30.909 --> 00:56:34.699
activities of human centered data scientists,
and I think we should all be human

685
00:56:34.739 --> 00:56:38.699
centered data scientists. And the first
one is go to the field and observed

686
00:56:38.860 --> 00:56:45.340
the data being generated. Without understanding
what a row in your data set means,

687
00:56:45.500 --> 00:56:50.809
without actually observing that happening, without
knowing what it's captured, what doesn't,

688
00:56:51.170 --> 00:56:54.409
what happens when something goes wrong,
they'll be very limited in the output

689
00:56:54.409 --> 00:56:58.449
that you can have. And also, if you do go and do that,

690
00:56:58.889 --> 00:57:02.920
you'll be so much better positioned to
ask questions that matter of their data.

691
00:57:04.400 --> 00:57:07.840
Without talking to those agents, we
wouldn't have asked that commission question of

692
00:57:08.000 --> 00:57:14.480
the data set. So going to
the field observing data being generated, is

693
00:57:14.840 --> 00:57:21.550
item number one. Item two is
design with, not for by, iterating

694
00:57:21.710 --> 00:57:27.750
on prototypes. So this process of
constant iteration, conversation with people who will

695
00:57:27.789 --> 00:57:31.179
actually be using the output, getting
there by in on the decisions that you're

696
00:57:31.219 --> 00:57:37.380
making, means that it's going to
be something that's useful when you finish the

697
00:57:37.460 --> 00:57:40.619
project, not I worked for three
months. Is this good for you?

698
00:57:42.539 --> 00:57:45.690
Oh, no, it's not,
or requires some major changes. It's how

699
00:57:45.769 --> 00:57:51.329
do we keep that process tighter and
sync so that we're actually building things that

700
00:57:51.369 --> 00:57:55.530
are useful, and we do that
with really low fidelity prototypes that were constantly

701
00:57:55.690 --> 00:58:02.199
testing. The third is to put
outcomes, not methods or outputs first.

702
00:58:02.880 --> 00:58:07.039
So that's really saying what is the
outcome? We cared about in our case

703
00:58:07.440 --> 00:58:13.349
it was the increase in adoption of
digital financial services. That's what we cared

704
00:58:13.389 --> 00:58:16.710
about and in particular, we thought
we could do that by improving the tools

705
00:58:16.750 --> 00:58:22.309
that mobile money agents had. So
our goal was to say, okay,

706
00:58:22.429 --> 00:58:28.300
the best outcome is for mobile money
agents to be making more transactions. That's

707
00:58:28.340 --> 00:58:30.539
what we want to measure and it
wasn't. How do we do the most

708
00:58:30.659 --> 00:58:37.900
interesting dimensionality reduction on this huge data
set that we have? The fourth item

709
00:58:37.300 --> 00:58:42.409
is to build consensus on metrics for
success, and I think this is one

710
00:58:42.449 --> 00:58:49.530
of the most difficult but most important
ones. Is You need to define up

711
00:58:49.690 --> 00:58:55.920
front what success means and you need
to get buy in from everyone on that

712
00:58:57.239 --> 00:59:00.440
being successful, and I think this
is one of the things that people assume

713
00:59:00.679 --> 00:59:06.360
they've got the same goals if they're
working on the same project from the get

714
00:59:06.480 --> 00:59:10.789
go. But until you have that
explicit discussion about what success means and what

715
00:59:10.949 --> 00:59:16.150
those metrics are, you won't be
optimizing for exactly the same thing. So

716
00:59:16.829 --> 00:59:22.429
those I think my called action really
is to take those and try to build

717
00:59:22.469 --> 00:59:25.420
them into your process as a data
scientist, other than becoming a human center

718
00:59:25.500 --> 00:59:30.980
data scientist, thinking about your users
and you using a more collaborative process.

719
00:59:31.219 --> 00:59:36.019
I would encourage people to come check
out a competition and driven data dot org.

720
00:59:36.340 --> 00:59:39.650
Got A lot of interesting social impact
projects happening there, or to check

721
00:59:39.690 --> 00:59:44.090
out one of the open source projects
that we had as part of this discussion

722
00:59:44.289 --> 00:59:50.210
that's crepet cutter data science and Dan
the ethics checklist package, and, of

723
00:59:50.289 --> 00:59:53.440
course, both of those projects and
engaging in competitions and all the other great

724
00:59:53.440 --> 00:59:58.599
stuff you do on a driven data
will help any butting data Scientis, are

725
00:59:58.599 --> 01:00:02.960
established data scientists, to doing more
human centered data work as well. Yeah,

726
01:00:04.000 --> 01:00:07.469
that's the goal, Peter. That's
been such a pleasure having you on

727
01:00:07.550 --> 01:00:09.789
the show. You know, thanks
for having me. I love chatting as

728
01:00:09.829 --> 01:00:20.349
always. Thanks for joining our conversation
with Peter About the importance of human center

729
01:00:20.429 --> 01:00:24.380
design in data science. We saw
the need for applying software engineering best practices

730
01:00:24.420 --> 01:00:29.619
in data work and the importance of
ethics checklists, along with driven Datas,

731
01:00:29.739 --> 01:00:34.219
Cookie Cutter Data Science and Don Projects, which respectively, step up to these

732
01:00:34.260 --> 01:00:38.170
issues. We also saw how the
process of human center design is a manifestation

733
01:00:38.289 --> 01:00:43.329
of the principle of empathy for users
in terms of iterating on data products,

734
01:00:43.449 --> 01:00:47.369
for example, that they both want
and find useful. Peter provided several illuminating

735
01:00:47.409 --> 01:00:52.400
examples, including driven data's work with
mobile money financial services in Tanzania to make

736
01:00:52.440 --> 01:00:57.880
the interaction between agents and customers one
that can help build trust in the mobile

737
01:00:57.960 --> 01:01:01.800
money system. Starting of what is
desirable is key, as opposed to what's

738
01:01:01.800 --> 01:01:07.869
feasible and then rapidly iterating through inspiration, ideation and implementation with the end user

739
01:01:07.909 --> 01:01:12.789
a longer way. And don't forget, if you'd like this interview, you

740
01:01:12.989 --> 01:01:15.590
might be interested in working with driven
data. Currently the team is looking for

741
01:01:15.670 --> 01:01:21.099
a software engineer who loves the idea
of building python applications for social impact.

742
01:01:21.380 --> 01:01:24.940
Will include a link in the show
notes. Also get pump the heck up

743
01:01:25.019 --> 01:01:30.059
for next week's episode a conversation with
Angela Busser about managing data science teams.

744
01:01:30.380 --> 01:01:35.690
Angela is director of data science at
I robot, where she leads the team

745
01:01:35.849 --> 01:01:40.969
through development of machine learning algorithms,
sentiment analysis and anomaly detection processes. I

746
01:01:42.170 --> 01:01:45.170
robot are the makers of consumer robots
that we all know and love, like

747
01:01:45.289 --> 01:01:50.679
the Rumba and the Brava, which
are, respectively, a robotic vacuum cleaner

748
01:01:50.800 --> 01:01:54.559
and a robotic mop Angela will talk
about how to get into data science management,

749
01:01:54.800 --> 01:01:59.880
the most important strategies to ensure that
your data science team delivers value to

750
01:01:59.920 --> 01:02:04.550
the organization, how to hire data
scientists and key points to consider as your

751
01:02:04.590 --> 01:02:07.789
data science team grows over time.
In addition, to the types of tradeoffs

752
01:02:07.789 --> 01:02:10.750
you need to make as a data
science manager and how you make the right

753
01:02:10.869 --> 01:02:15.420
ones. All this and much,
much more. I'm your host. You

754
01:02:15.539 --> 01:02:19.659
Go, bound Anderson. You can
follow me on twitter at Hugo bound and

755
01:02:19.739 --> 01:02:22.780
data camp at data camp. You
can find all our episodes and show notes

756
01:02:22.900 --> 01:02:27.340
at data campcom communities podcast