#49 Data Science Tool Building

WEBVTT

1
00:00:00.120 --> 00:00:04.040
Wow. In this episode of data
framed, a data camp podcast, I'm

2
00:00:04.120 --> 00:00:08.429
speaking with West mckinney, creator of
the pandas project for data analysis tools in

3
00:00:08.550 --> 00:00:13.109
Python and author of Python for data
analysis, among many other things. Where's

4
00:00:13.189 --> 00:00:16.829
and I will talk about data science
tool building, what it took to get

5
00:00:16.870 --> 00:00:21.460
Pandus off the ground and how he
approaches building human interfaces to data to make

6
00:00:21.620 --> 00:00:25.620
individuals more productive. On top of
this, we'll talk about the future of

7
00:00:25.699 --> 00:00:30.539
data science tooling, including the Apacheero
project and how it can facilitate this future,

8
00:00:30.820 --> 00:00:35.539
the importance of data frames that are
portable between programming languages and building tools

9
00:00:35.579 --> 00:00:41.009
that facilitate data analysis work in the
big data limit. Pandas initially arose from

10
00:00:41.009 --> 00:00:44.890
where's noticing that people were nowhere near
as productive as they could be due to

11
00:00:44.929 --> 00:00:47.969
a lack of tooling, and the
projects he is working on today, which

12
00:00:48.009 --> 00:00:52.119
will discuss, arise from the same
place and present a bold vision for the

13
00:00:52.159 --> 00:00:56.000
future. Where's also makes clear that
pandas is not a one human show,

14
00:00:56.280 --> 00:00:59.560
and when people thank him for his
work, he reminds them to thank Jeff

15
00:00:59.560 --> 00:01:03.280
Freeback, Joris van den Boscher,
Philip Cloud and Tom Augsburger, along with

16
00:01:03.349 --> 00:01:07.709
the other pandas core developers that have
really been driving the project forward over the

17
00:01:07.790 --> 00:01:14.150
last five years. I for one
want to thank all of them. Finally,

18
00:01:14.590 --> 00:01:17.829
and not to give too much away, will also discuss the challenges of

19
00:01:17.870 --> 00:01:22.819
open source software development and how where's
is approaching funding and resourcing. Oss with

20
00:01:22.939 --> 00:01:27.180
his most recent venture Ursa labs.
Find out all this and more, including

21
00:01:27.379 --> 00:01:32.659
how much of panders was developed in
a small ease village apartment that wears may

22
00:01:32.700 --> 00:01:36.689
or may not have cohabitated with mice. Well, I can't give too much

23
00:01:36.689 --> 00:01:41.090
away. This is just the opening
monolog I'm Hugo bound Anderson, a data

24
00:01:41.129 --> 00:01:49.840
signed as the data camp, and
this is data framed. Welcome to data

25
00:01:49.920 --> 00:01:53.680
framed, a weekly data camp podcast
exploring what data science looks like on the

26
00:01:53.719 --> 00:01:59.439
ground for working data scientists and what
problems that Consol I'm your host, Hugo

27
00:01:59.519 --> 00:02:02.230
bound Anderson. You can follow me
on twitter at Hugo bound and data camp

28
00:02:02.390 --> 00:02:07.189
at data camp. You can find
all our episodes and show notes at Data

29
00:02:07.269 --> 00:02:15.710
Campcom. Community podcast listeners. As
always, check out the show notes for

30
00:02:15.819 --> 00:02:19.699
more material on the conversation today.
I've also included a survey in the show

31
00:02:19.699 --> 00:02:23.780
notes and a linked to a form
where you can make suggestions for future episodes.

32
00:02:23.020 --> 00:02:27.300
I'd really appreciate it if you take
the survey so I can make sure

33
00:02:27.300 --> 00:02:30.530
that we're producing episodes that you want
to hear. Now back to our regularly

34
00:02:30.810 --> 00:02:38.810
scheduled programming in Python. Hi there, ware's, and welcome to data framed.

35
00:02:38.569 --> 00:02:42.050
Thank you. Thanks for having me. Great pleasure to have you on

36
00:02:42.129 --> 00:02:45.240
the show and I'm really excited to
have you here today to talk about open

37
00:02:45.319 --> 00:02:49.520
source and software development, to talk
about your work at Ursa labs, apartciarrow,

38
00:02:49.840 --> 00:02:52.439
a bunch of other things to do
with tool building, but first I'd

39
00:02:52.439 --> 00:02:54.400
like to find out a bit about
you. Perhaps you could open by telling

40
00:02:54.439 --> 00:02:58.750
me what you're known for in the
data community. Be Sure? Yeah,

41
00:02:58.830 --> 00:03:02.349
I mean I'm best known for for
being the the original author of the Python

42
00:03:02.590 --> 00:03:07.909
Pandas Project, which I understand that
a lot of people use. So I

43
00:03:07.030 --> 00:03:10.909
started building that as a closed source
library a little over ten years ago and

44
00:03:12.030 --> 00:03:15.819
I've been working on a lot of
different open source projects for for the Python

45
00:03:15.979 --> 00:03:20.699
data science world, world and beyond. I also wrote a book called Python

46
00:03:20.819 --> 00:03:23.860
for a data analysis, which is
now in its second edition, and I

47
00:03:23.939 --> 00:03:29.289
think that's become a pretty pretty ubiquitous
resource for people that are getting into the

48
00:03:29.330 --> 00:03:31.569
field of data science and are wanting
to learn how to use pandous and,

49
00:03:31.689 --> 00:03:35.930
you know, get their feet wet
with working with data and congrats on the

50
00:03:35.969 --> 00:03:38.569
second edition. That was in the
past year or so that that was released,

51
00:03:38.569 --> 00:03:39.689
right. Yeah, it was at
yeah, the end of it was

52
00:03:39.729 --> 00:03:42.919
just about a year ago, the
end of two thousand and seventeen. And

53
00:03:44.080 --> 00:03:47.080
how did you get into data Sion's
tool building originally, because I'm aware that

54
00:03:47.120 --> 00:03:52.800
your background wasn't in Incus Pacific.
Right. I was a mathematician, so

55
00:03:52.879 --> 00:03:57.229
I studied a peer math at MIT. I did a little bit of a

56
00:03:57.270 --> 00:04:01.590
little bit of computer science. I
had had some exposure to the world of

57
00:04:01.710 --> 00:04:05.189
machine learning in that I was aware
that it existed, but, you know,

58
00:04:05.389 --> 00:04:12.020
Mike didn't have a statistics program so
data analysis and Statistics wasn't very familiar

59
00:04:12.060 --> 00:04:16.139
to me when I was entering the
working world and I got a job at

60
00:04:16.180 --> 00:04:21.660
an investment manager called Aqr capital management, which is based in Greenwich Connecticut,

61
00:04:23.060 --> 00:04:27.209
and there were a number of MIT
grads that had gone to work there and

62
00:04:27.490 --> 00:04:30.850
some of them were math majors and
they kind of sold me on the idea

63
00:04:30.889 --> 00:04:33.810
of getting experience with applied math and
then maybe I would go back to Grad

64
00:04:33.930 --> 00:04:38.720
school later on. And I found
that in my job there that rather than

65
00:04:38.839 --> 00:04:43.199
doing very much applied math, that
I was really doing a lot of data

66
00:04:43.240 --> 00:04:46.000
munging. So I was writing sequel, I was using excel and really I

67
00:04:46.079 --> 00:04:51.759
just found that I wasn't as productive
and efficient working with the data as I

68
00:04:51.990 --> 00:04:56.029
felt like I should have been.
And part of it was like, well,

69
00:04:56.069 --> 00:04:58.750
okay, I'm just just starting out
my career, I'm twenty two years

70
00:04:58.750 --> 00:05:00.829
old, you know, what do
I know? But if I looked around

71
00:05:00.829 --> 00:05:03.550
it, even even at people who
were a lot more senior to me and

72
00:05:03.629 --> 00:05:08.019
a lot more experienced, and it
seemed like they weren't very productive either and

73
00:05:08.100 --> 00:05:11.379
they were spending a lot of time. You know, obviously their skill with

74
00:05:11.540 --> 00:05:15.139
excel and cell shortcuts and so forth, keyboard shortcuts, was a lot better

75
00:05:15.180 --> 00:05:19.420
than mine, but still it seemed
like there was just something missing to kind

76
00:05:19.420 --> 00:05:24.649
of working with data. And I
started to learn. I started to learn

77
00:05:24.649 --> 00:05:27.250
are at the end of two thousand
and seven, beginning of two thousand and

78
00:05:27.250 --> 00:05:29.889
eight, and there were at that
point in time, you know, the

79
00:05:29.970 --> 00:05:32.449
our ecosystem was a lot less mature
and it felt like, you know,

80
00:05:32.529 --> 00:05:38.079
an interesting language for, you know, valuable language or doing statistics and data

81
00:05:38.120 --> 00:05:42.959
analysis. But we also needed to
kind of build software, and so I

82
00:05:43.160 --> 00:05:46.000
learned a little bit of Python and
thought like wow, this is a really

83
00:05:46.240 --> 00:05:49.269
easy to use programming language. I
had done some Java programming and thought that

84
00:05:49.670 --> 00:05:51.949
I just wasn't very good at job, and so I thought, man,

85
00:05:51.990 --> 00:05:56.750
I'm just not cut out for building
software. Are But I decided to have

86
00:05:56.790 --> 00:06:01.550
a tinker with building some data manipulation
tools and python. That was march,

87
00:06:01.750 --> 00:06:05.660
April two thousand and eight and you
know, just went down the rabbit hole

88
00:06:05.740 --> 00:06:10.699
from there and once I had made
myself more productive working with data, I

89
00:06:10.860 --> 00:06:15.100
started evangelizing the tools I was building
to my colleagues and, you know,

90
00:06:15.139 --> 00:06:18.209
I kept pulling on one thread and
ended up becoming more of a software engineer

91
00:06:18.250 --> 00:06:23.329
has a finance or you know,
a finance or math person. Yeah,

92
00:06:23.370 --> 00:06:26.170
there are a lot of interesting touch
points there, for example your background and

93
00:06:26.209 --> 00:06:29.449
Puel math and that you're in Connecticut. I actually I was working in Puel

94
00:06:29.529 --> 00:06:32.600
math and ended up doing applied maths
in a biology labine. You Haven Connecticut,

95
00:06:32.639 --> 00:06:35.439
not not in Greenwich, but at
that point I actually started dealing with

96
00:06:35.519 --> 00:06:40.480
data a lot as well, and
that's when I started getting into data science

97
00:06:40.519 --> 00:06:43.600
also. It's also interesting that Pantis, when you first developed it, was

98
00:06:44.040 --> 00:06:46.629
closed source. But before we get
there, you've spoken a bit to you

99
00:06:46.709 --> 00:06:50.029
know why you why you chose Python. Could you explain a bit more about

100
00:06:50.029 --> 00:06:53.790
what was attractive about python then,
because, of course one of the lot

101
00:06:53.829 --> 00:06:58.790
of the attractive things for researchers and
data scientists now about python is the data

102
00:06:58.829 --> 00:07:00.430
science stack, you know, pandas, psyche, learn, numpile, all

103
00:07:00.500 --> 00:07:03.019
of these things. So what what
made you really like it back in the

104
00:07:03.060 --> 00:07:06.540
day? Yeah, I mean at
that at that point of time, you

105
00:07:06.579 --> 00:07:09.740
know, two thousand and seven,
two thousand and eight. In terms of

106
00:07:09.899 --> 00:07:15.339
doing statistical computing, python was not
you know, let's let's think of it

107
00:07:15.459 --> 00:07:19.889
as a promising world that has not
yet been terrorformed. So think that there

108
00:07:19.970 --> 00:07:24.930
were kind of the nuts and bolts
of a really interesting environment. You know,

109
00:07:25.050 --> 00:07:28.410
I learned about the EPYTHON project and
said, you know, okay,

110
00:07:28.410 --> 00:07:32.759
here's a here's a really nice interactive
shell where you can plot things and you

111
00:07:32.839 --> 00:07:38.199
know, has tab completion and you
know, really basic interactive affordances that really

112
00:07:38.240 --> 00:07:42.680
help out a lot you had the
nuts and bolts of the of doing all

113
00:07:42.759 --> 00:07:46.629
the analytical computing that you need to
do for data manipulation. NUMB PI had

114
00:07:46.670 --> 00:07:50.389
its one point, I release,
I think in two thousand and six and

115
00:07:50.470 --> 00:07:55.189
it had become a mature project and
the you know, scientific python world was

116
00:07:55.310 --> 00:07:59.860
defragmenting itself after the number a numeric
rift which had persisted for several years and

117
00:07:59.939 --> 00:08:03.620
you know, Travis Travis Ole fent
had sort of work to bring those communities

118
00:08:03.660 --> 00:08:07.579
together. But really I think what
attracted me to the language was the accessibility

119
00:08:07.819 --> 00:08:13.129
and the fact the that it was
really very suited for interactive and exploratory computing,

120
00:08:13.329 --> 00:08:16.250
that you could you didn't have to
set up an elaborate development environment,

121
00:08:16.449 --> 00:08:20.089
you know, an Idee, to
be able to get up and running doing

122
00:08:20.209 --> 00:08:24.850
some really basic things. And so, having had experience with Java, I

123
00:08:24.930 --> 00:08:28.399
think one of the things that put
me off about Java was the elaborateness of

124
00:08:28.560 --> 00:08:33.159
the environment that you need to really
be productive, like you really need to

125
00:08:33.200 --> 00:08:35.960
set up an ide and it's those
all this tooling that you need to do,

126
00:08:37.159 --> 00:08:41.240
whereas with python you could do,
you know, some pretty complex things

127
00:08:41.070 --> 00:08:45.309
with a few lines of code and
a text file and then you just run

128
00:08:45.350 --> 00:08:50.230
the script. So that kind of
like interactive scripting feel of doing exploratory computing

129
00:08:50.350 --> 00:08:54.429
was really compelling to me at the
time, but obviously it was python was

130
00:08:54.509 --> 00:09:00.299
missing a lot of tools and so
it was it was a bit daunting to

131
00:09:00.419 --> 00:09:03.539
start the process of building some of
those tools from scratch. Yeah, and

132
00:09:03.620 --> 00:09:07.860
you mentioned I plyth Numpine and Travis, and I suppose you know this is

133
00:09:07.899 --> 00:09:11.129
the time where John Hunter was working
a lot on Mat Lib and working with

134
00:09:11.210 --> 00:09:15.970
Fernando to incorporate it with with ipython. There was a lot of close collaboration.

135
00:09:16.129 --> 00:09:18.169
I suppose to speaks to the idea
of community as well. And did

136
00:09:18.210 --> 00:09:22.090
you find the scientific path in community
something that was also attractive? Yeah,

137
00:09:22.210 --> 00:09:26.840
well, I you know, I
didn't have much interaction with the community until

138
00:09:26.879 --> 00:09:30.759
much later, I think. The
first person, there's two people that I

139
00:09:30.919 --> 00:09:33.440
met from the Python Community who are
like my first point of contact with that

140
00:09:33.600 --> 00:09:37.000
world. So so one person is
Eric Jones, who is a founder of

141
00:09:37.080 --> 00:09:41.710
m thought, which is the original, like Python, sign of a computing

142
00:09:41.750 --> 00:09:46.110
company based on Austin, Texas,
and I also run the SCIPI conference.

143
00:09:46.230 --> 00:09:48.110
Yes, so they run SCIPIE and
and so and thought was doing a lot

144
00:09:48.149 --> 00:09:52.429
of it. was doing a lot
of consulting work in New York City with

145
00:09:52.509 --> 00:09:56.139
financial firms that were getting big into
python during that era, like training and

146
00:09:56.220 --> 00:10:01.259
kind of custom development. And I
got in touch with Eric sometime during two

147
00:10:01.299 --> 00:10:05.860
thousand and nine and and sort of
gave him like kind of the very first

148
00:10:05.899 --> 00:10:09.970
external demo of pandas. And this
was right around the time that we were

149
00:10:09.009 --> 00:10:13.889
getting ready to publish the pandas bits
on Pipi and so forth, kind of

150
00:10:13.929 --> 00:10:18.649
the first open source version of the
project. And then the second person I

151
00:10:18.730 --> 00:10:22.399
met was John Hunter, and himself
from from Matt Potleb. I met him

152
00:10:22.440 --> 00:10:26.240
in Chicago in January two thousand and
ten. You know, I at that

153
00:10:26.360 --> 00:10:30.000
point I was being around for like
how to engage with python world, having,

154
00:10:30.360 --> 00:10:33.039
you know, just open sourced pandas, and because Jack John was working,

155
00:10:33.600 --> 00:10:37.950
he worked for trade link and up
until his death in two thousand and

156
00:10:37.990 --> 00:10:43.230
twelve he was a quant they're having
been a neuroscientist and kind of had been

157
00:10:43.269 --> 00:10:46.629
building Matt Potlb for many years.
He he kind of took me under his

158
00:10:46.750 --> 00:10:50.100
wing as kind of I was his
he was my mentor for, you know,

159
00:10:50.179 --> 00:10:52.340
for a couple of years and kind
of helped me enter and get involved

160
00:10:52.340 --> 00:10:56.340
in the community, and so I
definitely feel that the I found found it

161
00:10:56.379 --> 00:11:01.179
a very warm and kind of inviting
community, very collaborative and collegial, and

162
00:11:01.340 --> 00:11:03.490
I think I was attracted to that, that feeling. You know, I

163
00:11:03.529 --> 00:11:07.649
didn't seem like a lot of people
competing with each other. It was really

164
00:11:07.850 --> 00:11:11.529
just a lot of pragmatic software developers
looking to kind of build tools that were

165
00:11:11.649 --> 00:11:16.409
useful and to help each other,
help each other succeed. Yeah, and

166
00:11:16.610 --> 00:11:18.480
you actually still get the sense of
that when you go to Sudipot in in

167
00:11:18.559 --> 00:11:22.879
Austin, Texas, every every July, every second you you you get still

168
00:11:22.919 --> 00:11:26.440
get a strong sense of community and
people just loving building the tools together.

169
00:11:26.440 --> 00:11:30.000
Yeah, yeah, totally, totally. I mean obviously the community has grown

170
00:11:30.039 --> 00:11:35.509
much bigger and I think the the
ratio of project developers, people working on

171
00:11:35.629 --> 00:11:41.429
the open source projects, to the
users, that ratio has certainly changed a

172
00:11:41.590 --> 00:11:43.990
lot and that there are a lot
more users now than there are developers.

173
00:11:45.110 --> 00:11:48.340
You know, I think the very
first SCIPI conference was probably the majority of

174
00:11:48.419 --> 00:11:52.460
people there were people who are the
developers of open source projects. But,

175
00:11:52.860 --> 00:11:54.860
you know, still I think it's
a great community and I think that's that's

176
00:11:54.899 --> 00:11:58.379
helping kind of continue to bring people
into the ecosystem. Yeah, and I

177
00:11:58.419 --> 00:12:03.049
actually I had Brian Granger on the
podcast recently and we discussed those. You

178
00:12:03.090 --> 00:12:05.409
know, several people are discussing at
the moment that when now kind of entering

179
00:12:05.450 --> 00:12:11.610
a phase transition from having individual users
spread across ords and spread across the globe

180
00:12:11.889 --> 00:12:16.200
of a lot of open source packages
to actually having large scale institutional adoption right.

181
00:12:16.240 --> 00:12:20.440
Yeah, yeah, for sure.
And I'm wondering in terms of pandas

182
00:12:20.440 --> 00:12:24.639
starting off as a project, I'm
under the impression it was started as a

183
00:12:24.679 --> 00:12:28.000
tool to be used in finance.
Is that the case? Yeah, I

184
00:12:28.080 --> 00:12:30.710
mean it was focused. So if
you look, you can go back and

185
00:12:30.750 --> 00:12:35.070
download pan to zero point one,
which was published a PIPI and December two

186
00:12:35.110 --> 00:12:39.190
thousand and nine, and see what
was in the library and compared with now,

187
00:12:39.990 --> 00:12:43.980
the functionality was a lot more geared
towards time series data and the kinds

188
00:12:45.019 --> 00:12:48.340
of problems that we were dealing with
back at a Qr. I wouldn't say

189
00:12:48.340 --> 00:12:52.940
that it necessarily is finance specific.
Is Very General data manipulation. As a

190
00:12:52.980 --> 00:12:56.299
pretty small project back then, but
it was just about dealing with tabular data,

191
00:12:56.539 --> 00:13:01.809
dealing with messy data, data munging, missing kind of data alignment,

192
00:13:01.850 --> 00:13:05.690
essentially kind of all those like kind
of really basic wrangling and data integration problems.

193
00:13:05.730 --> 00:13:09.809
It wasn't really until two thousand and
eleven, two thousand and twelve,

194
00:13:09.970 --> 00:13:13.080
that the project got built. Like
I built the project out and created a

195
00:13:13.159 --> 00:13:20.399
more comprehensive set relational algebra facilities.
Like didn't have complete joints, like all

196
00:13:20.440 --> 00:13:24.440
the different kinds of basic joins until
two thousand and eleven. So it's features

197
00:13:24.519 --> 00:13:28.389
that was certainly skewed by the use
cases that we had in front of us

198
00:13:28.590 --> 00:13:31.509
back at a q are. And
how did you get the project off the

199
00:13:31.549 --> 00:13:35.509
ground? I know that's a relatively
ill formed question, but just in terms

200
00:13:35.509 --> 00:13:41.139
of hours and people land resources.
Well, you know, you smelt the

201
00:13:41.220 --> 00:13:43.539
metal, or you kind of.
You forge two weapons. You have to

202
00:13:43.580 --> 00:13:48.940
get the the crucible really, really
hot. So we open source to project

203
00:13:48.019 --> 00:13:52.179
at the end of two thousand and
nine and I think we had deliberated kind

204
00:13:52.220 --> 00:13:54.340
of the whether or not to open
source at all for about six months or

205
00:13:54.340 --> 00:14:00.169
so and ultimately powers that be decided
that we would open source pandas and sort

206
00:14:00.210 --> 00:14:03.009
of see what would happen. I
gave my very first talk about pandas if

207
00:14:03.049 --> 00:14:09.009
you can still find online, at
icon two thousand and ten and in Atlanta,

208
00:14:09.090 --> 00:14:11.320
and it was about using the subject
of the talk was about using python

209
00:14:11.440 --> 00:14:16.759
and quantitative finance. But the the
project didn't really go anywhere after that.

210
00:14:16.200 --> 00:14:20.000
So it was on hosted on Google
Code. This was, you know,

211
00:14:20.279 --> 00:14:22.759
get have existed, but it was
kind of a ruby thing at that time.

212
00:14:22.519 --> 00:14:26.629
And I left Qr to go back
to Grad School. I went to

213
00:14:26.669 --> 00:14:31.830
Duke to start a PhD and Statistics, statistical science it's called there, and

214
00:14:31.470 --> 00:14:35.190
I continued to do a little bit
of contract work developing pandas for for a

215
00:14:35.190 --> 00:14:41.019
Qr and somewhere, I don't.
I think the catalyst for me was an

216
00:14:41.220 --> 00:14:45.379
early two thousand and eleven. I
started to get contacted by more companies that

217
00:14:45.460 --> 00:14:50.100
were exploring use python for data analysis
use cases and they had seen my talk

218
00:14:50.139 --> 00:14:56.090
at Picon and were interested in getting
my perspective on statistical computing and I just

219
00:14:56.210 --> 00:15:01.250
had this feeling that the ecosystem was
facing a sort of existential crisis about whether

220
00:15:01.250 --> 00:15:05.490
or not it was going to become
truly relevant for doing statistics. And it

221
00:15:05.809 --> 00:15:11.000
was clear to me that pandas was
promising but really had not reached a level

222
00:15:11.039 --> 00:15:16.919
of functional completeness or usefulness to be
the foundation of a statistical computing ecosystem and

223
00:15:18.000 --> 00:15:20.870
python. And so I guess I
felt that feeling so strongly that I,

224
00:15:22.389 --> 00:15:24.429
you know, I sort of had
like an epiphany where it wasn't quite like,

225
00:15:24.669 --> 00:15:28.309
you know, shouting Eureka, jumping
out of the bathtub, but I

226
00:15:28.429 --> 00:15:31.309
emailed my advisor and said, Hey, I would like to take take a

227
00:15:31.350 --> 00:15:35.659
year off from my phd and go
explore this python programming stuff and we'll see

228
00:15:35.659 --> 00:15:39.019
how it goes. And I had
some money saved from my first job and

229
00:15:39.139 --> 00:15:43.779
I moved back to New York into
a tiny apartment in the East village which

230
00:15:43.820 --> 00:15:46.940
had mice and stuff, really not
the best place I've ever lived, but

231
00:15:46.220 --> 00:15:50.929
I essentially was like I'm just going
to work full time on pandas for a

232
00:15:50.009 --> 00:15:54.730
while and build it out and see
see what happens. And I think that's

233
00:15:54.769 --> 00:15:58.610
when, as soon as started kind
of socializing the functionality of pandas and filling

234
00:15:58.690 --> 00:16:03.009
in future gaps, you know,
implementing joins and fixing some of the internal

235
00:16:03.759 --> 00:16:07.720
issues. Of course I created other
internal problems, but there were some there

236
00:16:07.799 --> 00:16:11.519
were definitely some design problems in the
early versions of pandas. That got fixed

237
00:16:12.080 --> 00:16:15.240
and the summer of two thousand and
eleven. But as soon as hands could

238
00:16:15.279 --> 00:16:21.029
read CSP files pretty reliably and could
do joins and a lot of the basic

239
00:16:21.110 --> 00:16:23.789
stuff that you need to be productive
working with multiple data sets, I think

240
00:16:23.830 --> 00:16:26.909
that's when it started to catch people's
eye toward the end of two thousand and

241
00:16:26.909 --> 00:16:30.269
eleven and starts to take off the
ground. So around the same time I

242
00:16:30.509 --> 00:16:36.259
catched the idea of a data analysis
book and Python to o'riley and they agreed

243
00:16:36.340 --> 00:16:38.059
to do a book which, thinking
back on it, was a bit risky

244
00:16:38.100 --> 00:16:41.139
because you know, who knows what
would have become of handas. Was Not

245
00:16:41.179 --> 00:16:45.340
at all a prompt, you know, not obviously going to be successful back

246
00:16:45.379 --> 00:16:48.210
in two thousand and eleven. So
they decided to take a bet, and

247
00:16:48.409 --> 00:16:51.409
so much so that, you know, I asked them later why they didn't

248
00:16:51.409 --> 00:16:53.570
put a panda on the cover,
but they said, well, we're saving

249
00:16:53.570 --> 00:16:56.690
the panda for something really big,
and so it wasn't even clear then that

250
00:16:56.809 --> 00:17:02.399
python and pandas and everything was going
to be a popular thing. So kind

251
00:17:02.440 --> 00:17:10.519
of it's important to kind of have
that perspective. We jump right back into

252
00:17:10.640 --> 00:17:15.750
interview with this after a short segment. Now it's time for a segment cold

253
00:17:15.910 --> 00:17:19.630
data, signs, best practices.
I'm here with Ben Scranka, an independent

254
00:17:19.710 --> 00:17:23.869
data science consultant. Hi Been Hey, Hugo, it's great to be back.

255
00:17:25.990 --> 00:17:29.710
So what are we discussing this week? Well, do you need to

256
00:17:29.829 --> 00:17:33.900
explain or to predict? What do
you mean? Breeman has this must read

257
00:17:33.900 --> 00:17:38.460
paper called statistical modeling, the two
cultures. He discusses that there are two

258
00:17:38.500 --> 00:17:45.210
modeling approaches, the ALGORITHMIC modeling culture
and the data modeling culture. By algorithmic

259
00:17:45.250 --> 00:17:48.609
modeling, he means that the machine
learning approach, which has largely been developed

260
00:17:48.609 --> 00:17:53.369
by computer scientists. He contrasts this
with the data modeling culture, which used

261
00:17:53.369 --> 00:18:00.920
data as stochastic and worries about modeling
the data generating process. Economists and statisticians

262
00:18:00.240 --> 00:18:06.279
tend to fall into the latter camp. Breman argues that machine learning is dominant

263
00:18:06.279 --> 00:18:10.680
in terms of performance, accuracy and
ease of use. Is that? Oh,

264
00:18:10.759 --> 00:18:14.109
there is to it? If only
life were that easy. For Large

265
00:18:14.190 --> 00:18:18.349
class of problems, he is right, and traditional data modelers are rushing to

266
00:18:18.390 --> 00:18:22.910
adopt these methods. But the ALGORITHMIC
approach fails for another large class of problems,

267
00:18:23.269 --> 00:18:27.220
those where you need to explain the
problem, Ie to understand a causal

268
00:18:27.259 --> 00:18:32.819
connection. Gal each Mueli has done
some great research in this regard. She

269
00:18:33.019 --> 00:18:36.619
points out that you need to choose
your method and approach based on whether you

270
00:18:36.740 --> 00:18:41.690
want quote to explain or to predict. I see so use machine learning to

271
00:18:41.730 --> 00:18:47.609
predict and statistics or econometry to explain. That is pretty much the case.

272
00:18:47.970 --> 00:18:51.890
If your problems focus on prediction,
then ml is the place to start and

273
00:18:52.049 --> 00:18:56.079
is incredibly powerful. Both shallow and
deep learning models are producing incredible results for

274
00:18:56.200 --> 00:19:02.680
predictive and perceptual problems. But often
we need to understand the drivers that affect

275
00:19:02.680 --> 00:19:06.519
a business problem. In this case, we must run an experiment or perform

276
00:19:06.640 --> 00:19:11.230
causal regression analysis to eliminate bias in
our estimated effects sizes. So how do

277
00:19:11.269 --> 00:19:15.150
you know which to use? An
experiment is the gold standard because of the

278
00:19:15.230 --> 00:19:18.670
magic of random assignment. But you
may not be able to run an experiment.

279
00:19:18.750 --> 00:19:22.509
It could be too expensive, take
too long or be dangerous, such

280
00:19:22.549 --> 00:19:26.539
as something a human subjects committee would
not allow. If you can't get experimental

281
00:19:26.579 --> 00:19:32.740
data, then you must use observational
data and perform a causal regression analysis which

282
00:19:32.740 --> 00:19:37.539
captures the key features of the data
to eliminate bias. So tell me more

283
00:19:37.579 --> 00:19:41.250
about bias. We really need to
talk about indigenety in depth at some point.

284
00:19:41.569 --> 00:19:45.970
In dodgeneity is all the different ways
bad stuff can be hidden in your

285
00:19:45.970 --> 00:19:49.809
er term, which bias as results
such as sample selection, emitted variable bias,

286
00:19:51.130 --> 00:19:55.799
simultaneity and measurement error. For example, if you want to reduce churn,

287
00:19:56.160 --> 00:20:00.279
you need to build an explanatory model
to determine how different levers affect churn.

288
00:20:00.680 --> 00:20:06.119
In this case, the data is
censored because some customers remain customers past

289
00:20:06.160 --> 00:20:10.230
the end of our study. We
never observe them churning. Consequently, we

290
00:20:10.750 --> 00:20:15.390
need to model the censoring process.
Machine learning will have a difficult time learning

291
00:20:15.390 --> 00:20:19.269
how to compensate for censoring. A
classical survival model, as censoring baked into

292
00:20:19.269 --> 00:20:23.859
the hypothesis base will almost surely work
better. Finally, I should add that

293
00:20:23.940 --> 00:20:29.779
including indigenous features in a machine learning
model can cause all kinds of problems.

294
00:20:30.180 --> 00:20:33.059
If you treat machine learning algorithms as
black boxes, you may regret it if

295
00:20:33.099 --> 00:20:37.329
your model needs to be retrained regularly
you may have this problem and all the

296
00:20:37.450 --> 00:20:42.170
other benefits to the data muddling approach. Another huge benefit of the data modeling

297
00:20:42.170 --> 00:20:48.529
approach is that you can do inference
either frequent disturbation. That means you can

298
00:20:48.650 --> 00:20:52.319
formally stay hypotheses about which levers matter
and then test how likely it is that

299
00:20:52.440 --> 00:20:57.440
the data supports your hypothesis. Thanks
been for explaining why prediction isn't a silver

300
00:20:57.559 --> 00:21:07.630
bullet and why we also need models
to explain now data after that, until

301
00:21:07.869 --> 00:21:14.349
time to jump back to do about
chat with West mckinney. So, when

302
00:21:14.990 --> 00:21:18.299
living in the East village supporting yourself
to build out the package, did you

303
00:21:18.420 --> 00:21:23.420
have any inkling that it would achieve
the growth and wide Scot adoption that it

304
00:21:23.500 --> 00:21:27.619
has? Not? Not Really,
I mean I believed that. I mean

305
00:21:27.700 --> 00:21:33.609
obviously I had the belief that python
ecosystem had a lot of potential and that

306
00:21:33.289 --> 00:21:40.369
projects like pandas were necessary to help
the language and the community realize the potential.

307
00:21:40.609 --> 00:21:44.849
Like I think there was a lot
of computational firepower in the Numpie world

308
00:21:44.890 --> 00:21:49.240
and all the tooling python and tools
for interoperability with native code and so so

309
00:21:49.880 --> 00:21:52.519
I just wanted to help realize that
potential, but I didn't really have a

310
00:21:52.599 --> 00:21:57.839
sense of where go. There were
some other significant kind of confluence of things

311
00:21:57.920 --> 00:22:03.509
that happened, particularly when you consider
the development of stats models and psychic learn,

312
00:22:03.670 --> 00:22:07.789
which brought meaningful analytical functionality to python. Like, I think if pandas

313
00:22:08.069 --> 00:22:11.150
you know, really the big thing
that may pandas successful was the fact that

314
00:22:11.829 --> 00:22:17.380
it could read CSV files reliably,
and so it became like a first port

315
00:22:17.420 --> 00:22:22.539
of entry for for data into python
and for kind of data cleaning and data

316
00:22:22.579 --> 00:22:26.779
preparation. And so if you wanted
to do machine learning and psychic learn,

317
00:22:26.900 --> 00:22:30.490
you wanted to use stats models for
statistics and econometrics, you needed to clean

318
00:22:30.529 --> 00:22:34.210
data first, and so using pandas
was the obvious choice for that. But

319
00:22:34.329 --> 00:22:37.410
it yeah, it wasn't. It
wasn't obvious. And you know, I

320
00:22:37.730 --> 00:22:41.170
cruated a couple of my former colleagues
from Qr, am Klein and Changsha,

321
00:22:41.369 --> 00:22:47.400
to work with me on pandas and
we explored starting a company around financial analytics

322
00:22:47.480 --> 00:22:51.720
and Python powered by pandas, but
we were focused on building out pandas as

323
00:22:51.720 --> 00:22:55.400
an open source project kind of while
we explored kind of that startup idea,

324
00:22:55.680 --> 00:22:59.509
ultimately we didn't pursue that start up, but we it was clear that by

325
00:22:59.630 --> 00:23:03.509
mid two thousand and twelve that we'd
sort of crossed the critical horizon of people

326
00:23:03.789 --> 00:23:08.630
being interested in python as a language
for data analysis. And since then you've

327
00:23:08.710 --> 00:23:15.059
found certain institutions which have employed you
in order to work on pandas. Right.

328
00:23:15.420 --> 00:23:18.460
I wouldn't say that outside of my
time at Aqr when I was building

329
00:23:18.660 --> 00:23:22.299
pandas kind of initially, I've never
been employed directly to work on pandas.

330
00:23:22.380 --> 00:23:26.099
I started a company called Data Pad
with Changsha and so it was a venture

331
00:23:26.099 --> 00:23:30.890
back company and we were building a
visual analytics product that was powered by pandas

332
00:23:30.930 --> 00:23:33.329
and other python the data padas acquired
by cloud are at the end of two

333
00:23:33.329 --> 00:23:37.690
thousand and fourteen, and so Chong
and I landed there to work on and

334
00:23:37.809 --> 00:23:44.359
my my role at cloud Arrow was
to look holistically at the big data world

335
00:23:44.599 --> 00:23:49.640
and figure out how to forge a
better path for python and data science tools

336
00:23:49.680 --> 00:23:53.599
in general in the context of the
big data world. That's the hoodoop ecosystem

337
00:23:55.200 --> 00:23:59.990
and spark and kind of all the
technology that which largely Java based, which

338
00:23:59.990 --> 00:24:03.309
had been developed since, you know, two thousand and six or two thousand

339
00:24:03.349 --> 00:24:07.670
and eight and so but I wasn't
working on Pantis in particular at that point

340
00:24:07.869 --> 00:24:12.259
and I sort of had taken stock
of the structural and kind of infrastructural problems

341
00:24:12.619 --> 00:24:15.339
that pant has had. I gave
a talk at the end of two thousand

342
00:24:15.380 --> 00:24:19.019
and thirteen at Pie data and New
York on the title of the talk was

343
00:24:19.460 --> 00:24:25.130
practical medium data, analytics and Python
and the subtitle of the talk was ten

344
00:24:25.250 --> 00:24:27.849
things I hate about pandas, I
remember. So I had this kind in

345
00:24:27.890 --> 00:24:34.650
the background, this feeling that pandas
was built on a fantastic platform for scientific

346
00:24:34.690 --> 00:24:41.559
computing and numerical computing. So if
you're doing particle physics or HPC work in

347
00:24:41.680 --> 00:24:45.279
a national lab with a super computer, you know python is really great and

348
00:24:45.359 --> 00:24:49.920
that's how the ecosystem developed in the
late s early but for statistical computing and

349
00:24:51.160 --> 00:24:56.150
big data and analytics, fact that
like strings and categorical data wasn't a first

350
00:24:56.150 --> 00:25:00.390
class citizen in that world made things
a Lott harder. Missing data was not

351
00:25:00.430 --> 00:25:03.829
a first class citizen, and so
there were a lot of problems that had

352
00:25:03.869 --> 00:25:07.140
accumulated, and so at that point
I started to look beyond panda says it

353
00:25:07.180 --> 00:25:11.660
was implemented then into kind of how
we could build technology to advance the whole

354
00:25:11.660 --> 00:25:17.339
ecosystem and beyond the python world as
well. So I think a through line

355
00:25:17.339 --> 00:25:21.809
in this is really encapsulated by a
statement you made earlier, which is you

356
00:25:21.849 --> 00:25:26.049
want to build technologies and tools that
are truly relevant for doing statistics or working

357
00:25:26.130 --> 00:25:29.490
with data, and I know as
a tool builder, you're committed to developing

358
00:25:29.529 --> 00:25:33.650
human interfaces to data to make individuals
more productive, and I think that actually

359
00:25:33.650 --> 00:25:36.440
provides a really nice segway into a
lot of what you're thinking about now,

360
00:25:36.559 --> 00:25:38.880
in particular the Apache Arrow project.
So I'm wondering if you can tell me

361
00:25:38.920 --> 00:25:45.160
about Apocciaro and how you feel it
con facilitype data science work. Yeah,

362
00:25:45.279 --> 00:25:48.240
so I got involved in in what
became the Apach Arrow project, you know,

363
00:25:48.279 --> 00:25:52.309
as part of my work at cloud
era. So one problem that had

364
00:25:52.390 --> 00:25:56.750
plagued me as a python programmer was
the fact that when you arrive at foreign

365
00:25:56.829 --> 00:26:02.430
data and foreign systems that you want
to plug into, whether those are other

366
00:26:02.470 --> 00:26:07.059
kinds of you know, ways of
storing data or accessing data or accessing computational

367
00:26:07.099 --> 00:26:11.700
systems, that we were in a
position of having to build custom data connectors

368
00:26:12.339 --> 00:26:17.740
for python, for pandas or kind
of whatever python library you're using, and

369
00:26:17.940 --> 00:26:22.769
so I felt that we were losing
a lot of energy to building custom connectors

370
00:26:22.809 --> 00:26:26.450
into all of these different things.
And this problem isn't unique to python.

371
00:26:26.769 --> 00:26:30.930
So if you look at all of
the the number of like different pairwise adapters

372
00:26:30.130 --> 00:26:37.279
that are available to convert between one
data formatted another or serialized data from one

373
00:26:37.400 --> 00:26:41.720
programming language to another programming language.
So sharing data was something that had caused

374
00:26:41.720 --> 00:26:45.519
me a lot of pain, and
also sharing code and Algorithms was a big

375
00:26:45.599 --> 00:26:49.750
problem. So the way that pan
has is implemented internally. It has its

376
00:26:49.789 --> 00:26:55.750
own custom way of representing data that's
layered on top of numpire rays. But

377
00:26:55.910 --> 00:27:00.900
we had to essentially reimplement all of
our own algorithms and data access layers from

378
00:27:00.900 --> 00:27:04.859
scratch. You know, we'd implemented
our own CSFEE reader, our own interfaces

379
00:27:04.900 --> 00:27:10.819
to hdfive files, our own interfaces
to Jason Data. We have pretty large

380
00:27:10.940 --> 00:27:15.289
libraries of code and pandas for doing
in memory analytics, aggregating arrays, performing

381
00:27:15.369 --> 00:27:19.609
group by operations, and if you
look across other parts of the big data

382
00:27:19.650 --> 00:27:22.730
world you see the same kinds of
things. Implemented in many different ways in

383
00:27:22.809 --> 00:27:26.890
many different programming languages. In are
you have the same thing, many of

384
00:27:26.930 --> 00:27:30.839
the same things implemented in our so
it I was kind of trying to make

385
00:27:30.960 --> 00:27:37.200
sense of all of that energy loss
to sharing data and sharing code and thinking

386
00:27:37.240 --> 00:27:41.920
about how I could help enable the
data world to become a lot less fragmented

387
00:27:41.279 --> 00:27:45.549
and people building systems who are people
like me, who build tools for people,

388
00:27:45.990 --> 00:27:48.789
how to make people like me who
are building tools a lot more productive

389
00:27:48.829 --> 00:27:52.670
and able to build that and more
efficient data processing tools in the future.

390
00:27:52.950 --> 00:27:56.309
And so this was just kind of
feelings that I had, and so I

391
00:27:56.470 --> 00:28:00.299
started to poke around cloud era and
see if other people felt the same way.

392
00:28:00.460 --> 00:28:03.740
And so I was working with folks
on the IMPOLIT team, people like

393
00:28:03.819 --> 00:28:07.420
Marcel Pornacker, who started the IMPAULA
project, Todd Lipcon, who started the

394
00:28:07.740 --> 00:28:11.859
Patchi Kudu project. It's now a
patchy, and Paula joined the Apache Foundation.

395
00:28:12.529 --> 00:28:15.970
So you know there are a lot
of people a cloud era that essentially

396
00:28:15.970 --> 00:28:18.890
agreed with me, and we sort
of thought about like what kind of technology

397
00:28:18.930 --> 00:28:25.009
we could build the help improve interoperability
and we sort of centered on the problem

398
00:28:25.130 --> 00:28:30.119
of representing data frames and tabular data. And as we kind of looked outside

399
00:28:30.119 --> 00:28:33.640
of cloud era, we saw that
there were other groups of developers who concurrently,

400
00:28:33.680 --> 00:28:37.839
we're thinking about the exact same problem. So we bumped into folks from

401
00:28:37.839 --> 00:28:41.589
the Apache drill project, which is
a sequel on hadop system, and they

402
00:28:41.630 --> 00:28:45.869
were also thinking about the tabular data
interoperability problem, like how can we move

403
00:28:45.910 --> 00:28:52.269
around tabular data sets and reuse algorithms
and code and data without so much conversion

404
00:28:52.390 --> 00:28:55.940
and and energy loss, and so
very quickly, you know, we got

405
00:28:56.019 --> 00:29:00.420
two thousand and twenty five people in
the room representing twelve or thirteen open source

406
00:29:00.500 --> 00:29:06.539
projects, with a general consensus that
we should build some technology to proverbially,

407
00:29:06.700 --> 00:29:08.299
you know, tie the room together. That became a patchy Arrow, but

408
00:29:08.380 --> 00:29:12.009
it took all of two thousand and
fifteen to put the project together. Now,

409
00:29:12.250 --> 00:29:17.210
how is all this relevant to data
science? Well, what Arrow project

410
00:29:17.289 --> 00:29:23.400
provides is a way of representing data
and memory that is language agnostic and standardized

411
00:29:23.440 --> 00:29:27.279
and portable. So you can think
of it as being like a language independent

412
00:29:27.319 --> 00:29:33.440
data frame. So if you create
Arrow based data frames and python, you

413
00:29:33.480 --> 00:29:37.799
can share are of them with any
system, whether that's written in see or

414
00:29:37.799 --> 00:29:41.549
see Pos plus or javascript or Java
or rust or go. As long as

415
00:29:41.589 --> 00:29:45.670
they implement the Aero Column Reformat,
they can interact with that data without having

416
00:29:45.710 --> 00:29:49.869
to convert it or serialize to some
kind of intermediate representation like you usually have.

417
00:29:51.230 --> 00:29:55.339
So the goal of the project,
in addition of providing high quality libraries

418
00:29:55.980 --> 00:30:02.059
for building data science tools and building
databases, is also to improve the portability

419
00:30:02.099 --> 00:30:06.740
of code and data between languages.
Outside of kind of the interoperability side of

420
00:30:06.859 --> 00:30:11.569
the project, there's also the goal, within the walls of a particular data

421
00:30:11.609 --> 00:30:18.009
processing system, to provide a platform
of algorithms in and tools for memory management

422
00:30:18.089 --> 00:30:22.559
and data access that can accelerate large
scale data processing. So we wanted the

423
00:30:22.839 --> 00:30:29.559
Arrow column reformat to support working with
much larger quantities of data, the single

424
00:30:29.640 --> 00:30:33.119
note scale data, that is,
particularly data that does not fit into memory.

425
00:30:33.640 --> 00:30:34.869
And I love this idea of,
you know, tying the room together,

426
00:30:34.910 --> 00:30:37.390
as you put it, because essentially
it speaks the idea of kind of

427
00:30:37.390 --> 00:30:41.390
breaking down the walls between all these
silos that exist as well. Right.

428
00:30:41.029 --> 00:30:44.910
Yeah, yeah, I know.
I think if you look across and just

429
00:30:45.029 --> 00:30:48.630
within the data science world, I
mean, even though functionally we're solving many

430
00:30:48.670 --> 00:30:52.700
of the same problems like there's very
little collaboration that happens between the communities,

431
00:30:53.099 --> 00:30:57.220
whether collaborating at the software design level
or at the code level, and as

432
00:30:57.259 --> 00:31:02.140
a result, people point fingers and
accused people of reinventing wheels or like not

433
00:31:02.220 --> 00:31:07.529
wanting to collaborate. But you know, really it's if your data is different

434
00:31:07.730 --> 00:31:11.890
in a memory, there's just no
basis for codesharing in most cases, and

435
00:31:12.049 --> 00:31:18.089
so the desire to create an open
standard for data frames is just if you

436
00:31:18.130 --> 00:31:22.880
want to share code, it's essential
you have to standardize the representation in Ram

437
00:31:22.920 --> 00:31:26.359
or on the GPU or essentially at
the bite or the bit level. Agreeing

438
00:31:26.440 --> 00:31:30.000
on how what the data looks like
once you load it off disc or at

439
00:31:30.039 --> 00:31:34.990
once you parse it out of a
CSV file is the basis of collaboration amongst

440
00:31:36.109 --> 00:31:41.269
multiple programming languages or amongst different data
science languages that are ultimately based in see

441
00:31:41.309 --> 00:31:45.950
or see Pul's plus. Yeah,
I remember. Actually phenando Perez spoke to

442
00:31:45.029 --> 00:31:48.779
this as well, in he cannote
way. You also kennote to the and

443
00:31:48.859 --> 00:31:52.339
overall Jupitericon, saying we welcome so
many contributions, but we need to agree

444
00:31:52.380 --> 00:31:56.980
on some things right, and these
are certain things that we will agreed upon.

445
00:31:56.019 --> 00:31:59.740
So if you're going to contribute,
let's build on these particular things.

446
00:31:59.779 --> 00:32:02.529
Right, right, yeah, I
know. I think the Jupiter project certainly

447
00:32:02.569 --> 00:32:08.289
socialize this idea of open standards by
developing the kernel protocol providing a way.

448
00:32:08.289 --> 00:32:12.650
It's like, you know, here's
the abstract notion of like a computational notebook

449
00:32:12.650 --> 00:32:15.359
and here's how, if you want
to build a kernel add a new language

450
00:32:15.359 --> 00:32:17.960
to the Jupiter ecosystem, you know, here's how you do it, and

451
00:32:19.279 --> 00:32:21.960
you know that certainly has played out, you know, beautifully with you know,

452
00:32:22.000 --> 00:32:25.039
I think it's like over forty languages
have kernel implementation for Jupiter. But

453
00:32:25.559 --> 00:32:30.150
you know, I think in general
I think people are appreciating more the value

454
00:32:30.470 --> 00:32:35.390
of having open standards where that our
community developed and that are developed on the

455
00:32:35.430 --> 00:32:38.269
basis of consensus and where there's just
like kind of broad buy in. It's

456
00:32:38.349 --> 00:32:44.109
not like one developer, one sort
of isolated group of people building some technology

457
00:32:44.150 --> 00:32:46.660
and then trying to get people to
adopt it. So I think Jupiter is

458
00:32:46.740 --> 00:32:50.539
somewhat is unique in the sense that
it's started out in the python world,

459
00:32:50.619 --> 00:32:52.059
but I think it's there. You
know, they set out with the goal

460
00:32:52.140 --> 00:32:58.259
of embracing a much broader community of
users and developers, and that's played out

461
00:32:58.339 --> 00:33:01.130
in really exciting ways. I really
like the descriptions you got, even kind

462
00:33:01.170 --> 00:33:05.369
of the inspiration behind the Arrow project
in particular. You know, the need

463
00:33:05.450 --> 00:33:08.410
for interrupt uperability, the Importus of
these portable dieta frames. I don't want

464
00:33:08.450 --> 00:33:12.289
to go to fought down the rabbit
hole. I can't really help myself,

465
00:33:12.329 --> 00:33:15.359
though, like you, just make
just a bit more to kind of your

466
00:33:15.440 --> 00:33:17.720
thoughts behind the challenge of working the
big diet a limit. I mean,

467
00:33:17.759 --> 00:33:21.680
for example, that we have computers
and how drives that can store a lot

468
00:33:21.720 --> 00:33:24.440
of stuff, but we don't actually
have languages that can interact with unless we

469
00:33:24.480 --> 00:33:29.509
parallelize it right, right, right. So a common thing that I've heard

470
00:33:29.509 --> 00:33:32.029
over the years from people will say
West like I just want to write pandas

471
00:33:32.069 --> 00:33:36.390
code, but I want it to
work with big data. So it's a

472
00:33:36.470 --> 00:33:39.549
complicated thing because the way that a
lot of these libraries are designed, the

473
00:33:39.589 --> 00:33:44.299
way that pandas is designed, and
a lot of libraries that are similar to

474
00:33:44.420 --> 00:33:49.460
pandas. It's the implementation and kind
of the evaluate like the computational model,

475
00:33:49.980 --> 00:33:54.259
like when computation happens, like what
are the semantics of the code that you're

476
00:33:54.299 --> 00:33:58.849
writing? There's a lot of built
in assumptions around, like the idea that

477
00:33:59.009 --> 00:34:01.730
data fits in memory, Marie,
and that you know when you write a

478
00:34:01.890 --> 00:34:07.289
plus B, that like a plus
B is evaluated immediately and materialized in memory.

479
00:34:07.569 --> 00:34:12.000
And so if you want to scale
out, kind of scale up commute

480
00:34:12.079 --> 00:34:15.880
computing to data frame libraries, you
essentially have to re architect around the idea

481
00:34:15.960 --> 00:34:22.199
of deferred evaluation and essentially defining kind
of a rich enough Algebra or kind of

482
00:34:22.320 --> 00:34:30.070
intermediate representation of analytical computation where you
can actually use a proper query engine or

483
00:34:30.230 --> 00:34:37.389
query planner to execute operations. And
so really what is needed is to make

484
00:34:37.789 --> 00:34:43.019
libraries like panda's internally more like analytic
databases. And if you look at all

485
00:34:43.099 --> 00:34:47.780
the innovation that has happened in the
analytic database world over the last twenty years,

486
00:34:47.820 --> 00:34:52.579
of column or databases and things that
have happened in the big data world,

487
00:34:52.659 --> 00:34:57.889
you know very little of that,
of that innovation in scalable data processing

488
00:34:57.929 --> 00:35:01.050
has made its way into the hands
of data scientists. So really we know.

489
00:35:01.409 --> 00:35:06.769
One of my major goals with working
my involvement in the Arrow project is

490
00:35:06.929 --> 00:35:13.360
to provide the basis for collaboration between
the database and analytic database world, in

491
00:35:13.400 --> 00:35:16.440
the data science world, which it
is not something that's happened before. Ultimately,

492
00:35:16.519 --> 00:35:22.750
the goal is to create like an
embedded analytic database that is language independent

493
00:35:22.829 --> 00:35:24.949
and can be used in python,
can be used in our that can work

494
00:35:25.030 --> 00:35:29.789
with much larger quantities of data.
But it's going to take like a different

495
00:35:29.829 --> 00:35:35.070
approach in terms of the user API, because I think the at this idea

496
00:35:35.150 --> 00:35:40.780
of like magically retro fitting pandas,
or essentially retro fitting pandas with the ability

497
00:35:40.900 --> 00:35:45.900
to work with hundreds of gigabytes a
data or tearabytes of data, I hate

498
00:35:45.900 --> 00:35:46.699
to say, it's a little bit
of a pipe tream. I think it's

499
00:35:46.699 --> 00:35:51.730
going to require some breaking changes and
some kind of some different approaches to the

500
00:35:51.889 --> 00:35:53.889
problem. That's not to say the
Pantis is going away. I mean Pandis

501
00:35:54.010 --> 00:35:59.610
is not going anywhere and I think
is certainly is occupying the the sweet spot

502
00:35:59.690 --> 00:36:04.559
of being like the ultimate Swiss army
knife for data sets under a few GIGABYTES.

503
00:36:05.639 --> 00:36:10.320
So does this conversation relate to the
murmurings we've heard of potential pendis to

504
00:36:10.519 --> 00:36:15.119
in the plotline. Yeah, so
we I think at the end of two

505
00:36:15.119 --> 00:36:21.110
thousand and fifteen I started a discussion
in the Pantas community and so just Fyi

506
00:36:21.230 --> 00:36:23.309
I think people, you know,
are often thanking my seat to see people

507
00:36:23.349 --> 00:36:25.269
out in the community, the like
West, you know, thanks so much

508
00:36:25.269 --> 00:36:28.710
for Pandus. I have to remind
them, like to go out of your

509
00:36:28.750 --> 00:36:31.750
way and thank Jeffrey back and yours
fan in Bosh and Philip cloud and Tom

510
00:36:31.860 --> 00:36:37.019
Ausburger and the you know, the
other Pantas court developers that have really been

511
00:36:37.099 --> 00:36:40.619
driving the project forward over the last
five years. I haven't been very involved

512
00:36:40.659 --> 00:36:44.579
in the day to day development since
some time in two thousand and thirteen,

513
00:36:45.099 --> 00:36:47.289
but at the end of two thousand
and fifteen I started spending some more time

514
00:36:47.409 --> 00:36:51.889
with the pandas developers said, been
building this project for it's seven year.

515
00:36:52.050 --> 00:36:53.769
You know, it's a little over
seven years old. The codebase. Are

516
00:36:53.849 --> 00:36:57.369
there things that we would like to
fix, like what are we going to

517
00:36:57.530 --> 00:37:01.119
do about the performance and memory use
and scalability issues? I can't remember,

518
00:37:01.119 --> 00:37:04.840
I don't think. At that point
I don't know that Dask data frame existed

519
00:37:05.320 --> 00:37:09.039
and so dask has provided kind of
an alternative route to scaling pandas by using

520
00:37:09.119 --> 00:37:15.230
pandas kind of as is, but
essentially reimplementing Pandus operations using a dask computation

521
00:37:15.349 --> 00:37:20.349
craft, but looking at the kind
of single node scale, kind of the

522
00:37:20.469 --> 00:37:22.710
in memory side of pandas. We
sort of looked at, you know,

523
00:37:22.789 --> 00:37:27.389
what we'd like to fix about the
pandas internals and that was what we know

524
00:37:27.469 --> 00:37:31.340
we described as the kind of pandas
to initiatives. And around that time we

525
00:37:31.460 --> 00:37:37.380
were just getting ready to kick off
the Apache Arrow project, and so I

526
00:37:37.420 --> 00:37:40.099
wouldn't say that we, you know, we reached kind of like a fully

527
00:37:40.179 --> 00:37:43.860
baked, you know, game plan
in terms of how to create a quote

528
00:37:43.860 --> 00:37:46.690
unquote, pandas too. But I
think we reached some consensus that we would

529
00:37:46.730 --> 00:37:52.409
like to build a an evolved data
frame library that is a lot simpler and

530
00:37:52.530 --> 00:37:57.690
its functionality, so shedding some of
the baggage of multi indexes and some of

531
00:37:57.730 --> 00:38:00.400
the things and pandas that are can
be a bit complex and also don't lend

532
00:38:00.440 --> 00:38:05.800
themselves very well to out of core, you know, on like very large,

533
00:38:06.079 --> 00:38:08.960
not don't fit into memory data sets, but something that's focused on dealing

534
00:38:09.000 --> 00:38:13.510
with the very large data sets at
a single node scale, so large out

535
00:38:13.510 --> 00:38:15.869
of core, just big data sets
on a laptop. So we are I

536
00:38:15.909 --> 00:38:19.949
mean we are working on that and
you know, I think the project itself

537
00:38:20.070 --> 00:38:22.989
is not going to be called hands
to just to kind of not confuse people.

538
00:38:23.190 --> 00:38:27.900
And the Pantas project is we all
got together, the pandas team,

539
00:38:28.219 --> 00:38:30.659
we all got together in Austin over
the summer and this is one of the

540
00:38:30.739 --> 00:38:34.539
topics that, you know, we're
going to continue to grow and kind of

541
00:38:34.579 --> 00:38:37.619
innovate and evolve. The current pandas
project kind of nut as it is right

542
00:38:37.619 --> 00:38:44.769
now. But my goal is to
grow a parallel kind of companion project which

543
00:38:44.809 --> 00:38:51.409
is powered by the pachero ecosystem and
provides the Pandus like user experience in terms

544
00:38:51.449 --> 00:38:55.239
of usability and functionality, but is
really focused on kind of powering through very

545
00:38:55.239 --> 00:39:04.920
large on disc data sets. We'll
jump right back. Can do I interview

546
00:39:05.000 --> 00:39:08.280
with with is after a short segment. Let's now dive into a segment called

547
00:39:08.519 --> 00:39:14.429
studies in interpretability. I'm here with
powder coil, machine learning engineer and one

548
00:39:14.469 --> 00:39:17.630
of the core developers of the open
source statistical modeling platform PIMC. Three.

549
00:39:19.110 --> 00:39:21.989
Great to have you on the show. Padder, thanks for having me here.

550
00:39:22.030 --> 00:39:25.420
Go. So we're here to talk
about interpretability in building machine learning models

551
00:39:25.460 --> 00:39:30.780
and in data science more generally.
interpretability is telling you why I model makes

552
00:39:30.820 --> 00:39:34.659
certain decisions, and this is important, but it's more important in some areas

553
00:39:34.739 --> 00:39:37.019
and others. Right, I mean
it will be more important in insurance and

554
00:39:37.139 --> 00:39:40.090
healthcare, for example, then in
at deck, a space that you've worked

555
00:39:40.130 --> 00:39:46.010
in. Yes, here it's fair
to say that interpretability batters less and that

556
00:39:46.170 --> 00:39:51.650
tech the cost of showing a wrong
add is very different to, say the

557
00:39:51.769 --> 00:39:54.639
cost of Miss Pricing Insurance Policy.
Can you speak to this a bit more

558
00:39:54.679 --> 00:39:59.079
from your perspective? Yeah, so
an attach. The models I worked on

559
00:39:59.159 --> 00:40:07.269
largely involved lots of clever feature engineering. Under deployed models were really logistic regression

560
00:40:07.309 --> 00:40:10.389
due to the fact that they are
easy to power. Allies. So an

561
00:40:10.429 --> 00:40:15.989
attack. We care more about things
like predictive accuracy, because that's tied directly

562
00:40:15.070 --> 00:40:20.989
to the economic impact. We don't
care as much about explaining the model to

563
00:40:21.150 --> 00:40:24.539
internal customers. Are Regulators. I
mostly agree. However, you could imagine

564
00:40:24.539 --> 00:40:30.219
an algorithm that shows wealthy teenagers adds
for colleges but shows minorities adds for bio

565
00:40:30.340 --> 00:40:35.380
bondsmen having said that, in finance
and insurance, being able to explain models

566
00:40:35.420 --> 00:40:39.610
matters a lot right right the cost
of a mistake in credit risk models is

567
00:40:39.690 --> 00:40:45.449
very high you've loaned, for example, to a customer or client who defaults.

568
00:40:45.329 --> 00:40:50.880
I think as as we see more
applications of AI RML and assurance,

569
00:40:51.039 --> 00:40:53.840
healthcare and all the regulated industries,
we need to be more mindful of that.

570
00:40:53.960 --> 00:40:59.079
So can you comment on some work
you've seen in those industries? Will

571
00:40:59.119 --> 00:41:02.239
Sadly, some of the best work
I've seen has been under a nondisclosure agreement

572
00:41:02.280 --> 00:41:07.389
or da what example I saw was
a model for protecting credit risk for loans.

573
00:41:07.949 --> 00:41:12.309
The model of self as a random
force would lie him on top of

574
00:41:12.389 --> 00:41:16.590
it. For those of you who
don't know, lying sounds for locally interpretable

575
00:41:16.710 --> 00:41:22.139
model agnostic explanations, and lying is
basically a tool box the allows you to

576
00:41:22.539 --> 00:41:29.260
at explainable iputs from your black box
models. In the credit risk model case,

577
00:41:29.780 --> 00:41:34.610
it was easy to build a framework
for handling customer requests and regards why

578
00:41:34.690 --> 00:41:37.010
they were flagged up as at risk
of default, and it was able to

579
00:41:37.130 --> 00:41:43.610
convert that information into actionable information,
such as pay off your credit card debt

580
00:41:44.130 --> 00:41:47.440
or pay off your student Loos.
And how about in insurance? Well,

581
00:41:47.519 --> 00:41:53.360
basically, insurance companies have to allocate
reserve capital to compensate it for future losses.

582
00:41:54.280 --> 00:41:59.039
There's a lot of historical work in
Act Royal Community, but it's quite

583
00:41:59.079 --> 00:42:06.030
mathematically basic. Mccooning leverage newer techniques
and using programming languages like are and stand

584
00:42:06.469 --> 00:42:09.630
to compete a loss ratue, that
is a total amount that will be lost

585
00:42:10.110 --> 00:42:15.380
by the Insurance Company the future claims. This is a great example of where

586
00:42:15.380 --> 00:42:22.860
a better modeling approach can help you
better understand your risk. A more complicated

587
00:42:22.940 --> 00:42:27.059
model is worth in this case,
since you use case involves so much risk

588
00:42:27.099 --> 00:42:31.889
and so much capital. This model
was interpretable. For example, one could

589
00:42:31.929 --> 00:42:37.849
see the nactually incorporated uncertainly in the
posterior distribution business knowledge was also incorporated.

590
00:42:38.730 --> 00:42:44.800
Furthermore, there was an increased confidence
in the model, and the explicit statement

591
00:42:44.880 --> 00:42:51.320
of assumptions improved interpretability. Therefore,
is clear that this modeling approach can be

592
00:42:51.360 --> 00:42:55.880
a useful addition into our toolbox and
also can provide insights the traditional machine learning

593
00:42:55.880 --> 00:43:00.269
methods can't provide. For users who'd
love to learn more about these baysing and

594
00:43:00.349 --> 00:43:06.750
techniques, then I recommend mix resources. Search for lost cares. Case Study

595
00:43:06.869 --> 00:43:10.909
on the stand case studies website,
or my course on probabilistic programming is also

596
00:43:10.989 --> 00:43:15.420
excellent. That's called probabilistic programming primer. Thanks for speaking today, you,

597
00:43:15.539 --> 00:43:30.449
Patter. Anything I can do to
help to listeners. Time to get stright

598
00:43:30.530 --> 00:43:36.250
back into our chat with West mckinney. I'd like to step back a bit

599
00:43:36.369 --> 00:43:39.250
and think about open source software development
in general. I suppose, spoiler alert,

600
00:43:39.289 --> 00:43:42.880
where I want this to go is
to talk about you're one of your

601
00:43:42.880 --> 00:43:45.320
latest benches, Su Lebs, but
I'm wondering in your mind, what the

602
00:43:45.360 --> 00:43:51.159
biggest challenges for open source software development
are at this point in time. Well,

603
00:43:51.400 --> 00:43:54.400
we can have a whole podcast just
about this topic and of course it

604
00:43:54.480 --> 00:43:58.630
depends on the stage of a project
and all of the the way that I

605
00:43:58.710 --> 00:44:02.070
frame the problem when I talk to
people is that I think open source projects

606
00:44:02.150 --> 00:44:07.389
face, you know, funding and
sustainability problems of different kinds depending on the

607
00:44:07.429 --> 00:44:12.380
stage of the project. So I
think in the early stages of projects,

608
00:44:12.900 --> 00:44:16.460
when you're building something new or you're
essentially solving a own problem in a different

609
00:44:16.460 --> 00:44:22.059
way, it can be hard to
get support from other develop oppers or financial

610
00:44:22.059 --> 00:44:27.210
support to sponsor individuals to work on
the project, because it's hard to build

611
00:44:27.250 --> 00:44:31.610
consensus around something new and there might
be like even competing approaches to the same

612
00:44:31.730 --> 00:44:37.369
problem. And so we're talking about
the kind of funding that can support full

613
00:44:37.449 --> 00:44:38.920
time software developers. You know,
it can be a lot of money,

614
00:44:38.960 --> 00:44:45.199
and so committing a lot of money
to support a risky venture into kind of

615
00:44:45.199 --> 00:44:49.039
building a new open source project which
may or may not come become successful,

616
00:44:49.400 --> 00:44:52.710
can be a tough pill to swallow. For potential financial backer later on is

617
00:44:52.789 --> 00:44:58.389
projects become, you know, wider
adopted, they start becoming particularly projects that

618
00:44:58.429 --> 00:45:01.710
are so they are foundational and you
can call them like, I think the

619
00:45:02.190 --> 00:45:07.539
popular term is like open source infrastructure. There was a report so naught,

620
00:45:07.579 --> 00:45:10.420
EA Eggball, wrote the report called
roads and bridges, about kind of open

621
00:45:10.420 --> 00:45:15.820
source infrastructure with the Ford Foundation and
sort of is about this idea of like

622
00:45:15.980 --> 00:45:19.699
thinking about open source software as like
a public good, roads and bridges and

623
00:45:19.780 --> 00:45:23.610
like public infrastructure that everyone uses.
And you know, with public infrastructure it's

624
00:45:23.650 --> 00:45:29.489
great because it's supported by tax dollars, but we don't exactly have a open

625
00:45:29.530 --> 00:45:31.489
source tax. I mean it,
I could get behind one, but you

626
00:45:31.570 --> 00:45:36.960
know, we don't have that kind
of same kind of mentality around funding critical

627
00:45:37.119 --> 00:45:42.679
open source infrastructure and I think that
as projects become really successful and they become

628
00:45:42.760 --> 00:45:46.119
something that people can't live without,
they end up facing the classic tragedy of

629
00:45:46.159 --> 00:45:51.590
the Commons problem, where people feel
like well, you know, they derive

630
00:45:52.269 --> 00:45:54.389
some they drive a lot of value
from the project, but because everyone use

631
00:45:54.469 --> 00:45:58.909
this as a project, they don't
want to foot the bill of supporting and

632
00:45:59.030 --> 00:46:01.949
maintaining the software project. So,
whether you're on the early side of a

633
00:46:01.989 --> 00:46:05.780
project or the late you know,
in the early stage or a late stage,

634
00:46:05.820 --> 00:46:09.539
I think there's different kinds of funding
and sustainability challenges and in all cases,

635
00:46:09.659 --> 00:46:14.500
I think open source developers end up, particularly as projects become more successful,

636
00:46:14.659 --> 00:46:17.809
you end up quite overburdened and,
you know, burnout risk and I

637
00:46:17.929 --> 00:46:22.489
know I've I've experienced burn out many
times and many other open source developers have

638
00:46:22.650 --> 00:46:28.969
have experienced periods of significant burnout.
So what can listeners who are, you

639
00:46:29.050 --> 00:46:34.079
know, working or asparring dito scientists
or data analysts in organizations or sea level

640
00:46:34.280 --> 00:46:37.920
people within organizations, do for the
open source what would you like to see

641
00:46:37.920 --> 00:46:42.800
them do more for the open source
well, I think I think users and

642
00:46:42.920 --> 00:46:45.960
other folks can help with so its
people like me, I guess. I've

643
00:46:45.119 --> 00:46:49.989
recently been kind of, you know, working on putting myself in a situation

644
00:46:50.190 --> 00:46:53.630
where I am able to raise money
and put to work money that is donated

645
00:46:53.949 --> 00:46:59.190
for direct open source development and so
and I think so the best way a

646
00:46:59.230 --> 00:47:04.659
lot of people can help is by
selling the idea of supporting and either through

647
00:47:04.940 --> 00:47:09.179
development work or through direct funding,
supporting the open source projects that you rely

648
00:47:09.340 --> 00:47:14.380
on. So I think a lot
of companies and a lot of developers are

649
00:47:14.420 --> 00:47:20.409
our passive participants in open source projects
ex and so finding a way to contribute,

650
00:47:20.489 --> 00:47:23.889
whether it's through money or time.
It is difficult because many open source

651
00:47:23.929 --> 00:47:30.519
projects, particularly ones that are systems
related to infrastructure, they don't necessarily lend

652
00:47:30.519 --> 00:47:35.119
themselves to casual, quote unquote,
casual contribution. So if it's your five

653
00:47:35.199 --> 00:47:38.280
percent project or your twenty percent project, it can be hard as an individual

654
00:47:38.360 --> 00:47:43.519
to make a meaningful contribution to a
project which may have a steep learning curve

655
00:47:43.519 --> 00:47:47.150
or just require a lot of intense
focus. And so think for a lot

656
00:47:47.190 --> 00:47:52.670
of organizations the best way to help
projects can be to to donate money directly.

657
00:47:52.829 --> 00:47:55.190
So I think something this provides a
nice Seguay into your work at USA

658
00:47:55.190 --> 00:47:59.420
labs. I'd love for you to
just give us a rundown of Ursa labs

659
00:47:59.500 --> 00:48:01.059
and, in particular how it frames, you know, the challenges of open

660
00:48:01.099 --> 00:48:07.219
source software development. Yeah, so
seversa labs is an organization. I partnered

661
00:48:07.260 --> 00:48:10.539
with Hadley Wickham from the our community
and in our studio to found Ursa labs

662
00:48:10.699 --> 00:48:15.769
earlier earlier this year. The the
kind of raisind Etra of a versa labs

663
00:48:15.929 --> 00:48:22.329
was to to build shared infrastructure for
data science and particular building out the aero

664
00:48:22.409 --> 00:48:25.769
ecosystem as it the Apache aero ecosystem
as it relates to to data science and

665
00:48:25.889 --> 00:48:30.800
making sure that we have high quality, consistent support for all of the that

666
00:48:30.000 --> 00:48:36.559
new technology in the python and our
world and beyond, and improving interoperability for

667
00:48:36.599 --> 00:48:40.320
data scientists that use all those programming
languages. But the particular logistical details of

668
00:48:40.360 --> 00:48:45.789
versa labs is that we wanted to
be able to effectively put together an industry

669
00:48:45.949 --> 00:48:52.750
consortium type model where we can raise
money from corporations and use that money to

670
00:48:52.869 --> 00:48:57.260
hire full time open source developers.
So at the moment you know so urslabs

671
00:48:57.300 --> 00:49:00.219
is being being supported by our studio, by to sigma, where I used

672
00:49:00.260 --> 00:49:04.179
to work right up until the founding
of versa labs, and it's now being

673
00:49:04.260 --> 00:49:07.059
funded by nvidia, the makers of
graphics cards, and so we're, you

674
00:49:07.099 --> 00:49:13.489
know, kind of atworking actively on
bringing in more sponsors to build a larger

675
00:49:13.769 --> 00:49:17.010
team of developers, and I think
it's really confronting that challenge that I think

676
00:49:17.130 --> 00:49:22.769
for an engineer at a company,
as a parttime contributor to an open source

677
00:49:22.809 --> 00:49:27.079
project, may not be as effective, or nearly as effective as a full

678
00:49:27.159 --> 00:49:30.079
time developer, and so I want
to make sure I'm able to build an

679
00:49:30.079 --> 00:49:37.199
organization that is full of outstanding engineers
who are working full time on open source

680
00:49:37.199 --> 00:49:39.469
software and making sure that we are
able to do that in a scalable and

681
00:49:39.630 --> 00:49:45.550
sustainable way and is kind of organized
for the benefit of the open source data

682
00:49:45.630 --> 00:49:50.590
science world. So anyway, and
I having been through the consoling path and

683
00:49:50.710 --> 00:49:55.739
the startup path and working for single
companies, I think a consortium type model,

684
00:49:55.739 --> 00:50:00.380
where it's being funded by multiple organizations
and where we we are not building

685
00:50:00.380 --> 00:50:04.260
a product of some kind. It's
kind of a new model for doing open

686
00:50:04.300 --> 00:50:07.289
source development, but one that I'm
excited to pursue and see things go.

687
00:50:08.130 --> 00:50:12.289
Yeah, I think it's really exciting
as well because it does approach a lot

688
00:50:12.329 --> 00:50:15.409
of the different challenges. One in
particular, it's a troupe. It's a

689
00:50:15.409 --> 00:50:20.809
common problem, right, of developers
being employed by organizations and being given a

690
00:50:20.849 --> 00:50:23.679
certain amount of time to work on
open source software development but that time being

691
00:50:23.719 --> 00:50:29.960
eight in away because of different incentives
within organization. Essentially, yeah, I

692
00:50:30.039 --> 00:50:31.760
mean it. You know, I
think there have been ton of contributions to

693
00:50:32.000 --> 00:50:36.670
pandas and too a patchy Arrow from
developers that work at corporations, and those

694
00:50:36.829 --> 00:50:40.550
contributions mean a lot. So definitely
still looking for companies to collaborate on the

695
00:50:40.590 --> 00:50:45.989
road map and to work together to
build kind of new computational infrastructure for data

696
00:50:45.030 --> 00:50:47.989
science. You know, I think
it's tough when you know developer might show

697
00:50:49.030 --> 00:50:52.340
up and be spending a lot of
time for a month or two and then,

698
00:50:52.380 --> 00:50:54.420
based on their priorities within where the
company where they work, they might

699
00:50:54.619 --> 00:50:58.940
disappear for six months and that's just
the nature of things. You know,

700
00:50:59.019 --> 00:51:01.579
I think the kinds of developers that
make a big contributions to open source can

701
00:51:01.659 --> 00:51:06.409
often be more senior or like going
to be very important developers and their respective

702
00:51:06.409 --> 00:51:12.050
organizations, and so frequently gets kind
of called into kind of prioritize closed source

703
00:51:12.090 --> 00:51:15.289
or internal projects. That's just kind
of thinking of the ABB and flow of

704
00:51:15.530 --> 00:51:19.920
corporate environment. So I've got a
relatively general question for you. What does

705
00:51:20.000 --> 00:51:23.519
the future of Ditosians tooling look like
to you? Well, speculative, of

706
00:51:23.639 --> 00:51:28.199
course, but you know, I
think by spending my time on Aero Project,

707
00:51:28.480 --> 00:51:30.480
you know my objective and what I
would like to see happen in data

708
00:51:30.480 --> 00:51:37.670
science tooling is a defragmenting of of
data and code, so to have increased

709
00:51:37.949 --> 00:51:42.869
standardization and and adoption of open standards
like the aero column their format, storage

710
00:51:42.869 --> 00:51:47.579
formats like parquet and or protocols for
messaging like g RPC. So I think

711
00:51:47.659 --> 00:51:52.099
that in the future, I believe
that things will be a lot more standardized

712
00:51:52.260 --> 00:51:54.900
and a lot less fragmented. kind
of a slightly crazy idea. I don't

713
00:51:54.900 --> 00:51:59.099
know how crazy it is, but
I think also in the future that programming

714
00:51:59.139 --> 00:52:06.929
languages are going to diminish an importance
relative to data itself and common computational libraries.

715
00:52:07.489 --> 00:52:09.530
This is kind of a self serving
opinion, but I do think that

716
00:52:09.929 --> 00:52:15.570
if to be able to leave data
in place and to be able to choose

717
00:52:15.760 --> 00:52:20.599
the user interface, namely the programming
language. Is the programming language that best

718
00:52:20.639 --> 00:52:25.280
suits your needs in terms of interactivity
or software development or or so forth.

719
00:52:25.519 --> 00:52:30.590
That's you know. You can use
multiple programming languages to build an application or

720
00:52:30.710 --> 00:52:35.550
pick the programming language that's you know
that you prefer, while utilizing common libraries

721
00:52:35.590 --> 00:52:40.030
of Algorithms Common Query engines for processing
that data. And so I think we're

722
00:52:40.110 --> 00:52:45.340
beginning to see kind of murmurings of
this defragmentation happening and I think the Arrow

723
00:52:45.380 --> 00:52:49.780
project is kind of kick along this
process and socialize the idea of what a

724
00:52:49.900 --> 00:52:54.260
more defragmented and more consistent user experience
for data sciences with that might look like.

725
00:52:54.659 --> 00:52:58.929
That's a very exciting future. So
my last question, fews, is

726
00:52:59.050 --> 00:53:01.090
do you have a final cool to
action for all leastans out there? Yeah,

727
00:53:01.090 --> 00:53:05.449
I would say my call to action
would be to find some meaningful way

728
00:53:05.570 --> 00:53:08.050
to, you know, contribute to
the open source world, whether it's sharing

729
00:53:08.090 --> 00:53:13.719
your ideas or sharing your use cases
about what parts of the open source stack

730
00:53:13.719 --> 00:53:15.599
or working well for you or what
parts you think could serve you better.

731
00:53:15.920 --> 00:53:21.039
If you are able to contribute to
projects, whether through discussions on mailing lists

732
00:53:21.119 --> 00:53:25.280
or Github or commenting on the road
map or so forth. That's all very

733
00:53:25.320 --> 00:53:30.750
valuable. I think a lot of
people think the code is the only real

734
00:53:30.829 --> 00:53:34.150
way to contribute to open source projects, but actually, you know, I

735
00:53:34.269 --> 00:53:37.590
spend a lot of my time is
not writing code, it's reviewing code and

736
00:53:37.750 --> 00:53:42.300
kind of steering discussions about design and
Road Map and future scope, and I

737
00:53:42.380 --> 00:53:45.739
think the more voices and the more
people involved to help build consensus and kind

738
00:53:45.739 --> 00:53:51.019
of help prioritize the work that's happening
in open source projects helps, you know,

739
00:53:51.139 --> 00:53:54.059
make healthier and more productive communities.
And if you do work in an

740
00:53:54.099 --> 00:53:59.369
organization that has the ability to donate
money to open source projects, you know,

741
00:53:59.369 --> 00:54:05.849
I would love to see worldwide corporations
effectively tithing a portion of profits to

742
00:54:05.969 --> 00:54:08.730
fund open source infrastructure. I think
if corporations gave, you know, a

743
00:54:08.849 --> 00:54:14.679
fraction of one percent of their profits
to open source projects, the funding and

744
00:54:14.760 --> 00:54:19.960
sustainability crisis that we have now would
essentially go away. And obviously I guess

745
00:54:20.000 --> 00:54:22.239
that's might be a lot to ask, but I can always hope. So

746
00:54:22.800 --> 00:54:28.230
any corporations can lead by example and
certainly if you do donate money to open

747
00:54:28.230 --> 00:54:30.710
source projects, you should make a
show of that and make sure that other

748
00:54:30.869 --> 00:54:36.309
corporations know that you're a good citizen
and you're helping support the work of open

749
00:54:36.309 --> 00:54:40.059
source developers. I couldn't agree more
with it's been an absolute pleasure having you

750
00:54:40.139 --> 00:54:50.659
on the show. Thanks, higs
and fun thanks for joining our conversation with

751
00:54:50.699 --> 00:54:53.889
where's about pandas, data analysis tooling
in general, the future of data science

752
00:54:53.929 --> 00:54:59.730
and the challenges of open source software
development. Where stated that he thinks in

753
00:54:59.809 --> 00:55:05.050
the future that programming languages are going
to diminish in importance relative to data itself

754
00:55:05.289 --> 00:55:08.440
and common computational libraries, and his
work on a partciarrow is es central to

755
00:55:08.519 --> 00:55:13.960
this vision. The concept of portable
data structures that are accessible from a variety

756
00:55:13.960 --> 00:55:19.360
of programming languages and that can leverage
the vast computational power we now have to

757
00:55:19.559 --> 00:55:22.909
work in the limit of at least
hundreds of gigabytes. Many popular data science

758
00:55:22.989 --> 00:55:28.670
tools, such as pandas in general, do not effectively leverage modern hardware.

759
00:55:28.789 --> 00:55:32.110
One of Ursa labs goals is to
empower and accelerate the work of data scientists

760
00:55:32.429 --> 00:55:37.860
through more efficient and scalable in memory
computing. We also discussed Ursa labs,

761
00:55:38.099 --> 00:55:43.739
which I am so excited about,
and how the model of open source software

762
00:55:43.780 --> 00:55:49.139
development here is to effectively put together
an industry consortium type model where they can

763
00:55:49.219 --> 00:55:53.010
raise money from corporations and use that
money to hire full time open source developers.

764
00:55:53.050 --> 00:55:59.369
Also, get ready for next week's
episode, our two thousand and eighteen

765
00:55:59.409 --> 00:56:05.480
season one finale, a conversation with
Cathy O'Neil, data scientists, investigative journalist,

766
00:56:05.639 --> 00:56:09.400
consultant, algorithmic auditor and the author
of the critically acclaimed book weapons of

767
00:56:09.559 --> 00:56:15.039
math destruction. Kathee and I will
disgust ingredients that make up weapons of map

768
00:56:15.119 --> 00:56:20.869
destruction. Which Algorithms and models that
are important in society secret and harmful?

769
00:56:21.309 --> 00:56:24.510
From models that decide whether you keep
your job, the credit card or insurance,

770
00:56:24.750 --> 00:56:29.989
or algorithms that decide how we're police, sentenced to prison or given parole?

771
00:56:30.389 --> 00:56:32.980
Cathie and I will be discussing the
current lack of fairness in artificial intelligence,

772
00:56:34.340 --> 00:56:38.940
how societal biases are perpetuated by algorithms
and how both transparency and auditability of

773
00:56:39.019 --> 00:56:44.179
algorithms will be necessary for a fairer
future. What does this mean in practice?

774
00:56:44.579 --> 00:56:49.090
Join US next week to find out. As Kathy says, fairness is

775
00:56:49.130 --> 00:56:52.489
a statistical concept. It's a notion
that we need to understand at an aggregate

776
00:56:52.570 --> 00:56:58.250
level. And, moreover, data
science doesn't just predict the future, it

777
00:56:58.449 --> 00:57:02.079
causes the future. You'd best tune
in for this, our final episode of

778
00:57:02.199 --> 00:57:06.960
season one of data framed. I
didn't intend for that to sound threatening,

779
00:57:07.119 --> 00:57:10.239
but it came across. So.
I'm your host, Hugo bound Anderson.

780
00:57:10.360 --> 00:57:14.880
You can follow me on twitter at
Hugo bound and data camp at data camp.

781
00:57:15.159 --> 00:57:21.150
You can find all our episodes and
show notes at Data Campcom community podcast.

782
-->