Jake Vanderplas: Data Science For Academic Research

WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 7/6/2024 5:37:09 PM
Duration: 2967.179
Channels: 1
1
00:00:13.905 --> 00:00:27.365
Hello, and welcome to podcast dot in it, the podcast about Python and the people who make it great. I would like to thank everyone who supports the show on Patreon. Your contributions help to make the show sustainable. When you're ready to launch your next project, you'll need somewhere to deploy it, so you should check out linode@podcastinit.com/linode
2
00:00:29.265 --> 00:00:54.750
and get a $20 credit to try out their fast and reliable Linux virtual servers for running your app. And now you can deliver your work to your users even faster with the newly upgraded 200 gigabit network in all of their data centers. If you're tired of cobbling together your deployment pipeline, then it's time to try out GoCD, the open source continuous delivery platform built by the people at Thoughtworks who wrote the book about it. With GoCD, you get complete visibility into the life cycle of your software from 1 location. To download it now, go to podcastinit.com/gocd.
3
00:00:56.650 --> 00:00:59.545
Professional support and enterprise plug ins are available for added
4
00:01:00.105 --> 00:01:01.804
peace of mind. You can visit the site at podcastinnit.com
5
00:01:02.585 --> 00:01:10.650
to subscribe to the show, sign up for the newsletter, and read the show notes. And if you have any questions, comments, or suggestions, I would love to hear them. You can reach me on Twitter at podcastinit
6
00:01:11.030 --> 00:01:12.490
or email me at host@podcastinit.com.
7
00:01:13.830 --> 00:01:28.220
To help other people find the show, please leave a review on Itunes or Google Play Music. Tell your friends and coworkers and share it on social media. Your host as usual is Tobias Macy. And today, I'm interviewing Jake Vander Plaas about data science best practices and applying them to academic sciences.
8
00:01:28.520 --> 00:01:30.620
So Jake, could you start by introducing yourself?
9
00:01:30.975 --> 00:01:33.135
Yeah. Yeah. So my name is Jake. I'm,
10
00:01:33.535 --> 00:01:43.590
an astronomer by training, but for the last 5 or so years, I've been more on the data science side of academia. I've been working at the University of Washington E Science Institute.
11
00:01:43.890 --> 00:01:45.759
And we have a a really
12
00:01:46.665 --> 00:01:48.845
a great team here and a and a really interesting,
13
00:01:50.104 --> 00:02:00.210
goal, which has been for the last 5 years to to kind of serve as a glue between all sorts of different scientific departments and and other other research departments on campus
14
00:02:00.715 --> 00:02:06.015
with these common tools that everybody seems to be using, whether it's, you know, r or Python
15
00:02:06.315 --> 00:02:06.635
or,
16
00:02:07.515 --> 00:02:16.299
machine learning algorithms or or data cleaning approaches and things like this. So it's it's been a really fun position, and I've gotten to branch out a little bit from my astronomy
17
00:02:16.599 --> 00:02:20.595
into into areas that I never imagined I'd be working on.
18
00:02:21.375 --> 00:02:38.135
It's funny seeing the ways that your sort of progression through particularly in technology can change so drastically purely by the unifying fact that you're writing software. Right. Right. I sort of got into the Python world when I was doing my PhD in astronomy. And,
19
00:02:38.535 --> 00:02:42.875
the reason was I was basically using NumPy and scipy to drive
20
00:02:43.430 --> 00:02:46.390
everything I was doing with with astronomy data. And,
21
00:02:47.030 --> 00:02:52.985
it turns out that those tool tools are pretty general. So I started started going to to Python meetups and Python,
22
00:02:53.765 --> 00:02:54.265
conferences
23
00:02:54.565 --> 00:02:57.285
and and talking with people in other fields and,
24
00:02:57.765 --> 00:03:05.330
really got bitten by the o open source bug and started contributing back to these packages. So that's sort of how I got into things here.
25
00:03:06.110 --> 00:03:06.610
And
26
00:03:06.989 --> 00:03:08.370
what has been your experience
27
00:03:09.095 --> 00:03:22.390
sort of seeing the difference between the scientific Python communities and the more general Python communities, particularly as it manifests between the scipy conferences and the PyCon series? Yeah. You know, so I've I've been going to these
28
00:03:22.769 --> 00:03:23.269
both,
29
00:03:23.810 --> 00:03:26.935
conference series, scipy and PyCon, since about 2012.
30
00:03:27.314 --> 00:03:31.095
And early on, they were they were very different conferences.
31
00:03:32.114 --> 00:03:32.775
You know,
32
00:03:33.155 --> 00:03:41.190
when I when I started at at PyCon, it was, like, there was not much science or or or data content in it. It was a lot of,
33
00:03:42.470 --> 00:03:46.445
web development stuff, you know, Django and and a lot of the
34
00:03:46.825 --> 00:03:48.285
core Python contributions
35
00:03:48.665 --> 00:03:54.845
and, you know, having fun with generator expressions and things like that. Whereas the SciPy conference has always been about showcasing
36
00:03:55.145 --> 00:03:56.170
research that,
37
00:03:56.730 --> 00:03:57.390
that happens
38
00:03:57.850 --> 00:04:08.935
using Python, that's enabled by Python. But there it seems like over the last 5 years, there's been kind of a a convergence a little bit. Both conferences have moved a little closer together and that they've
39
00:04:09.235 --> 00:04:14.100
they've both embraced data science to some extent. You know, you go to SciPy and there are
40
00:04:14.580 --> 00:04:17.080
tutorials on on pandas and,
41
00:04:18.260 --> 00:04:26.525
doing, you know, large large database queries with with pandas style operations and things like this that are not necessarily
42
00:04:27.065 --> 00:04:28.604
all that close to the
43
00:04:29.040 --> 00:04:41.365
to the the physical science roots of of sci fi, but are more in the in the data science sense of things. Now on the other hand, you see you have PyCon, which has moved into that realm as well. I was in 2013,
44
00:04:41.824 --> 00:04:42.324
2014,
45
00:04:42.625 --> 00:04:47.630
actually, almost every year for the past 4 years, I've been involved with the tutorial,
46
00:04:48.650 --> 00:04:49.150
evaluations
47
00:04:49.530 --> 00:04:53.655
and sometimes the tutorial selections in the for PyCon. And,
48
00:04:54.095 --> 00:04:55.715
I remember in in 2013,
49
00:04:56.175 --> 00:04:56.835
there was
50
00:04:57.534 --> 00:04:58.354
a a discussion
51
00:04:59.055 --> 00:05:01.155
among the committee because there were, like,
52
00:05:01.560 --> 00:05:16.965
3 data science tutorials that were submitted or maybe, like, 1 1 on scikit learn and 1 on, machine learning with some some other tool. And there's this discussion in the committee that, like, should we really devote 2 tutorial sessions to such a niche topic like data science?
53
00:05:17.800 --> 00:05:18.440
Right? And then,
54
00:05:18.920 --> 00:05:30.935
it it's just funny thinking back on that because this last year, if you look at the PyCon tutorials in 2017, they were, like, half of them were data science, maybe 3 quarters of them. So it's it's totally shifted over the last few years.
55
00:05:31.635 --> 00:05:41.240
Yeah. It's definitely interesting seeing the ways that the massive amounts of data that are being collected now are so drastically influencing the career availabilities
56
00:05:41.540 --> 00:05:48.945
and the ways that some of the different popular languages are being used that weren't really being thought of at the time when they were first being created.
57
00:05:49.565 --> 00:05:52.230
Yeah. And and you see that larger trend too. Like,
58
00:05:52.710 --> 00:06:00.410
5 years ago when when Python was all about web stuff, it was, you know, all the talk was whether Ruby on Rails is going to, you know, overtake
59
00:06:01.255 --> 00:06:01.755
overtake
60
00:06:02.295 --> 00:06:14.520
what what people are using Python for. Now you don't hear much about that. But you hear more about the r versus Python thing or, you know, or Julia versus Python in the in the sort of scientific numerics world.
61
00:06:15.300 --> 00:06:23.715
Right. And then in the systems area, it's a lot of talk about Go versus Python Yeah. Largely because of the packaging story and deployment. Mhmm. Mhmm.
62
00:06:24.095 --> 00:06:37.585
And you mentioned that your background was originally in astronomy Mhmm. And now you're working more in the data science and education space. I'm wondering how your background in astronomy informs and influences your current work.
63
00:06:37.885 --> 00:06:41.585
Yeah. Well, I still, fortunately, I still get to work on astronomy
64
00:06:41.885 --> 00:06:43.950
topics a little bit here and there. I've been
65
00:06:44.510 --> 00:06:52.110
I've been publishing maybe 1 paper a year in, in astronomy journals. So I've I've kept my kept my toes in there. But,
66
00:06:52.590 --> 00:06:57.635
the thing is that astronomy in a lot of ways was kind of an early adopter of these
67
00:06:58.095 --> 00:06:58.915
these really
68
00:06:59.295 --> 00:07:00.915
data driven approaches
69
00:07:01.455 --> 00:07:02.995
to, to science.
70
00:07:03.450 --> 00:07:05.470
You know, we had even going back
71
00:07:06.170 --> 00:07:11.070
15 years or so, there's with the start of the Sloan Digital Sky Survey, it was this
72
00:07:11.435 --> 00:07:14.395
mode of of scientific operation where you you basically
73
00:07:15.115 --> 00:07:17.215
what Sloan did was it it scans
74
00:07:17.675 --> 00:07:20.735
scanned the whole sky and and got kind of this big
75
00:07:21.130 --> 00:07:27.470
catalog of objects up in the sky and then went back and followed up on those objects with some detailed spectroscopic
76
00:07:27.930 --> 00:07:28.430
observations.
77
00:07:29.050 --> 00:07:36.035
And in the end, it's somewhere I think over the course of 10 years, they got a a couple million of these spectroscopic observations.
78
00:07:36.655 --> 00:07:38.515
And the idea there was that
79
00:07:39.560 --> 00:07:40.520
this this was,
80
00:07:41.000 --> 00:08:00.419
really revolutionary at the time. The idea was that, if you wanted to be an astronomer studying the sky, instead of doing the classic thing of going out and applying for telescope time and then pointing that telescope where he wants and then getting that data, they wanted, the mode of exploring the sky to be database queries. Right? You have this this huge database,
81
00:08:01.360 --> 00:08:22.415
this huge catalog of all these interesting objects in the sky and a public open database where you can you can query by by position, by object type, by, by all sorts of different datasets. So so astronomy has been doing this thing for a long time, and I feel like in some ways, the last 5 years of the you might call the data science revolution in
82
00:08:23.375 --> 00:08:25.075
in, in scientific research,
83
00:08:25.455 --> 00:08:29.315
it's been a lot of other fields catching up to that notion of
84
00:08:29.615 --> 00:08:30.835
large public datasets,
85
00:08:31.410 --> 00:08:31.910
of,
86
00:08:32.370 --> 00:08:37.590
working together in kind of an open fashion. And it's been really fun to see. And and
87
00:08:38.295 --> 00:08:41.915
to be honest, astronomy has prepared me really well for that.
88
00:08:42.775 --> 00:08:46.430
And in your current work at the University of Washington,
89
00:08:46.890 --> 00:08:55.150
you're working on educating students on some of the different best practices in data science, and my understanding as well is that you're teaching some of those same lessons.
90
00:08:55.495 --> 00:09:00.155
Yeah. What we're doing at at Esciences is sort of multipronged. We're trying to,
91
00:09:00.535 --> 00:09:03.915
we're trying to reach all of campus. Right? That because there's
92
00:09:05.180 --> 00:09:18.714
there's, of course, students that need to that need to learn these skills to be effective scientists in this in this era. And so 1 of the things we've done on the education side that's been interesting is we've been trying to influence the curriculums,
93
00:09:19.415 --> 00:09:31.375
the curricula of all these these departments around campus. And as you can imagine, if you've ever been in academia, you know that it's hard to change things. There's a lot of inertia. And if you say, for example, that,
94
00:09:32.155 --> 00:09:35.295
you think astronomers need to be learning about
95
00:09:35.675 --> 00:10:06.275
how to how to work with large datasets, how to write effective code, how to how to do software engineering, that sort of thing. You say, okay. Well, let's let's create a graduate class for that. But graduate students are already overworked and they already have a full curriculum. So if you're gonna create a class that's about data science methods, about software engineering, about statistical methods or something like that, you need to figure out what you're gonna drop from there. Correct? Right? And this is really hard because what are you gonna drop, stellar atmospheres? Are you gonna are you gonna
96
00:10:06.755 --> 00:10:29.375
drop the course on the interstellar dust? You know, because each each of those courses has someone who's devoted their career to studying this thing and thinks it's the most important thing of the in the world. So how could you how could you get a PhD in astronomy without having a course in that? So it's this really difficult thing. Right? Because it's this set of skills that's absolutely fundamental to doing science today.
97
00:10:29.850 --> 00:10:39.425
And departments are really slow to actually add that to the curriculum. So we've been pushing on that in in a number of different ways here and giving graduate students these, interdisciplinary
98
00:10:39.965 --> 00:10:42.545
electives that they can take, things like transcriptable
99
00:10:42.925 --> 00:10:57.710
options where you could if you take, some these extra courses, you could get a PhD in biology with an emphasis in data science. And that appears on your transcript, kinda like a a minor for an undergraduate degree. So we've been working with the administration,
100
00:10:58.165 --> 00:11:06.504
working with different departments to try to figure out how to get these sorts of skills into the main curricula. But at the same time, we've been doing, a lot on the
101
00:11:07.160 --> 00:11:16.540
more informal education side. So so programs like Software Carpentry have been amazing. We've been a partner with Software Carpentry for a number of years now and running
102
00:11:17.185 --> 00:11:52.779
workshops a few times a quarter. And that's just to get if you're not familiar with Software Carpentry, that basically takes people from ground 0 to, figuring out how to use the shell, how to work with Python or with R, how to manipulate data, how to use Git and GitHub in version control, and and these sorts of core skills that are so necessary these days in science. Another aspect of it that's been really fun is not so much working with the students, but working with the researchers that are already in place, whether they're post docs or research scientists or even faculty. And we have a number of ways through the through the eScience Institute where we can
103
00:11:53.524 --> 00:12:44.605
build connections with people around campus. And they they go from really simple things like just having open office hours. We have 1 of our 1 of our e science data scientists or research scientists is basically on duty all through the week. Each person does does about 2 hours or so. And so throughout the week, anyone from around campus can come in and just ask technical questions about their research and get get connected with someone who has expertise in the area they need, whether whether they need some sort of statistical modeling or whether they need to apply machine learning in some way or whether they need to scale up their their data and and use databases effectively, or whether they need to build a a web front end for some kind of public outreach aspect of their research. So we have we have all these ways of connecting with people around campus. And then for me as an academic researcher, those
104
00:12:44.985 --> 00:12:59.165
have often led to more long term collaborations. So now I have instead of just having papers in astronomy, you know, I have papers in geophysics or in transportation policy or things like that. And it's been been really fun. And
105
00:12:59.545 --> 00:13:14.190
in the work that you're doing both with students and with active researchers, what have been some of the common misconceptions or stumbling blocks that they come across in the process of trying to learn and understand and apply these different statistical
106
00:13:14.569 --> 00:13:16.110
and data science techniques?
107
00:13:16.715 --> 00:13:23.215
So it's it really varies from person to person because we we get a lot there's a lot of people who have
108
00:13:24.290 --> 00:13:28.550
there's a wide range of backgrounds when it comes to computing and to statistics.
109
00:13:29.250 --> 00:13:32.965
And it's because these topics have just are are not treated,
110
00:13:33.505 --> 00:13:36.725
in in a lot of undergraduate and graduate curricula. These
111
00:13:37.025 --> 00:14:25.269
these topics are not treated as first class citizens. So there's a huge range of of what people know coming in. But the the biggest challenge actually as far as as far as academic research goes is that the incentive structure for academics, you know, if people people are looking to basically get tenure eventually. The the incentive structure has nothing to do with with writing good code or writing good tools that other people can make use of. It's all about papers and citations. Right? So if I tell someone, hey, it's you'll you'll be much better in the long run if you can version control your code and document the code and write in a modular way so that the next time you come around, you can remember what you did and build on what you did. That doesn't help someone get a paper out in the next month that they need in order to,
112
00:14:26.095 --> 00:14:48.394
to appease their tenure committee. Right? So it's academia is in this weird place right now where it's kind of, in some ways, shot itself in the foot by by developing these incentive structures that don't match what makes research most effective in the long run. So a big part of what we've been doing here and what we've been working on with our colleagues at NYU and Berkeley, who are on
113
00:14:49.514 --> 00:14:52.730
who who we share a grant with, is trying to figure out how to
114
00:14:53.110 --> 00:14:58.810
how to nudge that how how to nudge academia in the right direction and think about how we can,
115
00:14:59.635 --> 00:15:03.015
make it so that the activities we think are valuable
116
00:15:03.555 --> 00:15:04.775
to science overall
117
00:15:05.154 --> 00:15:09.250
are recognized within the incentive structure. You know, they're recognized
118
00:15:10.190 --> 00:15:29.269
by hiring committees, recognized by funding committees. So that, you know, if I if I spend a week a week or 2 after publishing a paper, making sure that my software is something that can be used by other researchers to to reproduce my results, to extend my results, or to make their make their own research more effective
119
00:15:29.649 --> 00:15:47.009
to it's important to make sure that that's somehow recognized. Or if it's not recognized, then it's it's a waste of 2 weeks in terms of my academic career even though it might be incredibly valuable in terms of progress of science as a whole. Given the fact that there are all of these
120
00:15:47.550 --> 00:15:50.370
different tools and technologies and techniques
121
00:15:50.714 --> 00:15:59.375
that would benefit the researchers in the long run, and a lot of them have a fairly steep learning curve. What are some of the areas
122
00:15:59.800 --> 00:16:05.980
of improvement particularly in that tooling that you think would be most beneficial particularly to the,
123
00:16:06.360 --> 00:16:07.340
academic sciences?
124
00:16:08.035 --> 00:16:15.175
Yeah. There's a lot of that's a good question because there there are a lot of barriers to entry and there's pretty steep learning curves.
125
00:16:15.635 --> 00:16:16.695
1 thing that we
126
00:16:17.380 --> 00:16:19.160
we teach often is,
127
00:16:19.940 --> 00:16:25.435
Git for version control. And Git is it can be just so painful because there's,
128
00:16:25.995 --> 00:16:31.295
there's so many things to remember. There's this whole mental model of what's going on with your repository
129
00:16:32.075 --> 00:16:38.060
and the remote and the staging area and the the commit history and the directory,
130
00:16:38.440 --> 00:16:42.540
you know. And it takes a long time sometimes to get someone
131
00:16:42.875 --> 00:17:04.325
to to get someone familiar enough where Git is a a tool that helps their workflow rather than hinders their workflow. So that that's something that's pretty hard. I think it's it's sort of unfortunate that we, as a a data science community, have landed on Git and GitHub because I think Mercurial is, has a lot of the advantages of Git and and,
132
00:17:04.784 --> 00:17:15.985
makes things a little bit easier. But we've, you know, we've we've ended up in this place where where Git is sort of the lingua franca of code develop of open source development and of,
133
00:17:16.545 --> 00:17:17.045
reproducibility.
134
00:17:17.505 --> 00:17:21.665
So we have to we have to teach that if we want people to be able to to,
135
00:17:22.145 --> 00:17:29.240
work with that. So those sorts of things. And then there are other things too. Like, in in the Python worlds, the plotting thing,
136
00:17:30.179 --> 00:17:30.679
visualization
137
00:17:31.140 --> 00:17:41.875
is not all that easy. You know, matplotlib is, is a really incredible tool because it allows you to do almost anything you want as far as creating publication quality graphics. But it's,
138
00:17:43.030 --> 00:17:46.010
not all that great a tool for, like, quick exploratory
139
00:17:46.390 --> 00:17:53.985
visualization. You end up needing to do a lot of, you know, a lot of tweaking of axes and figure layouts and things like this
140
00:17:54.285 --> 00:17:57.425
that that impede your ability to to quickly
141
00:17:58.820 --> 00:17:59.720
explore data.
142
00:18:00.180 --> 00:18:00.680
So
143
00:18:01.140 --> 00:18:03.320
those kind of things are difficult and there's,
144
00:18:04.100 --> 00:18:14.015
I think there there are efforts on various fronts to to make make these tools easier to use. Oh, the the other thing I wanted to mention along that, the difficulties that academics have is,
145
00:18:14.735 --> 00:18:15.235
the
146
00:18:15.650 --> 00:18:17.030
packaging and installation
147
00:18:17.570 --> 00:18:18.070
and
148
00:18:18.450 --> 00:18:24.470
remembering, like, for Python, for example, remembering the difference between Unix paths and Python paths
149
00:18:24.875 --> 00:18:35.570
and Jupyter kernels and where they're pointing and all all this sort of stuff sometimes means that, you know, if if a student comes up to me and is using a Jupyter Notebook and says, I want to install
150
00:18:35.950 --> 00:18:45.375
Astropy so I can use it in this notebook. There's, like, this whole backtrack we need to do of, like, what kernel are you using? What Python version are you using? Are you using pipercondda?
151
00:18:46.075 --> 00:19:01.325
And how do you use pipercondda to install it in in that specific environment? You know, there's conda ends and there's virtual ends and all this sort of stuff. Like, if you if you take a little while to explain it and figure it out, it all makes sense. But for
152
00:19:01.945 --> 00:19:06.269
a a new grad student coming in who just wants to run some Python code, it's,
153
00:19:06.669 --> 00:19:09.010
the startup cost is pretty immense.
154
00:19:09.470 --> 00:19:10.929
So I think, fundamentally,
155
00:19:11.230 --> 00:19:13.335
what's going on there is you have this,
156
00:19:14.054 --> 00:19:16.955
you have tools written by, sort of,
157
00:19:17.414 --> 00:19:18.315
Linux hackers
158
00:19:18.855 --> 00:19:31.414
for Linux hackers. And then now you have people coming along who've who have only ever double clicked on icons in Windows that want to use this. And there's there's kind of a a lack of
159
00:19:31.875 --> 00:19:37.575
a lack of common experience and common knowledge by the people who design the API versus the the users.
160
00:19:38.140 --> 00:19:41.520
And then that's that's a real challenge. So it means that using these tools
161
00:19:42.460 --> 00:19:47.815
to start using these tools, you need to get people up to speed on using the UNIX command
162
00:19:48.275 --> 00:19:48.515
line, which,
163
00:19:49.715 --> 00:19:52.995
doesn't always feel like the best use of time for someone who's,
164
00:19:53.635 --> 00:20:10.765
you know, worried about funding next month and trying to get research out. So it's it it can be a little bit difficult. Yeah. The UNIX command line in particular is a long road to go down because I use it every day for my work as a systems administrator. And there are still things that are obtuse and sort of unintuitive
165
00:20:11.145 --> 00:20:16.890
for me and for people who have been doing it even for longer than I have because it's just a sort of amalgamation
166
00:20:17.190 --> 00:20:26.184
of different tools with a little bit of glue to, you know, to to bind them together. So it's interesting to see efforts such as the conch shell from Anthony Scopetz
167
00:20:26.645 --> 00:20:38.640
and how that could potentially play into trying to ease that on ramp for people who want to be able to work in this environment but not get exposed to some of the sharp edges of the overall UNIX ecosystem.
168
00:20:39.195 --> 00:20:46.155
Yeah. And there's been there been other I I like that you mentioned the conch shell. That that's a really, really nice project, and it's been
169
00:20:46.720 --> 00:21:04.054
it streamlined some of those things. Other other examples, I think, of people that are trying to smooth over those rough edge edges are the Jupyter project, you know, making kind of bringing everything through these notebooks or through this new JupyterLab interface to to sort of hide the the details of what are happening,
170
00:21:04.434 --> 00:21:05.495
for for beginners.
171
00:21:05.820 --> 00:21:14.934
And then also the the conda project, I think, has been really valuable because it lets people it it it takes away some of the installation headaches that dominated,
172
00:21:15.315 --> 00:21:19.174
the Python world in the in the 2000. You know? If you tried to
173
00:21:19.475 --> 00:21:23.335
install scipy from source even on Linux in, like, 2, 005,
174
00:21:24.019 --> 00:21:26.019
was a pain in the butt. Now you can,
175
00:21:26.419 --> 00:21:35.225
even if you're on Windows or Mac, you can type conda install scipy, and you you basically basically have it working. But 1 of the problems with that that I've been noticing in my teaching is that every time you create a new tool or a new
176
00:21:39.789 --> 00:21:47.705
a new layer that smooths out the rough edges of the layer below it, it means that everything works well 95%
177
00:21:48.485 --> 00:21:51.385
of the time, but the 5% of the time it breaks,
178
00:21:51.765 --> 00:22:00.840
your user has absolutely no vocabulary to figure out how to fix it. So the example that comes to mind is if you, if you're if you have a user using
179
00:22:01.220 --> 00:22:03.000
a using a Jupyter notebook,
180
00:22:03.654 --> 00:22:04.154
and,
181
00:22:04.855 --> 00:22:20.674
I I mentioned this before. If if you're in a Jupyter Notebook and you wanna install SciPy so that it works in your Jupyter Notebook, you need to know that the Jupyter Notebook has these kernels that are running in under it, and the kernel points to a Python environment. And a Python environment is built by virtualenv or by conda.
182
00:22:20.975 --> 00:22:47.400
And in order to install in that environment, you need to choose which installer to use, whether it's pip or conda install, and you need to make sure the environment is activated before you run it and that the the PIP that you're using points to the Python version that you're using in the Jupyter kernel. So there there's this whole, like, level of abstraction that you don't want people to worry about, and they usually don't. But then when something goes wrong, they're all of a sudden confronted with this tidal wave of
183
00:22:47.700 --> 00:22:50.435
of concepts that they need to know in order to just
184
00:22:50.755 --> 00:22:54.695
get sci fi working in their notebook. Right? And I I see that with,
185
00:22:55.395 --> 00:23:00.710
with a lot of these tools that are trying to trying to smooth things over. And I
186
00:23:01.090 --> 00:23:05.990
I debate what to do about that. Right? Because on on 1 hand, you could go the route of, like,
187
00:23:06.370 --> 00:23:16.065
let's just start with the command line and let's ignore all these tools so that we build up the knowledge people need in order to debug everything that comes up. But then you you spend 10 weeks,
188
00:23:16.524 --> 00:23:29.395
getting people to the point where they can run, you know, print hello world in a in a condo environment or something. Right? But on the other hand, if you if you try to smooth out all those and get people working with with their hands on data right away,
189
00:23:30.015 --> 00:23:38.500
maybe using a a VM that's set up perfectly to run Jupyter Notebook that they connect to over the network and and nothing's running on their own computer.
190
00:23:38.960 --> 00:23:46.095
You can do that, and you can get them analyzing data right away. But then at the end of the quarter, when that when that VM server turns off,
191
00:23:46.395 --> 00:23:58.250
they are no further along than they were at the beginning because they don't know how to use their computer. So this is the kind of question that I really have been struggling with lately is in terms of data science education is,
192
00:23:58.790 --> 00:24:01.955
what level of abstraction do you start with with the students,
193
00:24:02.335 --> 00:24:20.975
and and how much do you spend time teaching fundamentals like how to list directories in the Unix shell? And how much do you spend time relegating those questions to Stack Overflow and and teaching them how to manipulate and visualize data? It's a hard 1. The, layers of abstraction problem has been plaguing software
194
00:24:21.355 --> 00:24:23.620
since there was software, basically. So, you know,
195
00:24:24.580 --> 00:24:34.375
there's the law of leaky abstractions where every abstraction will leak at some point. And, you know, it's it's it's great until it isn't, and then it's horrible. We should all just be writing machine code. Right?
196
00:24:35.075 --> 00:24:38.695
1 of the projects that you're involved with right now is Altair,
197
00:24:39.130 --> 00:24:41.150
which you were mentioning some of the difficulties
198
00:24:41.610 --> 00:24:44.429
of being able to work with interactive visualizations,
199
00:24:45.049 --> 00:24:46.995
particularly in the context of Matplotlib.
200
00:24:47.455 --> 00:24:54.195
And my understanding is that Altair is an effort in that direction for being able to generate statistical visualizations
201
00:24:54.655 --> 00:24:55.715
with a strong
202
00:24:56.100 --> 00:25:04.919
integration with Pandas data frames. So I'm curious how the work that you're doing on that factors into your teaching, and what are some of your goals with that project overall?
203
00:25:05.475 --> 00:25:08.295
Yeah. This this project has been really interesting because,
204
00:25:09.235 --> 00:25:12.910
on my side, it's kind of a software engineering challenge, and all the conceptual
205
00:25:13.390 --> 00:25:18.450
parts of it are are sort of taken care of and baked in. So it's it's based on this,
206
00:25:19.310 --> 00:25:20.130
this visualization
207
00:25:20.430 --> 00:25:26.435
grammar called Vega Lite that comes out of the interactive data lab here at University of Washington. And
208
00:25:26.975 --> 00:25:29.075
that that lab is run by Jeff Haire.
209
00:25:29.600 --> 00:25:44.095
You may have heard of Jeff Haire because he and his grad student when he was at Stanford, his he and his grad student, Mike Bostock, created this thing called d 3, which drives a lot of interactive visualization on the web these days. But if you have ever,
210
00:25:44.655 --> 00:25:59.664
tried to use d 3 in a in a exploratory data analysis context, it's really painful. Right? Because d 3 is this incredibly imperative thing. You're like you're like drawing the axis and then drawing the tick marks and then drawing the data. And
211
00:25:59.965 --> 00:26:14.005
to create a a bar chart in d 3, it it's like a 100 lines of code or something like that. So d 3 is is awesome for these kind of polished finished products and making something that's available to the world through this this web API,
212
00:26:14.465 --> 00:26:16.725
but it's not great for the exploratory
213
00:26:17.105 --> 00:26:22.245
process. And so when when Jeff finished d 3, he he started working on
214
00:26:22.580 --> 00:26:37.775
these grammars of visualization that would kind of address that. So Vega is a is a declarative grammar of visualization that lets you say, instead of here's some code to draw the axis and then here's some code to draw some dots in the right place on those axis to visualize
215
00:26:38.090 --> 00:26:42.590
you know, create a scatter plot. Instead, what Vega does it says it says, here's
216
00:26:42.890 --> 00:26:51.855
here's the data and here's how I want the data to be visualized on the screen. It gives you a grammar to specify that. Vega itself is still a little bit complicated.
217
00:26:52.555 --> 00:27:08.054
So after Vega was more or less complete, the team started working on Vega Lite. And the idea of Vega Lite is is pretty cool. You're you're basically taking these I you're you're taking these concepts that have come out of the data visualization research over the past few years
218
00:27:08.355 --> 00:27:45.370
and creating a grammar that lets you express those concepts. So for example, if you wanna scatter plot some data on X versus Y, really fundamentally what you're doing is you're taking a column of data and mapping it to the X axis on the chart. And you're taking another column of data and you're mapping it to the y axis on the chart. And then the charting library should, once given that, should be able to generate the points and generate the axis and all these things. So all you really need to specify is, here's some data. I want x to be this column. I want y to be that column. And then the details are taken care of. So it's it's a really powerful way to,
219
00:27:45.690 --> 00:27:51.390
to quickly visualize data because it's you you don't have to worry about all these incidental details.
220
00:27:51.690 --> 00:27:57.015
And what the Alterra project does is it tries to it it creates a Python API
221
00:27:57.315 --> 00:27:58.375
around this
222
00:27:58.675 --> 00:27:59.735
grammar of visualization.
223
00:28:00.195 --> 00:28:03.735
The grammar is expressed in in the form of, JSON notation.
224
00:28:04.530 --> 00:28:07.110
And I don't like writing raw JSON dictionaries.
225
00:28:07.810 --> 00:28:10.870
So I'd rather write I'd I'd rather use some sort of
226
00:28:11.335 --> 00:28:25.580
object oriented API in Python that that generates those JSON structures. So that's what Altair tries to do. And, yeah, it's it's been a a really fun project because it allows you to it allows you to to really quickly build up complicated visualizations
227
00:28:25.880 --> 00:28:37.315
from these these core fundamental concepts. And what I'm really excited about Altera with for data science education is the fact that it number 1, it takes away the need to
228
00:28:37.695 --> 00:28:38.800
to learn this
229
00:28:39.280 --> 00:28:42.820
this very imperative boiler plate that you get in something like matplotlib
230
00:28:43.280 --> 00:28:45.540
and lets you basically just think
231
00:28:46.375 --> 00:28:48.154
about the data and the visualization.
232
00:28:48.534 --> 00:28:50.554
And also it, by design,
233
00:28:50.855 --> 00:28:51.595
it enforces
234
00:28:52.215 --> 00:28:52.715
visualization
235
00:28:53.095 --> 00:29:18.160
best practices in some ways Because you're not in an imperative language where you're building the plot element by element to the final to the final piece, you have to think at each element. Is this a good choice as far as what we know about effective data visualization? But in the case of, of Altair and Vega Lite, those good choices are being made by the code behind the declarative specification.
236
00:29:18.620 --> 00:29:20.405
So you you end up with
237
00:29:21.185 --> 00:29:21.765
with really
238
00:29:22.385 --> 00:29:22.885
useful,
239
00:29:23.665 --> 00:29:24.165
visualizations
240
00:29:24.705 --> 00:29:30.910
without having to think about about the details of that along the way. So my my colleague, Brian,
241
00:29:32.090 --> 00:29:42.275
Brian Granger, who's on the Jupyter team and he teaches down at Cal Poly San Luis Obispo. He's been using our early versions of Altair to teach data science and teach
242
00:29:42.655 --> 00:29:43.110
teach,
243
00:29:43.590 --> 00:29:45.530
like, the fundamentals of data visualization.
244
00:29:45.910 --> 00:29:52.250
And he it seems like it's been pretty successful for him because the API of the library kind of
245
00:29:52.575 --> 00:29:56.995
shifts how you think about data visualization. So you're you're thinking in terms
246
00:29:57.295 --> 00:30:04.880
of things like encoding like, champ map mapping columns to to channels, to encoding channels and and visualization
247
00:30:05.180 --> 00:30:07.840
channels. And these things that that help you
248
00:30:08.375 --> 00:30:16.795
help you make visualizations that are visualizations more effectively no matter what tool you're using. So it's sort of like, when when the a when the API
249
00:30:17.255 --> 00:30:20.780
mirrors best practice, you end up in a much better situation.
250
00:30:21.480 --> 00:30:22.620
And 1 of the utilities
251
00:30:23.000 --> 00:30:24.060
of data visualization
252
00:30:24.680 --> 00:30:27.420
in the context of doing data science
253
00:30:27.865 --> 00:30:30.285
is being able to quickly identify
254
00:30:30.985 --> 00:31:03.915
what the overall shape of the data is, identify outliers visually rather than having to painstakingly go through the underlying data or perform, you know, perform statistical analysis on the data that may or may not yield any useful results and then being able to guide the direction in which you are querying the data and trying to process it to see what are some of the underlying answers that you can gain from it. So I could definitely see how having Altair is a quick way of translating the data from the format that you already have it in your Python code to a format that you can see
255
00:31:04.375 --> 00:31:23.935
visually will drastically increase the rate at which people can explore and process that, underlying data. We still do have we we still do have a few technical challenges in it that we're working out. Like, for example, the the fact that the the Python code spits out this JavaScript notation that's then read by the JavaScript's
256
00:31:24.795 --> 00:31:35.840
Vega Lite library in order to visualize it in your browser. There are these because there are these multiple steps. Right? It's another another layer of abstraction. Right? And and we we run into issues where,
257
00:31:36.385 --> 00:31:42.405
if there's an error on the Python side, we know how to handle that. But if there's an error on the JavaScript side, it's sometimes
258
00:31:43.105 --> 00:31:47.940
hard for users to debug. You know, the initial release of Altair, if,
259
00:31:48.580 --> 00:31:52.920
you did something wrong, like, if you specified a column that doesn't exist in your data,
260
00:31:53.255 --> 00:32:05.120
the output would just be blank. Right? And it gives you no hints about the facts that you used a capital letter versus a lowercase letter in your column name. So we're still working through some of the usability
261
00:32:05.580 --> 00:32:22.350
issues of that. The other thing is that the the mode that it works in, if you have a Pandas data frame, is it serializes the data frame to JSON and then passes that whole JSON objects to the browser in order to visualize. So if you have, like, if you have a data frame with a 1, 000, 000 elements,
262
00:32:22.730 --> 00:32:28.190
then what it tries to do is serialize a a 1, 000, 000 rows into JSON, and you you end up with this
263
00:32:28.525 --> 00:32:29.265
this massive,
264
00:32:30.285 --> 00:32:37.565
massive glob of data that you're submitting to your browser. And it it can it can run make things run pretty slowly. So we're still,
265
00:32:38.205 --> 00:32:42.450
I'd say the library is still young and we're we're trying to iron out some of these,
266
00:32:42.910 --> 00:32:46.290
issues that come up. But I think I think long term as we
267
00:32:46.804 --> 00:32:51.625
work with this library with students and kinda figure out how to how to
268
00:32:52.085 --> 00:32:58.529
streamline some of those things and make make the errors easier to recover from, I think it could be a really useful library.
269
00:32:59.149 --> 00:33:05.554
And 1 of the other libraries that I'm familiar with that seems to be targeting a similar application
270
00:33:05.855 --> 00:33:09.715
is Bokeh from the folks at Continuum. So I'm wondering what was
271
00:33:10.100 --> 00:33:10.680
lacking or
272
00:33:11.620 --> 00:33:20.565
sort of difficult in that library that ins that that necessitated the work on Altair or they just targeting completely different use cases?
273
00:33:21.025 --> 00:33:29.040
Yeah. That's an interesting 1 because Bokeh start started out 5 years ago. They they really wanted to do kind of, like, a ggplot style thing for,
274
00:33:29.580 --> 00:33:40.125
for Python. I know Peter Wang, when he was starting Bokeh, he really was, he was digging into grammar of graphics and figuring out how to do those sorts of things in a in a more declarative manner.
275
00:33:40.685 --> 00:33:43.025
As as Bokeh developed, it became
276
00:33:43.485 --> 00:33:45.105
kind of more of an imperative
277
00:33:46.510 --> 00:33:47.730
engine that they wanted
278
00:33:48.350 --> 00:33:52.690
to wanted to build a a declarative wrapper on on later on.
279
00:33:53.070 --> 00:33:54.770
So if you use BookIt, it's
280
00:33:55.195 --> 00:33:58.335
it's an incredible library. You know, you can you can do these
281
00:33:58.715 --> 00:34:00.895
these amazing interactive visualizations
282
00:34:01.515 --> 00:34:04.550
online. You can target the browser. You can even do,
283
00:34:05.170 --> 00:34:17.315
really large datasets because they use HTML 5 Canvas rather than rather than SVG. So it allows you to do some bigger datasets sets than other web visualization tools. The thing that was lacking for me was the,
284
00:34:17.715 --> 00:34:18.935
the easy API.
285
00:34:19.315 --> 00:34:19.815
Bokeh,
286
00:34:20.369 --> 00:34:22.390
to to do a lot of things requires
287
00:34:22.690 --> 00:34:35.625
requires a lot of boilerplate code in general in my experience, and I know people might might get mad at me for saying that. But I I just don't I don't find it all that more intuitive than a tool like Matplotlib for your your daily,
288
00:34:36.245 --> 00:34:37.865
exploratory data analysis.
289
00:34:38.650 --> 00:34:45.950
That said, they have been working on this recently. There's a package called hollow views that started out as a separate visualization
290
00:34:46.250 --> 00:34:52.214
package, and it's now been wrapped into the Bokeh project. And it's gonna basically be Bokeh's declarative
291
00:34:52.515 --> 00:34:53.015
API.
292
00:34:53.555 --> 00:34:54.855
So that so it's
293
00:34:55.280 --> 00:34:57.940
it's sort of the exploratory data analysis
294
00:34:58.400 --> 00:35:13.025
or data visualization API built on top of this really powerful back end that's the Bokeh Live. I think I I I see a lot of potential there. I think that could be a a really nice tool as as time goes on. With Altair, we have a slightly
295
00:35:13.790 --> 00:35:15.170
different goal than
296
00:35:15.550 --> 00:35:29.105
Okay's goal. We want we want it to be an API that people can use. But key part of Altair is this Vega Lite back end. And Vega Lite is is more than just an API for visualization. It's a grammar
297
00:35:29.485 --> 00:35:31.280
of graphics serialization.
298
00:35:31.740 --> 00:35:39.359
So the idea is that if you create a plot in Altair, you can serialize it to Vega Lite. And Vega Lite is this compact
299
00:35:39.785 --> 00:35:43.724
JSON notation that you can pass to anyone that, in
300
00:35:44.345 --> 00:35:48.560
in principle tells you everything you need to know to reproduce the plot. So,
301
00:35:48.940 --> 00:35:52.880
my vision long term, what I would love to have is a matplotlib,
302
00:35:54.140 --> 00:35:58.365
reader for Vega Lite. So you can create something in Altair, send it to matplotlib,
303
00:35:58.665 --> 00:36:01.485
load it up in matplotlib, and then start doing matplotlib's
304
00:36:01.785 --> 00:36:08.930
annotations on top. Similarly, I could imagine, you know, sending sending this Vega Lite thing over to Bokeh or sending it over to ggplot,
305
00:36:09.550 --> 00:36:22.359
and having Vega Lite be this sort of lingua franca that operates between programming languages and between packages in those languages. Now that that that vision will take a long time and a lot of software engineering to realize, but,
306
00:36:22.920 --> 00:36:26.460
we're seeing a little bit of this buy in. Like for example,
307
00:36:27.240 --> 00:36:27.740
Wikimedia
308
00:36:28.280 --> 00:36:30.715
recognizes Vega Lite for visualizations
309
00:36:31.095 --> 00:36:32.955
now. So you can go on with Wikimedia
310
00:36:33.655 --> 00:36:35.515
and put a Vega Lite specification
311
00:36:35.815 --> 00:36:52.995
in there, and it will render it on the web page. And then if someone comes along and they want to modify that plot, they're not trying to reproduce someone's PNG with another package in order to add a new access label. Right? They can actually load the the Vega Lite and change the Vega Lite spec using,
312
00:36:53.935 --> 00:37:10.565
raw JSON editor. They could change it using a tool like Altair. Or, in the future, you might be able to change it with a tool like Bokeh or ggplot or matplotlib or hollow views or any of these other ones. So our vision for that is to really unify the visualization of information across languages
313
00:37:10.865 --> 00:37:11.345
and,
314
00:37:11.744 --> 00:37:32.860
using this this Vega Lite as a format. I should mention similarly that in the newer versions of Jupyter Notebook, Vega Lite is built in. So, you know, you no longer will have to load the Vega Lite library. Jupyter just knows how to knows how to handle Vega Lite. So whether you're using Jupyter from R, from Python, from Julia, if you have a visualization
315
00:37:33.240 --> 00:37:46.320
library that knows how to output these serializations, these Vega Lite specifications, then you can visualize it in Jupyter. You can visualize it on Wikimedia, and we're hoping other tools will adopt that as well. And it sounds like you've been involved in a lot of really
316
00:37:46.700 --> 00:37:48.720
interesting research and applications
317
00:37:49.180 --> 00:37:55.545
of data science. I'm wondering what are some of the most novel or noteworthy that you think are worth mentioning here?
318
00:37:55.845 --> 00:37:56.484
Yeah. So,
319
00:37:57.045 --> 00:38:02.460
there have been a lot of interesting things. My work over the past few years has been
320
00:38:02.840 --> 00:38:11.335
more along the lines of, like, the the software engineering side of things. So 1 of the things I really enjoy doing is is taking,
321
00:38:12.115 --> 00:38:15.575
algorithms and concepts that are kind of locked away in
322
00:38:16.035 --> 00:38:17.015
basically useless
323
00:38:17.315 --> 00:38:17.815
papers,
324
00:38:18.835 --> 00:38:26.000
and putting them in the form of software so that I can use them in my projects and so that others can use them in theirs. And so
325
00:38:26.380 --> 00:38:28.720
I think I think this is a a real
326
00:38:29.485 --> 00:38:32.385
strength of the kind of data science workflow that's,
327
00:38:32.685 --> 00:38:39.260
the reproducible workflow that's going on now. So I've I've been involved, for example, in the Astro Pi project, which is a,
328
00:38:39.820 --> 00:38:40.720
a a community
329
00:38:41.020 --> 00:38:44.400
collection of of astronomy routines written in Python
330
00:38:44.700 --> 00:38:45.200
and
331
00:38:45.525 --> 00:38:50.244
documented well, unit tested well, maintained well. And so it means that I can take,
332
00:38:50.724 --> 00:38:58.089
I can take algorithms that are out there in the literature that everybody has their own homegrown version of and create a really nice
333
00:38:58.390 --> 00:39:00.170
nicely implemented, nicely documented
334
00:39:00.549 --> 00:39:16.740
version of this in a Python package that then others can go out and use instead of wasting cycles, reinventing the wheel in their research. So that in in my personal work, that's been kind of the thing that I've been pretty passionate about is is doing these these sorts of things. From kind of, colleagues
335
00:39:17.295 --> 00:39:23.155
around the world and in in my institution, the 1 thing that I've been really excited about is seeing,
336
00:39:24.015 --> 00:39:24.675
the uptake
337
00:39:24.975 --> 00:39:27.970
of tools like the Jupyter Notebook for,
338
00:39:28.750 --> 00:39:29.970
publishing executable
339
00:39:30.270 --> 00:39:33.730
documents in in association with scientific projects.
340
00:39:34.215 --> 00:39:42.235
And the best example of that from the last couple years, I think, is the, the LIGO results, the these gravitational wave results.
341
00:39:42.600 --> 00:39:49.420
And there's it's amazing now that, you know, it's it's getting to the point that observing gravitational waves from a pair of colliding
342
00:39:49.805 --> 00:39:57.085
black holes is commonplace enough that it doesn't even get a press release anymore. You know, there was 1, I think, last week that sort of flew under the radar.
343
00:39:57.565 --> 00:39:57.885
And
344
00:39:59.070 --> 00:40:11.065
but but the team at LIGO is doing a great job of making the data available, making the simple analysis code available in the form of a Jupyter notebook with explanations and with code inputs and outputs.
345
00:40:11.445 --> 00:40:17.225
So you as, you know, average person on the Internet with, with Jupyter installed
346
00:40:17.670 --> 00:40:24.250
can go and download this and like see how they did their analysis and and see the statistics that go into
347
00:40:24.710 --> 00:40:25.210
extracting
348
00:40:26.055 --> 00:40:27.275
gravitational wave
349
00:40:27.575 --> 00:40:35.755
signals from this incredibly noisy data. And that sort of thing is, I I think is amazing. We're we're seeing some of the same stuff in in, journalism
350
00:40:45.224 --> 00:40:55.599
publishing Jupyter Notebooks that allow you to go in and check their assumptions and make sure that what they're doing is, you know, is is up to par. So that that sort of thing in the last
351
00:40:55.980 --> 00:40:57.339
5, 10 years has been
352
00:40:57.819 --> 00:41:01.475
5 years, I should say. Jupyter Notebooks are only about 5 years old, 6 years old.
353
00:41:01.775 --> 00:41:05.955
That sort of thing has been really cool to see, this this ability of of
354
00:41:06.255 --> 00:41:14.020
people to publish data, publish analysis scripts, and and the average technically literate person on the Internet can go in and,
355
00:41:14.580 --> 00:41:19.325
and check it and and validate it. Yeah. I think that that's 1 of the really valuable
356
00:41:20.105 --> 00:41:20.605
and
357
00:41:20.984 --> 00:41:21.484
interesting
358
00:41:22.025 --> 00:41:22.525
outcomes
359
00:41:22.905 --> 00:41:32.020
of this revolution that we've been having in terms of data science and open data is the fact that it does lower the barrier for citizen scientists
360
00:41:32.480 --> 00:41:37.194
and to get involved and also the advent of some of the more open
361
00:41:37.494 --> 00:41:44.020
journals that are making it easier to gain access to some of these publications and just have, you you know, somebody who's not necessarily
362
00:41:44.640 --> 00:41:54.405
working in academia be able to actually double check some of those results and get involved in the overall scientific process rather than having it be contained within these different universities
363
00:41:54.705 --> 00:41:56.005
or research institutions
364
00:41:56.305 --> 00:42:00.965
that, you know, have historically been harder to gain access to to the commonly person.
365
00:42:01.450 --> 00:42:15.185
It's been it's been really cool to see that and and to see it enabled by, you know, by Python tools and enabled by some of the software that my friends and colleagues are are spending their time on. It's it's great to see that impact in the real world.
366
00:42:15.485 --> 00:42:27.240
And on those lines, what are some of the other trends in data science and data analysis that you're most excited for and seeing how they play out in the future? I've already touched on some of the things that I'm most excited about is these
367
00:42:28.424 --> 00:42:29.244
I I mentioned
368
00:42:30.744 --> 00:42:31.244
the
369
00:42:31.625 --> 00:42:34.444
the visualization, like, the the Altair specification
370
00:42:34.744 --> 00:42:36.845
being this lingua franca between languages.
371
00:42:37.329 --> 00:42:39.270
I I think it's exciting to see
372
00:42:39.650 --> 00:42:44.630
that developing in other spaces too. Like, for example, Wes McKinney, who created Pandas,
373
00:42:45.135 --> 00:42:50.115
seems to be spending a lot of time recently on this sort of idea of
374
00:42:50.494 --> 00:42:52.035
translatable data structures
375
00:42:52.335 --> 00:42:55.700
in the case of data frames. You know, there's this there's this,
376
00:42:56.500 --> 00:43:00.180
feather format that both Python and R can read and,
377
00:43:00.740 --> 00:43:12.250
and these these kinds of ideas. I think, what I wanna see in the next couple of years is people stop having these language wars. You know, I I think it's it might be fun, but it's very unfruitful
378
00:43:12.550 --> 00:43:20.815
to argue about R versus Python versus whatever other tool you want. You know, these these tools are all really great and have their advantages and disadvantages.
379
00:43:21.195 --> 00:43:24.895
And if we can make the tools work together, work work at concert
380
00:43:25.260 --> 00:43:25.920
more easily,
381
00:43:26.220 --> 00:43:37.494
then I think we all win. Whereas if we if we spend our effort, like, trying to troll our users on Twitter or something like that, then it's then it's not as useful. So that that's sort of, more ecumenical
382
00:43:37.795 --> 00:43:39.414
approach to data science,
383
00:43:39.714 --> 00:43:52.915
I think is something I'm really excited about seeing. Yeah. I can definitely agree with the fact that turning the conversation from either or to yes and can have much more valuable outcomes and fosters the growth of larger communities
384
00:43:53.215 --> 00:43:57.715
than just the ones that are contained within a particular language or toolset. Yeah. Definitely.
385
00:43:58.069 --> 00:44:03.030
And are there any other topics that you think we should talk about that we didn't touch on yet? Oh, man. We,
386
00:44:03.829 --> 00:44:09.365
you know, we we covered all the stuff stuff I'm really excited about right now in the in the data science space.
387
00:44:09.744 --> 00:44:13.170
Yeah. I think the 1 thing that we didn't really cover because it deserves its
388
00:44:14.049 --> 00:44:17.510
own conversation in and of itself is the idea of repeatability
389
00:44:17.890 --> 00:44:28.685
within research. But as I said, that's a very long and involves topic that does not have any obvious solutions yet. But I think that a lot of the work that you and others are doing in the tooling and,
390
00:44:29.030 --> 00:44:36.810
sort of pedagogical techniques is sort of yielding fruit there to make it easier for that reproducibility of research.
391
00:44:37.205 --> 00:44:38.025
Yeah. The combination of
392
00:44:38.565 --> 00:44:42.825
of executable document formats like like Jupyter Notebook and R Markdown,
393
00:44:44.005 --> 00:44:49.380
with building of of software skills and the use of version control and,
394
00:44:50.080 --> 00:45:00.145
open publishing of code. I think that's really addressing that. 1 1 thing that hasn't been really well addressed in that space, I think, is reproducibility and repeatability
395
00:45:00.685 --> 00:45:05.700
on the scale of, like, 5 to 10 years versus the scale of 6 months to 1 year.
396
00:45:06.100 --> 00:45:09.480
This is something I think we, as scientists, are gonna have to address because
397
00:45:09.895 --> 00:45:18.235
packages evolve quickly. Right? So a package like NumPy, you know, you can you can do a lot of things in it. And if you install NumPy version 1.11,
398
00:45:18.880 --> 00:45:23.840
you can expect that you'll be able to do the same thing 10 years down the road. But,
399
00:45:24.560 --> 00:45:26.820
but thing features are deprecated. APIs
400
00:45:27.360 --> 00:45:33.115
are changed slowly over the course of years, and there are deprecation cycles so that, you know, if you move from NumPy
401
00:45:33.415 --> 00:45:42.460
111 to 1 dot 13, maybe something that that's going to be deprecated. There, you get a warning and you can fix it. But code that's just sitting in a repository
402
00:45:42.840 --> 00:45:47.615
for a decade, I have no confidence that I'll be able to run that code again,
403
00:45:47.994 --> 00:45:58.120
when I when I revisit it in 10 years. And that's that's a larger thing. And that goes beyond the packages, you know. Even if you even if you're able to record package versions and install legacy
404
00:45:58.420 --> 00:46:02.155
versions of Python and NumPy and and the whole tool chain, like,
405
00:46:03.095 --> 00:46:06.075
things like like VMs or Docker containers even
406
00:46:06.375 --> 00:46:14.710
are what you build that on top of. Your foundation is shifting enough that I'm not sure we'd be able to to work with it. So that that's a concern to me because,
407
00:46:15.090 --> 00:46:18.244
we're making such progress in in the reproducibility
408
00:46:18.545 --> 00:46:25.204
space right now on the, like, 1 to 3 year timeline that I think we're gonna be grappling with the reproducibility
409
00:46:25.665 --> 00:46:41.464
issue on 5 to 10 year timeline down the road. Yeah. And even to just the challenges of data preservation, particularly with these large datasets is where do you store it, who pays for it. Oh, yeah. That's a very complicated discussion and maybe 1 that we can find time for on another episode.
410
00:46:41.924 --> 00:46:49.030
Yeah. Yeah. Well, the Moore's law helps with that a little bit. Bit, you know, that like, I talked about the Sloan Digital Sky Survey that was huge
411
00:46:49.490 --> 00:46:54.230
15 years ago, but it's like, you know, 20 terabytes now, which you can buy for a few $100.
412
00:46:55.665 --> 00:46:57.125
So we're we're getting there.
413
00:46:57.585 --> 00:47:13.525
Well, I think this is probably a good place to wrap up. So for anybody who wants to get in touch with you or follow the work that you're up to, I'll have you add your preferred contact information to the show notes. And with that, I'll move us on to the picks. Okay. And my pick this week is going to be The Redwall Cookbook,
414
00:47:13.825 --> 00:47:20.609
which is just what it sounds like, for anybody who's not familiar with the Redwall series. They're this collection of novels that,
415
00:47:21.150 --> 00:47:25.890
are focused around this group of woodland creatures that have built an abbey and all of the different,
416
00:47:26.270 --> 00:47:27.970
adventures that they go on or,
417
00:47:28.675 --> 00:47:37.750
enemies that they have to battle. It's a really fun and interesting set of novels, and every 1 of them has a description of these vast and ornate feasts with very detailed
418
00:47:38.450 --> 00:47:59.579
discussions of the types of food. And so for a long time, readers of the series were asking for a cookbook, and so the author of that series eventually did release 1 that is also a story in and of itself that has the recipes that dispersed throughout. So it's a really fun book with some good recipes, and so I'll add a link to that in the show notes. And with that, I'll pass it to you, Jake. Do you have any picks for us this week?
419
00:48:00.040 --> 00:48:04.395
Oh, man. You know what I've been reading recently is a lot of nonfiction, and there's a
420
00:48:04.875 --> 00:48:05.935
historian from,
421
00:48:06.395 --> 00:48:11.295
Princeton named Kevin Kruse, who, I really, really appreciate his.
422
00:48:11.890 --> 00:48:15.089
He wrote a book called White Flight, and then the other 1
423
00:48:15.490 --> 00:48:20.069
I'm reading the next 1 right now, and I'm trying to remember what it's called. But, yeah, he he has some
424
00:48:20.395 --> 00:48:21.615
really great perspectives
425
00:48:21.994 --> 00:48:43.450
on how social trends over the last 100 years have influenced the political landscape of today. And, in light of what's going on in politics right now, I'm I'm finding it really interesting to have that sort of historical back. Definitely sounds like an interesting book. Well, I appreciate you taking the time out of your day today to join me and talk about the the work talk about the work that you're up to. It's definitely
426
00:48:43.750 --> 00:48:47.270
interesting and valuable endeavors, and I hope to,
427
00:48:47.670 --> 00:48:58.744
continue following your work because, you've already produced a lot of useful tools for myself and others. And so with that, I'd like to thank you again and I hope you enjoy the rest of your day. Yeah, thanks so much.