Brian Granger and Fernando Perez of the IPython Project

WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 7/6/2024 4:08:18 PM
Duration: 4908.044
Channels: 1
1
00:00:13.985 --> 00:00:15.525
Hello, and welcome to podcast.init.
2
00:00:16.470 --> 00:00:19.289
We're recording today on June 3, 2015.
3
00:00:20.150 --> 00:00:23.369
Your hosts as usual are Tobias Macy and Chris Patti.
4
00:00:23.945 --> 00:00:28.285
Today, we're interviewing Fernando Perez and Brian Granger about IPython.
5
00:00:29.385 --> 00:00:32.605
You can follow us on Itunes, Stitcher, or TuneIn Radio,
6
00:00:32.930 --> 00:00:37.030
and please give us feedback. You can leave us a review on Itunes or Stitcher,
7
00:00:37.410 --> 00:00:39.990
find us on Twitter, email us at hosts@podcastinit.com,
8
00:00:41.755 --> 00:00:43.535
or leave a comment on our show notes.
9
00:00:44.635 --> 00:00:47.055
Brian, Fernando, could you please introduce yourselves?
10
00:00:48.315 --> 00:00:48.815
Sure.
11
00:00:49.530 --> 00:00:55.710
Thank you folks for having us. My name is Fernando Perez. I am a staff scientist at Lawrence Berkeley National Laboratory,
12
00:00:56.465 --> 00:01:04.005
researcher at UC Berkeley, and I've been programming in Python since, 2001. I'm originally a particle physicist and applied mathematician.
13
00:01:04.465 --> 00:01:10.770
And A lot of what I do is really scientific open source software development around high content source tools.
14
00:01:11.790 --> 00:01:13.170
And I'm Brian Granger.
15
00:01:13.575 --> 00:01:16.635
I'm a physics professor at Cal Poly State University.
16
00:01:17.095 --> 00:01:22.075
It's a California state school on the central coast of California about 3 hours
17
00:01:22.590 --> 00:01:23.810
south of the Bay Area
18
00:01:24.429 --> 00:01:32.415
and I too am a physicist by training. Actually, Fernando and I were classmates in graduate school at the University of Colorado
19
00:01:33.195 --> 00:01:37.980
and originally I studied theoretical, atomic, molecular, and optical physics
20
00:01:39.180 --> 00:01:41.440
and I started with Python
21
00:01:42.220 --> 00:01:42.720
about
22
00:01:43.340 --> 00:01:44.800
I guess it was around 2000,
23
00:01:45.135 --> 00:01:46.035
late 2002,
24
00:01:46.815 --> 00:01:48.275
2003, a little bit after Fernando
25
00:01:48.655 --> 00:01:49.635
got into it.
26
00:01:50.495 --> 00:01:54.915
Very cool. It's kind of amazing how strong the connection is
27
00:01:55.920 --> 00:02:05.674
between physics and computer science. I mean, it goes just kind of beyond, well, physicists use computers. I cannot tell you how many people in my career that I've encountered
28
00:02:06.215 --> 00:02:07.115
who have now
29
00:02:07.655 --> 00:02:18.300
basically given up physics and gone full time computer science in 1 stripe or other and really just kinda never looked back. It's there seems to be some sort
30
00:02:18.680 --> 00:02:23.915
of kindred souls thing going on between people who study physics and people who end up working in the tech industry.
31
00:02:24.375 --> 00:02:37.090
It's it's kind of interesting. Yeah. And I'll say too that of the number of physicists who I've spoken with who did make the transition or at least who do programming on a regular basis, a lot of them tend to use Python for whatever reason.
32
00:02:38.925 --> 00:02:45.620
Yeah. I I think it's got the amazing numerical capabilities that Python brings to the table that other languages in this sphere kinda don't.
33
00:02:46.099 --> 00:02:48.599
So how did you folks get introduced to Python?
34
00:02:49.940 --> 00:02:51.220
In my case, it was,
35
00:02:51.700 --> 00:03:03.160
a suggestion actually from my office mate, another grad student in my case. I had a stack of complicated a complicated stack of many programming languages that I used to run my simulations when I was in grad school. A lot of Perl,
36
00:03:03.940 --> 00:03:07.560
babysit supercomputing runs, a lot of Oc, Bash, Sed,
37
00:03:07.860 --> 00:03:09.560
Big C codes, gnuplot,
38
00:03:09.860 --> 00:03:13.555
IDL, Mathematica, and it was just a complicated pile of codes.
39
00:03:14.175 --> 00:03:14.915
My officemate
40
00:03:15.295 --> 00:03:42.050
told me, hey. You should look into this Python language. It's kinda like Perl, but a lot simpler, and it can actually do a lot of numerics. It's also interactive. It's a lot simpler, and you might be able to replace many of those tools with something easier. And I 1 day, I looked at the stack of books I had on my desk, and I realized I was probably spending more time context switching between languages, between 6 or 7 languages than actually getting any work done. And I realized maybe if I could replace 5 or 6 of those with just 1 and reduce that stack maybe
41
00:03:42.670 --> 00:03:51.155
to see for the high performance stuff and maybe Mathematica because of all this, I might not get get more work done. I mean, little did I think that I would just spend all my time running Python tools,
42
00:03:51.535 --> 00:04:01.370
but, but it was true that I can't use these tools. And so and when I started learning Python and I realized after about 24 hours reading the Python tutorial
43
00:04:01.830 --> 00:04:02.390
that I had
44
00:04:03.105 --> 00:04:05.605
that I was able to do in Python
45
00:04:05.985 --> 00:04:12.780
things that about, that after a couple of years of writing heavy duty Pearl, I had no idea whether they were even possible or
46
00:04:13.080 --> 00:04:16.460
not. I realized that I would never write another line of Pearl in my life.
47
00:04:17.080 --> 00:04:17.580
So
48
00:04:17.960 --> 00:04:19.340
after that, I just
49
00:04:19.640 --> 00:04:24.134
stopped, and I I switched to Python, and I haven't stopped writing Python ever since.
50
00:04:25.794 --> 00:04:28.055
And, for me, I was actually
51
00:04:28.435 --> 00:04:32.690
as a postdoc at the time. I came back to visit Boulder for a conference,
52
00:04:33.950 --> 00:04:35.810
and Fernando was still in Boulder.
53
00:04:36.110 --> 00:04:37.330
We met and had dinner,
54
00:04:37.835 --> 00:04:42.015
and I think it was probably about a year after you started working in Python, Fernando.
55
00:04:42.875 --> 00:04:47.620
And I remember Fernando telling me about Python with this great excitement,
56
00:04:48.640 --> 00:04:51.780
and I went back to my post doc and
57
00:04:52.080 --> 00:04:54.980
1 afternoon sat down and, as an exercise,
58
00:04:55.764 --> 00:04:58.665
ported over 1 of my codes that I wrote as a grad student
59
00:04:59.125 --> 00:05:02.025
looking at phase transitions and traffic flows.
60
00:05:02.565 --> 00:05:03.065
And
61
00:05:03.605 --> 00:05:05.465
I used that to learn the language.
62
00:05:05.880 --> 00:05:07.180
And I remember within
63
00:05:07.960 --> 00:05:10.460
a few hour period, it was all in 1 afternoon,
64
00:05:11.240 --> 00:05:13.340
getting everything working with the visualization
65
00:05:14.395 --> 00:05:17.455
and just being blown away at the ease
66
00:05:18.155 --> 00:05:20.414
of both learning it and also how
67
00:05:21.354 --> 00:05:23.055
the flexibility of the language
68
00:05:23.919 --> 00:05:26.900
allowed me to explore physical questions that
69
00:05:27.919 --> 00:05:34.164
wouldn't have even occurred to me because of the limitations of previously I was doing everything in c plus plus.
70
00:05:34.625 --> 00:05:35.925
And I think seeing
71
00:05:37.264 --> 00:05:44.710
seeing how that flexibility of the language had a direct impact on my ability to do science was a huge selling point
72
00:05:45.250 --> 00:05:48.390
and over the next year, year and a half, I transitioned
73
00:05:49.490 --> 00:05:51.110
everything to, Python.
74
00:05:52.535 --> 00:05:55.755
It's very cool. It's it's really interesting how many scientists
75
00:05:56.854 --> 00:06:02.790
made that transition, made that leap from using Perl and, you know, being productive with it, but to
76
00:06:03.170 --> 00:06:06.950
moving to Python and finding that their productivity just kind of exploded.
77
00:06:07.330 --> 00:06:24.550
I certainly saw that happening at the Human Genome Project when I was there a number of years back, and people definitely regarded Perl as the sort of legacy code base that they were trying to root out wherever they had the opportunity to, and everything new was being written in Python, and they were much happier with it.
78
00:06:24.945 --> 00:06:25.445
Yeah.
79
00:06:27.585 --> 00:06:35.009
So it sounds like the 2 of you knew each other before you got involved in the IPython project. Can you tell us a little bit about how you guys
80
00:06:36.110 --> 00:06:44.354
met? Yeah. Yeah. 1st day of grad school, actually. I remember we were we yeah. We were both grad students, as Brian mentioned, in physics at CU Boulder,
81
00:06:44.735 --> 00:06:50.590
and sitting outside of the induction, 1st year grad students at in Boulder, Colorado in 1996,
82
00:06:51.289 --> 00:06:52.909
and we've been friends ever since.
83
00:06:53.689 --> 00:06:54.189
Great.
84
00:06:54.889 --> 00:06:59.565
So for anyone who may not have heard of or used IPython, can you describe what it is?
85
00:07:01.065 --> 00:07:11.760
Sure. IPython well, back in the day, IPython started its life as, as an interactive as an improving interactive shell for Python. I was a grad student in 2001,
86
00:07:12.325 --> 00:07:15.465
finishing my dissertation in in particle physics in in Boulder.
87
00:07:15.765 --> 00:07:43.940
And when I got when, at the behest of of my office mate, I I made the switch from Perl to Python. I realized, oh, wait a minute. I can actually use this thing and, and actually port a lot of my workflow to this tool, and I can add numerics, and I can add plotting, and I can actually run a lot of my codes in this tool, but I need a better interactive environment. If I'm going to actually use this tool, I wanna actually call my codes, I wanna access the file system, I want tab completion, I wanna run my scripts, I want to call out my figures,
88
00:07:44.400 --> 00:07:49.775
I want to actually run this as my environment. And I was I was used to tools like IDL,
89
00:07:50.155 --> 00:07:50.895
like Mathematica,
90
00:07:51.195 --> 00:07:57.879
like the Unix shell, and the interactive Python prompt is compared to those are very, very primitive tools. You can't do
91
00:07:58.259 --> 00:08:03.080
LS. You can't easily execute scripts and get really good trace packs. You can't easily do good debugging.
92
00:08:03.645 --> 00:08:08.945
There it it it just isn't a very, sophisticated tool for that. And so, obviously,
93
00:08:09.485 --> 00:08:20.315
a grad student facing the option of either finishing a dissertation and writing it up or playing with code, the the choice was easy. And so I dug into just building a a better interactive
94
00:08:20.715 --> 00:08:23.135
shell, and thus was born. I was born in Python.
95
00:08:23.515 --> 00:08:30.300
Eventually, my advisor kind of called me to order, and I I I went back to actually getting getting work done, and, and graduated.
96
00:08:30.699 --> 00:08:33.920
But I put out the project out there. Initially, it was just,
97
00:08:34.620 --> 00:08:41.725
as all open source projects are born, kind of a a a 1 man show for a while. It was just put, hosted on my my web page at the university.
98
00:08:42.105 --> 00:08:49.300
Then a company from Austin called Anthought offered me to put it up and host it. They were the company behind, behind the SciPy,
99
00:08:49.700 --> 00:08:51.960
library to host it on their CVS repositories
100
00:08:52.420 --> 00:09:06.690
and to host the mailing list, which today they continue to host the mailing list for the project to host it. And, they were the people behind the nascent SciPy Conferences and and the scientific Python community. And so at that point, IPython sort of became a project
101
00:09:07.150 --> 00:09:15.410
that became 1 piece in the puzzle of scientific Python and that's really what gave it life as a project and not just as a 1 off exercise.
102
00:09:15.764 --> 00:09:21.144
If it had been just that, it probably would have died as just a, okay, a 1 off little interactive exercise.
103
00:09:21.445 --> 00:09:27.839
But because it became part of other of the the workflow of other scientists and other scientists began to contribute,
104
00:09:28.220 --> 00:09:35.295
and there was a conference that at the time was held every year at Caltech for scientists building sci fi and then Matplotlib
105
00:09:35.835 --> 00:09:38.975
and numerical computing tools and other tools for scientific computing
106
00:09:39.275 --> 00:09:42.399
where we all congregated every year to build these tools,
107
00:09:43.260 --> 00:09:55.445
then, a community of users began to grow around, not just type Python, but all of these tools where we collaborated, where we exchange code, where we we kind of work together on trying to build an entire ecosystem of similarly
108
00:09:55.904 --> 00:10:20.200
developed open source tools for scientific computing, around the Python language. And today, this is now a large effort. And in fact, it was it was, I think, at the second or third of those conferences where Brian and I started collaborating on IPython because, I mean, Brian and I, obviously, as we mentioned, we knew each other well, but we hadn't we didn't start working on IPython at the very beginning, but it didn't take very long before we joined forces on the project, and it was precisely
109
00:10:20.740 --> 00:10:26.095
after me spending some time at 1 of these scientific Python conferences where we're doing courses on the project.
110
00:10:27.274 --> 00:10:40.460
Cool. That's very interesting to think of particularly for anybody who uses IPython as part of their daily work flow that if it weren't for mthaw offering to host the project that it may not have continued to exist as we know it.
111
00:10:41.475 --> 00:10:47.015
No. They were extremely they were extremely generous and and and I I we owe them a huge debt of gratitude
112
00:10:47.395 --> 00:10:51.710
because they they provided support for the project at the very beginning and throughout the project,
113
00:10:52.090 --> 00:10:55.550
and they they supported us at critical junctures. They actually funded
114
00:10:55.955 --> 00:10:57.235
Brian and I in,
115
00:10:57.795 --> 00:10:58.295
2, 009,
116
00:10:59.235 --> 00:10:59.735
2010
117
00:11:00.595 --> 00:11:03.735
at a critical point when we were developing the network protocols
118
00:11:04.115 --> 00:11:19.745
that actually allowed us to to build the machinery that eventually led to the QT console and the modern network architecture that supports the notebook and then the multi language, all of the modern features that really made possible kind of the renaissance of the project and its and its modern incarnation.
119
00:11:20.445 --> 00:11:22.625
All of that was thanks to, to MThought's
120
00:11:23.050 --> 00:11:24.589
support and financial contributions,
121
00:11:24.970 --> 00:11:31.949
and they've been a fantastic supporter of the community. They've supported the Sci Fi conference for many years, and so now we owe them a huge debt of gratitude.
122
00:11:33.875 --> 00:11:34.375
Another
123
00:11:34.754 --> 00:11:37.975
part of the history of the project that I was gonna add is that
124
00:11:38.355 --> 00:11:40.855
so Fernando and I started talking about
125
00:11:42.300 --> 00:11:45.360
some of the general ideas around interactive computing,
126
00:11:47.260 --> 00:11:48.080
in 2, 005
127
00:11:50.385 --> 00:11:51.525
and almost immediately,
128
00:11:53.265 --> 00:11:56.565
we started to think about a notebook like interface.
129
00:11:57.449 --> 00:12:00.350
And from 2, 005 until 2011,
130
00:12:01.129 --> 00:12:04.350
we actually made multiple attempts to build something like this.
131
00:12:04.810 --> 00:12:07.665
I don't what was it? 4 or 5 attempts, Fernando?
132
00:12:08.125 --> 00:12:14.390
Something like 5. I mean, in in in version 0 0 1 of the system, there's even references
133
00:12:22.630 --> 00:12:23.130
And
134
00:12:23.834 --> 00:12:30.415
And it it really took us all those different iterations of trying to build this, trying to understand the problem
135
00:12:31.329 --> 00:12:37.509
before we could do a good job of it. The other part was that a lot of the tools that we rely on such as XeroMQ,
136
00:12:37.889 --> 00:12:38.389
WebSockets
137
00:12:39.170 --> 00:12:40.149
just didn't exist.
138
00:12:41.235 --> 00:12:44.615
And the notebook came about sort of over 3 summers
139
00:12:44.995 --> 00:12:46.935
in 2009, 'ten, and 'eleven.
140
00:12:47.394 --> 00:12:51.510
We had funding in 2, 009 and 'ten to refactor
141
00:12:52.050 --> 00:12:56.685
some of the old code base to get it ready to build this more network based model.
142
00:12:57.145 --> 00:12:58.285
That was in 2009.
143
00:12:58.665 --> 00:13:02.605
And then in 2010, Fernando and I had the funding from NthoT
144
00:13:03.460 --> 00:13:07.240
to build the network architecture that enabled the qt console.
145
00:13:07.860 --> 00:13:09.400
And then in 2010,
146
00:13:10.020 --> 00:13:17.345
we built on top of that and and actually created the the current iteration of the notebook that it continues to exist today.
147
00:13:18.765 --> 00:13:22.769
Very cool. It struck me as you were as you were talking about its genesis,
148
00:13:23.070 --> 00:13:23.570
how
149
00:13:24.269 --> 00:13:25.490
how you as scientists
150
00:13:26.509 --> 00:13:29.730
had this need for a really great interactive environment
151
00:13:30.285 --> 00:13:37.770
more than just sort of a prompt with better editing capabilities and and things like that because a lot of languages have that. But
152
00:13:38.970 --> 00:13:46.510
IPython, from my perspective, is really 1 of the crown jewels of the Python ecosystem. It's something that's and IPython and IPython Notebook
153
00:13:46.935 --> 00:13:51.595
are really kind of unique. And I realize project Jupyter now encompasses other languages, but
154
00:13:51.975 --> 00:13:58.620
prior to that, there really was kinda nothing else like it that I've ever seen. And it really struck me that
155
00:13:59.160 --> 00:14:03.885
when when other people may ask in other programming language communities, well, why should we?
156
00:14:04.765 --> 00:14:16.790
What's what's the the gain for us in in sort of doing the the work to make it more make our programming language more accessible or more desirable for science work. And I would point to that and say,
157
00:14:17.090 --> 00:14:20.230
this is something that grew out of a need that only scientists
158
00:14:21.145 --> 00:14:30.840
would really have to that extent, and now the rest of the community can benefit from it because I'm not a scientist by any stretch of the imagination. I'm a I'm a tech worker
159
00:14:31.220 --> 00:14:34.040
and, an information worker, I guess you'd say.
160
00:14:34.420 --> 00:14:38.575
And the things that I can do with IPython Notebook are are just
161
00:14:39.535 --> 00:14:42.195
amazing from my, you know, limited perspective.
162
00:14:42.895 --> 00:14:46.435
So, that's really cool. Thank you for for telling us that story.
163
00:14:47.140 --> 00:14:50.600
How challenging was it to port IPython to Python 3?
164
00:14:51.460 --> 00:14:55.960
So the the the simple answer for Fernando and I is that it was completely trivial,
165
00:14:56.605 --> 00:14:59.265
but that's because he and I didn't do it. We,
166
00:15:00.605 --> 00:15:01.905
the summer of 2011,
167
00:15:02.445 --> 00:15:04.200
when we were building
168
00:15:04.740 --> 00:15:06.680
the the current version of the notebook,
169
00:15:07.220 --> 00:15:08.120
Thomas Clover,
170
00:15:08.580 --> 00:15:11.560
who's now 1 of the core developers on the project,
171
00:15:12.295 --> 00:15:15.435
came and started to to work on the Python 3 port.
172
00:15:15.975 --> 00:15:19.835
And he basically did the the initial version of that single handed
173
00:15:20.570 --> 00:15:23.310
and did an absolutely fantastic job with that.
174
00:15:23.770 --> 00:15:24.750
I I would say
175
00:15:25.690 --> 00:15:34.615
it was not a trivial port in the sense that most of the weeds importing to Python 3 have to do with strings and bytes and Unicode
176
00:15:35.475 --> 00:15:38.110
and a lot of what hypython
177
00:15:38.970 --> 00:15:40.910
deals with are strings.
178
00:15:41.529 --> 00:15:44.430
Maybe it's a string of code or a file name
179
00:15:44.810 --> 00:15:45.710
or output
180
00:15:46.145 --> 00:15:48.485
and so getting all of that working right
181
00:15:49.105 --> 00:15:55.525
was fairly non trivial in terms of the technical details though. Thomas is our resident expert on that.
182
00:15:57.810 --> 00:15:59.430
So what prompted the
183
00:16:00.769 --> 00:16:08.075
I guess it wasn't a name change from my Python to project Jupyter, but what what prompted the creation of project Jupyter as an extension to IPython,
184
00:16:08.615 --> 00:16:13.320
and were there any large changes associated with that in the IPython project
185
00:16:15.400 --> 00:16:19.740
itself? Well, the the, the the prompting was kind of just a a natural
186
00:16:20.280 --> 00:16:20.780
gradual
187
00:16:21.665 --> 00:16:22.965
realization as we were building
188
00:16:24.145 --> 00:16:24.965
the machinery
189
00:16:25.505 --> 00:16:48.904
for the system where we said, well, wait a minute. Everything that we're doing here when we looked at the protocol, when we looked at how the notebook and the console and the and the graphical console worked, and we said, wow, we're sending code, it's executed on the other side by a kernel and comes back, the results come back. There's nothing in here that really is very specific about Python other than at the other end, something takes that and happens to interpret
190
00:16:49.365 --> 00:16:51.610
that string as a blob of Python.
191
00:16:52.070 --> 00:17:11.870
It could be Perl or it could be Ruby or anything else for all it cares. Right? What if what if we just abstract that over, and then all of a sudden we have basically a generic protocol to do interactive computing in any language. Right? And what if we just clean we if we if we do that, we'll have to do a little bit of work because we've probably let,
192
00:17:12.250 --> 00:17:12.990
we've allowed
193
00:17:13.290 --> 00:17:17.914
kind of some hidden assumptions about there being Python creep into the system. But if we do that, we
194
00:17:18.554 --> 00:17:23.274
a unique system that will be that will allow us to reuse our architecture for
195
00:17:24.190 --> 00:17:27.090
And so once we kind of accepted to pay that price,
196
00:17:27.470 --> 00:17:29.090
it took some work. Yes.
197
00:17:29.870 --> 00:17:40.725
Once we accepted to pay to pay to pay the price of of abstracting over programming languages and to clean up any assumptions that we had about it being specific to the Python language,
198
00:17:41.184 --> 00:17:41.684
then
199
00:17:42.110 --> 00:17:46.610
we we saw the benefit because now all of a sudden people began actually implementing
200
00:17:47.149 --> 00:17:48.049
the back end,
201
00:17:48.590 --> 00:17:52.685
for other languages. And so we actually went through the our implementation.
202
00:17:53.225 --> 00:17:56.205
We cleaned we cleaned things up, and we defined
203
00:17:56.505 --> 00:18:01.320
how to implement what we call the kernel for other for other languages.
204
00:18:01.700 --> 00:18:07.355
And people began implementing kernel for other kernels for other programming languages, and we began seeing a Haskell
205
00:18:07.735 --> 00:18:14.315
kernel and then a Ruby kernel and then, kernels for other languages. But then, obviously, the questions
206
00:18:14.890 --> 00:18:23.149
began appearing. Why is it called IPython if I'm running the Julia kernel? And, obviously, the Julia community wanted to call it IJulia, not IPython.
207
00:18:23.495 --> 00:18:30.235
Why do I have to run at the command line, IPython dash dash Julia? Why do I have to run IPython dash dash profile Haskell?
208
00:18:30.680 --> 00:18:38.540
Right? And so we we wanted to come up with a name that would really indicate that the project is language agnostic. We may have a reference implementation
209
00:18:39.125 --> 00:18:46.505
of the kernel that may still be in Python. Much like in the Python language, CPython happens to be the reference implementation of the language,
210
00:18:47.570 --> 00:19:02.915
that happens to be written in c, but maybe at some point in the future. Who knows? PyPy may become the reference implementation in a few years if it if at some point it becomes better. So for us, IPython now is the reference implementation of our protocols. It is our default kernel. It is the 1 that we develop,
211
00:19:03.230 --> 00:19:12.929
and a lot of our tools are developed in Python. But Jupyter represents this larger project, which is about interactive computing. The name is inspired
212
00:19:13.455 --> 00:19:15.395
by Julia, Python, and R
213
00:19:15.855 --> 00:19:25.380
being the 3 what we consider to be the 3 open languages of data science. That name came to us thinking actually about a conversation that that I was having 1 evening at GitHub,
214
00:19:25.760 --> 00:19:28.419
with folks who were arguing about basically the
215
00:19:28.720 --> 00:19:35.255
those languages. And and at 1 point, I said, look. These languages are not enemies, of each other. We shouldn't be arguing amongst
216
00:19:35.635 --> 00:19:47.490
amongst ourselves. Julia, Python, and R are not enemies. The enemy is closed source science. These 3 languages are all partners in having open open source tools for scientific research.
217
00:19:48.190 --> 00:19:55.525
But it's not an acronym. It really is it really is meant to simply represent 3 languages that that that are part of an open source ecosystem
218
00:19:55.929 --> 00:20:03.070
for scientific for for scientific computing and for computing, in general. It doesn't have to be scientific research. It really is about interactive computing in general.
219
00:20:04.505 --> 00:20:05.005
So
220
00:20:05.305 --> 00:20:10.685
I know that there's a project that I came across recently called the Beaker Notebook, which
221
00:20:11.065 --> 00:20:11.965
seems to
222
00:20:12.470 --> 00:20:19.770
be based largely on IPython and project Jupyter, and it seems that it has the capability of executing multiple languages
223
00:20:20.070 --> 00:20:21.370
within the same notebook.
224
00:20:21.924 --> 00:20:22.505
Does product Jupyter
225
00:20:23.125 --> 00:20:24.265
itself have that capability?
226
00:20:27.044 --> 00:20:30.265
Well, IPython the IPython kernel can actually execute
227
00:20:30.670 --> 00:20:31.250
multiple notebooks,
228
00:20:31.710 --> 00:20:33.730
multiple languages. If you prefix
229
00:20:34.430 --> 00:20:34.930
individual
230
00:20:35.710 --> 00:20:36.210
cells
231
00:20:36.590 --> 00:20:40.450
with double percent for example, if you prefix a cell with double percent pearl,
232
00:20:40.945 --> 00:20:41.765
that cell
233
00:20:42.145 --> 00:20:42.965
can execute
234
00:20:43.505 --> 00:20:46.404
in Ruby. If you prefix a cell with double percent,
235
00:20:47.904 --> 00:20:50.164
r, that cell can be an r.
236
00:20:50.500 --> 00:20:56.280
But that is specific to the IPython kernel. Our design is that the basic protocol
237
00:20:56.660 --> 00:20:59.205
refers to 1 kernel at a time.
238
00:20:59.684 --> 00:21:03.304
That kernel can define semantics, whatever semantics it wants.
239
00:21:03.605 --> 00:21:06.745
The IPython kernel happens to define very sophisticated
240
00:21:07.130 --> 00:21:13.390
semantics that can include more than 1 language. And in fact, the IPython kernel is capable of juggling in 1 process,
241
00:21:13.770 --> 00:21:28.480
Python, Fortran loaded code, Julia, r, Bash, Ruby, you name it. You can have 20 languages juggling back and forth in the same process. So we do have that capability within the IPython kernel. But the design of the protocol is that the notebook speaks to 1 kernel at a time.
242
00:21:30.300 --> 00:21:38.794
Data scientists have adopted the use of IPython notebooks in their work at a large scale. What is it about notebooks that lend themselves to this particular problem domain?
243
00:21:41.250 --> 00:21:45.029
I think it's the fact that in, in the context of data science,
244
00:21:45.330 --> 00:21:49.909
a lot of what you're trying to do is to combine you're not trying to develop a software application.
245
00:21:50.404 --> 00:21:51.225
You're trying to understand
246
00:21:51.845 --> 00:21:53.144
a specific problem
247
00:21:53.524 --> 00:21:56.264
typically centered around a specific dataset,
248
00:21:56.644 --> 00:21:59.850
and you're trying to extract insight and communicate
249
00:22:00.870 --> 00:22:04.570
your understanding of that problem, your understanding of that insight
250
00:22:04.870 --> 00:22:20.170
as a narrative. You're trying to extract that something out of it to communicate, and that communication may be to yourself because you're gonna try to read that tomorrow or to someone else, whether it's your partner, whether it's a client, whether it's scientific research that you're going to publish.
251
00:22:20.550 --> 00:22:28.785
The point is you're executing code, but that code is not to build software. That code is to grind through your data using your tools, your tool your Python tools, your R tools, your Julia tools,
252
00:22:31.870 --> 00:22:38.910
to visualize your data, to apply an algorithm, to reduce your data, and then you will likely produce tools, produce,
253
00:22:39.310 --> 00:22:39.810
visualizations,
254
00:22:40.605 --> 00:22:56.180
summarize plots, produce summary statistics. And then, typically, you will actually introduce a narrative in English and or in whatever language you speak. You will introduce human narrative, and it's the combination of that. It's the combination of the code, the results of the code, and the human narrative that put together
255
00:22:56.675 --> 00:23:02.295
constitutes the output that you care about. And that combination together really is
256
00:23:03.235 --> 00:23:37.210
it's sort of the the perfect format for that. Right? And the ability to have all of those things together and to share that as a compact unit that you can share with anyone, that you can send to someone, and they can directly jump into it. They can re execute, and that someone can be yourself 3 weeks from now or your colleague or your audience. It's sort of the perfect atomic unit. And I think that's why it has sort of caught on like wildfire. And and as we said earlier, as Brian alluded to earlier, this is something that we we fully acknowledge we did not invent and we wanted it because we, as grad students, had used these tools in our own research and we knew how powerful these ideas were.
257
00:23:38.550 --> 00:23:42.330
So, yeah, the as you mentioned, the capability of including a narrative
258
00:23:43.065 --> 00:23:51.565
in line with the processing and display of the data that you're talking about is very powerful and I've actually seen
259
00:23:52.200 --> 00:24:01.500
at at well, a few instances, but 1 in particular of somebody actually using IPython as a platform for writing a book. And the 1 I'm thinking of in particular is
260
00:24:01.875 --> 00:24:05.735
Bayesian Methods for Hackers, I believe, is the title by Cameron Davidson Pylon.
261
00:24:06.675 --> 00:24:08.935
Yeah. And there there's been a number of
262
00:24:09.795 --> 00:24:11.335
books or book like
263
00:24:12.420 --> 00:24:13.800
compilations of notebooks.
264
00:24:14.500 --> 00:24:17.400
Another example of that is Jose Unpinko.
265
00:24:18.100 --> 00:24:20.760
Started out writing a series of blog posts
266
00:24:21.075 --> 00:24:23.414
on doing signal processing with Python,
267
00:24:24.034 --> 00:24:27.095
and then he he wrote those blog posts as notebooks
268
00:24:28.115 --> 00:24:29.735
and then worked with Springer
269
00:24:30.280 --> 00:24:32.300
to turn those notebooks into
270
00:24:32.680 --> 00:24:33.260
a traditional
271
00:24:33.800 --> 00:24:34.860
Springer book
272
00:24:36.120 --> 00:24:37.660
on signal processing.
273
00:24:38.945 --> 00:24:41.605
And we're starting to see a number of other books
274
00:24:42.225 --> 00:24:43.525
developed in that way.
275
00:24:44.145 --> 00:24:44.645
And
276
00:24:45.105 --> 00:24:45.605
another
277
00:24:46.150 --> 00:24:48.170
exciting development along these lines
278
00:24:49.590 --> 00:24:51.610
is that in the last month,
279
00:24:52.150 --> 00:24:53.210
O'Reilly Media
280
00:24:54.395 --> 00:24:55.935
has built support for
281
00:24:56.635 --> 00:24:57.135
notebooks
282
00:24:57.515 --> 00:25:00.495
into their publishing platform which is Atlas
283
00:25:01.070 --> 00:25:03.890
and so O'Reilly authors can now
284
00:25:04.590 --> 00:25:09.649
author their content that end up in O'Reilly Books and online posts
285
00:25:10.275 --> 00:25:14.695
as notebooks and then also distribute that content to users as notebooks.
286
00:25:15.315 --> 00:25:19.175
And so that's another example of these computational narratives
287
00:25:20.230 --> 00:25:24.010
being authored in different contexts and distributed in in different contexts.
288
00:25:26.630 --> 00:25:37.414
That's really very cool. I mean, I think that's that's something that's been missing. You know, when I when when you look at how younger people just aren't consuming books like a lot of us
289
00:25:37.830 --> 00:25:51.125
became used to sort of coming up in the field now. They're they're using online resources like blogs and and the like. I think 1 of the things that that is causing that, or at least contributing to it, is that lack of immediacy.
290
00:25:52.065 --> 00:25:53.525
And I think that
291
00:25:53.905 --> 00:25:56.565
having them be able to open up an ebook
292
00:25:57.159 --> 00:25:57.659
and
293
00:25:57.960 --> 00:25:59.740
bring up an IPython notebook
294
00:26:00.440 --> 00:26:08.835
with, if they're trying to learn functional programming or something, you know, a lesson in what are monads and how do I use them that's interactive that they can actually
295
00:26:09.294 --> 00:26:14.100
play with. I think that could really sort of that could save technical books.
296
00:26:16.080 --> 00:26:18.420
That that's something that we're very excited about.
297
00:26:18.800 --> 00:26:19.300
Yeah.
298
00:26:20.275 --> 00:26:29.095
So the IPython Notebook seems like a really great tool for educators in advanced fields. Have you seen widespread adoption in this area? And is it a focus for the project?
299
00:26:29.980 --> 00:26:32.400
Yeah. It's definitely a focus for the project,
300
00:26:32.860 --> 00:26:33.760
partially because
301
00:26:34.220 --> 00:26:35.679
we ourselves are educators,
302
00:26:36.220 --> 00:26:36.720
and
303
00:26:37.554 --> 00:26:40.135
we end up using the notebook a lot ourselves,
304
00:26:41.395 --> 00:26:45.174
to teach. So this is 1 area that we definitely dog food a lot.
305
00:26:45.700 --> 00:26:49.160
Just this quarter, actually today, I had the last class
306
00:26:49.620 --> 00:26:54.335
for this spring quarter of a computational physics course that I'm teaching here to undergrads,
307
00:26:55.195 --> 00:26:58.255
but we're seeing the notebook used for teaching
308
00:26:58.955 --> 00:27:05.640
across a wide range of fields ranging from bioinformatics to to physics to data science to computational fluid dynamics.
309
00:27:06.260 --> 00:27:07.800
And honestly, at this point,
310
00:27:08.180 --> 00:27:12.645
we're having trouble even keeping track of all the different people who are doing this now,
311
00:27:13.025 --> 00:27:13.525
and
312
00:27:14.065 --> 00:27:14.565
it
313
00:27:15.025 --> 00:27:19.205
it definitely is a focus for us in terms of development as well.
314
00:27:19.830 --> 00:27:22.890
What we're finding is that if you're
315
00:27:24.230 --> 00:27:24.730
writing
316
00:27:25.270 --> 00:27:27.289
a notebook as a single individual,
317
00:27:27.985 --> 00:27:31.845
the workflow is fairly decent now, and and the user experience,
318
00:27:32.625 --> 00:27:36.005
when you start to work with a large group of people,
319
00:27:36.610 --> 00:27:37.750
such as in a classroom,
320
00:27:38.530 --> 00:27:45.510
in a collaborative context where you want to distribute materials to the students, you want to gather materials back from the students,
321
00:27:46.095 --> 00:27:47.475
there's grading involved.
322
00:27:48.015 --> 00:27:51.875
There's still a lot of significant pain points associated with that
323
00:27:52.335 --> 00:27:55.510
and we're we actually have a a new project
324
00:27:56.050 --> 00:27:56.870
called nbgrader,
325
00:27:57.410 --> 00:27:58.390
as in notebookgrader,
326
00:27:59.410 --> 00:28:02.870
that is designed to help address some of those pain points.
327
00:28:03.355 --> 00:28:04.895
And, the lead developer
328
00:28:05.275 --> 00:28:08.655
of that is a grad student at Berkeley, Jess Hamrick,
329
00:28:09.435 --> 00:28:18.420
and she has been working on a course at Berkeley that's been using this since January, and then I've been using it for the last few months.
330
00:28:18.880 --> 00:28:21.835
And it definitely helps, but we still have a long ways
331
00:28:22.635 --> 00:28:25.774
to go and there's a lot of interesting work that I think
332
00:28:26.395 --> 00:28:27.135
in terms
333
00:28:28.475 --> 00:28:30.495
of really making the notebook practical
334
00:28:31.179 --> 00:28:31.840
for teaching,
335
00:28:32.380 --> 00:28:34.720
you know, we really need to address these points.
336
00:28:36.299 --> 00:28:39.225
That's very cool. I can I can totally see that
337
00:28:39.625 --> 00:28:40.765
as someone who,
338
00:28:41.225 --> 00:28:46.684
to be honest, I'm a really crappy student, to be just to be perfectly frank about it? And I think that,
339
00:28:47.490 --> 00:28:51.670
for me, 1 of the things that that made my classroom experience
340
00:28:51.970 --> 00:28:53.670
very difficult is that
341
00:28:54.130 --> 00:29:06.710
I definitely found there was a a real gulf when I sat there in class and just tried to absorb what was being taught to me and what I was being tested on. And so to have the opportunity to have this
342
00:29:07.170 --> 00:29:08.310
kind of interactive
343
00:29:09.090 --> 00:29:27.890
thing that I can noodle on and really sort of, like for me, I feel like I have a learning cliff, you know. I have to play with something and poke at it and prod at it and do it, do it, do it. And then all of a sudden, I really get it. And it sounds like using a notebook in the classroom context, and even in in terms of grading notebooks
344
00:29:28.270 --> 00:29:29.490
in a test environment
345
00:29:30.029 --> 00:29:31.695
sounds like something that I could
346
00:29:32.575 --> 00:29:34.514
really get behind. That's really exciting.
347
00:29:36.575 --> 00:29:37.075
Yep.
348
00:29:38.735 --> 00:29:41.154
So, GitHub recently added the ability
349
00:29:41.500 --> 00:29:43.680
to render notebooks in a repo.
350
00:29:44.300 --> 00:29:46.320
Did you work with them to build that integration?
351
00:29:49.260 --> 00:29:49.760
Yeah.
352
00:29:50.220 --> 00:29:50.720
Though,
353
00:29:51.405 --> 00:29:59.745
to be fair, the the bulk of the hard work was done by the GitHub team, but we did coordinate with them for a long time, and we discussed we we discussed extensively
354
00:30:00.480 --> 00:30:17.225
sort of what what would be required for it and how we could help, with it. But the vast majority of the work was obviously on on their internal architecture. But, yes, it was something that we were very keen on seeing happen, and it was wonderful to collaborate with the GitHub team. And they were they were great, they were great about it.
355
00:30:17.765 --> 00:30:22.030
Yeah. It definitely seems like something that will really drive even more awareness
356
00:30:22.330 --> 00:30:22.830
of
357
00:30:23.770 --> 00:30:27.230
the presence and capabilities of IPython to people
358
00:30:27.795 --> 00:30:30.615
outside of Python who may have never come across it before?
359
00:30:31.795 --> 00:30:34.455
Yeah. We're we're we're really excited about that and
360
00:30:35.660 --> 00:30:40.880
I think, you know, for a long time now, we have had the n b viewer site
361
00:30:41.660 --> 00:30:42.480
that allows
362
00:30:43.425 --> 00:30:47.045
users to see a static HTML rendering of a notebook
363
00:30:47.665 --> 00:30:50.885
and the way nb viewer works is you could pass it a URL
364
00:30:51.185 --> 00:30:52.005
or a GitHub
365
00:30:53.110 --> 00:30:54.330
repo and
366
00:30:54.630 --> 00:30:55.130
org
367
00:30:56.470 --> 00:30:58.409
and view notebooks in that context
368
00:30:58.870 --> 00:31:02.625
and that that's been a very popular platform for people to share notebooks,
369
00:31:03.645 --> 00:31:06.625
but I think part of the awkwardness is that
370
00:31:07.245 --> 00:31:13.070
a lot of people are already on GitHub and they're browsing through repos and notebooks in that context
371
00:31:14.170 --> 00:31:24.554
and it does get to be a little difficult when you click on a notebook on at least this is how it used to be click on a notebook on GitHub and all you see is raw JSON data
372
00:31:24.855 --> 00:31:28.480
and so to to be able to see on GitHub the rendered notebook
373
00:31:29.419 --> 00:31:32.480
is is a huge improvement in the user experience.
374
00:31:32.940 --> 00:31:34.080
And I I think
375
00:31:35.100 --> 00:31:36.960
part of what we're hoping is that
376
00:31:38.684 --> 00:31:41.024
this drives the discussion about
377
00:31:41.965 --> 00:31:43.105
open and reproducible
378
00:31:43.644 --> 00:31:44.144
science
379
00:31:44.684 --> 00:31:45.825
and journalism
380
00:31:46.880 --> 00:31:50.980
and we're starting to see more and more people who are using GitHub
381
00:31:51.600 --> 00:31:57.225
and the notebook as a platform for sharing their content associated with various types of publications
382
00:31:57.924 --> 00:32:04.250
in a way that allows users to come on and look at exactly what they've done and reproduce it if they want to.
383
00:32:06.570 --> 00:32:08.830
Very cool. Yeah, there's definitely
384
00:32:09.530 --> 00:32:17.034
I actually was just at the open data science conference here in Boston this past weekend, and Nice. There was a lot of discussion around
385
00:32:17.975 --> 00:32:27.169
open data and open access to data and exchange of ideas around that and it's definitely a narrative that I've been seeing pop up in a lot of places that
386
00:32:27.549 --> 00:32:28.049
people,
387
00:32:29.434 --> 00:32:34.815
even just the average citizen who may not have any interest or facility in technology
388
00:32:35.674 --> 00:32:36.174
are
389
00:32:36.809 --> 00:32:38.830
becoming more aware of
390
00:32:40.090 --> 00:32:45.070
the data that they produce and what that what sort of an impact that has on their lives,
391
00:32:45.675 --> 00:32:52.495
and also the fact that there's just a general need for them to become a little bit more well versed in
392
00:32:54.110 --> 00:32:56.290
how that data is used and how algorithms
393
00:32:57.230 --> 00:33:02.535
are present in everything that they do from when they wake up to when they go to bed if they live in any
394
00:33:03.495 --> 00:33:04.235
sort of a modern
395
00:33:04.615 --> 00:33:06.315
and developed nation.
396
00:33:06.695 --> 00:33:09.195
So it's very cool to see IPython
397
00:33:10.270 --> 00:33:13.570
being used as a tool to help facilitate some of that transparency.
398
00:33:18.025 --> 00:33:22.845
So what are some of the most interesting uses of IPython and IPython Notebooks that you have seen?
399
00:33:24.184 --> 00:33:24.825
Well, we,
400
00:33:25.625 --> 00:33:26.765
we've tried to
401
00:33:27.240 --> 00:33:28.140
sort of
402
00:33:29.400 --> 00:33:33.820
enable enable things with with some of with some of our infrastructure,
403
00:33:34.705 --> 00:33:35.205
and
404
00:33:35.585 --> 00:33:50.810
we we collect we collect a gallery. We have a gallery of of interesting notebooks that of, where we keep track of some of some of the interesting things that we see. It's it's on our Wiki. It doesn't have it doesn't have absolutely, obviously, absolutely everything
405
00:33:51.350 --> 00:33:51.670
because it's
406
00:33:52.715 --> 00:33:53.995
it it it only has,
407
00:33:54.395 --> 00:33:56.335
it's it's limited by our own bandwidth.
408
00:33:56.875 --> 00:34:13.765
And it's a manually it's a manually curated directory. But, but there are there are things like reproducible and and this is obviously biased by our sign sign of interest. The things like reproducible academic publications is a is a section that, as as Brian was mentioning already, is particular interest to us. There is, already a couple of dozen
409
00:34:14.065 --> 00:34:34.404
scientific papers that you can, go to and some of which are very high profile scientific papers that you can click on and are accompanied by an entire GitHub repository that has carefully curated collections of notebooks. Some of them have 10 IPython notebooks that you can click through and follow every single step of the process to reproduce
410
00:34:34.944 --> 00:34:38.530
every table and figure in the paper, which is really, really beautiful
411
00:34:38.830 --> 00:34:45.570
to see scientists sort of saying, this is how I did all of the work in that paper, follow through, and and get all of my results.
412
00:34:46.575 --> 00:34:47.875
You can see notebooks
413
00:34:48.415 --> 00:34:48.915
also
414
00:34:49.935 --> 00:34:52.515
that have purely educational value, notebooks
415
00:34:52.895 --> 00:34:55.010
on topics like sing signal processing,
416
00:34:55.390 --> 00:34:55.890
engineering,
417
00:34:56.830 --> 00:34:59.970
linguistics, machine learning, physics, chemistry, math,
418
00:35:00.270 --> 00:35:00.770
visualization,
419
00:35:01.390 --> 00:35:06.785
whimsical stuff like analyzing soccer soccer data, analyzing Wikipedia data, all kinds of topics.
420
00:35:07.265 --> 00:35:19.140
And and it's it's kind of an interesting collection for the public to peruse because you can learn about all kinds of things. And it's a very it's a very useful way of seeing the kinds of stuff that people are doing. There are also
421
00:35:20.160 --> 00:35:20.980
very notable
422
00:35:21.825 --> 00:35:32.680
projects that that that people have carried, that people have carried that that have kind of highly dedicated effort. For example, Lorena Auerbach, who is who is an educator that Brian mentioned earlier,
423
00:35:33.220 --> 00:35:34.680
recently put together
424
00:35:35.140 --> 00:35:35.800
a MOOC
425
00:35:36.180 --> 00:35:37.560
that was taught collaboratively.
426
00:35:37.940 --> 00:35:43.125
She's a professor in mechanical engineering at George Washington University, and she put together a MOOC
427
00:35:43.425 --> 00:35:47.285
taught between herself, a colleague of hers at the University of Southampton,
428
00:35:47.890 --> 00:35:51.990
and another colleague of hers in Chile. And the 3 of them put together
429
00:35:52.369 --> 00:35:55.025
a move taught on the Open edX platform,
430
00:35:56.305 --> 00:36:00.404
that is an a large collection of IPython Notebooks on numerical
431
00:36:00.865 --> 00:36:01.365
computing,
432
00:36:01.744 --> 00:36:02.724
all of it available
433
00:36:03.105 --> 00:36:04.440
online, all of it
434
00:36:05.079 --> 00:36:09.660
presented on the Open edX platform and that you can work through as basically,
435
00:36:10.040 --> 00:36:15.335
and execute the entire set to teach yourself numerical computing with Python end to end. Another,
436
00:36:15.715 --> 00:36:18.295
another highlight that has been extremely popular
437
00:36:18.675 --> 00:36:24.520
is Peter Norvig's collection of iPython Notebooks. Peter Norvig, who's the director of research at Google,
438
00:36:24.820 --> 00:36:29.320
periodically will tackle interesting problems that he finds that he's thinking about.
439
00:36:30.100 --> 00:36:34.765
And he will often post his solutions to to challenging problems as notebooks.
440
00:36:35.145 --> 00:36:37.645
And those notebooks tend to generate a lot of traffic
441
00:36:38.105 --> 00:36:48.895
online. We see them. We there are several of his posts are listed on on on our gallery, and they're very, very interesting. Recently, he posted 1 on the traveling salesman problem, for example, which is a fascinating
442
00:36:49.195 --> 00:37:00.660
exam discussion on how on dynamic programming and how to tackle the traveling salesman program, where he goes through and analyzes the problem very, very carefully. And it's a it's a great example precisely
443
00:37:01.040 --> 00:37:02.580
showing why the notebook
444
00:37:02.960 --> 00:37:31.325
is a good way is a good platform for this because he builds these little snippets of code where he explains what he's about to do. He builds a tiny little function, then he runs it. He shows the result and he builds another small function. The code is never very large, and you can follow the discussion and understand his train of thought. And and he tackles really, really complex logic and builds it gradually. And then finally, a very recent project that just came online a few days ago from a neuroscientist at Janelia Farm, a rising star neuroscientist
445
00:37:31.780 --> 00:37:32.440
in neuroscience called,
446
00:37:32.980 --> 00:37:34.320
Jeremy Freeman is,
447
00:37:34.740 --> 00:37:35.480
at notebooks.codeneuro.org.
448
00:37:38.100 --> 00:37:40.600
He is a demonstration that uses some of our technology,
449
00:37:41.005 --> 00:37:48.065
some of the Docker images that we built, and a project that comes out of, IPython called TempNB that allows you to basically spawn
450
00:37:48.365 --> 00:37:53.310
ephemeral notebooks notebooks in Docker containers. And Jeremy has put together,
451
00:37:53.850 --> 00:37:56.510
the TempNB infrastructure together with his libraries,
452
00:37:56.965 --> 00:38:00.165
libraries that he has built using Spark, the, the,
453
00:38:01.525 --> 00:38:09.849
the distributed processing framework together with his code for the analysis of time series data. And he's put this beautiful system dot called notebooks.codnoord.org
454
00:38:11.270 --> 00:38:15.130
to demonstrate the analysis of time series data taken
455
00:38:15.494 --> 00:38:18.714
from in in this case, you have some examples from mice and from zebrafish.
456
00:38:19.015 --> 00:38:25.529
And it's an entire demonstration that you can just open and click and and load up these container these Spark containers,
457
00:38:26.150 --> 00:38:31.609
with code and data with the entire Spark machinery ready to go and learn how to use these libraries
458
00:38:32.225 --> 00:38:33.525
and and run the demonstrations
459
00:38:33.905 --> 00:38:36.405
with real with, with real datasets
460
00:38:37.025 --> 00:38:39.840
running this code on on on very interesting scientific datasets.
461
00:38:40.400 --> 00:38:49.140
So it's a it's a it's a it's a very it's a very relevant and very timely timely example of using both our tools and the Spark framework as well as Jeremy's,
462
00:38:49.805 --> 00:38:51.105
Jeremy's research libraries
463
00:38:51.725 --> 00:38:52.705
to demonstrate
464
00:38:53.485 --> 00:38:54.225
it online.
465
00:38:55.885 --> 00:39:00.680
Yeah. That's 1 of the really kinda incredibly compelling things about IPython Notebook.
466
00:39:01.540 --> 00:39:07.234
Even as you as kinda dovetailing on what Tobias said for members of the general public. Right? Because
467
00:39:07.615 --> 00:39:11.234
lately, there has been kind of an increasing fascination with
468
00:39:12.340 --> 00:39:17.320
consuming data in in terms of these infographics that you see in places like
469
00:39:17.620 --> 00:39:20.600
wired and floating around the Internet and things like that.
470
00:39:20.915 --> 00:39:24.855
And what's kind of amazing about IPython Notebook, even for a layperson,
471
00:39:25.155 --> 00:39:25.655
is
472
00:39:26.035 --> 00:39:33.580
it's it's like, you know, going from a photo to virtual reality. It's that kind of a transition going from these static infographics
473
00:39:34.200 --> 00:39:42.415
into an IPython notebook because you could you the data is live. You can actually sort of slice it, dice it, manipulate it, view it differently,
474
00:39:43.035 --> 00:39:43.855
change it.
475
00:39:44.475 --> 00:39:47.740
It's a living, breathing thing that you can actually really
476
00:39:48.520 --> 00:39:53.805
sort of interact with and move through and, you know, make your own inferences about. It's it's
477
00:39:54.605 --> 00:39:55.585
really kind of
478
00:39:56.444 --> 00:40:02.145
it's really just to say amazing is not giving it enough praise. It's kinda mind boggling, actually.
479
00:40:04.000 --> 00:40:07.860
So are there any notable projects that use IPython as 1 of their components?
480
00:40:10.160 --> 00:40:10.880
Yeah. There's,
481
00:40:11.725 --> 00:40:16.705
there's actually, quite a few and we have a list on the website. We probably just wanna highlight
482
00:40:17.165 --> 00:40:23.310
a few right now. 1, I I wanna plug my plug my day job because it's a very interesting,
483
00:40:23.690 --> 00:40:24.190
scientific
484
00:40:24.570 --> 00:40:28.670
research project funded by the Department of Energy. It's a large platform called KBASE
485
00:40:29.055 --> 00:40:30.355
for computational biology.
486
00:40:30.815 --> 00:40:33.635
It's a project led by Lawrence Berkman National Laboratory,
487
00:40:34.415 --> 00:40:41.970
and that is also which is a large collaboration also with Argonne National Lab, Oak Ridge National Laboratory, and Brookhaven National Lab to build a platform
488
00:40:42.670 --> 00:40:43.170
for,
489
00:40:43.470 --> 00:40:43.970
bioinformatics,
490
00:40:44.830 --> 00:40:48.210
at scale that is meant to allow biologists
491
00:40:48.515 --> 00:40:49.975
to basically analyze,
492
00:40:50.355 --> 00:40:51.495
large scale datasets
493
00:40:52.115 --> 00:40:53.655
and and actually collaborate
494
00:40:54.035 --> 00:40:55.255
on large scale analysis
495
00:40:55.780 --> 00:41:00.840
from single organ from single suborganisms to microbial communities all the way to plants,
496
00:41:01.380 --> 00:41:03.800
and plants and and modeling plants in their environments
497
00:41:04.100 --> 00:41:07.215
and do complex and predictive modeling across,
498
00:41:07.615 --> 00:41:13.940
data that lives in the entire system and collaborate on that in the entire interface. And sort of the the whole
499
00:41:14.400 --> 00:41:16.900
system runs as a as a highly customized
500
00:41:17.680 --> 00:41:18.180
system,
501
00:41:18.720 --> 00:41:22.099
through IPython. So the the what is called the narrative interface
502
00:41:22.425 --> 00:41:24.845
is really a very complex and customized
503
00:41:25.224 --> 00:41:38.035
IPython interface, and that actually is my main my main research project, in my day job as a scientist at the Department of Energy. And this is a project that has roughly 50 people working both on it at at the Department Energy.
504
00:41:38.415 --> 00:41:38.915
So,
505
00:41:39.215 --> 00:41:40.835
it's currently in its 4th year.
506
00:41:41.295 --> 00:41:54.280
Another project that I wanna mention, which is a project that we've collaborated for a long time with, it's called Sage. It's an open source mathematics project, which is written in Python. It was started by a mathematician who used to be a professor at UCSD
507
00:41:54.740 --> 00:41:58.275
when he started it. He's now professor at the University of Washington in Seattle.
508
00:41:58.654 --> 00:42:05.635
Sage is a is a project that basically tries to unify many open source math projects
509
00:42:05.950 --> 00:42:09.810
as well as writing its own code, providing a lot of new code in Python,
510
00:42:10.830 --> 00:42:29.369
basically, for providing a a single unified environment for math. It's not so much for numerical computing. It's like NumPy and SciPy and Matloflib. Even though it uses NumPy, SciPy and Matloflib, it was really more designed for pure math it was originally especially, more designed for pure mathematics. The creator of Sage is William Stein. He's a number theorist.
511
00:42:29.910 --> 00:42:34.535
And it's a very rich project. It has used like Python since the beginning. The the original,
512
00:42:34.855 --> 00:42:36.954
William contacted me back in 2005.
513
00:42:37.575 --> 00:42:40.870
The terminal console for Sage has always been a
514
00:42:41.270 --> 00:42:42.170
a very customized
515
00:42:42.470 --> 00:42:47.690
version of IPython because Sage has its own custom syntax, which isn't quite Python. He actually preparses
516
00:42:48.390 --> 00:43:00.235
Python to to allow syntax, which is invalid Python. And so for him, the fact that hypython allowed a custom parcel is perfect because that's exactly what he was what he wanted to do. Sage has had a notebook
517
00:43:00.540 --> 00:43:20.720
since the beginning, and there's been a very kind of playful back and forth between the Sage notebook and the IPython notebook where we've learned from Sage and Sage has learned from us. And we've there's been kind of a competitive collaboration where we've learned from each other over the years. But there's been a very but we've collaborated for a long time. It's been a very fruitful relationship, and we're very good friends with the Sage team.
518
00:43:21.020 --> 00:43:21.520
Quantopian
519
00:43:21.900 --> 00:43:30.085
is a company, is a startup in, in the Boston area that has built a collaborative platform for financial modeling where the,
520
00:43:30.705 --> 00:43:32.485
the, environment allows
521
00:43:32.865 --> 00:43:35.285
people basically to design trading algorithms
522
00:43:36.100 --> 00:43:41.400
that run on the Quantopian platform. Quantopian offers people the data and the compute environment.
523
00:43:41.700 --> 00:43:44.755
And if people have algorithms that perform well, then they can basically,
524
00:43:45.155 --> 00:44:07.905
expose those algorithms to others and and and and market them. And that platform is actually built on top of the IPython machinery, running IPython kernels on top of their customized runtime and IPython notebooks. And Quantopian developers actually contribute to the project. They they have made fantastic contributions, and they've developed new infrastructure and and and new machinery for the project that they and we have a great relationship with them.
525
00:44:08.309 --> 00:44:10.569
Wakari, which is a hosted platform
526
00:44:11.030 --> 00:44:15.130
from a company called Continuum Analytics, which is another big player in the
527
00:44:16.125 --> 00:44:21.825
in the scientific computing ecosystem, is a company that offers effectively a completely hosted
528
00:44:22.205 --> 00:44:23.985
is a completely hosted IPython
529
00:44:24.610 --> 00:44:25.910
IPython notebook system
530
00:44:26.210 --> 00:44:37.484
that they've offered for a while. We should we should mention that a lot of the the hosted stuff that we've mentioned such as tempingb before and many of our services and a lot of the hosted notebooks that you see online,
531
00:44:37.944 --> 00:44:38.845
a lot of this
532
00:44:39.305 --> 00:44:41.880
is actually backed by the fact that,
533
00:44:42.500 --> 00:44:44.359
notebooks are available hosted
534
00:44:44.980 --> 00:44:51.175
by Rackspace. Many of our services are actually hosted by Rackspace because Rackspace actually develops
535
00:44:51.795 --> 00:45:02.800
a lot of our, hosts a lot of our services. Rackspace is a company that develops a lot of Python code. 1 of our core developers is a full time Python full time IPython developer and he's a full time Rackspace
536
00:45:03.245 --> 00:45:03.745
engineer.
537
00:45:04.125 --> 00:45:07.585
And, they host a full time temp this ephemeral
538
00:45:07.965 --> 00:45:09.745
Python service called TempNB
539
00:45:10.160 --> 00:45:11.540
is hosted by Rackspace,
540
00:45:11.840 --> 00:45:13.140
and a lot of this infrastructure
541
00:45:13.440 --> 00:45:14.980
is, has been built,
542
00:45:15.280 --> 00:45:19.140
with Rackspace resources. So these are some of the both open source
543
00:45:19.535 --> 00:45:20.035
government
544
00:45:20.415 --> 00:45:21.535
and, commercial,
545
00:45:21.934 --> 00:45:28.835
projects that in 1 form or another are are building on top of the the IPython machinery. I hope that answers the question.
546
00:45:29.760 --> 00:45:31.300
Very much so. It does.
547
00:45:31.600 --> 00:45:35.940
That is a pretty impressive list. So definitely a lot of very cool projects
548
00:45:36.320 --> 00:45:43.464
for people to check out. There are a couple I've heard of, but a number that I haven't heard of as well. So I guess I know what I'm doing for the rest of the evening.
549
00:45:45.230 --> 00:45:51.330
So where do you see project Jupyter going in the future, and are there any particular new features you'd like to see added?
550
00:45:53.025 --> 00:45:54.005
Yeah. So we
551
00:45:55.905 --> 00:45:58.085
have quite ambitious plans for the future.
552
00:45:58.464 --> 00:46:02.244
It's been a huge transition for us over the last 12 months
553
00:46:02.880 --> 00:46:03.860
moving from
554
00:46:04.680 --> 00:46:05.180
the
555
00:46:06.000 --> 00:46:08.420
more narrow focus of just Python
556
00:46:09.200 --> 00:46:12.744
to the broader focus of many different languages
557
00:46:13.845 --> 00:46:14.345
and
558
00:46:14.964 --> 00:46:19.385
I think 1 of the the biggest challenges we have right now is scaling up organization
559
00:46:20.370 --> 00:46:21.270
from a human perspective.
560
00:46:22.450 --> 00:46:25.110
Part of that is attracting new developers
561
00:46:25.410 --> 00:46:27.350
but honestly part of it is just
562
00:46:27.865 --> 00:46:28.365
finding
563
00:46:28.665 --> 00:46:33.965
good ways of working with the developers we have. We we have enough people now working on the project
564
00:46:35.240 --> 00:46:36.540
that it's really challenging
565
00:46:37.320 --> 00:46:41.020
to just get all of us working in a productive way.
566
00:46:41.320 --> 00:46:42.540
Part of that is
567
00:46:43.405 --> 00:46:44.625
working on getting
568
00:46:44.925 --> 00:46:47.025
more sustainable funding for the project
569
00:46:47.405 --> 00:46:49.505
and also building up the user community
570
00:46:49.885 --> 00:46:51.025
in useful ways.
571
00:46:51.349 --> 00:46:53.849
In terms of new features that we're looking
572
00:46:54.390 --> 00:46:56.970
at, 1 of the big areas is collaboration
573
00:46:57.990 --> 00:46:59.290
and the biggest
574
00:46:59.735 --> 00:47:04.075
feature that we're targeting right now in that front is real time collaboration
575
00:47:04.775 --> 00:47:07.355
similar to what you get in the context
576
00:47:07.815 --> 00:47:12.100
of Google Drive where multiple people can open up a document at the same time,
577
00:47:12.400 --> 00:47:14.740
edit it at the same time, see the edits.
578
00:47:15.360 --> 00:47:15.860
And
579
00:47:16.160 --> 00:47:16.660
about,
580
00:47:17.205 --> 00:47:19.385
I guess, it's about a year and a half ago now,
581
00:47:19.765 --> 00:47:24.425
some engineers at Google approached us with a very nice prototype
582
00:47:24.950 --> 00:47:25.849
of this capability
583
00:47:26.390 --> 00:47:27.210
that integrated
584
00:47:27.750 --> 00:47:28.490
the Google
585
00:47:28.869 --> 00:47:30.730
Cloud APIs for collaboration,
586
00:47:31.030 --> 00:47:33.130
the document storage with the
587
00:47:33.510 --> 00:47:34.570
IPython notebook
588
00:47:35.495 --> 00:47:39.595
and we currently have funding from Google for a postdoc
589
00:47:40.055 --> 00:47:42.635
at Berkeley that's helping us build that
590
00:47:42.935 --> 00:47:43.435
basically
591
00:47:44.590 --> 00:47:49.010
integrate that the work that they did into the Jupyter notebook
592
00:47:49.470 --> 00:47:50.210
more efficiently,
593
00:47:50.830 --> 00:47:55.545
and we have some other funding that's going to be coming online to help us with that work as well.
594
00:47:56.244 --> 00:47:59.065
A lot of the work that we're looking at
595
00:47:59.605 --> 00:48:00.505
over the next,
596
00:48:01.070 --> 00:48:02.690
I would say, 2 to 3 years
597
00:48:03.150 --> 00:48:06.450
has to do with making our user interface
598
00:48:07.230 --> 00:48:13.245
and architecture more modular. We're seeing a lot of people who want to take different components of the notebook
599
00:48:13.945 --> 00:48:15.950
and plug it into different contexts.
600
00:48:16.490 --> 00:48:17.230
For example,
601
00:48:17.930 --> 00:48:20.030
what O'Reilly Media has done
602
00:48:20.570 --> 00:48:23.310
is taken the back end, the architecture
603
00:48:23.610 --> 00:48:24.670
for running code,
604
00:48:25.325 --> 00:48:30.944
but their front end is not the the traditional notebook user interface. They have a custom user interface
605
00:48:31.724 --> 00:48:33.984
and they're using part of our JavaScript
606
00:48:34.450 --> 00:48:38.230
code but right now it's a little bit wonky to do that
607
00:48:38.609 --> 00:48:41.030
and so we're spending a lot of time
608
00:48:41.465 --> 00:48:44.285
redesigning our user interface to make it more modular
609
00:48:44.825 --> 00:48:47.885
and part of this gets back to the user experience that
610
00:48:48.345 --> 00:48:49.325
people have
611
00:48:50.010 --> 00:48:51.550
in the notebook right now.
612
00:48:51.850 --> 00:48:52.990
The latest version
613
00:48:53.530 --> 00:48:56.030
of the notebook also includes the text editor
614
00:48:56.730 --> 00:48:57.790
and a terminal
615
00:48:59.345 --> 00:49:04.725
And if you've used that, you'll quickly realize that you might want to have a notebook
616
00:49:05.025 --> 00:49:07.670
next to a text editor or next
617
00:49:08.549 --> 00:49:10.650
to a tabbed set of terminals.
618
00:49:11.589 --> 00:49:18.605
And so we're really pushing the limits on what's possible in the in the web browser in terms of
619
00:49:18.985 --> 00:49:21.805
like paneled layouts, paneled and tab layouts
620
00:49:22.505 --> 00:49:25.405
and we're we're going to be spending a lot of time
621
00:49:26.059 --> 00:49:27.440
working on the user interface.
622
00:49:29.020 --> 00:49:35.119
A lot of the things we're thinking about have to do with just the general usability and design of the notebook.
623
00:49:36.125 --> 00:49:40.385
While the current notebook definitely, I think, eases the pain,
624
00:49:40.925 --> 00:49:41.425
of
625
00:49:41.965 --> 00:49:43.345
a lot of usage cases,
626
00:49:43.809 --> 00:49:48.710
there's still a lot of pain points that our our users experience every day with our current notebook.
627
00:49:49.170 --> 00:49:53.190
And so we're gonna spend a lot of time thinking about the usability of it.
628
00:49:53.965 --> 00:49:55.505
I don't know, Fernando. They're
629
00:49:56.685 --> 00:49:58.705
I'm probably missing a lot here. You wanna
630
00:49:59.165 --> 00:50:00.465
jump in on some of this?
631
00:50:01.690 --> 00:50:07.950
No. No. I I think those are. That that's pretty good. That's a that's a pretty good summary of the of the highlights.
632
00:50:09.654 --> 00:50:16.954
Well, if we get nothing else, I'm sure people will be absolutely ecstatic with just the features that you listed above because those are
633
00:50:17.280 --> 00:50:19.619
pretty remarkable. Even just the collaborative
634
00:50:20.160 --> 00:50:21.380
real time editing
635
00:50:21.680 --> 00:50:25.460
of a notebook would be pretty amazing for a lot of people. I'm sure.
636
00:50:26.545 --> 00:50:29.444
Actually, 1 other 1 other general area that
637
00:50:29.825 --> 00:50:32.404
we see a lot of work on and that is
638
00:50:33.640 --> 00:50:35.980
multi user deployments of the notebook.
639
00:50:37.160 --> 00:50:40.060
We have a new sub project called JupyterHub
640
00:50:41.205 --> 00:50:43.225
that adds a multi user
641
00:50:43.925 --> 00:50:44.425
deployment
642
00:50:45.285 --> 00:50:48.185
system for the notebook that you can deploy to
643
00:50:48.760 --> 00:50:51.900
multiple users on a centralized server or servers and
644
00:50:53.160 --> 00:51:00.555
there are different people using that in different ways. Temp n b, which we host is 1 example of that,
645
00:51:01.095 --> 00:51:04.155
but there's also a lot of other companies, startups,
646
00:51:04.615 --> 00:51:05.755
scientific projects,
647
00:51:06.780 --> 00:51:09.280
universities leveraging that in different ways
648
00:51:09.740 --> 00:51:13.280
and that's a very active area of development for us.
649
00:51:14.405 --> 00:51:18.425
May maybe maybe maybe 2 2 very quick comments.
650
00:51:18.965 --> 00:51:23.145
1 is that the collaboration actually has is has multiple angles.
651
00:51:23.700 --> 00:51:31.400
The the real time live collaboration is is obviously a super important 1. But in the context of what Brian just said of of
652
00:51:31.755 --> 00:51:39.215
JupyterHub and and these multi user things, it's actually a much more nuanced problem because it really is about what does it mean to share
653
00:51:40.120 --> 00:51:43.020
projects that involve notebooks when notebooks are executable
654
00:51:43.320 --> 00:51:43.820
things.
655
00:51:44.360 --> 00:51:45.740
So when you share
656
00:51:46.200 --> 00:51:52.665
things on, say, Google Docs, you're just sharing static documents. When you share projects on something like, say, GitHub,
657
00:51:53.125 --> 00:51:54.425
you're sharing effectively
658
00:51:54.805 --> 00:51:55.305
repos
659
00:51:55.760 --> 00:52:00.500
where there's a model of there's the forking model of for repository contributions.
660
00:52:01.280 --> 00:52:05.665
Neither of those models quite work well when you actually have executable state.
661
00:52:06.205 --> 00:52:07.745
And so there's a lot of nuance
662
00:52:08.205 --> 00:52:33.340
that comes into play when you're talking about sharing and collaborating on things that have live effectively live executable state in memory. And we haven't completely figured out exactly how that's what that should look like. And so there's a lot of really interesting questions in that. Some of which have to do with with the version which is about live live collaboration, but others go beyond that. And so there's a lot of open questions there that we'll be hammering on in the next few years.
663
00:52:33.640 --> 00:52:38.474
And, and the other the other point that I wanted to mention is the fact that we're also exploring
664
00:52:38.935 --> 00:52:43.115
sort of the questions that go in the direction of what we call sort of
665
00:52:43.655 --> 00:52:44.635
software engineering
666
00:52:44.935 --> 00:52:45.755
in the notebook,
667
00:52:46.420 --> 00:52:48.599
which is a kind of a catchphrase
668
00:52:48.900 --> 00:52:49.720
for asking,
669
00:52:50.260 --> 00:52:53.079
well, if I'm if I'm developing code interactively,
670
00:52:53.700 --> 00:52:55.240
but I want to extract
671
00:52:55.595 --> 00:53:14.915
maybe something reusable out of it, what would that look like if I don't simply want to say copy it out into a file and then turn it just into a library file that I edit in text editor. If I actually want to maybe keep it in a notebook because I maybe want to be able to go back and still test it interactively. If I wanna keep something which remains live
672
00:53:15.295 --> 00:53:23.240
because I want to keep it with live examples or something, but I still want to be able to use it as a library, How could we make the notebook environment
673
00:53:23.779 --> 00:53:29.559
better support those use cases? And we're not quite sure what that should look like, but we feel that it's our responsibility
674
00:53:30.245 --> 00:53:31.225
to sort of explore
675
00:53:31.845 --> 00:53:41.260
basically where the notebook creeps a little bit into an IDE. And that may be anathema to some people, but we feel that it's our job to kind of at least push that push that boundary a little bit.
676
00:53:41.560 --> 00:53:42.060
Yeah.
677
00:53:43.480 --> 00:53:45.820
I I guess 1 1 other area is
678
00:53:46.635 --> 00:53:50.655
the Jupyter notebook supports interactive widgets that allow you to
679
00:53:50.955 --> 00:53:52.735
quickly build user interfaces
680
00:53:53.640 --> 00:53:58.700
in the notebook documents that are bound to data and objects in the back end kernels
681
00:53:59.559 --> 00:54:01.420
and right now those
682
00:54:02.575 --> 00:54:03.795
those user interfaces
683
00:54:04.335 --> 00:54:07.395
can really can be embedded in these notebook documents
684
00:54:08.255 --> 00:54:10.595
and there's a very strong demand
685
00:54:11.519 --> 00:54:14.660
for people to be able to develop those user interfaces
686
00:54:15.039 --> 00:54:15.940
in a notebook
687
00:54:16.720 --> 00:54:18.740
document but then deploy them
688
00:54:19.475 --> 00:54:21.415
in more of a dashboard context
689
00:54:21.715 --> 00:54:22.695
where the consumers,
690
00:54:23.875 --> 00:54:29.200
the viewers of that content may not want to see any code at all. Maybe all they want to see is
691
00:54:30.460 --> 00:54:35.520
a visualization widget has some sliders and a text box and some check boxes to
692
00:54:36.125 --> 00:54:39.425
specify what algorithms are being run or the options
693
00:54:39.965 --> 00:54:41.585
passed to various functions.
694
00:54:42.205 --> 00:54:43.265
And so the sort
695
00:54:43.725 --> 00:54:45.025
of viewing a notebook
696
00:54:46.350 --> 00:54:46.850
as
697
00:54:47.310 --> 00:54:48.690
a sort of web application
698
00:54:49.070 --> 00:54:50.050
slash dashboard
699
00:54:50.430 --> 00:54:51.170
type entity
700
00:54:51.630 --> 00:54:59.885
that can be deployed and used and reused in different contexts is something that we're we're thinking a lot about and there's a lot of interest in.
701
00:55:01.385 --> 00:55:08.260
That is so cool. I mean, just the the thought of being able to like, I'm thinking about in industry in my day to day work,
702
00:55:08.960 --> 00:55:13.175
I'm relatively new to Python, still I've only been using it for the last 6 or 7 months.
703
00:55:13.555 --> 00:55:16.215
And to be able to go to 1 of the more senior
704
00:55:16.515 --> 00:55:19.895
Python devs and say I'm having trouble with this piece of code
705
00:55:20.480 --> 00:55:29.220
and not having to completely, if they're working from home or something, deal with Skype or whatever the case may be, to be able to be able to share an IPython notebook
706
00:55:29.885 --> 00:55:30.545
with them
707
00:55:30.925 --> 00:55:33.985
with the code that they that we can then collaboratively
708
00:55:35.165 --> 00:55:37.400
edit and play with and evolve.
709
00:55:38.339 --> 00:55:39.559
The the opportunities
710
00:55:40.020 --> 00:55:42.359
with that are just amazing to me.
711
00:55:43.059 --> 00:55:46.955
A few weeks ago, we interviewed Jonathan Slenders who wrote pt Python.
712
00:55:47.335 --> 00:55:53.670
I don't know if you folks have encountered that project, but it brings IDE like capabilities to interactive Python.
713
00:55:54.290 --> 00:55:56.950
So if you're at the prompt and you have,
714
00:55:57.490 --> 00:56:03.365
an object defined or even just, let's say, like string, you can say string dot and hit tab,
715
00:56:03.825 --> 00:56:04.725
and ptpython
716
00:56:05.105 --> 00:56:06.005
will give you,
717
00:56:06.465 --> 00:56:07.845
in your text prompt,
718
00:56:08.190 --> 00:56:10.930
a list of completions that you can then choose
719
00:56:11.310 --> 00:56:13.730
to invoke the method that you wanna invoke.
720
00:56:14.270 --> 00:56:17.490
Have you ever considered including this kind of capability in in IPython?
721
00:56:18.575 --> 00:56:25.715
There there's a thread there's a thread already with the author on the mailing list this week. So if you check our archives, like, 2 days ago, there's a thread with him already.
722
00:56:26.750 --> 00:56:36.785
That's great. I I I wonder if, you know, I perhaps concededly, I'm wondering if we had some small part in that because when we interviewed him a few weeks ago, he said he had done a shell
723
00:56:37.164 --> 00:56:41.505
or, you know, he had done some work to allow IPython to be used
724
00:56:41.890 --> 00:56:42.950
from within PTPython,
725
00:56:43.330 --> 00:56:44.870
but the 2 kernels were incompatible.
726
00:56:45.730 --> 00:56:53.565
And I I said to him at that point in time, well, you know, it sounds to me like maybe it might be worth talking to those guys because
727
00:56:53.945 --> 00:56:55.885
if you could add this capability
728
00:56:56.345 --> 00:56:58.365
to IPython, it would be
729
00:56:58.900 --> 00:57:00.119
a really incredible
730
00:57:00.500 --> 00:57:12.464
opportunity to to improve, you know, to raise the bar once again. That's great. I'm glad it's being thought about and potentially worked on. Now we're looking into it and partly because apparently there's there there seems to be really fundamental
731
00:57:13.165 --> 00:57:13.665
incompatibilities
732
00:57:14.045 --> 00:57:15.345
between Python 3.5
733
00:57:15.645 --> 00:57:22.660
and the library that we need on Windows to support the the terminal called PyReadLine. Apparently, it doesn't work at all on Python 3.5.
734
00:57:23.255 --> 00:57:29.755
And, it apparently, the thing that this guy is using may work on Windows out of the box and it's pure Python dependency.
735
00:57:30.460 --> 00:57:33.599
So we may swap out we may swap out Pyreeline
736
00:57:33.900 --> 00:57:42.065
with his thing, and it may actually give us sort of better capabilities than ReadLine and kind of, and actually support a richer experience.
737
00:57:42.765 --> 00:57:55.345
ReadLine, even though it seems to be a simpler dependency because it apparently has shipped, it actually turns out to be a really thorny 1 because on on OSX, it's actually not shipped at fault because it is a it's a fake read line. You get to look at it, which
738
00:57:55.645 --> 00:58:02.200
is has its own set of problems. And on Windows, it doesn't even exist. And what we have is is this fake thing called PyredLine,
739
00:58:02.580 --> 00:58:12.375
which apparently on 3 5 doesn't even run because of changes that were made to Python. And it's not clear at all that will ever be fixed. And so we're we're we're actually looking seriously at the same.
740
00:58:13.075 --> 00:58:19.950
That's great. I wouldn't know We don't we don't know what'll what'll happen, but but it is a possibility even for pretty serious technical reasons.
741
00:58:20.730 --> 00:58:27.065
Very cool. I wouldn't normally recommend that 1 set of guests go back and listen to an interview with another set of guests, but
742
00:58:27.605 --> 00:58:32.025
you might actually consider take giving a listen to our interview with Jonathan because
743
00:58:32.340 --> 00:58:33.400
he kind of explains
744
00:58:34.180 --> 00:58:35.960
how some of it works, and
745
00:58:36.260 --> 00:58:38.680
it really seems like a very elegantly
746
00:58:39.540 --> 00:58:40.760
well thought through
747
00:58:41.140 --> 00:58:41.640
implementation.
748
00:58:42.405 --> 00:58:43.785
It really honestly,
749
00:58:44.405 --> 00:58:49.145
talking with him really kind of impressed me. So I'm I'm glad to hear that you folks are looking at that. And
750
00:58:49.685 --> 00:58:50.505
and, honestly,
751
00:58:51.125 --> 00:58:51.705
it has
752
00:58:52.320 --> 00:58:57.060
moved me to go really browse some of his code because his implementation is very,
753
00:58:57.440 --> 00:59:00.055
very interesting. I think it can t it's it's nice
754
00:59:00.535 --> 00:59:03.275
in a way kind of a simple use case for asyncio,
755
00:59:03.575 --> 00:59:07.915
a very understandable use case. So it makes for some really interesting code reading.
756
00:59:08.375 --> 00:59:09.675
Martin here Martin here.
757
00:59:10.200 --> 00:59:12.620
Yeah. It's very cool having this connection
758
00:59:13.000 --> 00:59:14.460
between guests that we
759
00:59:14.760 --> 00:59:20.414
have interviewed and are interviewing. So just very cool to see how that comes about in such
760
00:59:20.954 --> 00:59:24.494
a broad and welcoming community as Python. So very cool.
761
00:59:26.154 --> 00:59:32.059
So what are some of the features of IPython and IPython Notebooks that an average user might not know about?
762
00:59:33.960 --> 00:59:40.375
So I'm gonna take a little bit of liberty and stretch the definition of the word average.
763
00:59:41.075 --> 00:59:41.575
Absolutely.
764
00:59:42.275 --> 00:59:43.895
And I think the
765
00:59:44.570 --> 00:59:45.550
the most important
766
00:59:45.850 --> 00:59:46.910
point that
767
00:59:47.450 --> 00:59:49.150
in this front is that
768
00:59:50.010 --> 00:59:51.070
the the architecture
769
00:59:53.075 --> 00:59:55.415
that all of these tools are built on
770
00:59:56.115 --> 00:59:59.095
is a very open flexible architecture
771
01:00:00.290 --> 01:00:02.710
and the the main building blocks of this,
772
01:00:03.010 --> 01:00:04.550
1, is the notebook format.
773
01:00:05.410 --> 01:00:06.870
So the notebook documents
774
01:00:07.250 --> 01:00:08.845
are just a JSON
775
01:00:09.224 --> 01:00:11.964
data structure that is stored on the file system
776
01:00:12.904 --> 01:00:13.404
and
777
01:00:14.025 --> 01:00:15.724
there's a lot of flexible
778
01:00:16.184 --> 01:00:16.845
of flexibility
779
01:00:17.860 --> 01:00:20.360
around that notebook document. For example,
780
01:00:21.300 --> 01:00:27.400
it's really easy to read and write these notebook documents from just about any programming language you could imagine.
781
01:00:28.015 --> 01:00:28.995
So for example,
782
01:00:29.695 --> 01:00:31.795
you may have some system that's not
783
01:00:32.575 --> 01:00:34.835
the Jupyter notebook where there's content.
784
01:00:35.609 --> 01:00:36.349
It would be,
785
01:00:36.730 --> 01:00:39.069
I mean, a few lines of code
786
01:00:39.529 --> 01:00:40.029
to
787
01:00:40.809 --> 01:00:44.829
import and export data from this other system to notebooks.
788
01:00:45.994 --> 01:00:49.535
Another example is the network protocol that we have
789
01:00:49.915 --> 01:00:51.535
for communicating with kernels
790
01:00:51.915 --> 01:00:54.895
where kernels are the separate process that we start
791
01:00:55.260 --> 01:00:57.440
that runs code in a particular language
792
01:00:59.340 --> 01:01:02.320
and there's a lot of people thinking about
793
01:01:02.780 --> 01:01:05.775
and working on various, I would say,
794
01:01:06.795 --> 01:01:07.295
instantiations
795
01:01:07.915 --> 01:01:10.015
of interactive computing systems
796
01:01:10.875 --> 01:01:16.070
and what we're seeing is that the the people who take the time to understand our architecture
797
01:01:17.170 --> 01:01:19.030
end up getting, like, superpowers
798
01:01:19.490 --> 01:01:22.950
in terms of what they can do in a really short period of time.
799
01:01:24.055 --> 01:01:26.795
1 example of that is the Atom
800
01:01:27.174 --> 01:01:29.674
text editor that came out of GitHub
801
01:01:30.535 --> 01:01:31.434
and recently
802
01:01:33.310 --> 01:01:36.050
someone came up with an Adam plugin called hydrogen
803
01:01:37.150 --> 01:01:38.609
that can run code,
804
01:01:38.990 --> 01:01:40.849
that you have in the Adam text editor
805
01:01:41.335 --> 01:01:43.435
using any of the Jupyter kernels
806
01:01:44.295 --> 01:01:44.795
and
807
01:01:45.175 --> 01:01:49.994
the person who developed this took the time to understand our network protocol and how this worked
808
01:01:51.260 --> 01:01:57.120
and what they got from that is that their plug in could all of a sudden run code in any of the
809
01:01:57.660 --> 01:02:00.080
approximately 40 languages that we support
810
01:02:00.845 --> 01:02:01.585
and so
811
01:02:02.204 --> 01:02:04.785
with a very minimal amount of work
812
01:02:05.405 --> 01:02:07.424
compared to what it would take to
813
01:02:07.760 --> 01:02:11.140
get something talking to 40 different languages from scratch.
814
01:02:11.840 --> 01:02:16.654
And so I I think the thing that I would encourage people is that, you know, if
815
01:02:16.954 --> 01:02:19.694
you're interested in in taking what we've done
816
01:02:20.154 --> 01:02:28.920
and using it in different ways, it will be really worth your time to to spend the time understanding the different building blocks that we have,
817
01:02:29.220 --> 01:02:30.359
the network architecture,
818
01:02:31.059 --> 01:02:36.815
and all of this And there's a lot of potential for reassembling these building blocks in new and interesting ways.
819
01:02:38.555 --> 01:02:42.390
So is there anything in particular that you would like to ask our listeners to help with?
820
01:02:45.730 --> 01:02:46.630
Yeah. I mean,
821
01:02:47.085 --> 01:02:53.105
obviously, first of all, as as every open source project is join us as kind of as developers.
822
01:02:53.724 --> 01:02:57.890
We're we're an open source project. There's lots of work to do on multiple
823
01:02:58.190 --> 01:02:58.690
areas.
824
01:02:59.069 --> 01:03:02.049
There's front end work to do in the notebook
825
01:03:02.605 --> 01:03:05.265
On the, on the UI, there's lots of JavaScript
826
01:03:05.724 --> 01:03:08.385
work to do. There's also work in the back end.
827
01:03:08.924 --> 01:03:14.000
There is, the the Brian already mentioned kind of the the document publishing pipeline,
828
01:03:14.859 --> 01:03:27.605
the the idea of writing notebooks and turning them into documents, into websites, into blog posts, into papers, into books. It's a complex pipeline, but it's a really interesting problem that has many different aspects and many targets.
829
01:03:27.960 --> 01:03:35.175
And and that can be actually a pretty fascinating 1 that, and that is somewhat isolated from other aspects of the project.
830
01:03:35.655 --> 01:03:38.475
And so it it doesn't necessarily intersect with,
831
01:03:39.255 --> 01:03:55.745
with everything else. There's a piece of the project that that is somewhat technical but also well isolated, which we haven't talked too much about today, which is all of the parallel computing stuff, a little bit more technical, a little bit more on the scientific end, but also something where people can contribute. But an important and completely different 1 is that,
832
01:03:56.225 --> 01:04:00.325
as the community grows and as the project grows, 1 part that doesn't scale
833
01:04:00.800 --> 01:04:01.460
is us.
834
01:04:01.760 --> 01:04:08.420
We cannot be everywhere at once. We we cannot fly everywhere. We can't be in multiple locations.
835
01:04:08.974 --> 01:04:09.954
And so organizing
836
01:04:10.655 --> 01:04:11.635
meetups, organizing
837
01:04:11.935 --> 01:04:20.480
kind of community events in other places. And 1 thing that we do have a little bit of is resources. We can if if community if folks in the community want to organize
838
01:04:20.859 --> 01:04:23.440
a meetup and they want to basically run a small
839
01:04:23.900 --> 01:04:25.920
community event, we can probably
840
01:04:26.300 --> 01:04:27.905
send your way some resources,
841
01:04:28.205 --> 01:04:29.585
even some financial resources
842
01:04:29.965 --> 01:04:44.815
to support those events because we are getting some fun some, some funding that will allow us to support to the community in that fashion. And that is 1 way that actually helps us because it means we don't have to be the ones who do who do have to get on a plane to be everywhere.
843
01:04:45.755 --> 01:04:46.494
And also
844
01:04:46.955 --> 01:04:51.780
be a little bit patient with us in the sense that we know that our documentation in some aspects of the project
845
01:04:52.260 --> 01:04:55.720
are not ideal. We are working hard, to improve them.
846
01:04:56.260 --> 01:04:56.500
We,
847
01:04:57.540 --> 01:04:58.440
we'll get there.
848
01:04:59.845 --> 01:05:05.065
So is there anything that we didn't ask you about or didn't talk about that you would like to bring up?
849
01:05:06.244 --> 01:05:09.520
Yeah. I think we we do wanna make sure that we that we credit,
850
01:05:09.820 --> 01:05:12.080
sort of 2 aspects of the project.
851
01:05:13.100 --> 01:05:14.400
1 is our,
852
01:05:15.075 --> 01:05:25.910
the rest of our core developers. Even though Brian and I are here and have spoken today a lot, the project is not is doesn't exist of us. It really exists because of the rest of our core developers.
853
01:05:26.609 --> 01:05:31.750
We have we have a number of folks who work either full time or nearly full time on the project
854
01:05:32.135 --> 01:05:36.155
and who are absolutely fantastic. And they're really the ones who make all the magic happen.
855
01:05:36.455 --> 01:05:37.595
Jonathan Frederick,
856
01:05:38.135 --> 01:05:39.035
Kyle Kelly,
857
01:05:39.369 --> 01:05:42.750
Minragan Kelly, Thomas Kloiber, Matthias Busonnier,
858
01:05:43.530 --> 01:05:45.310
Jess Hamrick, Damian Avila,
859
01:05:45.690 --> 01:05:48.109
Sylvain Corley, Jason Grout,
860
01:05:49.595 --> 01:05:51.295
Ryan am I forgetting someone?
861
01:05:52.235 --> 01:05:53.775
Some of the folks at Google?
862
01:05:54.235 --> 01:05:56.655
Yes. Tester Tong, in particular,
863
01:05:57.270 --> 01:05:57.770
Kyra
864
01:05:58.310 --> 01:05:58.810
Patel.
865
01:05:59.590 --> 01:06:01.130
And I I may be forgetting,
866
01:06:01.830 --> 01:06:02.730
Paul Ivanoff,
867
01:06:03.430 --> 01:06:08.875
who is a little bit less active lately but was a very active core developer for a long time in the project.
868
01:06:09.655 --> 01:06:10.395
And also
869
01:06:11.255 --> 01:06:11.835
our institutional,
870
01:06:12.535 --> 01:06:26.414
sponsors, not, not just the the institutions that I happen to employ Brian and myself, Cal Poly and UC Berkeley slash LBL, but the fact that we have institutions who support our work by funding it, the Alfred P. Sloan Foundation,
871
01:06:27.035 --> 01:06:28.575
the Gordon and Betty Moore Foundation,
872
01:06:29.115 --> 01:06:32.740
Rackspace in particular has supported us enormously. Microsoft,
873
01:06:33.520 --> 01:06:35.140
Google, the Simons Foundation,
874
01:06:35.760 --> 01:06:36.660
Continuum Analytics,
875
01:06:36.960 --> 01:06:37.460
MThought.
876
01:06:37.839 --> 01:06:43.125
If it wasn't, if it weren't for for these, for these entities, the project just wouldn't exist.
877
01:06:43.425 --> 01:06:44.485
Without their resources,
878
01:06:44.945 --> 01:06:49.605
it wouldn't exist. We also wanna give a shout out to Numfocus. Numfocus is
879
01:06:50.050 --> 01:06:59.590
a 5 0 1 c 3. It is a foundation that actually supports the project, in a slightly different manner. It's a foundation that, Brian is in the board of directors.
880
01:06:59.955 --> 01:07:07.655
I'm an ex member of the board of directors. I'm actually 1 of its founders. And it's kind of a parallel entity, if you will, to the Python Software Foundation,
881
01:07:07.955 --> 01:07:12.730
but that was created by the scientific computing community to really serve
882
01:07:13.109 --> 01:07:14.410
as a as an umbrella
883
01:07:14.869 --> 01:07:15.369
foundation
884
01:07:15.750 --> 01:07:19.130
for the open source scientific computing community
885
01:07:19.795 --> 01:07:30.079
to serve the needs of scientific projects that are that are open source open source project. And it is the project that hosts the IPython slash Jupyter project
886
01:07:30.460 --> 01:07:31.680
as a as a 501c3.
887
01:07:32.619 --> 01:07:40.285
And, and Enfocus is our fiscal is our fiscal sponsor. It's our fiscal, it's our fiscal home. It's what allows us to receive to receive
888
01:07:41.705 --> 01:07:42.445
tax deductible
889
01:07:42.825 --> 01:07:43.565
tax deduct
890
01:07:44.185 --> 01:07:45.725
tax deductible donations.
891
01:07:46.310 --> 01:07:46.710
And,
892
01:07:47.110 --> 01:08:02.685
and, and that's a and it's a very important project in the in the scientific, in the opens in the open science community. It also hosts it's not just a Python project. It it hosts the Julia project. It hosts the r open science community. It hosts the software carpentry project and the data carpentry project.
893
01:08:03.270 --> 01:08:06.150
And without all of these people and and these projects,
894
01:08:06.550 --> 01:08:07.690
we wouldn't be anywhere.
895
01:08:08.390 --> 01:08:08.890
Yeah.
896
01:08:09.910 --> 01:08:23.860
That's great. It's it's really heartening to see the amount of community support and contribution that goes into a project like this and just the the amount of time and effort and dedication that everybody puts into it.
897
01:08:24.400 --> 01:08:27.865
That's great. At this point in time, we usually do the PIX.
898
01:08:28.505 --> 01:08:30.605
So, Tobias, why don't you get us started?
899
01:08:31.065 --> 01:08:36.925
Sure. So my first pick today is the Dayworlds trilogy by Philip Jose Farmer.
900
01:08:37.389 --> 01:08:39.250
It's a sci fi trilogy
901
01:08:40.030 --> 01:08:41.170
set in the future
902
01:08:41.630 --> 01:08:42.130
where
903
01:08:42.750 --> 01:08:47.614
population has grown to the point where it's not possible for everyone to be
904
01:08:47.994 --> 01:08:49.935
awake and active at the same time.
905
01:08:50.554 --> 01:08:58.929
So people are divided up into specific days of the week during which they're active and they're put into suspended animation during the rest of the week.
906
01:08:59.310 --> 01:09:00.690
And so houses
907
01:09:01.230 --> 01:09:03.570
are are outfitted with a basement
908
01:09:03.949 --> 01:09:04.449
with
909
01:09:06.414 --> 01:09:12.434
the suspended animation pods for the various families who live in that household during the different days of the week.
910
01:09:12.800 --> 01:09:20.100
And in this world, the most heinous crime is to be what they call a day breaker, which is somebody who is awake all days of the week.
911
01:09:20.675 --> 01:09:25.495
And the protagonist of the story happens to be a day breaker, and it is just
912
01:09:25.875 --> 01:09:26.614
a very
913
01:09:26.915 --> 01:09:28.455
compelling and thought provoking
914
01:09:30.350 --> 01:09:31.570
exploration of
915
01:09:31.870 --> 01:09:33.090
the effects of population
916
01:09:33.710 --> 01:09:39.535
as it expands, and it's just very well written, very entertaining, highly recommend it.
917
01:09:40.715 --> 01:09:43.295
My next pick is a site called readruler.com,
918
01:09:44.920 --> 01:09:46.940
which is a site that will
919
01:09:47.240 --> 01:09:49.660
link up to your Pocket account
920
01:09:50.200 --> 01:09:51.980
and analyze all of the articles,
921
01:09:52.440 --> 01:09:53.160
and it will
922
01:09:53.915 --> 01:10:00.335
it it allows you to tag them with the length of time that it will take you to read them. And it actually has a,
923
01:10:01.120 --> 01:10:10.420
in the settings, you can go in. It has a reading sample so that you can calculate your reading speed so that it can more accurately tag the various articles. So
924
01:10:10.725 --> 01:10:19.225
I have a large number of articles in my pocket account that I have been meaning to get to, and I found this is a really interesting way of being able to
925
01:10:19.560 --> 01:10:25.900
pick up the low hanging fruit and see, okay, these are the very short articles without necessarily having to go through and look at them individually.
926
01:10:26.895 --> 01:10:30.435
And I am going to leave it at that for today. So, Chris, please take it away.
927
01:10:30.895 --> 01:10:31.395
Cool.
928
01:10:32.015 --> 01:10:38.829
I'm gonna do commit a a little bit of heresy here On a Python podcast, I'm going to recommend a Ruby
929
01:10:39.210 --> 01:10:39.710
thing.
930
01:10:40.170 --> 01:10:44.429
I'm gonna recommend it because it is so good that I think that
931
01:10:44.865 --> 01:10:53.285
you can learn something about programming by watching these. It doesn't matter what programming language you're in. Yes. They're they're oriented towards Ruby,
932
01:10:53.890 --> 01:11:04.304
but the fundamental concepts that Avdi puts forth are really interesting, could be really could be useful to anyone, I think. There's a series of screencasts called Ruby Tapas.
933
01:11:05.085 --> 01:11:10.784
And what's amazing about them, I know everybody and their uncle has started producing paid screencasts these days,
934
01:11:11.300 --> 01:11:14.920
is they are the most perfect exemplar from my perspective
935
01:11:15.300 --> 01:11:15.800
of
936
01:11:16.820 --> 01:11:24.365
just little itsy bitsy, exactly what they say, tapas, little tidbits, little, like, 3 to 5 minute videos that show you
937
01:11:24.905 --> 01:11:25.805
1 concept
938
01:11:26.185 --> 01:11:31.840
and only 1 concept, and and do it really clearly, really in a really straightforward
939
01:11:32.220 --> 01:11:37.715
way with a little bit of humor. They're just they're phenomenal. I I still subscribe to them
940
01:11:38.095 --> 01:11:40.195
even though I don't use Ruby much anymore.
941
01:11:40.495 --> 01:11:49.210
I've kinda transitioned to Python in my day job. I still write Ruby with Chef, but I I still keep giving Avdi Man money because they're just that good. So there.
942
01:11:50.309 --> 01:11:56.094
The next pick that I have, we've mentioned her or rather I've mentioned her a bunch in the podcast, but 1 of our readers
943
01:11:56.475 --> 01:12:03.190
mentioned that I had not actually picked her work yet and so it hadn't been in our show notes. I just wanted to pick codenewbies.com.
944
01:12:04.050 --> 01:12:07.750
It's it's a site and associated gaggle of communities
945
01:12:08.290 --> 01:12:13.955
put together by Saran Yitbarak, we hope to have on our show at some point in the future. She is a a a,
946
01:12:15.055 --> 01:12:16.195
1 woman dynamo,
947
01:12:16.975 --> 01:12:23.570
when it comes to helping people learn how to code and get their 1st job in the in in the technology field.
948
01:12:23.950 --> 01:12:25.810
Really, it's it's an amazing resource,
949
01:12:26.190 --> 01:12:30.345
both obviously for people who are trying to learn how to code and even from my perspective,
950
01:12:30.805 --> 01:12:32.265
even if you're an old hand,
951
01:12:32.645 --> 01:12:36.185
it's a really great community. Her podcast is amazing.
952
01:12:36.620 --> 01:12:44.060
Her forums are great. You can help new people out and and probably learn something while you're doing it. It's, it's it's definitely worth
953
01:12:44.940 --> 01:12:52.145
if not looking into for your own sake, it's definitely worth being aware of just because it's amazing work that she's doing and deserves recognition.
954
01:12:53.370 --> 01:12:56.750
And, for my last pick, I'm going to pick a,
955
01:12:57.050 --> 01:12:58.110
a Twitter client
956
01:12:58.490 --> 01:13:00.510
for Max called tweetbot.
957
01:13:01.785 --> 01:13:07.645
And it's 1 of those pieces of software that really just like hits the spot. The UI is beautiful.
958
01:13:08.105 --> 01:13:09.165
It is featureful.
959
01:13:09.545 --> 01:13:12.430
It does everything that I need it to do and more.
960
01:13:13.130 --> 01:13:14.270
It's very intuitive.
961
01:13:14.890 --> 01:13:16.510
The authors are very responsive.
962
01:13:17.450 --> 01:13:28.275
It works great on the Mac and on my iOS devices and syncs my tweets in between so that, you know, I can stop reading at work and or at home and then start reading on the subway and,
963
01:13:28.815 --> 01:13:37.230
it it keeps my place. It's just it's, you know, it's it's well worth its weight in gold, worth its price. I forget how much it is, but it's pretty cheap.
964
01:13:37.610 --> 01:13:39.070
So, yeah. That.
965
01:13:39.435 --> 01:13:41.695
And I think that's about it for me this week.
966
01:13:42.395 --> 01:13:44.735
Brian, why don't you give us your picks?
967
01:13:45.675 --> 01:13:50.140
Yeah. So there's actually 2 books that I'm really enjoying right now.
968
01:13:50.680 --> 01:14:02.795
The first is a new book out by Joel Gruss. I think that's how you say his his last name. Apologies if not and in the book it's an O'Reilly book called Data Science from Scratch
969
01:14:04.829 --> 01:14:08.210
and part of what's fantastic about Python is that
970
01:14:09.150 --> 01:14:10.530
there's all of these libraries
971
01:14:11.525 --> 01:14:13.864
for doing data science such as Pandas,
972
01:14:14.165 --> 01:14:14.665
statsmodels,
973
01:14:15.045 --> 01:14:16.344
scikit learn and
974
01:14:18.570 --> 01:14:22.909
those libraries are great but I think they have a weakness in that
975
01:14:23.449 --> 01:14:24.590
they end up being
976
01:14:25.130 --> 01:14:27.070
very easy to use black boxes
977
01:14:28.975 --> 01:14:30.035
that don't necessarily
978
01:14:30.575 --> 01:14:33.315
help someone who's learning about data science
979
01:14:33.855 --> 01:14:37.155
to understand what the algorithms are actually doing.
980
01:14:37.720 --> 01:14:40.460
And I think getting that deeper knowledge of
981
01:14:40.840 --> 01:14:43.580
what's actually going on is really important.
982
01:14:44.200 --> 01:14:44.700
And
983
01:14:45.295 --> 01:14:47.795
the the philosophy of data science from scratch
984
01:14:48.415 --> 01:14:52.115
is that he he walks through how to implement
985
01:14:52.640 --> 01:14:54.580
a lot of the core algorithms
986
01:14:55.360 --> 01:14:56.500
in data science
987
01:14:57.440 --> 01:14:59.885
from scratch without relying on
988
01:15:00.284 --> 01:15:02.704
scikit learn, for example, or pandas.
989
01:15:03.324 --> 01:15:05.425
And it's at a really nice level
990
01:15:05.804 --> 01:15:09.590
where someone who knows the basics of Python and NumPy
991
01:15:10.130 --> 01:15:10.870
and matplotlib
992
01:15:11.410 --> 01:15:22.585
is going to be able to pick this up and learn a lot about the internal details, and so I've I've been really enjoying that especially as I think about how to teach data science to undergrads.
993
01:15:23.285 --> 01:15:26.025
And then the second book is William Cleveland's
994
01:15:26.710 --> 01:15:28.170
Elements of Graphing Data.
995
01:15:28.630 --> 01:15:32.570
I got this a few months ago and I've been spending time with it and
996
01:15:33.510 --> 01:15:36.364
it's just an extremely practical
997
01:15:36.744 --> 01:15:38.045
nice book about
998
01:15:38.505 --> 01:15:39.005
visualizations
999
01:15:39.625 --> 01:15:40.284
and plotting
1000
01:15:40.905 --> 01:15:41.405
and
1001
01:15:41.864 --> 01:15:42.364
it's
1002
01:15:42.960 --> 01:15:46.900
it's a type of book that, you know, I spend 5 minutes with it and
1003
01:15:47.600 --> 01:15:48.500
I feel like
1004
01:15:49.120 --> 01:15:52.885
a lot of things that I had fuzzy thinking about
1005
01:15:53.264 --> 01:15:54.324
in terms of visualization
1006
01:15:55.025 --> 01:15:58.724
all of a sudden became crystal clear, like simple questions of
1007
01:15:59.270 --> 01:16:04.650
should your should the tick marks on your axes be pointing inside or outwards?
1008
01:16:05.670 --> 01:16:07.610
And, he says, well,
1009
01:16:07.995 --> 01:16:10.495
you have data on the inside of the frame
1010
01:16:10.875 --> 01:16:12.815
and if your tick marks point inwards,
1011
01:16:13.675 --> 01:16:20.160
there's a good chance that the data is going to overlap the tick marks, making it difficult to see them so they should point out loops.
1012
01:16:20.860 --> 01:16:29.655
And obviously it's not a universal rule that you always must follow but it's a very practical way of thinking about choices like that that otherwise
1013
01:16:30.755 --> 01:16:34.930
in the past I think I would have approached from a more of a visual aesthetic perspective
1014
01:16:35.810 --> 01:16:37.830
and so it it's a wonderful book.
1015
01:16:39.330 --> 01:16:40.950
Yeah. I I guess,
1016
01:16:41.330 --> 01:16:45.165
kind of a little bit less pragmatic right now of a mindset.
1017
01:16:45.784 --> 01:16:46.685
In terms of
1018
01:16:47.065 --> 01:16:51.645
books, a book that has resonated a lot lately with me is Republic Lost
1019
01:16:52.170 --> 01:16:53.390
from Lawrence Lessig.
1020
01:16:53.850 --> 01:16:55.150
It's really a fantastic
1021
01:16:55.610 --> 01:16:56.110
read,
1022
01:16:56.890 --> 01:16:58.910
I think, for today's world
1023
01:16:59.505 --> 01:17:06.005
from Lessig. Lessig kind of took a step back away from this. A lot of his work had been about code
1024
01:17:06.370 --> 01:17:15.570
and the the culture of the Internet. And this is really a book about Congress and about the modern world of politics. And what I find fascinating about it is that it's it's a book about how,
1025
01:17:16.645 --> 01:17:18.745
sort of how humans with good intentions
1026
01:17:19.685 --> 01:17:21.625
and, and, fundamentally
1027
01:17:22.005 --> 01:17:25.145
sort of trying in many ways trying to do the right thing,
1028
01:17:25.830 --> 01:17:27.030
can can do,
1029
01:17:27.830 --> 01:17:30.010
things that are so problematic to society.
1030
01:17:30.630 --> 01:17:33.350
And I think it's it's a book that has very enlightened lessons,
1031
01:17:34.385 --> 01:17:36.165
I think to teach many of us,
1032
01:17:36.545 --> 01:17:40.325
and and and asks really important questions about society.
1033
01:17:41.025 --> 01:17:47.770
And the other 1 is actually an author more than a specific book from Colombia. I've I've heard a lot of,
1034
01:17:48.230 --> 01:17:50.250
in the in in the geek world,
1035
01:17:50.755 --> 01:17:56.675
because I'm I'm a Colombian. In the in the geek world, there are a lot of I hear a lot of people are,
1036
01:17:57.555 --> 01:17:59.175
in the US, like, actually,
1037
01:17:59.780 --> 01:18:03.800
a Latin American author called Jorge Luis Borges because a lot of his poetry
1038
01:18:04.100 --> 01:18:08.625
and a lot of his short stories have a very kind of logical mathematical bent
1039
01:18:09.185 --> 01:18:09.845
to it.
1040
01:18:10.385 --> 01:18:13.365
And, Columbia has an author that in my mind
1041
01:18:13.745 --> 01:18:14.625
that it was very,
1042
01:18:15.185 --> 01:18:17.125
was much less well known than Orjes,
1043
01:18:17.590 --> 01:18:18.890
but who I would recommend,
1044
01:18:19.510 --> 01:18:26.705
because there's echoes of Borges in in him, but, but with a little bit of a Garcia Marquez list. And his name is Alvaro Multis,
1045
01:18:27.345 --> 01:18:34.245
and he has both short stories and poetry that, that has kind of some echoes of Borges with a lot of really crazy
1046
01:18:34.785 --> 01:18:35.445
kind of
1047
01:18:36.500 --> 01:18:39.640
character who has who is a wanderer traveler
1048
01:18:40.100 --> 01:18:41.640
through the jungles of Colombia,
1049
01:18:42.180 --> 01:18:45.695
who is a very interesting character called called the Macrolle Gabriel.
1050
01:18:46.315 --> 01:18:49.514
And I would, I would highly recommend his novels to people who,
1051
01:18:49.994 --> 01:18:51.855
to to people who have kind of
1052
01:18:52.990 --> 01:18:54.770
a a love a love for Moriches'
1053
01:18:55.390 --> 01:19:08.085
logical mindset, but at the same time, were interested in in a venture. So, again, Alvaro Maltis. He is available actually in English, but he's not very well known. And so I figured I would make, some, kind of slightly oddball picks.
1054
01:19:10.040 --> 01:19:23.075
Great. Well, thank you very much, and thank you both for taking the time to come on the show with us today. So for anybody who is interested in following you and the things that you guys are doing, what would be the best way to do that?
1055
01:19:26.150 --> 01:19:27.690
Twitter or email in my case?
1056
01:19:28.310 --> 01:19:34.585
Yeah. Honestly, Twitter these days has become an invaluable way where we keep in touch with users,
1057
01:19:34.965 --> 01:19:36.265
we hear about things,
1058
01:19:37.205 --> 01:19:44.210
so that that's a great place to to go for that. Okay. So what Twitter handle should should users be following to to do that?
1059
01:19:45.790 --> 01:19:46.290
So,
1060
01:19:46.590 --> 01:19:47.810
the Twitter handle
1061
01:19:48.190 --> 01:19:49.810
for Jupyter is
1062
01:19:50.495 --> 01:19:51.075
at projectjupyter
1063
01:19:51.695 --> 01:19:52.195
and
1064
01:19:54.335 --> 01:19:57.795
the 1 for IPython is IPython dev
1065
01:19:59.470 --> 01:20:00.190
And then,
1066
01:20:00.830 --> 01:20:03.010
my Twitter handle is Ellison BG
1067
01:20:04.110 --> 01:20:05.570
and Fernando Moses.
1068
01:20:07.605 --> 01:20:08.745
At, fperez_org.
1069
01:20:13.125 --> 01:20:13.625
Excellent.
1070
01:20:15.619 --> 01:20:29.945
So, again, thank you very much for coming on the show. It has been a wonderful discussion, and I learned a whole lot about the IPython and Jupyter projects that I never knew before. And I definitely think I have a lot of things to read up about. So
1071
01:20:30.410 --> 01:20:35.949
thank you very much. Me too. And I and I and I also just wanna really quickly, in addition to thanking Brian and Fernando,
1072
01:20:36.250 --> 01:20:40.935
wanna thank Travis Oliphant for actually hooking us up with these 2 guys because
1073
01:20:41.315 --> 01:20:45.095
the IPython notebook project is 1 of the first things that
1074
01:20:45.540 --> 01:20:48.840
when Tobias and I were saying, hey, we should do a Python podcast,
1075
01:20:49.460 --> 01:20:55.865
we both sort of batted around as, you know, in my opinion, this is 1 of the coolest things in the Python ecosystem.
1076
01:20:56.245 --> 01:20:58.885
So we really wanted to talk to you from the very beginning. And,
1077
01:20:59.285 --> 01:21:02.505
once again, thank you, Travis, for for for hooking us up.
1078
01:21:04.760 --> 01:21:11.180
Thanks for having us. We really appreciate it. And, obviously, Travis is another 1 of our long standing friends,
1079
01:21:11.495 --> 01:21:15.515
supporters, colleagues, and sort of heroes of our community. So thanks, Travis.
1080
01:21:16.215 --> 01:21:19.035
Yeah. Thank thanks so much for having us on. It's been a pleasure.