#44: Project Jupyter and IPython

WEBVTT
00:00:00.001 --> 00:00:03.160
One of the fastest growing areas in Python is scientific computing.
00:00:03.160 --> 00:00:07.700
In scientific computing with Python, there are a few key packages that make it really special.
00:00:07.700 --> 00:00:11.420
These include NumPy, SciPy, and the related packages.
00:00:11.420 --> 00:00:16.800
But the one that brings it all together, visually, is IPython, now known as Project Jupyter.
00:00:16.800 --> 00:00:20.780
And that's the topic of episode 44 of Talk Python To Me.
00:00:20.780 --> 00:00:25.540
You'll learn about the big split, plans for the recent $6 million in funding,
00:00:25.540 --> 00:00:30.700
Jupyter at CERN and Large Hadron Collider with Min Arkay and Matthias Boutonnier.
00:00:54.080 --> 00:01:01.200
Welcome to Talk Python To Me, a weekly podcast on Python, the language, the libraries, the ecosystem, and the personalities.
00:01:01.200 --> 00:01:05.320
This is your host, Michael Kennedy. Follow me on Twitter where I'm @mkennedy.
00:01:05.320 --> 00:01:09.200
Keep up with the show and listen to past episodes at talkpython.fm.
00:01:09.200 --> 00:01:11.800
And follow the show on Twitter via at Talk Python.
00:01:11.800 --> 00:01:15.380
This episode is brought to you by Hired and SnapCI.
00:01:15.380 --> 00:01:22.080
Thank them for supporting the show on Twitter via at Hired underscore HQ and at Snap underscore CI.
00:01:23.080 --> 00:01:25.200
Hi, folks. No news this week.
00:01:25.200 --> 00:01:29.100
I do have a big announcement coming, and I'm really looking forward to sharing it with you all,
00:01:29.100 --> 00:01:31.900
but I'm not quite ready to talk about it yet, so stay tuned.
00:01:31.900 --> 00:01:36.280
For now, let's get right to the interview with the Project Jupyter core devs,
00:01:36.280 --> 00:01:38.200
Min Arkay and Matthias Boutonnier.
00:01:38.200 --> 00:01:40.840
Matthias, Min, welcome to the show.
00:01:40.840 --> 00:01:41.500
Thanks.
00:01:41.500 --> 00:01:43.120
Thanks, Mike, for having us here.
00:01:43.540 --> 00:01:49.660
Yeah, I'm really super excited to talk about Python intersected with science in this thing called
00:01:49.660 --> 00:01:51.780
IPython, or what's become Project Jupyter.
00:01:51.780 --> 00:01:53.800
So that's going to be really great.
00:01:53.800 --> 00:01:58.000
And before we get to that, though, let's just talk about how you got involved.
00:01:58.000 --> 00:01:59.320
How do you get into programming?
00:01:59.320 --> 00:02:03.040
How do you get involved with IPython and all that stuff?
00:02:03.040 --> 00:02:03.740
What's your background?
00:02:03.740 --> 00:02:04.760
Min, you want to go first?
00:02:04.760 --> 00:02:05.200
Sure.
00:02:05.200 --> 00:02:11.520
Yeah, so I was an undergrad in physics at Santa Clara University, working with Brian Granger,
00:02:11.520 --> 00:02:13.980
one of the founders of the IPython project.
00:02:13.980 --> 00:02:21.720
And I was interested in computing and simulation and things, and ended up working on the interactive
00:02:21.720 --> 00:02:28.640
parallel computing part of IPython as my undergrad thesis, and started doing my numerical simulation
00:02:28.640 --> 00:02:34.000
homework stuff in Python, even though the classes were taught in MATLAB and Octave and things,
00:02:34.000 --> 00:02:39.980
and enjoying the scientific Python ecosystem of NumPy and Matplotlib and things.
00:02:39.980 --> 00:02:44.360
And that's kind of how I came to the project and scientific Python in general.
00:02:44.360 --> 00:02:45.280
Yeah, that's really cool.
00:02:45.280 --> 00:02:48.000
And was IPython already a thing when you got started?
00:02:48.000 --> 00:02:48.660
Yeah.
00:02:48.660 --> 00:02:54.260
Fernando created IPython in 2001, and I was doing my undergrad a few years after that.
00:02:54.260 --> 00:02:59.440
And so I joined the project after it had been around for about five years in 2006, and I've
00:02:59.440 --> 00:03:02.000
been working on it for the past 10 years, I guess, now.
00:03:02.000 --> 00:03:03.480
Yeah, about 10 years.
00:03:03.480 --> 00:03:04.560
So how time flies.
00:03:04.560 --> 00:03:05.700
Matthias, how about you?
00:03:05.700 --> 00:03:11.600
Oh, so I've been to the project much later than Min.
00:03:11.600 --> 00:03:16.620
I actually started programming a long time ago and came across one of the huge refactoring
00:03:16.620 --> 00:03:18.100
of IPython, MinDead.
00:03:18.100 --> 00:03:22.140
I think it finished in summer 2011.
00:03:22.920 --> 00:03:27.820
Just after that, they released the Qt console, so the current IPython team at the time.
00:03:27.820 --> 00:03:35.080
And the project was much more friendly and in a good shape for beginners Python programmers.
00:03:35.080 --> 00:03:42.560
At the time, I was beginning my PhD in biophysics in Paris, and I started contributing to the project.
00:03:42.760 --> 00:03:45.560
It was my first big contribution to an open source project.
00:03:45.560 --> 00:03:52.540
And I started to spend my night and weekend doing my PhD, improving IPython, which was helping
00:03:52.540 --> 00:03:55.140
me really a lot for my PhD.
00:03:55.140 --> 00:04:00.940
And I quickly became a core contributor, and I've stayed in the team since then.
00:04:01.100 --> 00:04:07.500
Maybe a good place to start talking about this whole project is maybe we can start with the
00:04:07.500 --> 00:04:07.760
history.
00:04:07.760 --> 00:04:12.060
Originally, this project was called IPython and IPython Notebooks, right?
00:04:12.400 --> 00:04:19.440
Yeah, IPython was around for a good 10 years before we got a version of the notebook out, although
00:04:19.440 --> 00:04:24.960
we had been working on various versions of notebooks for about five years that most attempts kind
00:04:24.960 --> 00:04:25.940
of didn't go anywhere.
00:04:25.940 --> 00:04:27.780
So what did it look like?
00:04:27.780 --> 00:04:31.620
What did the product look like in the early days?
00:04:31.620 --> 00:04:38.020
Yeah, so initially, Fernando created IPython as just a better interactive shell for Python,
00:04:38.360 --> 00:04:44.980
so giving you some better tab completions, nice colorful tracebacks, things like that.
00:04:44.980 --> 00:04:50.560
Also, Python's a nice, verbose language, but when you're doing interactive stuff, some of
00:04:50.560 --> 00:04:55.320
the bash shell syntax is nicer to type when you're doing LS and CD and everything.
00:04:55.320 --> 00:05:01.280
So one of the things that Fernando did early on was add this notion of magics for extending
00:05:01.280 --> 00:05:06.060
the Python language to give convenient commands for interactively typed things.
00:05:06.060 --> 00:05:13.340
like you can type CD in IPython, which you can't type in a regular Python environment.
00:05:13.340 --> 00:05:20.360
And then there are magics that are particularly useful for the scientific visualization things
00:05:20.360 --> 00:05:27.880
and things like the time at magic for profiling and good map.lib integration for the event loop
00:05:27.880 --> 00:05:28.500
and things like that.
00:05:29.000 --> 00:05:31.280
I know what the shell looks like today.
00:05:31.280 --> 00:05:35.820
You know, you can load it up and it kind of looks like a shell and you type in there,
00:05:35.820 --> 00:05:41.300
but you can do things like plot graphs and those will pop up into separate windows and things like that, right?
00:05:41.300 --> 00:05:45.280
Yeah, and most of that is provided by tools like MapPotlib,
00:05:45.280 --> 00:05:50.880
but often those tools need a little bit of help to make sure that the terminal stays responsive.
00:05:50.880 --> 00:05:58.020
And that's one of the things that IPython helps with in terms of what Python calls the input hook
00:05:58.020 --> 00:06:04.300
to ensure that the terminal remains responsive while a GUI event loop is also running.
00:06:04.300 --> 00:06:05.620
Right, yeah, very cool.
00:06:05.620 --> 00:06:12.340
So how did it go from that to the more, I want to think of it as like articles or published style,
00:06:12.460 --> 00:06:18.280
these notebooks that you can use to communicate almost like finished work rather than something interactive?
00:06:18.280 --> 00:06:24.700
Notebook style interfaces, since a lot of the IPython folks come from a physics background.
00:06:24.700 --> 00:06:32.520
So Brian and Fernando, who started the project, were doing their graduate work in physics at Boulder at the same time.
00:06:32.520 --> 00:06:34.520
And then I was Brian's physics student.
00:06:34.520 --> 00:06:37.800
And notebook environments are pretty common.
00:06:37.980 --> 00:06:43.940
There are various commercial and non-commercial products that have kind of notebook processing environments,
00:06:43.940 --> 00:06:47.400
especially for math analysis.
00:06:47.400 --> 00:06:51.460
So often code is not the best representation of math,
00:06:51.460 --> 00:06:54.420
but there are rich, you know, rendered mathematical expressions that are nice.
00:06:54.420 --> 00:07:00.600
Brian and Fernando knew that they wanted a notebook type interface fairly early on,
00:07:00.600 --> 00:07:04.220
but the tools just weren't there to build it.
00:07:04.220 --> 00:07:07.120
And IPython wasn't in a shape to really support it.
00:07:07.260 --> 00:07:10.840
So slowly at first, and then it kind of picked up speed.
00:07:10.840 --> 00:07:14.400
We added the pieces for putting that together.
00:07:14.400 --> 00:07:20.180
But it was kind of in, it was on the horizon for many years before it actually happened.
00:07:20.180 --> 00:07:20.600
Sure.
00:07:20.600 --> 00:07:23.280
It took a while to build the maturity into it.
00:07:23.280 --> 00:07:27.820
What are some of those building blocks that it was waiting on?
00:07:27.820 --> 00:07:36.820
Yeah, I think that web technology and web socket were one of the technologies that was missing to actually use a notebook.
00:07:36.820 --> 00:07:44.800
If I remember correctly, one of the latest prototypes that we did not release was using AJAX polling.
00:07:45.820 --> 00:07:53.320
But the ability to actually push a result to the web front end once, as soon as the kernel gets a result,
00:07:53.320 --> 00:07:59.000
is one of the key factors that pushed the notebook forward and allowed us to do the notebook.
00:07:59.000 --> 00:08:08.660
Actually, the notebook that you know nowadays, the first prototype, was actually using still draft of web sockets that stayed in draft state a long time.
00:08:08.660 --> 00:08:15.600
And so we were really bleeding edge on this technology and adopting a lot of everything, everything in browser,
00:08:15.600 --> 00:08:19.720
and everything to just rely on what current browser can do for the notebook.
00:08:19.720 --> 00:08:27.200
There is, by the way, we can put that in the note of the podcast, a really nice blog of Fernando that recaps the history of IPython.
00:08:27.200 --> 00:08:38.100
And even 150 lines of Python, which is a version of IPython when it was like a few weeks old, which is IPython 0.1,
00:08:38.100 --> 00:08:43.620
that we can dig up for people who are interested in trying really early prototype.
00:08:43.620 --> 00:08:45.660
Yeah, go back and see the history.
00:08:45.660 --> 00:08:54.340
That's a really interesting point, Matthias, because it's easy to think of the web as being this very rich, powerful, capable platform,
00:08:54.340 --> 00:08:58.140
because it has been for the last five years or so.
00:08:58.140 --> 00:09:03.600
But 10 or even more than 10 years ago, it was not, right?
00:09:03.600 --> 00:09:07.080
It was basically just documents on the web, right?
00:09:07.080 --> 00:09:10.340
You had a little bit of JavaScript, and that was about it, right?
00:09:10.340 --> 00:09:11.940
Yeah, I think so.
00:09:11.940 --> 00:09:13.840
I haven't used that much.
00:09:13.840 --> 00:09:17.580
I was not developing on the web that much 10 years ago.
00:09:17.580 --> 00:09:20.320
I was more a C, C++ person.
00:09:20.320 --> 00:09:23.900
You mean maybe it was more developing web at the time?
00:09:23.900 --> 00:09:33.780
Yeah, I wrote an early version of the web-based notebook for IPython during the summer of 2006, 2007, I think.
00:09:33.780 --> 00:09:37.840
Even then, the tools available really weren't...
00:09:37.840 --> 00:09:41.620
It was not a particularly pleasant thing to work with.
00:09:42.120 --> 00:09:43.340
I bet it wasn't.
00:09:43.340 --> 00:09:52.540
Did you end up in a lot of situations where you're like, oh, this only works in Firefox, and this one only works in IE, and just partly working in a lot of places?
00:09:52.540 --> 00:09:54.580
It's frankly still like that.
00:09:54.580 --> 00:09:56.740
It still never works in IE.
00:09:56.740 --> 00:09:58.860
Yeah.
00:09:58.860 --> 00:10:01.520
Yeah, it's hard to love IE, I know.
00:10:02.060 --> 00:10:16.320
Well, but I mean, recent versions of Internet Explorer are actually really nice and have good standards implementations and everything, but the reputation of IE6 kind of overshadows.
00:10:16.840 --> 00:10:18.720
Yeah, it definitely casts a long shadow.
00:10:18.720 --> 00:10:30.200
And, you know, Microsoft, I think just last week, possibly, like very recently, just ended support for all versions of IE other than, I think, in 11 and onward.
00:10:30.520 --> 00:10:33.840
Maybe 10 and onward, but certainly knocked out a whole bunch of them.
00:10:33.840 --> 00:10:38.580
And, you know, once that kicks in, that's going to be a good day for everyone that has to work on the web.
00:10:38.580 --> 00:10:41.380
Microsoft has done a lot of things recently.
00:10:41.380 --> 00:10:51.020
Last week also, if I remember correctly, they did release as open source the JavaScript engine that will power the next version of their browser.
00:10:51.660 --> 00:11:10.360
So Google has V8, which both power Chrome and Node.js, which is actually one of the technologies that helped the notebook become reality because JavaScript was painfully slow 10 years ago and is now really, really fast thanks to V8.
00:11:10.360 --> 00:11:20.140
And so it's really nice to see nowadays Microsoft actually releasing open source software and contributing to the community.
00:11:20.620 --> 00:11:35.560
And I hope that in the next few years, Microsoft will lose some fact that everybody is complaining about IE and everything and get actually nice software, not that many security bugs and so on and so forth.
00:11:35.560 --> 00:11:44.500
It will be really nice if that comes along because a lot of people run their software and it would be, you know, the world would be a better place if it works really well.
00:11:44.500 --> 00:11:46.900
I certainly think they're on the right path.
00:11:46.900 --> 00:11:48.200
I think it's pretty interesting.
00:11:48.480 --> 00:11:57.920
So one thought I had while you guys were talking about this is how does, what's the cross-platform story or IPython and Jupyter in general?
00:11:57.920 --> 00:12:05.120
Does it work kind of equally well on Windows, Linux, OS X or are there places that are more equal than others?
00:12:05.120 --> 00:12:11.880
Linux and OS X are a little bit more equal than Windows, but it should work.
00:12:12.380 --> 00:12:13.580
It should work everywhere.
00:12:13.580 --> 00:12:28.400
And even though all of our developers and everything are working exclusively on Linux and OS X, when we do user surveys and things, we find that roughly half or even slightly more than half of our users are running Windows.
00:12:28.820 --> 00:12:46.880
So even though it often doesn't work quite as well or we frequently during the development process will introduce bugs that we don't notice for a while, Windows really is a first-class platform for the kind of local desktop app that happens to use a web browser for UI case of the notebook.
00:12:46.880 --> 00:12:55.480
There are certain aspects of installation that are often more challenging on Windows, especially in terms of installing kernels other than the Python one.
00:12:55.480 --> 00:13:01.020
So installing multi-language kernels is more challenging on Windows.
00:13:01.020 --> 00:13:05.240
And I think that's not necessarily a specific deficiency of Windows.
00:13:05.240 --> 00:13:10.020
It's more just the kind of developer maintainers don't tend to use Windows.
00:13:10.020 --> 00:13:15.760
So the documentation and education often just don't cover what you need to do for Windows as well.
00:13:15.760 --> 00:13:16.540
Right.
00:13:16.540 --> 00:13:25.700
If you don't develop and test deploying your packages in the underlying compilers that have to make them go, well, you're more likely to run into problems, right?
00:13:25.700 --> 00:13:26.360
Yeah.
00:13:26.360 --> 00:13:31.560
I would say also that Trevis CI, so continuous integration, is often on Linux only.
00:13:31.560 --> 00:13:33.800
Setting up on Windows is painful.
00:13:33.800 --> 00:13:41.180
So we catch up bugs with continuous integration, much often with continuous integration on Linux.
00:13:41.180 --> 00:13:44.820
So less prone to bug on Linux.
00:13:44.820 --> 00:13:55.640
And the other thing is, I don't always like to say good things about half proprietary tools, but Konda changed a lot of things for the last few years.
00:13:55.900 --> 00:13:59.240
It was really painful to install Python on many systems.
00:13:59.240 --> 00:14:10.280
And now it's one of the solutions, especially at Software Carpentry Bootcamp, where we ask people to just install Konda and Konda install Jupyter, which now even come vended in it.
00:14:10.280 --> 00:14:14.980
And it's almost always works out of the box.
00:14:14.980 --> 00:14:19.440
And especially for beginners, it's a really, really nice tool.
00:14:19.660 --> 00:14:26.560
Yeah, Konda has really moved the bar for how easy it is to get set up, especially on Windows.
00:14:26.560 --> 00:14:32.920
There are lots of different ways to install things on Unix-y platforms that work fairly reliably.
00:14:32.920 --> 00:14:42.680
But the binaries provided by Konda and Anaconda are extremely valuable for beginners, especially on Windows, where people don't tend to have a working compiler set up.
00:14:42.680 --> 00:14:47.880
And a lot of the scientific packages won't build on people's Windows machines.
00:14:48.880 --> 00:14:51.500
So having binaries is extremely important.
00:14:51.500 --> 00:14:58.800
And the binaries provided by Konda and Anaconda have been extremely valuable, especially for people getting started in scientific Python.
00:14:58.800 --> 00:15:07.380
Yeah, I still think I have scars from the vcvars.bat was not found sort of errors trying to do stuff on Windows.
00:15:07.380 --> 00:15:14.580
And we had Travis Oliphant on Show 34, who is behind Konda and Continuum and all that.
00:15:14.780 --> 00:15:26.080
And I think it's a really cool thing that those guys are doing, sort of taking that build configuration step and just pre-building it and shipping the binaries, like you say.
00:15:26.080 --> 00:15:29.260
That really helps people when they're getting started, I think.
00:15:29.660 --> 00:15:47.980
Yeah, it's made a huge difference, especially, as Matthias mentioned, in the workshop, the kind of software carpentry and Python boot camp type environments, which often, you know, just a few years ago, where you spend the first day on installation, basically.
00:15:48.280 --> 00:15:51.340
Which is a high price to pay in a two-day workshop.
00:15:51.340 --> 00:15:53.860
And now it's often down to an hour.
00:15:53.860 --> 00:15:54.580
It's awesome.
00:15:54.580 --> 00:15:55.780
It's a super high price to pay.
00:15:55.780 --> 00:15:58.540
And it's also super discouraging, right?
00:15:58.540 --> 00:16:01.700
People come not because they want to learn how to configure their compiler.
00:16:01.700 --> 00:16:04.140
They want to come build something amazing, right?
00:16:04.140 --> 00:16:08.080
And they've got to, like, plow through all these nasty configuration edge cases.
00:16:08.480 --> 00:16:09.480
And, yeah, very, very cool.
00:16:09.480 --> 00:16:19.640
So, before we move farther, you know, just the other day, I was trying to describe IPython as somebody in, like, one or two sentences.
00:16:19.640 --> 00:16:21.920
And I didn't do a super job, I think.
00:16:21.920 --> 00:16:29.420
Could you guys maybe give me your elevator pitch for what is Jupyter or IPython, which becomes Jupyter?
00:16:29.420 --> 00:16:30.560
It's really tough.
00:16:30.560 --> 00:16:32.740
Have you seen the Lego movie?
00:16:32.740 --> 00:16:35.560
Do you know the song Everything is Awesome?
00:16:35.560 --> 00:16:36.660
Yes.
00:16:37.940 --> 00:16:39.580
That would be my pitch.
00:16:39.580 --> 00:16:42.000
Yeah.
00:16:42.000 --> 00:16:43.400
Everything is awesome.
00:16:43.400 --> 00:16:43.720
Okay.
00:16:43.720 --> 00:16:44.520
Yeah.
00:16:44.520 --> 00:16:58.520
So, I would say it's IPython and Jupyter projects together provide tools for interactive computing and reproducible research and software-based communication.
00:16:58.520 --> 00:16:59.700
Okay.
00:16:59.700 --> 00:17:01.360
It's kind of the high-level gist.
00:17:01.360 --> 00:17:06.440
It's fairly different than a lot of what's out there from a programmer's perspective.
00:17:06.660 --> 00:17:08.680
So, it does take a little explaining, doesn't it?
00:17:08.680 --> 00:17:10.060
Yeah.
00:17:10.060 --> 00:17:16.280
So, we have things like an environment in which to do the interactive programming and do the exploratory work.
00:17:16.940 --> 00:17:23.360
And then we also have things like the notebook document format, which are for distributing the communication and sharing it with other people.
00:17:23.720 --> 00:17:26.240
So, those are kind of the two aspects.
00:17:26.240 --> 00:17:26.240
So, those are kind of the two aspects.
00:17:26.240 --> 00:17:31.220
And Fernando likes to say we have tools for the life cycle of a computational idea.
00:17:31.220 --> 00:17:33.080
That's a very cool way to put it.
00:17:33.080 --> 00:17:34.080
It's a very cool tagline.
00:17:34.080 --> 00:17:34.620
I like it.
00:17:34.620 --> 00:17:39.340
We're talking about IPython because that's the historical place.
00:17:39.340 --> 00:17:42.840
And we're talking about Jupyter because that's the present and the future.
00:17:42.840 --> 00:17:46.320
Could you guys maybe talk about how it went from one to the other?
00:17:46.400 --> 00:17:47.080
What's the story there?
00:17:47.080 --> 00:17:47.940
Yeah.
00:17:47.940 --> 00:18:03.940
So, when we started working on building these UIs with rich media displays, the first one of which was the Qt console, the first step of that was separating the front end from what we call the kernel, which is where code runs.
00:18:04.360 --> 00:18:10.520
That meant essentially establishing a network protocol for a REPL, basically.
00:18:10.520 --> 00:18:19.440
And with that, we have the ability, an expression of, okay, I'm going to send an execute request that has some code for the kernel to evaluate.
00:18:19.440 --> 00:18:25.440
And then the kernel sends messages back that are display formats of various types.
00:18:25.440 --> 00:18:28.980
So, it can send back PNGs or HTML or text.
00:18:29.460 --> 00:18:44.040
We realized, not entirely on purpose, this wasn't what we set out to do, but we realized when we had this protocol that there was nothing Python-specific about it, that any language that understands a REPL can talk this protocol.
00:18:44.040 --> 00:18:52.860
And because the UI and the code execution were in different processes, there's no reason that the two need to be in the same language.
00:18:53.860 --> 00:19:07.160
Communities like, the first big one was the Julia language community, essentially saw the UI, specifically the notebook UI, and said, you know, we like that, we want to use that, we'd rather not reimplement it.
00:19:07.160 --> 00:19:09.680
So, what they implemented was the protocol.
00:19:09.680 --> 00:19:12.980
And once they implemented the protocol, they got the UI for free.
00:19:13.640 --> 00:19:25.760
The result of that, since we didn't set out to design that, there were a bunch of rough edges where we had assumed Python, but they were kind of incidental, smaller assumptions to work around.
00:19:25.760 --> 00:19:40.420
And so, since that started, we've been kind of refining protocols and things to remove Python and IPython assumptions, so that the UI is separate from the language in which execution happens.
00:19:40.620 --> 00:19:49.780
Because we don't really, you know, a lot of the benefits of the protocol and the display stuff, there's no reason it should be confined to code executing in Python.
00:19:49.780 --> 00:19:53.220
Yeah, that's a really happy coincidence, isn't it?
00:19:53.220 --> 00:19:54.200
That's excellent.
00:19:54.200 --> 00:19:54.640
Yeah.
00:20:05.520 --> 00:20:07.700
This episode is brought to you by Hired.
00:20:07.700 --> 00:20:13.300
Hired is a two-sided, curated marketplace that connects the world's knowledge workers to the best opportunities.
00:20:13.300 --> 00:20:20.800
Each offer you receive has salary and equity presented right up front, and you can view the offers to accept or reject them before you even talk to the company.
00:20:20.800 --> 00:20:26.520
Typically, candidates receive five or more offers within the first week, and there are no obligations, ever.
00:20:26.520 --> 00:20:28.100
Sounds awesome, doesn't it?
00:20:28.100 --> 00:20:29.760
Well, did I mention the signing bonus?
00:20:30.040 --> 00:20:33.140
Everyone who accepts a job from Hired gets a $1,000 signing bonus.
00:20:33.140 --> 00:20:35.940
And as Talk Python listeners, it gets way sweeter.
00:20:35.940 --> 00:20:41.660
Use the link Hired.com slash Talk Python To Me, and Hired will double the signing bonus to $2,000.
00:20:41.660 --> 00:20:43.800
Opportunity's knocking.
00:20:43.800 --> 00:20:47.240
Visit Hired.com slash Talk Python To Me and answer the call.
00:20:53.240 --> 00:20:55.780
Matthias, where did Jupyter come from?
00:20:55.780 --> 00:20:57.620
It used to be called IPython.
00:20:57.620 --> 00:21:00.260
Obviously, that doesn't make sense if you're not using Python.
00:21:00.260 --> 00:21:11.180
We've been thinking about renaming part of the project for much longer times than when we actually announced that we will be renaming to Jupyter.
00:21:11.180 --> 00:21:16.640
Of course, we were aware that users, especially non-Python users, were confused.
00:21:16.640 --> 00:21:20.280
Like, I want to use a notebook with R.
00:21:20.280 --> 00:21:22.060
Why should I install IPython?
00:21:22.060 --> 00:21:30.040
And you have to understand that many users even don't make the difference between Python and IPython.
00:21:30.040 --> 00:21:34.740
And many users also write IPython with lowercase i, and everyone knows that it's uppercase i.
00:21:34.740 --> 00:21:37.500
It's not made by Apple.
00:21:37.500 --> 00:21:37.920
Come on.
00:21:37.920 --> 00:21:38.480
Yeah.
00:21:38.480 --> 00:21:50.280
And so, yeah, we were searching for another name to actually something that is easy to Google, that is not already taken, where we can get a domain name.
00:21:50.280 --> 00:21:54.980
And that would have a connotation, a scientific connotation.
00:21:54.980 --> 00:22:03.800
And we wanted to do thanks to the Astro community that has been using IPython for a long, long time, almost since the beginning.
00:22:04.620 --> 00:22:12.480
And I still remember one day, Fernando wrote a mail to the whole team and said, hey, I just found this name.
00:22:12.480 --> 00:22:14.000
What do you think?
00:22:14.000 --> 00:22:16.860
And it was everybody agreed.
00:22:16.860 --> 00:22:26.080
And almost in a couple of days, we decided to grab all the domain name and start working on actually separating the project and everything.
00:22:26.080 --> 00:22:29.260
It has been a really tough transition.
00:22:29.840 --> 00:22:33.060
People were really, really confused about the renaming.
00:22:33.060 --> 00:22:35.020
People are still confused.
00:22:35.020 --> 00:22:42.900
But especially for new users, the distinction Jupyter-IPython is really, really useful.
00:22:42.900 --> 00:22:47.340
And also, it allowed Jupyter to become something slightly bigger.
00:22:47.340 --> 00:22:49.700
That was also in our mind in the back.
00:22:49.700 --> 00:22:56.400
Which was like, Jupyter is more a specification that you have a protocol and you have a set of tools.
00:22:56.800 --> 00:23:05.160
What is part of Jupyter is much broader and it can allow anybody to basically say, hey, I implement the Jupyter protocol.
00:23:05.160 --> 00:23:10.440
And so, it's easier to say, hey, I have a Jupyter Atom plugin.
00:23:11.580 --> 00:23:17.700
There are also legal issues around that, that using trademarks that are really close to Python is difficult.
00:23:17.700 --> 00:23:24.320
And Jupyter, being a brand new namespace, and you know that namespaces are great, we should use more of them.
00:23:24.320 --> 00:23:34.420
Allow people to use that and say that they are multi-language in a much better way than when saying we are compatible with IPython.
00:23:34.680 --> 00:23:38.060
Because IPython is also highly connected as a shell.
00:23:38.060 --> 00:23:40.960
And Jupyter is more than just a notebook.
00:23:40.960 --> 00:23:46.380
So, having Jupyter is much better and we are happy with that.
00:23:46.380 --> 00:23:48.080
Yeah, it makes perfect sense.
00:23:48.080 --> 00:23:54.180
I'm sure the transition was a little confusing for people who have been doing IPython or they've heard about IPython.
00:23:54.180 --> 00:23:55.300
They were going to look into it.
00:23:55.300 --> 00:23:56.500
Now it's this other thing.
00:23:57.180 --> 00:24:01.020
But there's more than just a couple of languages that are supported, right?
00:24:01.020 --> 00:24:01.920
How many are supported?
00:24:01.920 --> 00:24:06.420
It depends on when you want to be supported.
00:24:06.420 --> 00:24:17.420
We have a wiki page which is still on the IPython repository, which lists, if I remember correctly, 50 or almost 60 languages.
00:24:17.420 --> 00:24:22.000
It means that you can have languages that have many kernels.
00:24:22.000 --> 00:24:29.260
It means that someone at some point wrote a kernel or a toy kernel that works with IPython.
00:24:29.260 --> 00:24:32.800
And if I remember correctly, we have around 60.
00:24:32.800 --> 00:24:34.020
60.
00:24:34.020 --> 00:24:40.200
That means probably if you have a language you care about, it probably works with Jupyter, right?
00:24:40.200 --> 00:24:41.300
Or it's very edge.
00:24:41.300 --> 00:24:44.180
Most kernels won't have all the features.
00:24:44.700 --> 00:24:53.420
I would say that the one I know works with most of the features are the Python one because we maintain it.
00:24:53.420 --> 00:24:55.820
So you can see it as a reference implementation.
00:24:55.820 --> 00:25:02.900
There are other Python ones, like toys that are only a few hundred lines to show you how to implement that.
00:25:02.900 --> 00:25:07.540
The Julia kernel is a pretty feature complete.
00:25:07.540 --> 00:25:18.240
It's actually many of the features that we have in the IPython kernel were actually, by the Julia team, moved into the Julia language itself.
00:25:18.240 --> 00:25:30.200
So actually having implemented the protocol, having seen the Nullbook UI, allowed them to make much better abstraction for the Julia language and actually improve performance in some small area.
00:25:30.400 --> 00:25:32.400
So that's almost a thing.
00:25:32.400 --> 00:25:38.080
The Haskell kernel also have a really good maintainer and have really nice features.
00:25:39.120 --> 00:25:47.840
Like if you write some code in Haskell and you can rewrite it in a more compact form, the Haskell kernel will tell you that after running your code.
00:25:47.840 --> 00:25:49.840
They say, hey, you can rewrite it this way.
00:25:49.840 --> 00:25:52.560
It will be more compact and more readable by someone who does Haskell.
00:25:53.440 --> 00:25:57.280
So Ruby had some activity at some point.
00:25:57.280 --> 00:26:00.580
I'm not sure now how much activity there is.
00:26:00.580 --> 00:26:03.440
And we definitively have people from the R kernel.
00:26:03.440 --> 00:26:08.160
The R kernel was created by Thomas Clover, who is now back in the UK, still working with us.
00:26:08.160 --> 00:26:20.480
And I've been taken over by some R people who are actively contributing to the R kernel and are also reporting bugs and fixing bugs a lot in IPython itself.
00:26:20.660 --> 00:26:33.680
Yeah, and another active kernel author community is the Calico project, which is from the CS department of Bryn Mawr by Doug Blank, where it's kind of a multi-language.
00:26:33.680 --> 00:26:37.040
The kernel itself is a multi-language environment.
00:26:37.040 --> 00:26:39.440
They can actually switch between different runtimes.
00:26:39.440 --> 00:26:41.900
That does some pretty cool stuff.
00:26:41.900 --> 00:26:46.080
And they've been very helpful with implementation and protocol testing and things.
00:26:46.080 --> 00:26:46.920
Oh, that's cool.
00:26:47.040 --> 00:26:58.420
Yeah, and that's kind of related to what I was going to ask you next is if I want to write something in Python and then something in C++ and then something in R, can I do that in like one notebook and have the data work together?
00:26:58.420 --> 00:27:00.240
Well, yes.
00:27:01.420 --> 00:27:03.600
So there are a couple things to that.
00:27:03.600 --> 00:27:10.760
One is we have chosen in the notebook to associate one notebook with one kernel.
00:27:10.760 --> 00:27:16.020
So there's one process determining how to interpret the code cells and produce output as a result.
00:27:16.260 --> 00:27:34.400
There's another project derived from IPython called Beaker notebook that doesn't do this, that associates each cell with a kernel and then defines a data interchange for moving data around that allows running code and passing data around from JavaScript to R, Python, and like this.
00:27:34.940 --> 00:27:43.380
However, a kernel, from Jupyter perspective, a kernel can itself define semantics for running code in other languages.
00:27:43.380 --> 00:27:49.780
And IPython, this is where sort of the distinction between IPython and Jupyter comes up.
00:27:49.780 --> 00:27:53.660
That as far as Jupyter is concerned, there's one kernel associated with the notebook.
00:27:53.660 --> 00:28:09.980
But the IPython kernel can define these things called cell magics that say this is shorthand for actually compiling a block of C++ code with Cython and then running that, or the R magic that actually hands off code to an R interpreter.
00:28:09.980 --> 00:28:17.460
As far as Jupyter is concerned, there's only one kernel per notebook, but the kernels themselves can actually provide some of this multi-language functionality.
00:28:17.460 --> 00:28:18.800
And IPython does.
00:28:18.800 --> 00:28:31.540
Yeah, to extend on what Min said, there is another kernel, one of the Calico kernels, actually, which actually is one kernel that implements many languages, won't go exactly into details.
00:28:31.540 --> 00:28:41.780
And there is this nice distinction that a kernel is not always one language, it can be many languages, and in particular the Calico kernel uses triple person syntax to say,
00:28:41.780 --> 00:28:46.120
hey kernel, change how you parse the next string.
00:28:46.660 --> 00:28:50.820
And so you can actually switch in between three or four languages.
00:28:50.820 --> 00:28:51.920
I don't exactly remember.
00:28:51.920 --> 00:28:55.560
There is Python, there is Scheme, and something like that.
00:28:55.560 --> 00:28:57.220
And it's really interesting.
00:28:57.220 --> 00:29:01.440
And you have different ways of actually sharing data between languages.
00:29:01.440 --> 00:29:07.440
One of the examples I can try to dig, which is really interesting in multi-language integration,
00:29:07.440 --> 00:29:12.600
is the Julia Python magics demonstration.
00:29:12.600 --> 00:29:19.760
You can actually move data and actually have a tight integration between Julia and Python.
00:29:19.760 --> 00:29:22.160
It's only on Python 2, unfortunately.
00:29:22.160 --> 00:29:26.460
We need to update the code for Python 3, but I'm not fluent enough in Julia.
00:29:26.840 --> 00:29:35.860
You can define the Fibonacci function recursively in Julia, calling Fibonacci n-1 in Python and Fibonacci n-2 in Julia.
00:29:35.860 --> 00:29:41.620
And the Python versions that call Fibonacci n-1 in Julia and Fibonacci n-2 in Python.
00:29:41.620 --> 00:29:50.820
And you can ask for a number, and you will actually get a cross-language talk or a stack trace where you have a layer cake of each language,
00:29:51.080 --> 00:29:53.220
which is really, really impressive.
00:29:53.220 --> 00:30:01.140
You can even create from Julia, you can import Matplotlib from the Python side,
00:30:01.140 --> 00:30:09.040
create a figure in Julia with a function which takes sine, for example, from the Julia standard library,
00:30:09.040 --> 00:30:12.620
and cosine from the Python standard library on MPI,
00:30:12.620 --> 00:30:17.760
and plot that on the Matplotlib figure and get back into Python
00:30:17.760 --> 00:30:21.740
and annotates the figure from the Python side without copying the figures.
00:30:21.740 --> 00:30:24.300
It means that the two interpreters are actually sharing memory.
00:30:24.300 --> 00:30:31.940
So it shows you that you can do some really, really advanced cross-language integration
00:30:31.940 --> 00:30:34.860
without having to copy data back and forth.
00:30:34.860 --> 00:30:39.660
That sounds really interesting and useful for scientists.
00:30:39.660 --> 00:30:43.840
You know, maybe they've got something they've done in R or some other language.
00:30:43.840 --> 00:30:44.200
Yeah.
00:30:44.200 --> 00:30:46.340
Some little bit of processing, and they're like,
00:30:46.340 --> 00:30:50.640
I really just want to plug this in over here, but it's the wrong language, right?
00:30:50.640 --> 00:30:52.660
And that sounds like it makes it kind of possible.
00:30:52.660 --> 00:30:57.260
Yeah, and it's been, so Julia being a very young language community,
00:30:57.260 --> 00:31:01.760
it's been extremely valuable to them to build this bridge,
00:31:01.760 --> 00:31:05.600
largely to Python, but also to C and things.
00:31:05.600 --> 00:31:11.820
That Julia didn't, in order to come up to speed with what scientific programmers expect,
00:31:11.820 --> 00:31:15.060
with things like Matplotlib and stuff, they didn't need to write,
00:31:15.060 --> 00:31:18.520
okay, here's the Julia plotting library, just so that people could do anything.
00:31:18.520 --> 00:31:22.700
They could start out by saying, well, we'll just use Python libraries.
00:31:22.700 --> 00:31:28.180
And because of these really, really slick layers that let Julia talk to Python
00:31:28.180 --> 00:31:30.840
and Python talk to Julia in a really native way,
00:31:30.840 --> 00:31:35.740
Julia basically gets the entire Python library ecosystem for free,
00:31:35.940 --> 00:31:39.380
and then can kind of re-implement as needed.
00:31:39.380 --> 00:31:43.000
And as they find more idiomatic Julia ways to do things,
00:31:43.000 --> 00:31:44.800
they can start building those libraries.
00:31:44.800 --> 00:31:48.140
But they didn't have to start from zero just because it was a new language.
00:31:48.140 --> 00:31:52.440
For new languages in general, I think being able to interoperate with other languages
00:31:52.440 --> 00:31:56.840
is a really important, really valuable way to start kind of hit the ground running.
00:31:57.660 --> 00:32:04.660
That is super interesting because, you know, there's 10, 15 years of super solid data science stuff happening in Python.
00:32:04.660 --> 00:32:09.540
And if you can just go, we'll just start from there, rather than from zero, that makes all the difference.
00:32:09.540 --> 00:32:15.240
The other cross-language thing that a few people think about is actually the language in the kernel
00:32:15.240 --> 00:32:19.080
and the language of the front end, which is JavaScript and HTML.
00:32:19.080 --> 00:32:25.900
And the notebook allows you to do like these cross-language bindings really easily, especially with widgets.
00:32:25.900 --> 00:32:30.080
And one of the examples I give is you can have interactive 4chan.
00:32:30.080 --> 00:32:34.580
You can actually plot something in 4chan and display something in JavaScript
00:32:34.580 --> 00:32:38.160
and have a slider that you move and change the result of your 4chan computation.
00:32:38.160 --> 00:32:40.020
Wow, that's really awesome.
00:32:40.880 --> 00:32:46.580
One of the things I wanted to spend a few minutes talking with you guys about was what you referred to as the big split.
00:32:46.580 --> 00:32:51.200
So it used to be that IPython was like one giant GitHub repository, right?
00:32:51.200 --> 00:32:54.900
And now you've broken it into many smaller pieces, yeah?
00:32:54.900 --> 00:32:57.060
Yeah, I went from one to about a dozen.
00:32:57.060 --> 00:32:58.500
That sounds like a lot of work.
00:32:58.500 --> 00:33:02.660
Yeah, that was my spring, pretty much.
00:33:02.660 --> 00:33:09.140
It took three to six months to kind of get it all split up.
00:33:09.560 --> 00:33:13.860
But since we knew we eventually wanted to do something like that,
00:33:13.860 --> 00:33:20.900
we had, for the most part, already organized IPython into kind of sub-packages of dedicated functionality.
00:33:20.900 --> 00:33:26.320
So it wasn't too crazy difficult to break it up.
00:33:26.320 --> 00:33:31.520
But the tricky bits were like the common utilities that we use all over the place and how to deal with that.
00:33:31.520 --> 00:33:40.580
Yeah, it's the little interdependencies that don't seem big, but when they're woven in between all the pieces, all of a sudden it gets harder and harder, right?
00:33:40.580 --> 00:33:41.140
Yeah.
00:33:41.140 --> 00:33:46.140
So it was tricky to execute the big split in a variety of ways.
00:33:46.840 --> 00:34:00.560
So we wanted to preserve history, but we also didn't want to duplicate the already large IPython repo so that installing from Git would mean that it now is 12 times as big as getting IPython was.
00:34:00.560 --> 00:34:10.860
So we had to do some, on the new repos, some clever Git history rewriting to kind of prune out the history for the files that didn't survive.
00:34:10.860 --> 00:34:12.160
That's really interesting.
00:34:12.160 --> 00:34:17.200
You talked about that in your blog called The Big Split, which I'll link to in the show notes.
00:34:17.200 --> 00:34:21.220
And you had to use some funky commands to sort of make that happen, right?
00:34:21.220 --> 00:34:23.220
Be really careful about how you move the history over.
00:34:23.760 --> 00:34:35.640
Yeah, and this is what, there are some, I think, in the various version control tools, the fact that you can rewrite history in Git is both really scary and weird and gross, but also really useful sometimes.
00:34:35.640 --> 00:34:50.860
But it lets you do things like we did, which is selectively preserve history, which has been nice that you get, you know, you get the history of the notebook work in the notebook repo, even past the creation of the notebook repo.
00:34:50.860 --> 00:34:52.920
But you don't get the baggage, right?
00:34:52.920 --> 00:34:56.640
Yeah, but we don't get the history of all the rest of IPython.
00:34:56.640 --> 00:35:01.880
Yeah, because normally you delete a file out of the repo and it's just, it doesn't show up, but you're still moving it around, right?
00:35:01.880 --> 00:35:06.800
To give you an idea, I will just share, I will send you a link.
00:35:06.800 --> 00:35:13.900
Someone made a graph visualization of the dependencies into IPython before the Big Split.
00:35:14.460 --> 00:35:26.600
And on the same blog post, you have comparison with Django, Twisted, Flask, requests, so that you can get an idea of what the complexity of the entanglement was.
00:35:26.600 --> 00:35:30.640
It's on grokcode.com and it's blog post 864.
00:35:30.640 --> 00:35:34.340
And I will give you the link so you can put it in the notes.
00:35:34.880 --> 00:35:35.560
Oh yeah, thanks.
00:35:35.560 --> 00:35:37.800
It's pretty seriously entangled.
00:35:37.800 --> 00:35:44.100
There aren't dependency cycles and crazy loops of depending on each other.
00:35:44.100 --> 00:35:50.320
It's kind of a tree of dependencies, but there are many nodes on the graph.
00:35:50.740 --> 00:35:51.680
Sure, sure.
00:35:51.680 --> 00:36:16.020
SnapCI is a continuous delivery tool from ThoughtWorks that lets you reliably test and deploy your code through multi-stage pipelines in the cloud without the hassle of managing hardware.
00:36:16.580 --> 00:36:22.680
Automate and visualize your deployments with ease and make pushing to production an effortless item on your to-do list.
00:36:22.680 --> 00:36:28.560
Snap also supports Docker and M-browser debugging, and they integrate with AWS and Heroku.
00:36:28.560 --> 00:36:37.240
Thanks SnapCI for sponsoring this episode by trying them with no obligation for 30 days by going to snap.ci slash talkpython.
00:36:45.500 --> 00:36:48.960
Another thing I wanted to talk about with you guys is this thing called JupyterHub.
00:36:48.960 --> 00:36:50.940
What's the story of the JupyterHub?
00:36:50.940 --> 00:36:57.780
So JupyterHub came out of the value of Jupyter Notebooks and things in teaching context.
00:36:57.780 --> 00:37:04.160
So a lot of people, whether in workshops or classes, are using notebooks to present.
00:37:04.160 --> 00:37:09.400
This is the material that we're talking about and the example code and running, you know, doing live demos and things.
00:37:10.400 --> 00:37:20.940
You want your students to be able to follow along, and this is one of those cases where, you know, installing scientific Python stack is as much as Conda has made that easier.
00:37:20.940 --> 00:37:23.820
It still can be, you know, a significant bar to get over.
00:37:24.820 --> 00:37:36.820
So we wanted, and people were building kind of, people were building tools for kind of hacks around IPython at the time to deploy notebooks on behalf of users.
00:37:37.400 --> 00:37:49.860
And we wanted to provide kind of an official implementation of hosting notebooks on behalf of a group of users in the context of a research group.
00:37:49.860 --> 00:37:55.740
So you've got a machine that a bunch of, you know, half a dozen or so research scientists or students have access to,
00:37:55.800 --> 00:38:00.220
or you have a class of 10 or 50 or 100 students.
00:38:00.220 --> 00:38:03.680
And you say, all right, I've got these users.
00:38:03.680 --> 00:38:10.140
I can install packages for them and then point them at this URL and they can log in and run their notebook.
00:38:10.140 --> 00:38:20.400
So basically take away the installation problem by saying, I'm going to control the installation and host the notebooks and everything.
00:38:21.180 --> 00:38:24.780
And we wanted to kind of create the simplest, smallest version of that.
00:38:24.780 --> 00:38:25.720
And that's JupyterHub.
00:38:25.720 --> 00:38:26.460
Okay, nice.
00:38:26.460 --> 00:38:32.180
From the technical side, we asked before how, which technology were necessary for the notebook.
00:38:32.180 --> 00:38:34.320
And we spoke about WebSockets.
00:38:34.320 --> 00:38:44.580
One of the things which is important is we were using really recent technology and WebSockets was really new at the time.
00:38:44.960 --> 00:39:01.100
And one of the problems is many proxies or web servers were unable to correctly redirect WebSockets or even to update proxy rules without actually restarting the proxy or server.
00:39:01.100 --> 00:39:04.720
And that's one of the requirements we had for the notebook.
00:39:04.820 --> 00:39:18.180
If you want to spawn someone notebook without cutting the connection of the other, we had to have a dynamic proxy, which was able to respond to, for example, REST requests, like changes through there without dropping the WebSockets.
00:39:19.140 --> 00:39:24.840
And before JupyterHub, only a handful of prototypes of projects were able to do that.
00:39:24.840 --> 00:39:36.700
And actually, Min wrote one such an HTTP proxy using Node.js to suit this specific need that no other tools require.
00:39:37.020 --> 00:39:40.140
And that's why you actually need JupyterHub to run something.
00:39:40.140 --> 00:39:43.800
And JupyterHub needs to be one of the most front-facing software.
00:39:43.800 --> 00:39:45.800
I think that no Nginx can do it, too.
00:39:45.800 --> 00:39:50.040
You cannot, for example, use Apache or something else.
00:39:50.040 --> 00:39:54.060
Or it's really much more difficult to have many notebook servers running.
00:39:54.060 --> 00:39:59.060
It's interesting that a lot of the web servers weren't really built for that, right?
00:39:59.060 --> 00:40:03.180
Because I guess they came before WebSockets anyway.
00:40:03.180 --> 00:40:06.720
And that probably was not a super important criteria for them, right?
00:40:06.720 --> 00:40:13.520
Yeah, the web servers were actually surprisingly slow to adopt, to provide WebSocket implementations.
00:40:13.520 --> 00:40:21.360
Nginx didn't take too long, but it was quite a while before you could reasonably expect an Apache installation to support WebSockets.
00:40:21.360 --> 00:40:26.020
They do now, so there are notebook deployments behind both Apache and Nginx.
00:40:26.020 --> 00:40:34.740
But, yeah, we put together the configurable HTTP proxy as this kind of super simple proxy that you can update.
00:40:34.740 --> 00:40:40.780
You can update the routing table without relaunching or without losing existing connections or anything.
00:40:41.340 --> 00:40:45.980
The more you can make it easier for people to set up these environments, the better.
00:40:45.980 --> 00:40:52.380
Because it's not always going to be like some web server admin or a really experienced web developer doing this, right?
00:40:52.380 --> 00:40:56.260
It could just be a scientist who just wants this thing for their class, right?
00:40:56.260 --> 00:40:57.820
They don't want to deal with Nginx.
00:40:57.820 --> 00:41:00.180
Yeah, and that's something we're working on right now.
00:41:00.240 --> 00:41:05.580
Because the way JupyterHub is put together, it has two primary extension points.
00:41:05.580 --> 00:41:07.460
One is authentication.
00:41:07.460 --> 00:41:22.720
So how users log in, and you can kind of drop in any implementation of logging in, whether it's just local authentication with the system, with the password, or using OAuth with GitHub, or your campus sign-on, that kind of stuff.
00:41:22.720 --> 00:41:28.480
And then the other is the spawning, how it actually allocates resources for the single-user servers.
00:41:28.480 --> 00:41:40.040
But because there are so many choices for how to do that, it actually, one thing we're working on is making kind of more of a turnkey version that you can say,
00:41:40.040 --> 00:41:46.240
I want to use this authentication system and this spawning mechanism, and people can just deploy that.
00:41:46.240 --> 00:41:55.560
Because there are, from the very simple default behavior of just that works out of the box for, I've just got a shared machine that's on the internet,
00:41:55.560 --> 00:41:59.500
and I want to give all the users who already have accounts on that machine access.
00:41:59.500 --> 00:42:08.860
That's pretty trivial right now, all the way to a deployment last year that Jess Hamrick did at UC Berkeley
00:42:08.860 --> 00:42:22.580
for a couple hundred students in psychology using Docker Swarm and Nginx and a big multi-node deployment for a large number of users
00:42:22.580 --> 00:42:25.580
and using Ansible to automate all that deployment.
00:42:25.580 --> 00:42:26.700
That sounds awesome.
00:42:26.700 --> 00:42:28.260
Is that documented somewhere?
00:42:28.260 --> 00:42:30.100
Like, is there an article or something on this?
00:42:30.100 --> 00:42:30.280
Yeah.
00:42:30.280 --> 00:42:36.700
So she wrote a blog post for the Rackspace developer blog because the hosting for that class was all provided by Rackspace.
00:42:37.260 --> 00:42:43.400
So she wrote a blog post covering that, and her Ansible setup is just a repo on GitHub that we can link to.
00:42:43.400 --> 00:42:44.120
Okay, awesome.
00:42:44.120 --> 00:42:45.460
Yeah, we'll put that in the show notes.
00:42:45.460 --> 00:42:50.480
So I can go to Dropbox and get, like, storage as a service.
00:42:50.480 --> 00:42:54.840
I can go to Google Apps and get Word Processor as a service.
00:42:54.840 --> 00:42:57.220
Can I do that for Jupyter somewhere?
00:42:57.220 --> 00:43:02.120
Can I just go and, like, pay $5 a month and get, like, Jupyter?
00:43:02.120 --> 00:43:05.580
Yeah, there are a few companies hosting Jupyter Notebooks.
00:43:06.460 --> 00:43:11.980
So IBM has their, I believe it's called a workbench, data science workbench.
00:43:11.980 --> 00:43:17.600
Continuum Analytics has Wokari, which hosts Notebooks.
00:43:17.600 --> 00:43:20.520
I'm trying to think how many others there are.
00:43:20.520 --> 00:43:23.480
There's Domino Data Lab.
00:43:23.480 --> 00:43:26.100
So there's William Stein with SageMath.
00:43:26.100 --> 00:43:31.860
Yeah, SageMath Cloud is probably the primary, the one we're most connected to.
00:43:31.860 --> 00:43:32.940
Okay, cool.
00:43:32.940 --> 00:43:33.760
That's good to hear.
00:43:34.480 --> 00:43:38.280
Yeah, so there are a variety of these hosted notebook things.
00:43:38.280 --> 00:43:44.800
Yeah, one thing which is slightly related, I don't know if we might talk to that later.
00:43:44.800 --> 00:44:00.500
It's mybinder.org, which has been set up by Jeremy Freeman from Janelia Labs, where basically you set up a GitHub repository, your notebooks, a requirement file, some extra metadata if needed.
00:44:01.340 --> 00:44:03.720
And you link to mindbinder.org.
00:44:03.720 --> 00:44:05.360
I will give you the link.
00:44:05.360 --> 00:44:14.800
And it will actually, just for you, spawn a Docker instance with the requirements and give you a temporary notebook online.
00:44:15.100 --> 00:44:20.820
So if you have an article that you want to be reproducible, you can just post it on GitHub.
00:44:20.820 --> 00:44:25.700
It's basically like NB viewer, for those who know, but back by a kernel.
00:44:26.260 --> 00:44:28.540
And it's paid directly out of Jeremy's pocket.
00:44:28.540 --> 00:44:31.420
And huge thanks to him.
00:44:31.420 --> 00:44:41.860
If ever someone from Google hears that and likes the project, you would be grateful if you could give some free resource to Jeremy to host that.
00:44:42.800 --> 00:44:44.280
Wow, that's really cool.
00:44:44.280 --> 00:44:47.440
You know, I was thinking about how Docker might fit into it here.
00:44:47.440 --> 00:44:50.800
And that's certainly a really nice use of it.
00:44:50.800 --> 00:45:03.720
We have TempNB, which is written by Kyle Kelly at Rackspace, which drives, if you go to try.jupiter.org, you'll get temporary notebook server to kind of try out Jupyter with a few kernels installed.
00:45:03.720 --> 00:45:05.600
But it's very locked down.
00:45:05.600 --> 00:45:08.680
You can't get your own work in there or do network things.
00:45:09.360 --> 00:45:22.440
And so Binder basically combined the NB viewer idea with TempNB to kind of create a workspace where you can actually install packages and get everything running and preload it with notebooks.
00:45:22.440 --> 00:45:28.600
So we're kind of we're working on a variety of these kind of applications.
00:45:28.600 --> 00:45:33.760
A lot of them are Docker based of deploying notebooks on in various various contexts.
00:45:34.040 --> 00:45:44.860
So one thing I wanted to make sure we talk about before we wrap things up is not too long ago, I guess in July, your project got $6 million in funding.
00:45:44.860 --> 00:45:47.060
And that's just so awesome.
00:45:47.060 --> 00:45:54.740
I mean, I can't think of another open source project that's not deeply tied to a company that got that kind of money.
00:45:55.180 --> 00:46:06.240
Yeah, we've been extremely fortunate with the funding over the last few years, thanks to a huge amount of work from Brian Granger and Fernando Perez in chasing that funding down.
00:46:06.240 --> 00:46:13.180
And which is what has given me a job for the last few years and Matthias as well.
00:46:13.180 --> 00:46:15.480
So, yeah, that's really great.
00:46:15.480 --> 00:46:18.640
I mean, what kind of difference does that make for what you guys are doing?
00:46:18.740 --> 00:46:21.980
Obviously, a job, but what are the goals for that money?
00:46:21.980 --> 00:46:33.120
Really to expand, kind of expand the coverage of the Jupyter project that has a lot to do with education and kind of the document and publication pipeline.
00:46:33.600 --> 00:46:41.580
So with NB convert, we can convert notebooks to various formats, but that's a very young project that has a lot of a lot of room for improvement.
00:46:41.580 --> 00:46:52.620
And so integrating into publication pipelines and things like that and converting other formats to notebooks in addition to converting what we already do, which is converting notebooks to rendered formats.
00:46:52.780 --> 00:47:00.720
And then the other side is in the building web applications with the tools that we have.
00:47:00.720 --> 00:47:05.380
So live collaboration, real time collaboration in the notebook is going to be a big part of it.
00:47:05.380 --> 00:47:08.700
And then building applications other than the notebook.
00:47:08.700 --> 00:47:13.380
So we've got this protocol for execution that we've talked about as kind of the building block.
00:47:13.380 --> 00:47:17.020
You can use that without the notebook document.
00:47:17.760 --> 00:47:26.900
So you can build a web application that involves running code and producing output, but it may not necessarily be a document editor type application.
00:47:26.900 --> 00:47:31.480
It could be something on the other side that's not the notebook app, basically.
00:47:31.480 --> 00:47:43.780
O'Reilly built something, basically built a version of this called Phoebe that drives a few demos and some interactive code-based blog posts on the O'Reilly website.
00:47:43.780 --> 00:47:51.620
But if you look at how Phoebe works, and Phoebe also itself relies on some TempNB-based Docker deployment stuff.
00:47:51.620 --> 00:47:59.780
The way the JavaScript side of the code is organized and also to some degree the server side, we don't make it easy.
00:48:00.480 --> 00:48:09.340
So while it's technically true that you can use the kernel execution display stuff without the notebook document piece, we didn't make it easy to do that.
00:48:09.340 --> 00:48:12.940
So Phoebe kind of has to hack around a few of the choices we made.
00:48:12.940 --> 00:48:22.620
And right now we're in the middle with the help of a lot of volunteer to contribute work from Continuum Analytics and Bloomberg and other folks.
00:48:23.120 --> 00:48:31.800
We're working on refactoring the client side of the code into small JavaScript packages, just like we broke up the Python side into the various pieces.
00:48:31.800 --> 00:48:33.120
We're breaking up the JavaScript side.
00:48:33.120 --> 00:48:39.240
So you can say, I want to install just the client side code that lets me run code and get output.
00:48:40.280 --> 00:48:46.780
And with that, then you can build an application that is a web app that maybe you don't even show any code to the user.
00:48:46.780 --> 00:48:50.080
You just show sliders and buttons and everything.
00:48:50.300 --> 00:48:55.360
But the way your application works actually involves this executing code and getting output with the display protocol.
00:48:55.360 --> 00:48:58.400
So you don't have to rewrite all of that logic yourself.
00:48:59.160 --> 00:49:09.200
It's kind of similar to the way Julia and Haskell drove us to revise our messaging protocol to be more useful outside of Python.
00:49:09.200 --> 00:49:24.200
Things like Phoebe and other applications are driving us to kind of rethink how our JavaScript code works so that you can actually, you can use it kind of pieces at a time in contexts other than the ones that we've already thought about.
00:49:24.500 --> 00:49:32.140
Yeah, it sounds like there's going to be a lot of innovation and cool new stuff built on top of this now that you've broken out all the building blocks.
00:49:32.140 --> 00:49:37.540
Beyond this just purely technical, we have money for the project and actually write code.
00:49:37.540 --> 00:49:39.660
I first want to do thanks.
00:49:39.660 --> 00:49:45.020
I mean, the MZ Moore and Sloan Foundation are really people who are nice to work with.
00:49:45.020 --> 00:49:47.780
I mean, they just don't ask us for reports.
00:49:47.780 --> 00:49:48.860
It's really a back and forth.
00:49:48.860 --> 00:49:53.560
And the money is allowing us to do much more things than just being employed.
00:49:53.560 --> 00:50:01.140
We, for example, now will be able to host a Jupyter Day where we can actually have talks and invite speakers.
00:50:01.140 --> 00:50:04.440
We already had one in November in New York.
00:50:04.440 --> 00:50:07.420
And there is, no, it's in November.
00:50:07.420 --> 00:50:07.960
Anyway.
00:50:07.960 --> 00:50:13.380
And one next month, so in February in Chicago, there is a Jupyter Day.
00:50:13.380 --> 00:50:17.720
And without this sound, it would be difficult to actually bootstrap that.
00:50:18.220 --> 00:50:26.220
And the other thing is now we are around 20 people across the globe working on the project.
00:50:26.220 --> 00:50:37.180
People that are actually on their day job or at Continuum and are at Workspace or IBM, Google, Microsoft are actually allowed during their work day to contribute to the project.
00:50:37.800 --> 00:50:39.000
And this helps a lot.
00:50:39.000 --> 00:50:45.920
And one of the things that this fund will allow us is to actually gather twice a year, all of us, to actually meet each other.
00:50:46.560 --> 00:50:52.320
And once you meet people, you can do much more work than only interacting through the Internet.
00:50:52.320 --> 00:50:55.120
And that's something that is really, really great.
00:50:55.120 --> 00:50:59.940
And that will change a lot of things, I think, for the project.
00:51:00.100 --> 00:51:06.300
I mean, at least last year, I think the NumPy developer had never met all together.
00:51:06.300 --> 00:51:10.300
And they got some funding to get them together at SciPy.
00:51:10.300 --> 00:51:14.920
And apparently, it was really great to motivate everyone and relaunch the project.
00:51:14.920 --> 00:51:19.280
And I'm really looking forward because some people that are working with us have never met in person.
00:51:20.380 --> 00:51:27.480
And I would really, in a few weeks, we should meet together thanks to some of these funds.
00:51:27.480 --> 00:51:30.260
And that's something I'm really looking forward to.
00:51:30.260 --> 00:51:33.440
That's a really interesting point about the community.
00:51:33.440 --> 00:51:35.900
There's obvious technical stuff.
00:51:35.900 --> 00:51:41.080
But like you said, building the community both from the outside and the inside, that's definitely going to be powerful.
00:51:41.080 --> 00:51:41.860
Yeah.
00:51:41.960 --> 00:51:51.040
And another aspect of the building and maintaining the community that is facilitated by this funding is being able to,
00:51:51.040 --> 00:51:58.000
and one of the things that a lot of open source projects, our own included, really suffer on is documentation.
00:51:58.000 --> 00:52:02.060
That especially a fast-moving project, the documentation can really suffer.
00:52:02.060 --> 00:52:08.540
Because especially when you're working with volunteers and things, it's a lot more fun to work on code than docs.
00:52:08.540 --> 00:52:09.080
Yeah.
00:52:09.160 --> 00:52:15.160
Would you rather write a feature, a new feature, or would you rather talk about somebody else's feature that's going to be outdated anyway, right?
00:52:15.160 --> 00:52:16.040
Yeah.
00:52:16.040 --> 00:52:29.400
And I think it is the responsibility of funded projects like ours to devote some of those resources to maintaining communication with the community and building good docs.
00:52:29.640 --> 00:52:42.780
Because it's often easier to find people who can build good docs that you can pay to do that than it is people who can build good docs who are willing to volunteer a bunch of their time.
00:52:42.780 --> 00:52:50.300
And so being able to compensate people for contribution to documentation is, I think, an important thing for funded projects to do.
00:52:50.300 --> 00:52:57.800
And so we've recently hired a couple people to specifically emphasize improving documentation and things.
00:52:57.800 --> 00:53:04.960
And I think that's a really valuable thing that the funding provides for not just us, but for the community.
00:53:04.960 --> 00:53:08.160
Good documentation and tutorials and samples.
00:53:08.160 --> 00:53:11.380
That really makes it much easier to start using a project.
00:53:11.380 --> 00:53:31.560
One point about contribution that has followed me since I started the project, and I felt was great, especially in the Python community, and made me stuck to iPython, is most of the core developers, if you are a beginning Python programmer and you want to contribute to the project,
00:53:31.560 --> 00:53:39.000
We will likely spend a few hours with you on GitHub or even on Skype, if you ask, to help you build a feature.
00:53:39.000 --> 00:53:47.520
Even if it would take us five minutes, I will personally take a few hours to teach you how to do it if you want to contribute.
00:53:47.520 --> 00:53:51.800
So don't be afraid to come and ask and say, I don't know.
00:53:51.800 --> 00:53:52.900
I want to build that.
00:53:52.900 --> 00:53:54.220
I have the time.
00:53:54.220 --> 00:53:55.100
I have the motivation.
00:53:55.100 --> 00:53:57.380
It's okay if it takes you four weeks.
00:53:57.380 --> 00:53:58.760
It's okay to be wrong.
00:53:58.760 --> 00:53:59.640
It's okay to ask.
00:54:00.280 --> 00:54:17.200
Unlike some other community who are known for having a BDFL, which is not really nice on the meaning list, we will take time for you to help you, to teach you Git, because we really think that you should feel welcome.
00:54:17.200 --> 00:54:23.880
And that's how we had some people that stayed here, because contributing is a project where we help you.
00:54:23.880 --> 00:54:27.160
It's not, let me do it and I'm done in five minutes later.
00:54:27.880 --> 00:54:32.300
We prefer for people to take four weeks to do something than doing it ourselves.
00:54:32.300 --> 00:54:36.600
It's all about teaching people to fish rather than getting them a fish, right?
00:54:36.600 --> 00:54:41.200
You can just do so much more if you can get more people brought into the project and comfortable.
00:54:41.200 --> 00:54:43.080
And that's really, it's a great message.
00:54:43.080 --> 00:54:43.380
Thanks.
00:54:43.960 --> 00:54:48.440
So I basically have time for one more question for you guys, and then we'll call it a show.
00:54:48.440 --> 00:54:56.200
Min and I were talking about some really cool places using Jupyter before we started recording.
00:54:56.200 --> 00:55:06.600
And I just want to ask you each, maybe Matthias first, what's the coolest use or example of people using Jupyter or IPython that you know of?
00:55:07.140 --> 00:55:12.700
Peter Norvig posts on NB viewer are some of the most awesome things I've seen.
00:55:12.700 --> 00:55:20.740
He is really busy and he had some typos on his notebook and I sent him a patch and he took the time to reply to me.
00:55:20.820 --> 00:55:27.500
So he is both someone who wrote awesome notebooks and someone who is really friendly.
00:55:27.500 --> 00:55:33.780
And I also like a lot Jake Van Der Plass' blog when he's blogging.
00:55:33.960 --> 00:55:37.740
He has a blog engine that publish directly in the book.
00:55:37.740 --> 00:55:38.260
So it's Pelican.
00:55:38.260 --> 00:55:45.980
And if you can add a plugin and you can directly like wrote your notebook and they're published directly as blogs.
00:55:45.980 --> 00:55:49.720
And also my blog, which is Nicholas, is just notebook.
00:55:49.720 --> 00:55:51.920
I just push notebook on the GitHub repository.
00:55:51.920 --> 00:56:00.260
And Travis CI take care of compiling them to HTML and publish, which makes publishing really easy, I find.
00:56:00.260 --> 00:56:02.760
Oh, that's, yeah, those are really cool examples.
00:56:02.760 --> 00:56:03.440
Min?
00:56:03.440 --> 00:56:22.200
Yeah, so the examples for me are Thomas and I got to visit CERN in December where they're working on some big Jupyter deployments for the scientists working on the LHC and other experiments at CERN.
00:56:22.200 --> 00:56:39.040
So that's definitely a really cool thing to see, you know, Jupyter and IPython, as is someone whose degrees are in physics, seeing Jupyter and IPython valuable in that level of physics experiment and theory research has been really cool.
00:56:39.580 --> 00:56:47.160
The other things that I think are really cool are where it's been adopted in educational contexts.
00:56:47.160 --> 00:56:57.540
So, for instance, Lorena Barba has her AeroPython course teaching aerodynamics with Python is all in notebooks on GitHub.
00:56:58.420 --> 00:57:06.120
And then Doug Blank has his computer science courses all with notebooks in a JupyterHub instance for his students.
00:57:06.120 --> 00:57:14.860
And Jess Hamrick at Berkeley with her large deployment teaching cognitive computational models for cognition.
00:57:15.860 --> 00:57:28.100
And some of these are not programming classes, but they're classes meant to teach scientific principles where programming just happens to be a useful way to illustrate and explore the ideas.
00:57:28.360 --> 00:57:43.700
And the fact that notebooks are proving useful in kind of just learning, both learning how to do science for students at the undergraduate and graduate level, and then also for actual scientists doing, you know, doing real cool science.
00:57:43.700 --> 00:57:47.320
Both of those, I think, make me feel really good about working on the project.
00:57:47.320 --> 00:57:50.600
Yeah, I'm sure those are really motivational.
00:57:50.600 --> 00:57:53.820
You wake up, you're like, wow, I'm building software for all of this.
00:57:53.820 --> 00:57:54.960
This is fantastic.
00:57:55.300 --> 00:58:06.820
On show 29, I had Kyle Cranmer from the Atlas Experiment at Large Hedron Collider on, and he talked about how they were using IPython and stuff like that there as well, which is very cool.
00:58:06.820 --> 00:58:10.620
All right, gentlemen, this has been super interesting.
00:58:10.620 --> 00:58:22.500
It sounds like IPython has become Jupyter, and it has been such a success, but it seems like it's on the brink of breaking out to be way much more than what it has been.
00:58:22.500 --> 00:58:24.920
So, it's an exciting time.
00:58:24.920 --> 00:58:26.480
Thanks for having us.
00:58:26.480 --> 00:58:27.500
Yeah, thanks for having us.
00:58:27.500 --> 00:58:28.240
Yeah, you bet.
00:58:28.240 --> 00:58:29.040
Thanks for being on the show.
00:58:29.040 --> 00:58:30.080
Talk to you later.
00:58:30.080 --> 00:58:34.660
This has been another episode of Talk Python To Me.
00:58:34.660 --> 00:58:41.260
Today's guest was Matthias Boussignier and Min RK, and this episode has been sponsored by Hired and SnapCI.
00:58:41.260 --> 00:58:43.420
Thank you guys for supporting the show.
00:58:43.420 --> 00:58:45.940
Hired wants to help you find your next big thing.
00:58:45.940 --> 00:58:54.800
Visit Hired.com slash Talk Python To Me to get five or more offers with salary and equity presented right up front and a special listener signing bonus at $2,000.
00:58:54.800 --> 00:58:58.800
SnapCI is modern, continuous integration and delivery.
00:58:58.800 --> 00:59:05.340
Build, test, and deploy your code directly from GitHub, all in your browser with debugging, Docker, and parallelism included.
00:59:05.340 --> 00:59:08.700
Try them for free at snap.ci slash Talk Python.
00:59:08.700 --> 00:59:14.720
You can find the links from today's show at talkpython.fm/episode slash show slash 44.
00:59:14.720 --> 00:59:16.980
Be sure to subscribe to the podcast.
00:59:16.980 --> 00:59:19.180
Open your favorite podcatcher and search for Python.
00:59:19.180 --> 00:59:20.300
We should be right at the top.
00:59:20.300 --> 00:59:24.260
You can also find the iTunes and direct RSS feeds in the footer of the website.
00:59:24.260 --> 00:59:28.840
Our theme music is Developers, Developers, Developers by Corey Smith, who goes by Smix.
00:59:28.840 --> 00:59:31.840
You can hear the entire song on talkpython.fm.
00:59:32.340 --> 00:59:37.780
Oh, and don't forget to check out the podcast t-shirt at talkpython.fm/shirt.
00:59:37.780 --> 00:59:41.420
Get yours and share your love for Python with the whole world.
00:59:41.420 --> 00:59:43.560
This is your host, Michael Kennedy.
00:59:43.560 --> 00:59:45.920
As always, I really appreciate you listening to the show.
00:59:45.920 --> 00:59:47.980
Smix, take us out of here.
00:59:49.900 --> 01:00:08.960
I'll see you next time.
01:00:08.960 --> 01:00:09.960
Bye.
01:00:09.960 --> 01:00:11.960
.