Full Transcript for #285: Dask as a Platform Service with Coiled

00:02:06.840 --> 00:02:07.400 Hi, Michael. 00:02:07.400 --> 00:02:08.040 Hey, Michael. 00:02:08.040 --> 00:02:10.060 Thanks for having us again after a couple of years. 00:02:10.060 --> 00:02:12.260 Yeah, it's fabulous to have you both back. 00:02:12.260 --> 00:02:14.180 You all were guests previously. 00:02:14.180 --> 00:02:17.940 Hugo, you were on a really popular episode 139, 00:02:17.940 --> 00:02:20.600 Paths into a Data Science Career, 00:02:20.600 --> 00:02:22.900 which I think is great helping people do that. 00:02:22.900 --> 00:02:27.240 And Matthew, you were 207, Paralyzing Computation with Dask. 00:02:27.240 --> 00:02:28.460 Oh, I think you'll beat me. 00:02:28.460 --> 00:02:30.360 You beat me on the show, Hugo. 00:02:30.360 --> 00:02:31.400 It's not a competition. 00:02:31.400 --> 00:02:33.020 But if it was, you're right. 00:02:33.020 --> 00:02:33.620 You would have won. 00:02:33.620 --> 00:02:34.780 Yes, that's right. 00:02:34.780 --> 00:02:39.140 But I think this talk is going to be a really interesting extension 00:02:39.140 --> 00:02:41.320 of the one that you were on, Matthew, 00:02:41.320 --> 00:02:45.340 where we talked about the technical stuff of Dask and the computation. 00:02:45.340 --> 00:02:47.700 And now, you know, it's really grown a lot. 00:02:47.700 --> 00:02:49.000 That was February 2019. 00:02:49.000 --> 00:02:51.480 That was a time in the world when you could go places. 00:02:51.480 --> 00:02:55.600 Like you could just go out and you could go to a place where they made food 00:02:55.600 --> 00:02:57.360 and you could physically touch another person 00:02:57.360 --> 00:02:59.000 and then just take it and you wouldn't worry about it. 00:02:59.000 --> 00:02:59.600 It was amazing. 00:02:59.600 --> 00:03:00.180 That's right. 00:03:00.180 --> 00:03:00.520 Yeah. 00:03:00.520 --> 00:03:02.860 The actual remote experience of developing software 00:03:02.860 --> 00:03:03.840 hasn't changed a ton. 00:03:03.840 --> 00:03:07.780 I think actually people in our space have more or less the same workflow. 00:03:07.780 --> 00:03:10.700 We just, you know, don't see each other at conferences any longer. 00:03:10.700 --> 00:03:11.000 Yeah. 00:03:11.000 --> 00:03:11.180 Yeah. 00:03:11.180 --> 00:03:15.300 If we were accused of being antisocial as developers, well, we've taken it up a notch 00:03:15.300 --> 00:03:16.500 or everyone's just come. 00:03:16.500 --> 00:03:19.400 Other people have come to join us in this weird world. 00:03:19.400 --> 00:03:25.360 But yeah, actually, I think software is pretty social as, you know, even though it gets categorized, 00:03:25.360 --> 00:03:26.280 not so much that way. 00:03:26.280 --> 00:03:28.100 So it's great to have you both back. 00:03:28.100 --> 00:03:32.680 I'm looking forward to talking about Dask, but especially Coiled and this new company you 00:03:32.680 --> 00:03:37.500 are starting and this new platform you're starting sort of based on taking that to the next level. 00:03:37.500 --> 00:03:39.040 So it's going to be a really fun conversation. 00:03:39.040 --> 00:03:39.340 Yeah. 00:03:39.340 --> 00:03:40.120 I'm looking forward to it. 00:03:40.120 --> 00:03:40.520 As am I. 00:03:40.520 --> 00:03:41.280 Yeah, absolutely. 00:03:41.280 --> 00:03:41.840 Yeah. 00:03:41.840 --> 00:03:45.640 Now, before we get into it, though, you both were on before and I asked how you got into 00:03:45.640 --> 00:03:47.840 programming and how you got into Python. 00:03:47.840 --> 00:03:51.180 So I might have a really, you know, maybe keep it short since there's two of you and 00:03:51.180 --> 00:03:54.320 we did cover some of the stuff a little before, but let's try a different question. 00:03:54.320 --> 00:03:55.880 How did you get into data science? 00:03:55.880 --> 00:03:57.180 You go, you want to go first? 00:03:57.180 --> 00:03:57.600 Sure. 00:03:57.600 --> 00:04:01.860 I was working in applied math in cell biology and biophysics. 00:04:01.860 --> 00:04:05.560 And my job was ostensibly to do a bunch of mathematical modeling. 00:04:05.560 --> 00:04:11.920 But all my collaborators were generating loads of data that I needed to figure out how to 00:04:11.920 --> 00:04:16.780 analyze to collaborate with them and even, you know, play a big role in the iterative design 00:04:16.780 --> 00:04:18.500 of experiments and all of this type of stuff. 00:04:18.500 --> 00:04:20.840 This was 2011, 2012. 00:04:20.840 --> 00:04:22.540 Excel didn't seem like a great answer. 00:04:22.540 --> 00:04:23.900 That's where I started. 00:04:23.900 --> 00:04:28.400 And then I went to R and then I went to Python and kind of learned them concurrently because 00:04:28.400 --> 00:04:29.540 I was working with biologists. 00:04:29.540 --> 00:04:34.820 R was quite popular, but the IPython notebook was becoming more and more popular. 00:04:34.820 --> 00:04:38.480 It was a great resource for teaching oneself and teaching others. 00:04:38.480 --> 00:04:43.920 So I jumped in then and worked in research, worked in education a lot, and then moved to 00:04:43.920 --> 00:04:49.180 industry, to tech, and started working in ed tech to educate people around all of these 00:04:49.180 --> 00:04:49.680 tools as well. 00:04:49.680 --> 00:04:49.960 Yeah. 00:04:49.960 --> 00:04:50.580 Excellent. 00:04:50.580 --> 00:04:51.600 Matthew, how about yourself? 00:04:51.600 --> 00:04:55.820 I think for me, that probably the culmination point or the inflection point was just after 00:04:55.820 --> 00:04:57.140 I graduated from university. 00:04:57.140 --> 00:04:59.640 I graduated with like a physics and astronomy degree. 00:04:59.640 --> 00:05:01.360 And I was looking at job applications. 00:05:01.360 --> 00:05:04.440 I didn't like any of the jobs that I was qualified for. 00:05:04.440 --> 00:05:05.540 But the jobs that I loved. 00:05:05.540 --> 00:05:08.360 I had the same feeling when I studied math. 00:05:08.360 --> 00:05:11.120 I'm like, I love this stuff, but what am I going to do with this?

00:05:11.120 --> 00:05:12.700 And none of this looks that great to me. 00:05:12.700 --> 00:05:13.800 But I didn't know how to code. 00:05:13.800 --> 00:05:16.740 I had been coding since I was like a kid on an old TI calculator. 00:05:16.740 --> 00:05:21.300 And all of the programming mixed with science jobs, those were fascinating. 00:05:21.300 --> 00:05:24.860 So I actually went into graduate school for computer science, scientific computing, and that 00:05:24.860 --> 00:05:26.840 was a really good fit for me. 00:05:27.180 --> 00:05:29.140 I love that I got to touch all sorts of different domains. 00:05:29.140 --> 00:05:33.580 But that was probably the inflection point is looking at job applications after having 00:05:33.580 --> 00:05:35.920 already graduated college and realizing, ah, I think I made a mistake. 00:05:35.920 --> 00:05:37.720 Maybe I should have turned left earlier. 00:05:37.720 --> 00:05:41.740 That's really interesting because a lot of people, they look around and they see folks 00:05:41.740 --> 00:05:46.960 who have successfully created these amazing open source libraries and projects, initiatives, 00:05:46.960 --> 00:05:48.000 you know, like Dask. 00:05:48.000 --> 00:05:51.520 And they're like, the people who get to do that, they've been programming since they were 00:05:51.520 --> 00:05:55.880 five and they got a computer science degree and they just knew they wanted to do that. 00:05:55.940 --> 00:05:59.560 And I think that's not actually as true as often people think it is. 00:05:59.560 --> 00:05:59.780 Right. 00:05:59.780 --> 00:06:01.560 I think none of us got computer science degrees. 00:06:01.560 --> 00:06:02.180 Is that fair? 00:06:02.180 --> 00:06:03.240 Yeah, I didn't. 00:06:03.240 --> 00:06:08.000 I got a computer science book after I decided I didn't want to work in math anymore and started 00:06:08.000 --> 00:06:10.820 like, you know, C++ book and started going on that. 00:06:10.820 --> 00:06:12.100 That's not the same as a degree though. 00:06:12.100 --> 00:06:12.520 Good plan. 00:06:12.520 --> 00:06:16.240 I feel like there's an analog in the data science world as well with respect to, you know, 00:06:16.240 --> 00:06:19.940 all these boot camps that happen now and now masters in data science at prestigious 00:06:19.940 --> 00:06:25.980 universities and relatively few people I know at least have taken the university track. 00:06:25.980 --> 00:06:29.560 I think boot camp to junior data scientist positions far more prevalent. 00:06:29.560 --> 00:06:33.000 But for more on that, you can check out the previous episode that I recorded with Michael. 00:06:33.000 --> 00:06:35.640 Yes, exactly. 00:06:35.640 --> 00:06:36.840 We've got a whole show on that one. 00:06:37.560 --> 00:06:38.580 So, all right. 00:06:38.580 --> 00:06:44.480 Now, the last time we spoke, Hugo, you were at data camp doing stuff with teaching people 00:06:44.480 --> 00:06:46.280 online Python stuff. 00:06:46.280 --> 00:06:46.780 That was awesome. 00:06:46.780 --> 00:06:48.760 And Matthew, yeah, yeah, cool. 00:06:49.140 --> 00:06:54.480 And Matthew, you were just moving to do a little bit of collaborative stuff with NVIDIA, but 00:06:54.480 --> 00:06:58.780 now you both have, you know, teamed up, joined forces to do something else, right? 00:06:58.780 --> 00:06:59.640 What are you doing nowadays? 00:06:59.640 --> 00:07:06.080 So, we're building a company called Coiled, essentially to productize for the enterprise 00:07:06.080 --> 00:07:09.440 a lot of features of the PyData ecosystem and Dask in particular. 00:07:09.440 --> 00:07:12.760 So, the V1 of our product, we're just about to launch. 00:07:12.760 --> 00:07:16.720 So, it's mid-August 2020, we're about to launch, is managing Dask and distributed compute 00:07:16.720 --> 00:07:21.740 in the cloud for you and handling security, conda doc environments, team management, this 00:07:21.740 --> 00:07:22.300 type of stuff. 00:07:22.300 --> 00:07:24.940 And that doesn't really say what we do on a day-to-day basis. 00:07:24.940 --> 00:07:29.960 I think I could list what we don't do on a day-to-day basis more quickly than what we do. 00:07:29.960 --> 00:07:30.240 Yeah, sure. 00:07:30.240 --> 00:07:35.060 But having, you know, incorporated the company in February, it's kind of waking up every day 00:07:35.060 --> 00:07:38.800 and seeing what the most impactful stuff we can do actually is. 00:07:38.800 --> 00:07:39.880 And that's really exciting. 00:07:39.880 --> 00:07:43.000 And it's a great, I'm really excited to work with Matt and the whole Coiled team. 00:07:43.000 --> 00:07:43.820 Matthew, how about you? 00:07:43.820 --> 00:07:44.700 What do you do day to day? 00:07:44.880 --> 00:07:46.720 You've joined up with Hugo on Coiled. 00:07:46.720 --> 00:07:47.100 Yeah. 00:07:47.100 --> 00:07:49.640 So, day-to-day, you know, I still help maintain Dask. 00:07:49.640 --> 00:07:53.040 So, there's a ton of GitHub conversations, you know, Discord, Stack Overflow. 00:07:53.040 --> 00:07:54.580 But now, we're also making a company. 00:07:54.580 --> 00:07:56.840 So, we're, you know, figuring out sales. 00:07:56.840 --> 00:07:57.600 We're figuring out legal. 00:07:57.600 --> 00:08:00.980 There's a ton of sort of nuts and bolts of running a company. 00:08:00.980 --> 00:08:01.820 That is quite difficult. 00:08:01.820 --> 00:08:05.720 As Hugo was saying, listening to what you can do is different from what you can't. 00:08:05.720 --> 00:08:07.120 Yeah, exactly. 00:08:07.120 --> 00:08:16.500 So, you went to school for physics and astronomy, and then you got a master's degree in computer science, and then you did some really interesting data science around Dask. 00:08:16.500 --> 00:08:24.180 I bet none of that taught you about marketing funnels or accounting or taxation law or any of that fun stuff, right? 00:08:24.280 --> 00:08:25.180 You'd actually be surprised. 00:08:25.180 --> 00:08:26.080 Like, running... 00:08:26.080 --> 00:08:26.760 Like, yeah. 00:08:26.760

--> 00:08:29.880 Maintaining an open source project includes lots of those things. 00:08:29.880 --> 00:08:30.220 Okay. 00:08:30.220 --> 00:08:31.980 So, there's actually a lot of, like, we're hiring. 00:08:31.980 --> 00:08:33.520 We do a lot of community management. 00:08:33.520 --> 00:08:37.340 There's a lot of, like, employer-employee kinds of relationships that occur. 00:08:37.340 --> 00:08:37.620 Right. 00:08:37.620 --> 00:08:38.800 People are new to the project. 00:08:38.800 --> 00:08:39.200 Okay. 00:08:39.280 --> 00:08:39.440 Yeah. 00:08:39.440 --> 00:08:42.780 You need to have relationships with large companies like NVIDIA or Google or Microsoft. 00:08:42.780 --> 00:08:47.800 Like, running a sizable open source project is actually not so dissimilar from running a business. 00:08:47.800 --> 00:08:48.160 Yeah. 00:08:48.160 --> 00:08:52.480 Now, taxation laws and legal agreements, there's certainly more of them now. 00:08:52.480 --> 00:08:53.080 Probably so. 00:08:53.080 --> 00:08:57.000 But, yeah, you guys probably wear a lot of hats as you're getting this stuff off the ground. 00:08:57.000 --> 00:08:57.460 Indeed. 00:08:57.460 --> 00:08:58.060 Very much so. 00:08:58.060 --> 00:08:58.140 Yeah. 00:08:58.140 --> 00:09:04.220 And to speak to Matt's point, when we first started working together, he was like, okay, what's the call to action on the landing page? 00:09:04.220 --> 00:09:05.960 And I was like, oh, great. 00:09:05.960 --> 00:09:07.520 Let's have that conversation. 00:09:07.680 --> 00:09:17.040 And his, you know, appreciation of design ethics, sorry, design aesthetics and marketing funnels, I think is actually, I was pleasantly surprised. 00:09:17.040 --> 00:09:17.360 Yeah. 00:09:17.360 --> 00:09:22.600 Well, I mean, so there are probably, you know, 10 to 15 to 20 people who work on desk at different companies. 00:09:22.600 --> 00:09:25.040 You know, many of whom we've, like, tried to get jobs. 00:09:25.040 --> 00:09:27.040 So, you know, we do marketing. 00:09:27.040 --> 00:09:27.760 We do sales. 00:09:27.760 --> 00:09:30.360 We do all of those kinds of things you do inside of companies. 00:09:30.360 --> 00:09:36.740 We just do it without paying anybody directly or without having any actual control over everything. 00:09:37.140 --> 00:09:39.320 Running with responsibility is kind of like running a company without any power. 00:09:39.320 --> 00:09:40.680 It's a lot of soft power. 00:09:40.680 --> 00:09:42.000 Sounds like product management. 00:09:42.000 --> 00:09:43.180 Yeah, I can imagine. 00:09:43.180 --> 00:09:44.340 Sure. 00:09:44.340 --> 00:09:44.820 It does. 00:09:44.820 --> 00:09:45.160 I don't know. 00:09:45.160 --> 00:09:45.820 All right. 00:09:45.820 --> 00:09:46.140 Cool. 00:09:46.140 --> 00:09:50.140 And I guess also just throw out there really quick, Hugo, you did the DataFrame podcast for a while. 00:09:50.140 --> 00:09:52.660 So people may also know you from there as well, right? 00:09:52.660 --> 00:09:53.360 Are you still doing that? 00:09:53.360 --> 00:09:54.480 No, no, I'm not. 00:09:54.480 --> 00:09:56.260 And I miss podcasting a great deal. 00:09:56.340 --> 00:09:58.760 I'm very much enjoying going on podcasts. 00:09:58.760 --> 00:10:04.880 But Matt and I have some secret plans to start podcasting in the nearish future. 00:10:04.880 --> 00:10:07.220 So that may be a bit of a teaser there. 00:10:07.220 --> 00:10:10.180 But yeah, that was a really fun podcast to put out weekly. 00:10:10.180 --> 00:10:13.540 And of course, I came on your show, on this show before doing that. 00:10:13.740 --> 00:10:25.040 I just want to let everyone know that Michael was actually incredibly helpful in getting the podcast up and running and letting me know all the things that he learned along the way that, you know, I still learned a lot along the way. 00:10:25.040 --> 00:10:28.060 But there are several things that Michael helped so wonderfully with. 00:10:28.060 --> 00:10:29.140 So thank you for that, Michael. 00:10:29.340 --> 00:10:30.100 Oh, you're really welcome. 00:10:30.100 --> 00:10:31.800 I was glad to see you having success there. 00:10:31.800 --> 00:10:36.100 And I'm looking forward to whatever secret project you guys have in mind. 00:10:36.100 --> 00:10:36.740 As am I. 00:10:36.740 --> 00:10:37.540 Awesome. 00:10:37.540 --> 00:10:47.900 So we're going to talk about Dask and open source and where maybe some sort of company making this a little more powerful or amplifying the message there might come from. 00:10:47.900 --> 00:10:52.620 But I wanted to start a conversation a little more broad and philosophical. 00:10:52.620 --> 00:10:55.700 So I'm going to talk about two things real quick. 00:10:55.900 --> 00:11:02.480 First of all, we saw in 2017, Stack Overflow published an article called The Incredible Growth of Python. 00:11:02.480 --> 00:11:08.200 And it showed Python being pretty flat for many years, relatively, along with a bunch of other languages. 00:11:08.200 --> 00:11:14.400 And then around 2012, you know, somebody kicked the derivative up or the second derivative. 00:11:14.400 --> 00:11:15.860 And it just took off. 00:11:15.860 --> 00:11:16.400 Right. 00:11:16.400 --> 00:11:17.120 And it went up and up. 00:11:17.120 --> 00:11:18.320 And they said, look at this. 00:11:18.320 --> 00:11:19.780 It's about to pass these other languages. 00:11:19.780 --> 00:11:21.960 And if we predict out, look how crazy it's going to get. 00:11:21.960 --> 00:11:23.860 And, you know, that was three years ago. 00:11:23.920 --> 00:11:28.900 We can look back, you know, I'll actually link to the Stack Overflow trends now. 00:11:28.900 --> 00:11:33.940 And those predictions underestimated how popular Python was going to be. 00:11:33.940 --> 00:11:36.880 But more so, they underestimated the stability of the other languages. 00:11:36.880 --> 00:11:42.680 And they're actually going down more than the flatness that they predicted, which is, I think all those kinds of things is really

interesting. 00:11:42.840 --> 00:11:49.900 My theory is a lot of folks came into Python because of the data science stack around that timeframe. 00:11:49.900 --> 00:12:01.520 And so I guess my question to you all is, you know, how much is this incredible growth of Python, the incredible growth of data science in Python versus more broad stories around there? 00:12:01.520 --> 00:12:07.380 I think there's a lot of, I think the data science or scientific stack inside of Python certainly played a large role in that. 00:12:07.380 --> 00:12:15.540 And, you know, I think that's because we understood, I think that the scientific stack targeted very much scientific users, which were a good proxy for data science users today. 00:12:15.540 --> 00:12:15.800 Yeah. 00:12:15.840 --> 00:12:20.900 And so we had the same combination of performance and accessibility that was necessary to meet that need. 00:12:20.900 --> 00:12:22.600 So we're sort of ahead of the game by accident. 00:12:22.600 --> 00:12:25.540 Those are why a lot of the maintainers have a science background. 00:12:25.540 --> 00:12:25.780 Right. 00:12:25.860 --> 00:12:37.040 What I would say, though, is that Python is really only powerful today because of that union of the data science stack with the web stack, with the visualization stack, with the system operations stack. 00:12:37.040 --> 00:12:46.340 And it's really the fact that we can do all of those things together, which make Python sort of the standard default place to build advanced applications today. 00:12:46.340 --> 00:12:52.360 You know, if we were only MATLAB, for example, and we couldn't do web servers, we would be sort of a fringe language. 00:12:52.360 --> 00:12:53.260 We would be a niche. 00:12:53.740 --> 00:12:58.180 If we were only, you know, Scala and could do only Spark, we would also be sort of a niche. 00:12:58.180 --> 00:12:59.540 Python can do everything. 00:12:59.540 --> 00:13:02.480 And that is where you can build these really rich and awesome applications. 00:13:02.480 --> 00:13:02.860 Yeah. 00:13:02.860 --> 00:13:04.000 I think that's a really good point. 00:13:04.000 --> 00:13:12.660 I mean, people talk about that's why Node.js had the interest that it did, because you could sort of solve the problem in all the places you had to work. 00:13:12.660 --> 00:13:20.580 And I feel like that's kind of the story as well for Python on the scientific computation side and to some degree on the website as well. 00:13:20.580 --> 00:13:22.200 I mean, we all have to write HTML and stuff. 00:13:22.360 --> 00:13:31.360 But, you know, the way I think of it is, you know, if there's a biologist who's coming in and she has to do a little bit of computation, she's like, okay, I just need to do this. 00:13:31.360 --> 00:13:32.560 I need to make this graph. 00:13:32.560 --> 00:13:34.140 This data is too big for whatever. 00:13:34.380 --> 00:13:44.160 If I can write these 10 lines of code, not even in a function, just top to bottom, straight down, write these five lines, 10 lines of code, get this amazing output by using all these libraries. 00:13:44.160 --> 00:13:46.980 Eventually, the idea of like, well, maybe I need to pass different data. 00:13:46.980 --> 00:13:48.120 So maybe I need to write a function. 00:13:48.120 --> 00:13:49.860 And then, you know, you get a little bit farther. 00:13:49.860 --> 00:13:51.320 No, now I'm going to reuse this. 00:13:51.320 --> 00:13:52.520 I'm going to make this package. 00:13:52.760 --> 00:13:53.120 Right. 00:13:53.120 --> 00:13:56.740 You know, two years later, you look back and it's like, how did I become a programmer? 00:13:56.740 --> 00:13:57.760 I thought I was a biologist. 00:13:57.760 --> 00:13:59.980 I never, ever intended to be that. 00:13:59.980 --> 00:14:04.020 So I feel like Python brings a lot of people in through that sort of gravity. 00:14:04.160 --> 00:14:13.820 But because it'll solve the problems that are more advanced and has this rich sort of top end, you don't have to abandon it like you would MATLAB and go learn something else. 00:14:13.820 --> 00:14:14.800 You just get to stay there. 00:14:14.800 --> 00:14:17.600 And I think people are, it's sticky in that regard. 00:14:17.600 --> 00:14:18.140 I agree. 00:14:18.140 --> 00:14:21.420 This is a really common theme in how we think about data science. 00:14:21.420 --> 00:14:22.560 There's this on-ramp, right? 00:14:22.560 --> 00:14:25.080 You know nothing or you know how to use Excel. 00:14:25.080 --> 00:14:26.400 You can use a little bit of pandas. 00:14:26.400 --> 00:14:27.880 You can use a little bit of Dask on one machine. 00:14:27.880 --> 00:14:29.420 You can go out to scale on a cloud. 00:14:29.420 --> 00:14:32.560 You can run on the world's biggest supercomputers, right? 00:14:32.560 --> 00:14:34.480 And it's that smooth experience that we think about. 00:14:34.480 --> 00:14:36.560 We're getting ahead of myself a little bit talking about Dask. 00:14:36.560 --> 00:14:45.340 But that smooth experience of starting from nothing and working up to being amazing is what I think a lot of the Python ethos is about. 00:14:45.340 --> 00:14:46.340 Yeah, I agree. 00:14:46.340 --> 00:14:47.220 I agree with all of that. 00:14:47.220 --> 00:14:53.480 And I do think it's the rise of an entire community and network of tooling in the Python landscape. 00:14:53.480 --> 00:15:01.420 It's also the rise of tireless educators such as the carpentries, data carpentry and software carpentry bringing this stuff around the world. 00:15:01.420 --> 00:15:07.240 I'm very humbled to have played a small role in my work at DataCamp to spread the gospel of Python. 00:15:07.240 --> 00:15:08.780 I've never used that term before. 00:15:08.780 --> 00:15:10.120 We'll see how that lands. 00:15:10.120 --> 00:15:20.640 The other thing I think that I've been really excited about is seeing the wider Python community embrace data science as a fundamental part of the Python landscape now. 00:15:20.900 -->

00:15:27.680 I don't know if anyone's done a data analysis of, let's say, data science talks and keynotes and tutorials at PyCon over the past 10 years. 00:15:27.680 --> 00:15:30.720 But at least anecdotally, we've seen a huge embrace there. 00:15:30.720 --> 00:15:33.820 I mean, it was where we first met in your hometown of Portland, right? 00:15:33.820 --> 00:15:34.200 Oregon. 00:15:34.200 --> 00:15:34.760 Yeah. 00:15:34.760 --> 00:15:35.180 Right. 00:15:35.360 --> 00:15:39.220 Catherine Hough and Jake Vanderplus were two keynotes several years ago. 00:15:39.220 --> 00:15:39.820 And I was like, oh, wow. 00:15:39.820 --> 00:15:42.540 Yeah, they talked about the mosaic and it was beautiful. 00:15:42.540 --> 00:15:45.120 And we're having keynotes from the Python community. 00:15:45.120 --> 00:15:45.600 Exactly. 00:15:45.600 --> 00:15:46.380 Yeah. 00:15:46.380 --> 00:15:47.200 Super interesting. 00:15:47.720 --> 00:15:54.540 So I do think that that's a lot of the magic of Python is that it welcomes people from these different areas. 00:15:54.540 --> 00:15:56.640 And there was a really interesting survey. 00:15:56.640 --> 00:16:08.920 I think it was the JetBrains PSF combined survey that asked something like, how many you are data scientists and what percentage of the Python community do you believe data scientists are? 00:16:09.500 --> 00:16:22.080 And data scientists made up almost like the equal partition of web, which were the two biggest groups, but they thought they were much less represented because they, I think they felt like they were working more individually, but there were just so many of them. 00:16:22.080 --> 00:16:26.700 So it was really interesting that they believed that they were not as big of a part as they actually were. 00:16:26.700 --> 00:16:28.960 But I think that that perception is starting to change. 00:16:28.960 --> 00:16:33.500 I'd be interested in what type of like self-identification bias there is with respect to data science. 00:16:33.560 --> 00:16:40.880 Like I almost get the impression everyone has to identify as a data scientist these days, even like whatever they do, even if they're a gardener. 00:16:40.880 --> 00:16:41.880 It's true. 00:16:41.880 --> 00:16:43.140 It's true. 00:16:43.140 --> 00:16:44.960 Have you, have you touched pandas? 00:16:44.960 --> 00:16:45.460 Then you are. 00:16:45.460 --> 00:16:51.880 This portion of Talk Python To Me is brought to you by Brilliant.org. 00:16:51.880 --> 00:16:59.520 Brilliant has digestible courses in topics from the basics of scientific thinking all the way up to high-end science like quantum computing. 00:17:00.140 --> 00:17:05.340 And while quantum computing may sound complicated, Brilliant makes complex learning uncomplicated and fun. 00:17:05.340 --> 00:17:09.480 It's super easy to get started, and they've got so many science and math courses to choose from. 00:17:09.480 --> 00:17:14.500 I recently used Brilliant to get into rocket science for an upcoming episode, and

it was a blast. 00:17:14.500 --> 00:17:21.880 The interactive courses are presented in a clean and accessible way, and you could go from knowing nothing about a topic to having a deep understanding. 00:17:21.880 --> 00:17:26.440 Put your spare time to good use and hugely improve your critical thinking skills. 00:17:26.440 --> 00:17:29.900 Go to talkpython.fm/brilliant and sign up for free. 00:17:29.900 --> 00:17:34.460 The first 200 people that use that link get 20% off the premium subscription. 00:17:34.460 --> 00:17:39.740 That's talkpython.fm/brilliant, or just click the link in the show notes. 00:17:41.660 --> 00:17:49.480 Another thing that I want to ask you in this sort of philosophical idea is that I mostly do web stuff with Python. 00:17:49.480 --> 00:17:54.560 So I run my online stuff, my online training company, and various things with Python. 00:17:54.560 --> 00:17:56.680 But obviously, I'm interested in all of it. 00:17:57.000 --> 00:18:12.080 When I look at the web world, I feel like once Python 3 really got fully adopted, or at least became the default thing to do, a whole bunch of older ideas were all of a sudden still relevant, but they became, you know, that's neat. 00:18:12.080 --> 00:18:14.020 But let's rethink this now that we can. 00:18:14.100 --> 00:18:24.660 So I'm thinking of things like FastAPI, where they use type annotations to mean stuff, or, you know, things like that, or we're going to build this from the ground up with async and await, and so on. 00:18:24.660 --> 00:18:30.300 So there were just so many different web frameworks and other tools that just came out of nowhere. 00:18:30.440 --> 00:18:32.700 And some of them are not really maintained anymore. 00:18:32.700 --> 00:18:34.220 But all these flowers bloomed. 00:18:34.220 --> 00:18:36.120 And it was just a really interesting thing to see. 00:18:36.120 --> 00:18:39.620 I don't have the same visibility in the data science side for that. 00:18:39.620 --> 00:18:40.740 Did that happen? 00:18:40.740 --> 00:18:41.960 And what was it like? 00:18:41.960 --> 00:18:43.920 I think to a certain extent, it did. 00:18:43.920 --> 00:18:53.580 I think probably larger is the widespread adoption of tools and not necessarily the creation of new tooling and the development of specific tooling as well. 00:18:53.580 --> 00:18:54.700 But there are several tools. 00:18:54.700 --> 00:18:59.360 I think the first one that really is so important is the Jupyter ecosystem. 00:18:59.360 --> 00:19:06.220 I referred to IPython notebooks, and I did that accidentally, but also intentionally, because they were IPython notebooks back then, right? 00:19:06.220 --> 00:19:07.460 That was super cool. 00:19:07.460 --> 00:19:13.260 And I feel like people forget about IPython, but it's so important in the Jupyter ecosystem. 00:19:13.260 --> 00:19:24.260 But yeah, since 3.6, we've seen two years ago, like JupyterLab 1.0, which I think brings the possibility of so many new people to use the entire ecosystem.

00:19:24.260 --> 00:19:32.000 I always think JupyterLab does for Python, I think, what the RStudio IDE does for the R ecosystem. 00:19:32.000 --> 00:19:35.540 It reduces the barrier to entry for so many people. 00:19:35.540 --> 00:19:41.240 But then, as we've noted in other conversations, tools like Streamlit are really cool. 00:19:41.520 --> 00:19:51.800 And I think this is going to be the next evolution of data tooling is bridging the gap between local computation and iterative science in Python and productionized science, right? 00:19:51.800 --> 00:19:52.740 And machine learning. 00:19:52.740 --> 00:19:59.740 So the ability to kind of build a machine learning product quite quickly is wonderful there. 00:19:59.740 --> 00:20:08.680 Yeah, Streamlit was definitely one of the things that came to mind for me is like, this is really a different take on how this works, but it's kind of a neat, modern way to do it. 00:20:08.680 --> 00:20:09.140 Exactly. 00:20:09.140 --> 00:20:12.900 So for me, it wasn't so much the change of Python 3.6. 00:20:12.900 --> 00:20:19.540 I think the scientific stack, the PyData stack actually adopted Python 3 like two or three years before the web stack. 00:20:19.540 --> 00:20:22.060 Yeah, they definitely had a head start there. 00:20:22.060 --> 00:20:22.600 That was great. 00:20:22.600 --> 00:20:25.400 So I think that wasn't as transformational. 00:20:25.400 --> 00:20:27.400 We had a much more smooth transition, I think. 00:20:27.740 --> 00:20:34.140 I think around the same time, though, there was this huge influx of new people and new actors in the system. 00:20:34.140 --> 00:20:49.560 So there's like, you know, as we shifted from being this sort of fringe scientific computing language to becoming the de facto standard language for ML AI workloads, with the option of like TensorFlow and PyTorch, you've got, you know, Google and Facebook acting, you've got Nvidia in there now. 00:20:49.720 --> 00:20:53.000 And you have just, you know, a 10x in the user growth. 00:20:53.000 --> 00:21:00.920 That is really, I think, our like, change moment is the like the TensorFlow engagement of companies for better or for worse. 00:21:00.920 --> 00:21:01.180 Yeah. 00:21:01.460 --> 00:21:11.540 What about enterprises and large corporations seeing the PyData stack as legitimately the thing that they could adopt over, I don't know, saying they have to do with Java or something like that? 00:21:11.540 --> 00:21:13.380 Like, it seems like that's changed as well. 00:21:13.380 --> 00:21:19.660 Yeah, there's a huge acceptance of Python in both in the data science machine learning space, but also in production, right? 00:21:19.740 --> 00:21:24.980 As sort of Docker came up and as Kubernetes came up, like Python became a thing you could easily deploy into PenPom. 00:21:24.980 --> 00:21:27.080 And that's new as of the last decade. 00:21:27.080 --> 00:21:32.640 And with that comes a bunch of money to be made, which again, brings in this like other slew of actors in this space. 00:21:32.640 --> 00:21:32.900 Yeah. 00:21:32.900 --> 00:21:34.280 Both good and bad. 00:21:34.280 --> 00:21:34.480 Yeah. 00:21:34.480 --> 00:21:42.780 I think this is something which started for the most part in tech, where we saw massive, massive wins in open source adoption and then bled out slowly. 00:21:42.980 --> 00:21:49.960 I mean, finances is a great example, like seeing how much places like JPMC and Capital One and these places use Python now. 00:21:49.960 --> 00:21:51.680 But then we're seeing it in retail. 00:21:51.680 --> 00:21:56.880 I mean, one of the great use cases of Dask and Rapids and XGBoost is at Walmart. 00:21:56.880 --> 00:22:04.120 Going back to tech, the fact that Netflix runs like hundreds of thousands of Jupyter notebook batch jobs a day or something like that. 00:22:04.120 --> 00:22:04.860 And they're hiring. 00:22:04.860 --> 00:22:07.620 I mean, they've got a bunch of Jupyter core devs, right? 00:22:07.620 --> 00:22:08.840 Yeah, their whole paper mill thing. 00:22:08.840 --> 00:22:09.180 Yeah. 00:22:09.180 --> 00:22:11.080 Yeah, that's really interesting. 00:22:11.480 --> 00:22:21.920 I guess the early start on Python 3, because a lot of the tooling was improving and there wasn't as much like legacy Python data science. 00:22:21.920 --> 00:22:25.340 I think that was part of the key to being able to just say, you know what? 00:22:25.340 --> 00:22:29.760 We're just going to start something new on these new set of tools, new machine learning library. 00:22:29.760 --> 00:22:33.920 Let's just use Python 3 as opposed to, well, we're still on this web framework. 00:22:33.920 --> 00:22:35.160 We can't change versions. 00:22:35.160 --> 00:22:36.500 And that thing depends on this. 00:22:36.500 --> 00:22:39.120 So we're stuck, you know, seven or eight years in the past. 00:22:39.120 --> 00:22:40.500 So, yeah, pretty cool. 00:22:40.940 --> 00:22:41.980 I mean, it was still a huge effort. 00:22:41.980 --> 00:22:46.320 But I think there's just like there was less production use of the data science stack. 00:22:46.320 --> 00:22:48.600 It seemed like there was production use of the web stack. 00:22:48.600 --> 00:22:49.860 And that's what's really hard to change. 00:22:49.860 --> 00:22:56.380 And I'll also add that data scientists require flexibility and nimbleness and agility in their tooling. 00:22:56.380 --> 00:22:59.720 And it's such the questions they're answering are changing so quickly. 00:22:59.720 --> 00:23:03.300 The way these questions integrate into making business decisions. 00:23:03.300 --> 00:23:07.720 And that interface is changing so quickly that the tooling is kind of moving alongside that. 00:23:07.720 --> 00:23:12.480 And there's this wonderful co-evolution between the tooling and the techniques and the questions that we're required to answer. 00:23:12.480 --> 00:23:13.740 Yeah, that's a good point. 00:23:13.740 --> 00:23:14.200 All right. 00:23:14.200 --> 00:23:19.120 So let's set the

foundation for this project, this company that you all are working on. 00:23:19.120 --> 00:23:21.620 And that would be Dask. 00:23:22.000 --> 00:23:23.200 Matthew, you want to kick us off? 00:23:23.200 --> 00:23:27.800 Tell us, you give us the summary of Dask and all the cool things that it does. 00:23:27.800 --> 00:23:31.240 I mean, we did talk about it at WAGA, but that was over a year and probably not everyone heard it anyway. 00:23:31.240 --> 00:23:31.920 Yeah, sure. 00:23:32.120 --> 00:23:37.400 So Dask is an open source Python library that was designed to parallelize other Python libraries. 00:23:37.400 --> 00:23:40.000 So we first started Dask at Anaconda. 00:23:40.000 --> 00:23:48.000 And the goal was to parallelize out NumPy and Pandas and scikit-learn to sort of like the foundation of the sort of scientific Python Python Python stack. 00:23:48.000 --> 00:23:53.120 What we found pretty quickly is that about half users we were targeting, that's exactly what they wanted. 00:23:53.120 --> 00:23:56.620 They had a big table or a big array of numbers. 00:23:56.620 --> 00:24:01.100 And they wanted a bigger version of those things with the same APIs and the same feel. 00:24:01.260 --> 00:24:02.200 And so Dask gave that to them. 00:24:02.200 --> 00:24:06.140 But about half of our users wanted to do something totally different. 00:24:06.140 --> 00:24:11.080 They wanted to use the internals of Dask to parallelize out some other crazy thing that they were building. 00:24:11.080 --> 00:24:14.440 As we mentioned, in Python, people were building all sorts of new libraries. 00:24:14.440 --> 00:24:18.500 And they wanted to add a little parallelism, a little bit of scalability into their library. 00:24:18.500 --> 00:24:18.760 Yeah. 00:24:18.760 --> 00:24:21.040 And the internals of Dask allowed them to do that. 00:24:21.040 --> 00:24:25.240 So Dask is, at its core, a general purpose library for parallel computing. 00:24:25.240 --> 00:24:30.140 You can think of it in the same way you think of the threading module or the concurrent futures module 00:24:30.400 --> 00:24:32.440 or anything like that in the standard library. 00:24:32.440 --> 00:24:35.760 But it runs at scale and gives you full scalability. 00:24:35.760 --> 00:24:36.000 Right. 00:24:36.000 --> 00:24:39.840 When you say at scale, you mean you could have a cluster of 20 machines. 00:24:39.840 --> 00:24:40.100 Yeah. 00:24:40.100 --> 00:24:40.620 Or 200. 00:24:40.620 --> 00:24:41.040 Yeah. 00:24:41.040 --> 00:24:43.920 Something that looks like a Pandas data frame. 00:24:43.920 --> 00:24:45.700 And you ask it to do computation. 00:24:45.700 --> 00:24:48.080 And that computation happens on all those machines. 00:24:48.080 --> 00:24:48.500 Exactly. 00:24:48.500 --> 00:24:48.860 Yeah. 00:24:48.860 --> 00:24:51.000 And it's really smart about not moving data. 00:24:51.340 --> 00:24:55.300 Because if you've got a lot of data, the movement of it around might actually be the slowest 00:24:55.300 --> 00:24:55.760 part, right? 00:24:55.760 --> 00:24:55.980 Yeah. 00:24:55.980 --> 00:24:57.140 That and like 20 things. 00:24:57.140 --> 00:24:57.360 Yeah. 00:24:57.360 --> 00:25:01.780 So we think very, very deeply about how to run complicated task graphs at scale. 00:25:01.780 --> 00:25:04.660 And that allows for us to do things like big pandas. 00:25:04.660 --> 00:25:08.540 But also, you could use it in the same way you use Celery behind a web application. 00:25:08.540 --> 00:25:12.500 Or you could use it backing a big machine learning application. 00:25:12.980 --> 00:25:18.060 So Dask is a very flexible tool on which many people have built these sort of more special 00:25:18.060 --> 00:25:19.760 purpose scalability tools. 00:25:19.760 --> 00:25:23.520 You can think of it kind of like a tool that you would use to build Spark or the tool that 00:25:23.520 --> 00:25:26.740 you would use to build a parallel airflow, for example. 00:25:26.740 --> 00:25:29.040 I'll just build on this briefly by quoting Matt. 00:25:29.040 --> 00:25:30.420 And I always quote him on this. 00:25:30.420 --> 00:25:32.840 So he's heard me quote him many times on this. 00:25:32.840 --> 00:25:36.740 But this is a blog post he wrote called A Brief History of Dask, which you can find on 00:25:36.740 --> 00:25:39.120 coil.io forward slash blog. 00:25:39.260 --> 00:25:45.400 But he speaks to three goals that the original idea that him and several other people at Anaconda 00:25:45.400 --> 00:25:46.140 were thinking about. 00:25:46.140 --> 00:25:50.220 Two technical goals, which were to harness the power of all the cores on your laptop in 00:25:50.220 --> 00:25:50.540 parallel. 00:25:50.540 --> 00:25:53.160 A second one to support larger than memory computation. 00:25:53.160 --> 00:25:55.020 And these we know about. 00:25:55.020 --> 00:25:57.900 But he also mentions a social goal, which is to invent nothing. 00:25:57.900 --> 00:26:02.840 And I quote, we wanted to be as familiar as possible to what users already knew in the 00:26:02.840 --> 00:26:03.880 Pi data stack. 00:26:03.880 --> 00:26:08.640 And I think as the Pi data stack grew and garnered a lot of adoption, the fact that there was 00:26:08.640 --> 00:26:13.340 a social goal to invent nothing allows pandas and NumPy users then to use it immediately. 00:26:13.340 --> 00:26:18.740 And Matt says, you know, doesn't have an API because it uses the API, the packages that 00:26:18.740 --> 00:26:20.500 it allows people to do distributed compute with. 00:26:20.500 --> 00:26:25.080 Yeah, that's a really good goal because it doesn't matter how amazing the thing you came 00:26:25.080 --> 00:26:25.540 up with. 00:26:25.540 --> 00:26:31.700 If you say, OK, well, you used to talk to databases, but now use our special API that's 00:26:31.700 --> 00:26:34.180 like a database, but it's not exactly because it's better. 00:26:34.180 --> 00:26:37.840 And we have this graphing thing and it's like Altair, but it's not really Altair. 00:26:38.020

--> 00:26:40.060 It's just motivated by it. 00:26:40.060 --> 00:26:42.500 So you can just redo all your stuff and it's going to be better. 00:26:42.500 --> 00:26:44.840 And, you know, a lot of times it's like, you know what? 00:26:44.840 --> 00:26:48.820 The tools that I have really work well and I don't want to reinvent my world. 00:26:48.820 --> 00:26:49.280 Exactly. 00:26:49.280 --> 00:26:49.740 Yeah. 00:26:49.740 --> 00:26:52.280 I call this the principle of minimal creativity. 00:26:52.280 --> 00:26:53.660 I love it. 00:26:53.660 --> 00:26:54.560 It's like productive. 00:26:54.560 --> 00:26:55.820 That's a beautiful one. 00:26:55.980 --> 00:27:01.860 It's similar to the productive, like laziness of developers or something like that. 00:27:01.860 --> 00:27:02.080 Right? 00:27:02.080 --> 00:27:02.420 Sure. 00:27:02.420 --> 00:27:03.000 Yeah. 00:27:03.000 --> 00:27:06.780 This thing that we thought was great, creativity, you know, building a new wonderful thing is 00:27:06.780 --> 00:27:07.400 actually horrible. 00:27:07.400 --> 00:27:11.240 Like you really don't want to impose your thoughts on others. 00:27:11.240 --> 00:27:12.380 Building boring tools. 00:27:12.380 --> 00:27:12.680 Yeah. 00:27:12.680 --> 00:27:12.960 Right. 00:27:12.960 --> 00:27:13.980 Building boring tools. 00:27:13.980 --> 00:27:14.200 Yeah. 00:27:14.200 --> 00:27:14.760 Yeah. 00:27:15.020 --> 00:27:17.040 Otherwise known as getting stuff done, right? 00:27:21.180 --> 00:27:24.700 This portion of Talk Python To Me is sponsored by Monday.com. 00:27:24.700 --> 00:27:30.260 Monday.com is an online platform that powers over 100,000 teams daily work. 00:27:30.460 --> 00:27:35.800 It's an easy to use, flexible and visual teamwork platform beautifully designed to manage any 00:27:35.800 --> 00:27:37.800 team organization or online process. 00:27:37.800 --> 00:27:42.720 Now, for most of us, we missed our chance to build the first apps ever in the mobile app 00:27:42.720 --> 00:27:43.020 stores. 00:27:43.020 --> 00:27:47.420 It was a once in a lifetime opportunity, but it's one that's coming around again. 00:27:47.420 --> 00:27:53.220 Monday.com is launching their marketplace and running a contest for the best new apps featured 00:27:53.220 --> 00:27:54.220 right from the get go. 00:27:54.220 --> 00:27:57.740 Want to be one of the first in the Monday.com apps marketplace? 00:27:57.740 --> 00:27:59.220 Start building today. 00:27:59.220 --> 00:28:06.080 They're even giving away $184,000 in prizes, including three Teslas, 10 MacBooks, and more. 00:28:06.080 --> 00:28:10.220 Build your idea for an app and get in front of hundreds of thousands of users on day one. 00:28:10.220 --> 00:28:15.320 Start building today by visiting monday.com slash Python, or just click the link in your 00:28:15.320 --> 00:28:16.540 podcast player's show notes. 00:28:16.540 --> 00:28:22.700 My understanding is that Dask is used for all these different projects. 00:28:22.700 --> 00:28:27.040 And like sometimes you'll find Dask being used and you're not even really aware that 00:28:27.040 --> 00:28:30.680 Dask is somehow powering maybe it's parallelism or whatever. 00:28:30.680 --> 00:28:36.600 So maybe give us a couple of examples of places you found Dask being used that surprised you or 00:28:36.600 --> 00:28:37.420 you're proud of. 00:28:37.420 --> 00:28:38.060 Yeah, sure. 00:28:38.060 --> 00:28:41.160 I'll maybe talk about maybe Pangio first. 00:28:41.160 --> 00:28:45.040 So I think I mentioned Pangio at the end of our last podcast. 00:28:45.040 --> 00:28:45.640 Yeah. 00:28:45.680 --> 00:28:47.340 So Pangio is amazing. 00:28:47.340 --> 00:28:51.620 Pangio is a collaboration of earth scientists, so like climate scientists, meteorologists, 00:28:51.620 --> 00:28:57.220 oceanographers, and a bunch of open source software developers who teamed up to make a new software 00:28:57.220 --> 00:28:59.340 stack to solve a lot of climate science problems. 00:28:59.340 --> 00:29:04.580 We combined things like JupyterHub, Dask, Kubernetes, and other libraries on top of that, like Xarray, 00:29:04.580 --> 00:29:06.260 intake, all sorts of things. 00:29:06.260 --> 00:29:06.520 Yeah. 00:29:06.520 --> 00:29:11.340 And we just like, we revolutionized the way that that sector computes. 00:29:11.340 --> 00:29:16.500 There were sort of decades old software stacks that we just immediately showed were not nearly 00:29:16.500 --> 00:29:20.740 as powerful as we were able to do together, both in terms of computation and in terms of 00:29:20.740 --> 00:29:21.100 accessibility. 00:29:21.620 --> 00:29:25.500 It's at the point where, you know, if you were an undergraduate student in the Philippines, 00:29:25.500 --> 00:29:31.020 you can go and you can look at cloud-based data sets that are many terabytes large and 00:29:31.020 --> 00:29:34.300 see, you know, what climate change will do with sea level rise, for example. 00:29:34.300 --> 00:29:37.060 And that's something that we were all, that happened in like six or 12 months. 00:29:37.060 --> 00:29:37.540 It was amazing. 00:29:37.540 --> 00:29:40.840 And that's such an important problem that we need to solve. 00:29:40.840 --> 00:29:46.160 And so to know that some of the software you helped create is central to making that happen. 00:29:46.160 --> 00:29:46.820 That's really cool. 00:29:46.820 --> 00:29:48.280 Me and like a thousand other people. 00:29:48.280 --> 00:29:48.800 Yeah, yeah, yeah. 00:29:48.800 --> 00:29:49.180 Of course. 00:29:49.180 --> 00:29:50.260 Absolutely. 00:29:50.260 --> 00:29:50.660 Yeah. 00:29:50.940 --> 00:29:55.160 But it was a great, it was a great example of a bunch of different kinds of people all 00:29:55.160 --> 00:29:55.780 working together. 00:29:55.780 --> 00:29:57.540 It was a bunch of like computer scientists working together. 00:29:57.540 --> 00:29:58.780 We wouldn't have done it correctly. 00:29:58.780 --> 00:30:01.240 Once the oceanographers are working together, it wouldn't have done

it all together. 00:30:01.240 --> 00:30:02.600 It needed to be a collaboration. 00:30:02.600 --> 00:30:03.060 Right. 00:30:03.060 --> 00:30:03.300 Yeah. 00:30:03.300 --> 00:30:05.740 How many oceanographers are like, you know what? 00:30:05.740 --> 00:30:09.960 I'm going to set up a Kubernetes cluster so we can do automatic scale out, but then scale 00:30:09.960 --> 00:30:10.340 back down. 00:30:10.340 --> 00:30:10.780 You'd be surprised. 00:30:10.780 --> 00:30:10.960 You know what? 00:30:10.960 --> 00:30:12.420 No, no, I know. 00:30:12.420 --> 00:30:13.600 You'd be surprised. 00:30:13.600 --> 00:30:17.620 They ended up having to learn those skill sets over time, which is partially why we made Coiled, 00:30:17.620 --> 00:30:20.060 is to stop that from having to be the case. 00:30:20.120 --> 00:30:21.020 They're also sharp folks. 00:30:21.020 --> 00:30:22.980 I mean, people who are in that space are quite sharp. 00:30:22.980 --> 00:30:25.780 Another example might be so Rapids. 00:30:25.780 --> 00:30:28.100 So when we last talked, I had just joined NVIDIA. 00:30:28.100 --> 00:30:33.920 So at the time, NVIDIA was building Rapids data science suite that is GPU accelerated. 00:30:33.920 --> 00:30:38.020 So they have like NumPy, Pandas, CycleLearn equivalents that all run on the GPU. 00:30:38.260 --> 00:30:40.580 It's a little bit like what Dask does for parallelism. 00:30:40.580 --> 00:30:45.060 They're somewhat attempting to do that for the CUDA cores and GPU, right? 00:30:45.060 --> 00:30:45.720 Yeah, exactly. 00:30:45.720 --> 00:30:50.020 So, you know, Dask allowed you to do from Pandas to parallel Pandas. 00:30:50.020 --> 00:30:54.100 Rapids allows you to go from Pandas to GPU accelerated Pandas. 00:30:54.100 --> 00:30:58.940 And then Rapids and Dask together allow you to go to, you know, GPU accelerated Pandas across 00:30:58.940 --> 00:30:59.720 the cluster of machines. 00:31:00.840 --> 00:31:02.360 Yeah, that's a great way to put it together. 00:31:02.360 --> 00:31:02.680 Yeah. 00:31:02.680 --> 00:31:06.580 The really great result we had out there, which is super surprising to me, was they did a 00:31:06.580 --> 00:31:11.740 benchmark, the TPC XPB benchmark, where there's like a bunch of sort of data science and business 00:31:11.740 --> 00:31:12.840 analytics queries. 00:31:12.840 --> 00:31:18.500 And they got something like 40% faster and like seven times cheaper than the next solution, 00:31:18.500 --> 00:31:21.220 which is like Spark or MapReduce on a bunch of Dell machines. 00:31:21.420 --> 00:31:21.580 Yeah. 00:31:21.580 --> 00:31:24.140 And so in some sense, that's like 400, right? 00:31:24.140 --> 00:31:28.280 Because it's, you know, the 40X faster and the cheaper, right? 00:31:28.280 --> 00:31:31.620 Because normally you think I can get a lot faster, but I spend a lot of more money or I can get 00:31:31.620 --> 00:31:32.980 cheaper, but it goes a lot slower. 00:31:32.980 --> 00:31:35.920 But if you get it to expand in both axes, that's

awesome. 00:31:35.920 --> 00:31:41.120 No, to be honest here, it's a more expensive machine, but running for a much shorter amount 00:31:41.120 --> 00:31:41.400 of time. 00:31:41.400 --> 00:31:41.660 Yeah. 00:31:41.660 --> 00:31:45.100 So it's not 400X, it's genuinely 40X faster and only 7X cheaper. 00:31:45.100 --> 00:31:46.440 But yeah, no, it's amazing. 00:31:46.440 --> 00:31:51.560 And I think what I love about that is that there are all of these sort of old guard companies 00:31:51.560 --> 00:31:53.620 building out solutions for this benchmark. 00:31:53.620 --> 00:31:59.040 You've got Dell, HHP, and they're all using like monolith software projects like Spark. 00:31:59.040 --> 00:32:00.800 And this benchmark was different. 00:32:00.800 --> 00:32:06.060 It was NVIDIA and it was people at Coiled and it was people at Blazing SQL, another startup 00:32:06.060 --> 00:32:06.540 company. 00:32:06.540 --> 00:32:11.520 There are a bunch of small startups and a bunch of small software projects that all collaborated 00:32:11.520 --> 00:32:14.840 together to just smash this benchmark, right? 00:32:14.840 --> 00:32:18.340 All the other companies were fighting over sort of 10%, 20% gains. 00:32:18.340 --> 00:32:21.960 And we come in with like a 40X speed improvement. 00:32:21.960 --> 00:32:26.560 And so it just sort of, we just wiped everyone out, which was great. 00:32:26.560 --> 00:32:28.040 Yeah, that's a super cool project. 00:32:28.040 --> 00:32:31.920 And Rapids, I mean, it seems like that's a pretty early stage project and it's probably 00:32:31.920 --> 00:32:32.560 just going to grow. 00:32:32.560 --> 00:32:33.700 Early stage of moving fast. 00:32:33.700 --> 00:32:38.640 I mean, that's an example of a company investing in open source with a team of 50 people. 00:32:38.640 --> 00:32:39.680 It's impressive. 00:32:39.900 --> 00:32:44.000 And then the last one I might mention is Napari, which is kind of like Pangeo in that 00:32:44.000 --> 00:32:48.660 they are using Dask to advance another field, in this case, biomedical imaging. 00:32:48.660 --> 00:32:49.960 But their approach is really different. 00:32:49.960 --> 00:32:54.980 A lot of lab bench scientists, people who are looking at microscopes, don't know how to program. 00:32:54.980 --> 00:32:58.260 And so we can't give them a Jupyter notebook that they can play with. 00:32:58.660 --> 00:33:00.960 Instead, they're used to sort of point and click interfaces. 00:33:00.960 --> 00:33:04.720 And so Napari is an image viewer for large images. 00:33:04.720 --> 00:33:08.100 With a point and click interface, you sort of pan and zoom around these, you know, picture 00:33:08.100 --> 00:33:10.160 of a cell or picture of some sort of cancer thing. 00:33:10.160 --> 00:33:12.840 And what I love about this is that they use other parts of Python. 00:33:12.840 --> 00:33:14.940 They use the whole QT side of Python. 00:33:14.940 --> 00:33:17.100 So that's a QT based application. 00:33:17.100 --> 00:33:19.060 And they're not targeting data scientists. 00:33:19.060 --> 00:33:21.160 They're targeting actual scientists. 00:33:21.160 --> 00:33:26.460 And I think that's like a great space for growth of Python in the future is to build applications. 00:33:26.540 --> 00:33:27.500 Yeah, yeah, yeah. 00:33:27.500 --> 00:33:29.520 Because it's one thing to say we build it for data scientists. 00:33:29.520 --> 00:33:35.120 We're kind of like Python programmers in general, but they have this special visualization data 00:33:35.120 --> 00:33:41.560 handling skill set as opposed to the person who took your x-rays or whatever, right? 00:33:41.560 --> 00:33:42.500 Yeah, exactly. 00:33:42.500 --> 00:33:46.040 And also there's an order of magnitude growth that we can have. 00:33:46.040 --> 00:33:46.280 Yeah. 00:33:46.280 --> 00:33:51.240 And I'll also add to that that I've spoken with several users of Napari recently and the way 00:33:51.240 --> 00:33:55.020 it changes their approach to science and their scientific flow. 00:33:55.020 --> 00:33:56.820 I mean, they can do a bunch of imaging. 00:33:56.820 --> 00:34:00.220 And whereas previously they'd have to wait until the next day to check out the images, 00:34:00.220 --> 00:34:05.020 they can literally go away for 15 to 30 minutes, come back, check out the images, then plan the 00:34:05.020 --> 00:34:05.680 next experiment. 00:34:05.680 --> 00:34:11.100 So it changes their absolute flow state of scientific research and the rapid iterative 00:34:11.100 --> 00:34:11.740 cycle for them. 00:34:11.740 --> 00:34:12.980 Yeah, that's really awesome. 00:34:12.980 --> 00:34:17.800 I mean, it's got to feel great to work on that project and just see, you know, it's making a true 00:34:17.800 --> 00:34:19.840 impact for everyday people. 00:34:19.840 --> 00:34:20.560 Yeah, that's cool. 00:34:20.560 --> 00:34:26.580 The one other thing I'll add, which I think we hinted at before, is that Dask is famously used for 00:34:26.580 --> 00:34:31.940 leveraging clusters and parallelization, but it also does it locally. 00:34:31.940 --> 00:34:36.600 So before you scale out to your cluster or whatever, you can scale up to do out-of-core 00:34:36.600 --> 00:34:38.980 computing on larger data sets yourself. 00:34:39.360 --> 00:34:44.940 So it allows a lot of people to do more locally than they'd otherwise be able to across all 00:34:44.940 --> 00:34:47.740 the types of questions we've been discussing, which is pretty cool. 00:34:47.740 --> 00:34:48.320 Yeah, absolutely. 00:34:48.320 --> 00:34:53.720 Well, you know, the MacBook that I'm sitting here with, the MacBook Pro is a core i9 with 00:34:53.720 --> 00:34:55.660 six hyper-threaded cores. 00:34:55.660 --> 00:34:56.940 So 12, basically. 00:34:57.300 --> 00:35:02.120 And if I go and do Python stuff really hard on it, I get one twelfth the CPU consumption 00:35:02.120 --> 00:35:02.820 of what's available. 00:35:02.820 --> 00:35:06.780 And my gaming machine is, you know, 16 cores, right? 00:35:06.780

--> 00:35:11.440 It's like, if that stuff is just mostly idle in the Python world, unless you find a way to 00:35:11.440 --> 00:35:12.120 take advantage of it. 00:35:12.120 --> 00:35:13.840 And so that actually really surprised me about Dask. 00:35:13.840 --> 00:35:17.120 I thought of Dask as, you know, the way we sort of opened the conversation. 00:35:17.120 --> 00:35:22.720 It's this way to take huge data that has to fit into a cluster and not just onto my little 00:35:22.720 --> 00:35:26.400 laptop and run it distributed in terms of distributed machines. 00:35:26.400 --> 00:35:30.660 But the fact that it also runs and sort of scales up and takes better advantage of your 00:35:30.660 --> 00:35:31.780 local hardware is pretty awesome. 00:35:31.780 --> 00:35:34.460 The median cluster size is one, which is just a laptop. 00:35:34.460 --> 00:35:36.320 And so we optimize for that case. 00:35:36.320 --> 00:35:36.700 Yeah. 00:35:36.700 --> 00:35:37.800 Yeah, that's really cool. 00:35:37.800 --> 00:35:41.800 I was going to add, the other thing to note is the interoperability with a lot of other 00:35:41.800 --> 00:35:42.200 packages. 00:35:42.200 --> 00:35:47.120 So if you're using scikit-learn and you want to parallelize your hyperparameter search or 00:35:47.120 --> 00:35:50.820 whatever it is, you can use the, I think it's the njobs quag or something like that. 00:35:50.820 --> 00:35:52.640 And you can use Dask in the backend there. 00:35:52.640 --> 00:35:54.960 Recently, I was so excited to see Matt. 00:35:54.960 --> 00:35:59.100 We're doing these YouTube live streams weekly of people using Dask and a bunch of other 00:35:59.100 --> 00:35:59.380 stuff. 00:35:59.380 --> 00:36:02.160 So subscribe to our YouTube channel if you're interested. 00:36:02.160 --> 00:36:09.960 But marketing aside, somebody from Grubhub, Alex Egg, used Snorkel for weak supervision, which 00:36:09.960 --> 00:36:15.140 is a clever way to label your data for supervised learning without actually hand labeling it. 00:36:15.520 --> 00:36:20.280 And I saw Matt discover a sub-module of Snorkel called Snorkel Dask. 00:36:20.280 --> 00:36:24.760 So seeing, you know, the whole array, the whole ecosystem start to incorporate Dask is really 00:36:24.760 --> 00:36:25.420 exciting, right? 00:36:25.420 --> 00:36:25.780 Yeah. 00:36:25.780 --> 00:36:26.280 Yeah, that's cool. 00:36:26.280 --> 00:36:27.520 I had no idea they'd integrated it. 00:36:27.520 --> 00:36:29.240 I was like, oh, I just learned something today. 00:36:29.240 --> 00:36:31.860 They're using this stuff. 00:36:31.860 --> 00:36:32.120 Great. 00:36:32.120 --> 00:36:33.620 That's really, really neat. 00:36:33.620 --> 00:36:37.580 And yeah, I'll put a link to your YouTube channel there so people can check it out. 00:36:37.640 --> 00:36:40.660 I think it's really interesting that people are doing stuff on Twitch and programming, 00:36:40.660 --> 00:36:45.820 all these live streaming places that I never associated with programming

for the longest 00:36:45.820 --> 00:36:46.240 time. 00:36:46.240 --> 00:36:50.860 It was always, you know, World of Warcraft or some other sort of gaming thing that was in 00:36:50.860 --> 00:36:51.500 a different world. 00:36:51.500 --> 00:36:53.100 It all starts with gaming, man. 00:36:53.100 --> 00:36:55.280 Gaming's ahead of the curve, right? 00:36:56.780 --> 00:36:59.220 Switching, putting on my for-profit hat just for a second. 00:36:59.220 --> 00:37:04.120 Like now in COVID times, like marketing is completely the Wild West again. 00:37:04.120 --> 00:37:06.660 Like we've got to figure out how to reach our audience. 00:37:06.660 --> 00:37:08.580 I used to go to conferences and just know everybody. 00:37:08.580 --> 00:37:10.000 Now, how do we do that? 00:37:10.000 --> 00:37:12.420 You know, Twitch or live streaming is like a thing to play with. 00:37:12.420 --> 00:37:14.520 Probably not the right thing, but we're going to find out. 00:37:14.520 --> 00:37:19.320 And experimenting and finding out the right way to engage people today is actually a really 00:37:19.320 --> 00:37:22.000 fun and interesting problem for which there are no right answers yet. 00:37:22.000 --> 00:37:22.500 Yeah, yeah. 00:37:22.500 --> 00:37:25.760 It definitely is the Wild West and it's a fun time, right? 00:37:25.760 --> 00:37:29.900 Like you said, it's unknown, but it's really cool to be able to just explore all these 00:37:29.900 --> 00:37:30.460 different ideas. 00:37:30.460 --> 00:37:32.000 Hey, Hugo, marketing idea for you. 00:37:32.000 --> 00:37:37.220 Let's get on all the cool Python podcasts and see if they can cross-market to coiled things. 00:37:37.220 --> 00:37:38.480 I think that's a great idea. 00:37:38.480 --> 00:37:39.660 Let me just note that down. 00:37:39.660 --> 00:37:40.660 Yeah, great. 00:37:40.660 --> 00:37:44.300 Yeah, well, I don't really know which ones you should talk to, but we talk after. 00:37:44.300 --> 00:37:45.100 We embrace them after. 00:37:45.100 --> 00:37:46.700 Who should we talk Python to? 00:37:46.700 --> 00:37:47.780 I know, I know. 00:37:47.780 --> 00:37:52.720 All right, so let's take it over to the business side a little bit. 00:37:52.720 --> 00:37:53.900 We have DASK. 00:37:53.900 --> 00:37:55.820 We talked about some of the things that it does. 00:37:55.820 --> 00:38:00.740 One of the interesting challenges people often run into when they have these successful projects 00:38:00.740 --> 00:38:07.140 is how do I take this really amazing and powerful thing that I built and allow me to keep working 00:38:07.140 --> 00:38:10.380 on it by somehow getting paid to do so, right? 00:38:10.720 --> 00:38:17.540 And I think probably the first step is we've seen a really big adoption of open source software on the enterprise. 00:38:17.540 --> 00:38:20.140 It used to be, oh, what are you going to be doing? 00:38:20.140 --> 00:38:22.680 Are you doing Java and Oracle? 00:38:22.680 --> 00:38:24.360 Or are you doing Microsoft? 00:38:25.140 -->

00:38:27.300 Which type of company are you at, right? 00:38:27.300 --> 00:38:36.300 Like there's a lot of, there's even like Gardner reports talking about how open source software has not just lowered the cost for enterprises, 00:38:36.300 --> 00:38:40.900 but actually increased the quality at the same time, which is like one of these double wins as well. 00:38:41.260 --> 00:38:41.880 How do you guys see it? 00:38:41.880 --> 00:38:50.260 So I'll add one more piece of the puzzle to the question in the sense that not only are we seeing a lot of people in enterprise adopting open source, 00:38:50.260 --> 00:38:55.420 we're seeing, I'm actually going to paraphrase Brian Granger, who said this a couple of years ago at Jupiticon. 00:38:55.420 --> 00:39:04.640 We're seeing a phase transition from a lot of people using open source individually in enterprise to large scale enterprise adoption of open source. 00:39:04.640 --> 00:39:12.920 So having individuals use it within an organization, I think is, is something where the open source provides nearly all the value they, they need there. 00:39:12.920 --> 00:39:18.420 But for an institution to actually adopt OSS at scale across an organization, 00:39:18.420 --> 00:39:23.820 there are other moving parts that need to be in place that I think commercial companies can solve for. 00:39:24.320 --> 00:39:27.280 So having said that, maybe Matt can talk about why we're starting Coiled. 00:39:27.280 --> 00:39:27.660 Yeah. 00:39:27.660 --> 00:39:30.540 I love the Brian phrase, phase transition language. 00:39:30.540 --> 00:39:32.300 That sounds exactly like Brian, who's a physicist. 00:39:32.300 --> 00:39:32.720 Physicist. 00:39:32.720 --> 00:39:34.540 Physicist who then made Jupyter notebooks. 00:39:34.540 --> 00:39:35.180 Yeah. 00:39:35.180 --> 00:39:39.120 The way I think of it is that, sorry, so open source software one, right? 00:39:39.120 --> 00:39:42.080 So like no new company is installing Oracle or SAS today. 00:39:42.080 --> 00:39:43.620 They're installing open source. 00:39:43.620 --> 00:39:49.980 But as we tossed out all of these sort of enterprise software companies and their enterprise software stacks, 00:39:49.980 --> 00:39:54.120 we accidentally tossed out all the things they sold us that weren't software, right? 00:39:54.220 --> 00:39:57.060 So like they would, Oracle would sell you, you know, their database things. 00:39:57.060 --> 00:39:58.560 Like service level agreements. 00:39:58.560 --> 00:39:59.200 Exactly. 00:39:59.200 --> 00:40:02.300 How do, how do we get help if we're stuck and it does not working? 00:40:02.300 --> 00:40:03.500 All that kind of stuff, right? 00:40:03.500 --> 00:40:06.720 The sort of training safety blanket of commercial software. 00:40:06.720 --> 00:40:06.920 Yeah. 00:40:06.920 --> 00:40:12.360 Training, the safety blanket of enterprise support, but also, you know, hooking into enterprise office systems 00:40:12.360 --> 00:40:17.300 or network security or sort of plugging

all of those things together, integrating with other technologies. 00:40:18.080 --> 00:40:22.620 And now that all these companies are adopting this open source stack, all of that stuff is missing. 00:40:22.620 --> 00:40:24.380 And so it is again, the wild west. 00:40:24.380 --> 00:40:28.900 And it's sort of a, it's an awkward place as a company to be adopting this technology. 00:40:28.900 --> 00:40:35.800 I was in a conversation once with folks at NASA and they were saying, hey, look, what hosted notebook solution should we use? 00:40:35.860 --> 00:40:37.760 At the time there was, there was Domino Data Labs. 00:40:37.760 --> 00:40:39.640 There was Anaconda Enterprise was selling something. 00:40:39.640 --> 00:40:41.520 There was AWS SageMaker. 00:40:41.520 --> 00:40:43.780 And I said, hey, have you considered just using JupyterHub? 00:40:43.780 --> 00:40:48.200 It's actually a really pleasant experience and it's free and it's easy to use. 00:40:48.200 --> 00:40:50.260 And they said, no, but like we need to buy something. 00:40:50.260 --> 00:40:52.820 Like we're not going to manage JupyterHub. 00:40:52.820 --> 00:40:55.360 We need to buy something because we're NASA. 00:40:55.360 --> 00:40:58.380 And the open source thought is just, it's free. 00:40:58.380 --> 00:40:59.700 Why don't you just take it? 00:40:59.700 --> 00:41:00.820 It's not so hard. 00:41:00.820 --> 00:41:04.260 But of course for them, they don't want another puppy. 00:41:04.260 --> 00:41:04.880 Right, exactly. 00:41:04.880 --> 00:41:07.380 A thing they got to take care of and walk even in the rain. 00:41:07.380 --> 00:41:07.720 Yeah. 00:41:07.720 --> 00:41:12.120 And so that, I think filling in those gaps and providing all of that sort of supporting infrastructure, 00:41:12.120 --> 00:41:17.720 which is both technological and cultural infrastructure, that I think is a great place where for-profit 00:41:17.720 --> 00:41:23.260 companies can augment the open source software stacks that we have spent so all this time building 00:41:23.260 --> 00:41:26.280 and have really just out-competed the proprietary software stacks. 00:41:26.280 --> 00:41:29.380 But of course, doing those two together requires a lot of nuance. 00:41:29.380 --> 00:41:34.480 I think we haven't yet figured out the right models to be a good actor in the open source 00:41:34.480 --> 00:41:38.660 community that we love and love, that's highly principled, and also build a successful business. 00:41:38.660 --> 00:41:38.940 Yeah. 00:41:38.940 --> 00:41:39.460 And that's fun. 00:41:39.460 --> 00:41:41.220 And that's what we're excited about experimenting with. 00:41:41.220 --> 00:41:43.920 Well, and I think it's about time, right? 00:41:43.920 --> 00:41:50.000 The fact that these companies, for lack of a better word, they've started to feel the pain. 00:41:50.000 --> 00:41:52.660 They've decided they want to take on open source. 00:41:52.660 --> 00:41:57.100 They see the value and they're going to move there, but they're like, but there's still these 00:41:57.100 --> 00:41:59.640 problems or these new problems that we don't know how to solve. 00:41:59.640 --> 00:42:04.060 And so I think there's opportunities for people to come along, knowledgeable in open source stuff 00:42:04.060 --> 00:42:10.900 to help give them the support they need so that they can go to the CTO or whatever, say, 00:42:10.900 --> 00:42:15.340 we can build on this and here's how we're going to get the support that we need. 00:42:15.340 --> 00:42:17.820 Not, you know, Sarah's pretty good with Dask. 00:42:17.820 --> 00:42:19.880 She could probably fork it and fix whatever. 00:42:19.880 --> 00:42:23.020 Well, they've done that for a while and they're into problems. 00:42:23.020 --> 00:42:23.760 Yes, I know. 00:42:23.760 --> 00:42:26.260 To be clear, though, I think that isn't on them. 00:42:26.260 --> 00:42:28.560 You said it's about time as though there's sort of blame. 00:42:28.560 --> 00:42:30.660 But I think that blame is on us. 00:42:30.660 --> 00:42:32.260 It's on the open source community. 00:42:32.260 --> 00:42:35.200 Like we needed to learn how to speak corporation. 00:42:35.200 --> 00:42:36.400 And like we're learning that. 00:42:36.400 --> 00:42:39.720 We're learning how to write legal agreements that they can actually sign. 00:42:39.900 --> 00:42:44.200 We're learning how to give them institutions that they can engage with on a peer-to-peer basis. 00:42:44.200 --> 00:42:44.780 Right. 00:42:44.780 --> 00:42:46.980 You know, Ford can't sign a contract with Dask. 00:42:46.980 --> 00:42:50.560 Dask is not an entity that you can sue if they break your company. 00:42:50.560 --> 00:42:52.080 Coiled, you can sue. 00:42:52.080 --> 00:42:52.880 Like, great. 00:42:52.880 --> 00:42:54.100 They can sign a contract with Coiled. 00:42:54.100 --> 00:42:55.860 But I think that's really on us. 00:42:55.860 --> 00:43:00.940 Our community needs to figure out how to build more of those institutions to engage better with 00:43:00.940 --> 00:43:05.560 companies and eventually pull money out of them to feed back the open source innovation 00:43:05.560 --> 00:43:08.300 that we've all been building really successfully over the last decade. 00:43:08.300 --> 00:43:12.860 I think another part of the value commercial entities like Coil can add is, as Matt mentioned, 00:43:12.860 --> 00:43:18.160 the PyData ecosystem is, for the most part, developed by a huge number of scientists and 00:43:18.160 --> 00:43:20.440 people who are answering scientific questions themselves. 00:43:20.440 --> 00:43:27.020 So the PyData ecosystem is very good at helping scientists and data scientists answer scientific 00:43:27.020 --> 00:43:29.240 research or industry-based questions. 00:43:29.440 --> 00:43:33.900 It doesn't necessarily solve for all the cultural challenges that occur within organizations 00:43:33.900 --> 00:43:35.120 and the enterprise. 00:43:35.120 --> 00:43:36.540 Now, I actually haven't... 00:43:36.540 -->

00:43:40.340 This isn't as well thought out as I hope it to be in the future. 00:43:40.340 --> 00:43:45.120 But these types of cultural concerns, I think, are about making sure everyone across an organization 00:43:45.120 --> 00:43:49.980 is happy, such as IT, checking all the boxes from IT that I mentioned before, like network 00:43:49.980 --> 00:43:54.560 security, auth, usage controls, making sure management is on board with everything that's happened. 00:43:54.560 --> 00:43:59.680 So advanced telemetry for management so that they can enable collaboration and make sure 00:43:59.680 --> 00:44:01.940 they're aware of costs and usage and that type of stuff. 00:44:01.940 --> 00:44:06.460 So not necessarily solving for the nodes of the network of an organization, but solving for 00:44:06.460 --> 00:44:11.860 the edge as well, I think, is something key that entities like Coil can help solve for. 00:44:11.860 --> 00:44:12.520 Yeah, absolutely. 00:44:12.520 --> 00:44:17.660 I definitely think that the donate button is not the right answer, right? 00:44:17.660 --> 00:44:22.600 In the corporate world, making a donation, it doesn't even make sense. 00:44:22.600 --> 00:44:26.280 How are the shareholders gaining value from this donation or whatever? 00:44:26.280 --> 00:44:27.700 It just doesn't even make sense. 00:44:27.700 --> 00:44:30.220 You can get pizza money pretty easily. 00:44:30.220 --> 00:44:30.440 Yeah. 00:44:30.440 --> 00:44:31.600 But going beyond, that's hard. 00:44:31.600 --> 00:44:35.220 But yeah, having a career or a business around it is really tricky. 00:44:35.220 --> 00:44:41.000 So I think you're right that the speaking corporate while keeping the ethos of open source 00:44:41.000 --> 00:44:41.940 is pretty important. 00:44:41.940 --> 00:44:48.000 So it sounds to me like you guys decided, let me guess, November to December time, you 00:44:48.000 --> 00:44:53.960 decided there's this gap and you guys can start a company, which you founded in February, called 00:44:53.960 --> 00:44:59.900 Coil to start addressing some of these problems in the data science, distributed computing world, 00:44:59.900 --> 00:45:01.300 people doing task stuff. 00:45:01.300 --> 00:45:02.180 That's right. 00:45:02.180 --> 00:45:02.480 All right. 00:45:02.480 --> 00:45:03.040 Yeah. 00:45:03.380 --> 00:45:05.360 So tell us, what is Coil? 00:45:05.360 --> 00:45:10.860 What slice of this enterprise challenge or corporate challenge are you taking on? 00:45:10.860 --> 00:45:11.100 Yeah. 00:45:11.100 --> 00:45:16.720 So Coil is a for-profit company based around scaling Python or sort of based around Dask, 00:45:16.720 --> 00:45:19.760 whose mission is to make computing accessible, right? 00:45:19.760 --> 00:45:24.900 So we care very deeply about enabling individuals to scale out comfortably and also extending that 00:45:24.900 --> 00:45:26.180 capabilities into corporations. 00:45:26.180 --> 00:45:31.580 We interacted with hundreds of companies who are using Dask and

all had more or less the 00:45:31.580 --> 00:45:32.120 same needs. 00:45:32.440 --> 00:45:35.580 We mentioned them before, all the needs that you have around adopting open source software, 00:45:35.580 --> 00:45:40.520 training, support, deployment problems, security off IT issues. 00:45:40.520 --> 00:45:44.640 We were more or less begged by enough companies to make this thing that we decided to make it. 00:45:44.640 --> 00:45:44.880 Yeah. 00:45:44.880 --> 00:45:46.320 That's a good place to be. 00:45:46.320 --> 00:45:52.340 Data scientists, as I mentioned before, require a lot of nimbleness and flexibility in the tools 00:45:52.340 --> 00:45:52.820 they use. 00:45:52.820 --> 00:45:57.480 And at the moment, they need to do all types of crazy stuff from, let's say they want to build 00:45:57.480 --> 00:45:58.240 something at scale. 00:45:58.240 --> 00:46:02.340 They need to, as we mentioned before, figure out Kubernetes and Docker containers. 00:46:02.340 --> 00:46:06.640 And then like battle with AWS and all of this stuff. 00:46:06.640 --> 00:46:08.720 So do a whole bunch of DevOps-y stuff, right? 00:46:08.720 --> 00:46:10.360 That they shouldn't be doing. 00:46:10.360 --> 00:46:10.600 Yeah. 00:46:10.600 --> 00:46:14.640 And it probably makes their managers nervous because, oh yeah, all we had to do to make 00:46:14.640 --> 00:46:18.900 this work is we created the Kubernetes cluster and then we put the data into this S3 bucket 00:46:18.900 --> 00:46:20.720 and we were able to just talk to it and it was great. 00:46:20.720 --> 00:46:22.060 Like, wait, wait, wait, wait, wait. 00:46:22.060 --> 00:46:30.500 What's the access level or what's the publicity versus lockdown state of that S3 bucket? 00:46:30.500 --> 00:46:32.760 Are we going to end up on the news next week because of this? 00:46:32.760 --> 00:46:34.040 There's a story around Pangeo. 00:46:34.040 --> 00:46:35.020 We mentioned Pangeo before. 00:46:35.020 --> 00:46:38.400 The sort of Earth Science Collaboration Climate Science Group. 00:46:38.400 --> 00:46:42.200 That was actually a really eye-opening experience for me because I really started to understand 00:46:42.200 --> 00:46:43.420 what enterprise software looked like. 00:46:43.740 --> 00:46:48.180 So what we did, I think we were giving a talk at the American Meteorological Society. 00:46:48.180 --> 00:46:51.460 And over the weekend, a bunch of us hacked this thing together. 00:46:51.460 --> 00:46:54.740 It was JupyterHub on the cloud with Dask enabled. 00:46:54.740 --> 00:46:57.600 And so we computed this big data set. 00:46:57.600 --> 00:47:01.220 And then we showed that everyone in the room could do the exact same computation on the cloud. 00:47:01.220 --> 00:47:03.220 So I did this fantastic thing on stage. 00:47:03.220 --> 00:47:05.240 And then we said, hey, everyone, open up your laptops. 00:47:05.240 --> 00:47:06.800 You can do the same thing too. 00:47:06.800 --> 00:47:08.160 And that blew up. 00:47:08.160 --> 00:47:14.080 I think within about six months, we went from three groups in Columbia University, NCAR, 00:47:14.080 --> 00:47:16.020 and Anaconda to about 50 groups. 00:47:16.020 --> 00:47:18.740 So everybody was excited about this. 00:47:18.740 --> 00:47:21.840 We sort of revolutionized how we do Earth System Science. 00:47:21.840 --> 00:47:23.960 But then that's when the nightmare started. 00:47:23.960 --> 00:47:27.580 Because suddenly we have all these groups trying to use this public access thing. 00:47:27.580 --> 00:47:30.000 We had to figure out like, hey, wait a minute, this is costing us money. 00:47:30.000 --> 00:47:32.160 So we had to implement user access controls. 00:47:32.160 --> 00:47:33.960 Okay, who's whitelist on this thing? 00:47:33.960 --> 00:47:34.600 Who can use it? 00:47:35.000 --> 00:47:39.100 Suddenly, there were different groups in different continents who wanted to use different regions on the cloud. 00:47:39.100 --> 00:47:42.780 Now we have to have not one Kubernetes cluster, but a dozen Kubernetes clusters. 00:47:42.780 --> 00:47:45.460 Everyone wanted their own software to be installed. 00:47:45.460 --> 00:47:49.760 The oceanographers have a different software stack from the satellite imagery people, it turns out. 00:47:49.760 --> 00:47:53.760 And so they were all asking me, hey, can you just pip install this one more package, please? 00:47:53.760 --> 00:47:56.620 And eventually, you can't add all those packages together. 00:47:56.620 --> 00:47:59.260 So now we had to make software environment switchers. 00:47:59.260 --> 00:48:01.000 It turns out that no one had... 00:48:01.000 --> 00:48:04.260 We couldn't figure out how to do AWS credentials. 00:48:04.260 --> 00:48:06.720 So people couldn't store their data on the cloud. 00:48:06.720 --> 00:48:07.280 It was read... 00:48:07.280 --> 00:48:11.040 You could read any data, but you couldn't then transform it and then store it into your own buckets. 00:48:11.040 --> 00:48:14.420 And so figuring out auth and passing that around. 00:48:14.420 --> 00:48:20.040 And then security, it was open for an embarrassing amount of time before Bitcoin folks showed up. 00:48:20.040 --> 00:48:21.500 It was all of those problems. 00:48:21.500 --> 00:48:22.880 Eventually, I stepped back. 00:48:22.880 --> 00:48:27.120 And as you sort of predicted, all the oceanographers started taking over Kubernetes. 00:48:27.120 --> 00:48:28.400 And that was what happened. 00:48:28.580 --> 00:48:41.660 And that experience really showed me personally all of the challenges around doing this kind of science or data science at scale, both on a technical perspective, but also on a social cultural perspective. 00:48:41.660 --> 00:48:44.760 There's a ton more than just algorithms work to do. 00:48:44.760 --> 00:48:46.660 Yeah, it seems so easy just to set it up. 00:48:46.660 --> 00:48:48.980 And then there's just one more thing and one more thing. 00:48:48.980 --> 00:48:52.800 And eventually, yeah, you've got oceanographers doing

Kubernetes, which is fine. 00:48:52.800 --> 00:48:53.380 They can do it. 00:48:53.380 --> 00:48:56.680 But you shouldn't have to understand how your car works just to go get groceries, right? 00:48:56.680 --> 00:48:57.720 Or to go to work, right? 00:48:57.720 --> 00:49:00.720 If that's not your intent, it should just get out of the way. 00:49:00.820 --> 00:49:05.300 I mean, some of them then did become auto mechanics as a result, which is a fine career change. 00:49:05.300 --> 00:49:06.400 Yeah, absolutely. 00:49:06.400 --> 00:49:07.380 If you want it. 00:49:07.380 --> 00:49:09.540 But if you're trying to do something else, you don't want to work on your car. 00:49:09.540 --> 00:49:10.220 You just want to get there. 00:49:10.220 --> 00:49:10.860 Awesome. 00:49:10.860 --> 00:49:19.000 Okay, so tell us exactly what is it that Coiled is doing so we can sort of understand where it fits in trying to solve this enterprise, open source business story. 00:49:19.000 --> 00:49:22.100 If I want to be your customer, what can I do? 00:49:22.100 --> 00:49:23.900 Is it a SaaS service? 00:49:23.900 --> 00:49:26.220 Is it like hosted notebooks? 00:49:26.220 --> 00:49:27.340 What do I do? 00:49:27.340 --> 00:49:29.180 Yeah, so Coiled sells many things. 00:49:29.480 --> 00:49:33.600 Mostly, we focus on the Coiled product, which is a hosted Dask solution. 00:49:33.600 --> 00:49:37.840 So we solve all of the infrastructure problems around hosting Dask. 00:49:37.840 --> 00:49:42.780 So if you want to, if you're on our beta, for example, you could actually, I invited you just half an hour ago, Mike. 00:49:42.780 --> 00:49:48.600 Yeah, and I'm sorry, I didn't get to it quick enough to sign up and play with it, but I wish I had. 00:49:48.600 --> 00:49:51.940 So if you did, you could pip install Coiled wherever you run Python. 00:49:51.940 --> 00:49:55.660 It could be in a Python session, could be in Jupyter Notebook, could be on the cloud. 00:49:55.660 --> 00:49:57.640 And then you would ask for a cluster. 00:49:57.640 --> 00:49:59.400 You'd say Coiled.cluster. 00:49:59.400 --> 00:50:02.800 And that would allocate a bunch of machines for you on the cloud somewhere. 00:50:02.800 --> 00:50:07.940 You would then connect to those machines in a super secure, super credentialized, very nice way. 00:50:07.940 --> 00:50:09.620 And you'd be off to the races. 00:50:09.620 --> 00:50:16.400 So if you start my beta, we sort of try to minimize the time of signing up to running computations on the cloud. 00:50:16.400 --> 00:50:18.620 Currently, the number is around two or three minutes. 00:50:18.620 --> 00:50:20.900 We can get it down to about one minute, I think. 00:50:20.900 --> 00:50:22.760 Just enough time to make a cup of coffee. 00:50:22.920 --> 00:50:24.080 Just enough time to make a cup of coffee. 00:50:24.080 --> 00:50:24.580 Yeah. 00:50:24.580 --> 00:50:25.320 Yeah. 00:50:25.320 --> 00:50:29.840 If you have a Keurig with the pre-made filter thing, if you got to get the coffee

out, maybe it won't be nice. 00:50:29.840 --> 00:50:30.680 Yeah. 00:50:30.680 --> 00:50:33.020 So from your perspective, it just looks like Dask. 00:50:33.020 --> 00:50:34.320 We didn't have a set of Kubernetes. 00:50:34.320 --> 00:50:37.180 But there's a bunch of other stuff around that that we manage. 00:50:37.180 --> 00:50:38.680 So first, manage Dask clusters. 00:50:38.680 --> 00:50:41.040 Second, customizable software environments. 00:50:41.260 --> 00:50:44.900 You want to use your favorite software libraries. 00:50:44.900 --> 00:50:49.480 And we're going to help you manage that between your local environment and also in Docker images on the cloud. 00:50:49.480 --> 00:50:51.180 We also help you share that with your colleagues. 00:50:51.180 --> 00:50:55.180 So often we find a sort of one person who manages software and they share it with other folks. 00:50:55.180 --> 00:50:58.640 And then third, user management, cost management, and telemetry. 00:50:58.640 --> 00:51:01.740 So that coil will help you understand what's going on. 00:51:01.740 --> 00:51:06.800 Or more likely, we'll help your boss understand what you're doing and make sure that you don't break the bank. 00:51:06.800 --> 00:51:07.080 Yeah. 00:51:07.080 --> 00:51:12.340 I've definitely got to some very high cloud computing bills before. 00:51:12.340 --> 00:51:15.040 Although in my defense, I said, this is going to be very expensive. 00:51:15.040 --> 00:51:16.300 We should use this other service. 00:51:16.300 --> 00:51:16.960 No, no. 00:51:16.960 --> 00:51:19.900 We have a contract in agreement with this cloud service. 00:51:19.900 --> 00:51:20.940 So we're going to use it. 00:51:20.940 --> 00:51:22.340 Then I got a message. 00:51:22.340 --> 00:51:25.120 Why did we spend $15,000 on that last month? 00:51:25.120 --> 00:51:25.380 Yeah. 00:51:25.380 --> 00:51:26.520 Because you told me to use it. 00:51:26.520 --> 00:51:28.000 It would have been $500 over there. 00:51:28.000 --> 00:51:30.680 But these surprises are unwelcome. 00:51:30.900 --> 00:51:30.980 Yeah. 00:51:30.980 --> 00:51:33.100 And so just elevating that to you visibly, right? 00:51:33.100 --> 00:51:34.920 There's a page which shows all the running clusters. 00:51:34.920 --> 00:51:37.760 It shows how much they're costing per hour and how much they've cost total. 00:51:37.760 --> 00:51:39.700 You can aggregate those costs across users. 00:51:39.700 --> 00:51:41.220 You can set policies. 00:51:41.220 --> 00:51:45.220 Hey, I mean, by default, if you leave a cluster running, this is a super common mistake. 00:51:45.220 --> 00:51:47.460 We'll just shut it off after 20 minutes for you. 00:51:47.460 --> 00:51:51.740 So there's lots of things that you can do that save you a ton of money if you're doing them correctly. 00:51:51.740 --> 00:51:58.480 So Coil is just our way of running Python at scale in the cloud in a way that's a bit opinionated. 00:51:58.780 --> 00:52:01.540 It is the way that I find to be the best way to run, to run

that. 00:52:01.540 --> 00:52:03.760 And it also allows you to be opinionated as a user. 00:52:03.760 --> 00:52:07.260 You can do it from the comfort of wherever you do your data science, right? 00:52:07.260 --> 00:52:12.820 So if you like doing it in JupyterLab or Jupyter Notebooks or IPython, you can do it all from there. 00:52:12.820 --> 00:52:15.260 So we're trying to meet end users where they are. 00:52:15.500 --> 00:52:18.380 So we will spin up all the AWS stuff for you. 00:52:18.380 --> 00:52:19.720 And you don't need to go there. 00:52:19.720 --> 00:52:33.560 And I think one of the biggest pain points, which Matt hit the nail on the head with, is making sure that you can move seamlessly from local computation to cluster computation, where your environments and your data are all matched up, which is super exciting. 00:52:33.560 --> 00:52:42.140 Maybe I'm doing some work in Jupyter, working with Dask to try to take advantage of the 16 cores that my machine has or whatever. 00:52:42.140 --> 00:52:46.100 And at some point, I decide we're going to give it more data. 00:52:46.100 --> 00:52:47.280 We're going to productionize it. 00:52:47.280 --> 00:52:50.320 And it's going to actually need to be faster or work with more data. 00:52:50.320 --> 00:52:53.900 I can basically switch where I'm talking to. 00:52:53.900 --> 00:52:56.860 I can import coil, say, give me a cluster and go here. 00:52:56.860 --> 00:52:59.160 And that's pretty much what that switch looks like. 00:52:59.160 --> 00:52:59.340 Yeah. 00:52:59.340 --> 00:53:00.460 You can run coil from anywhere. 00:53:00.460 --> 00:53:01.240 You can run Python. 00:53:01.240 --> 00:53:03.560 In that story, you're actually probably not starting with Dask. 00:53:03.560 --> 00:53:06.220 You're probably starting with pandas on a very small sample of data. 00:53:06.220 --> 00:53:09.740 In that same notebook, in that same process, you say, hey, wait, it's time to scale up. 00:53:09.740 --> 00:53:10.640 Import Dask. 00:53:10.640 --> 00:53:11.700 Use Dask locally. 00:53:11.700 --> 00:53:13.280 You know, use your 16 cores. 00:53:13.280 --> 00:53:15.680 Say, wait, actually, this is running a little slow. 00:53:15.680 --> 00:53:16.940 I want to kick it up a notch. 00:53:16.940 --> 00:53:20.000 I want to go to the cloud or go to your local Kubernetes cluster or whatever. 00:53:20.000 --> 00:53:24.400 And then, you know, you import coiled, hit a button, and now you're running your computation 00:53:24.400 --> 00:53:28.340 elsewhere on some remote cluster of honor machines, all in the same user flow. 00:53:28.340 --> 00:53:30.480 You mentioned before, like, hey, are you hosting notebooks? 00:53:30.480 --> 00:53:32.200 And actually, intentionally, we are not. 00:53:32.200 --> 00:53:38.540 We do not want to own the user's environment because your user environment is going to look 00:53:38.540 --> 00:53:40.220 so much different from everybody else's. 00:53:40.220 --> 00:53:41.280 Maybe you don't use Jupyter. 00:53:41.280 --> 00:53:43.140 Maybe you use PyCharm or VS Code. 00:53:43.140 --> 00:53:45.540 Maybe you're actually just like running this in a cron job. 00:53:45.540 --> 00:53:47.200 We don't want to own everything. 00:53:47.200 --> 00:53:53.120 We just want to sprinkle in robust, secure parallelism and scalability into your existing 00:53:53.120 --> 00:53:54.120 workflows, right? 00:53:54.120 --> 00:53:58.060 So this is, again, in sort of the Dask ethos of invent nothing and being minimally creative. 00:53:58.060 --> 00:53:58.860 Minimal invitation. 00:53:58.860 --> 00:53:59.580 Yeah. 00:53:59.580 --> 00:54:02.500 Minimal creativity. 00:54:02.500 --> 00:54:03.020 Yeah. 00:54:03.020 --> 00:54:04.320 The principle of minimal creativity. 00:54:04.320 --> 00:54:05.060 Yeah. 00:54:05.060 --> 00:54:07.900 So you don't have to do very much to switch to using coiled. 00:54:07.900 --> 00:54:09.220 It's a small change. 00:54:09.220 --> 00:54:13.240 So you talk about the different places I might, like, I could be using JupyterLab, which would 00:54:13.240 --> 00:54:13.640 be cool. 00:54:13.760 --> 00:54:16.820 I could be using VS Code or PyCharm, and that would be cool. 00:54:16.820 --> 00:54:21.820 Another place that seems like it might be really interesting to use this is I'm in a fast 00:54:21.820 --> 00:54:26.760 API backend, and I want to answer quick with a lot of data, right? 00:54:26.760 --> 00:54:29.000 It could be, I guess what I'm getting at is like production. 00:54:29.000 --> 00:54:29.800 Yeah, absolutely. 00:54:29.800 --> 00:54:32.200 You can use Coil from anywhere you can use Python. 00:54:32.200 --> 00:54:33.300 What's the story of data? 00:54:33.420 --> 00:54:38.140 So, you know, one of the reasons I might switch away from just running Pandas or Dask would 00:54:38.140 --> 00:54:39.580 be I have a lot of data. 00:54:39.580 --> 00:54:44.100 At the same time, how do I get the data to you guys, to your clusters? 00:54:44.100 --> 00:54:48.020 Because that, like I said, could be the slow thing moving around a cluster. 00:54:48.020 --> 00:54:52.400 If you got to, like, get that out of one data center to another, that could even make it worse. 00:54:52.400 --> 00:54:52.720 Yeah. 00:54:52.720 --> 00:54:57.200 Well, we often find is that if you have, you know, gigabytes of data on your machine, it's 00:54:57.200 --> 00:54:59.940 more likely that you've already downloaded the data from somewhere else. 00:54:59.940 --> 00:55:03.960 You had your data on the cloud or on your local data center, it would have been better 00:55:03.960 --> 00:55:04.880 just not to download it. 00:55:04.880 --> 00:55:09.820 And so Coil really helps you run your computations in that remote space very comfortably. 00:55:09.820 --> 00:55:14.240 So, you know, if you have data on AWS S3, you know, there's no reason to download it, open 00:55:14.240 --> 00:55:16.640 up Coil, and you can just run on the data that's there. 00:55:16.640 --> 00:55:16.940 Right. 00:55:16.940 -->

00:55:21.760 Get an authenticated URL, like a signed URL over to that S3 bucket or something. 00:55:21.760 --> 00:55:23.740 I mean, you already have the URL, right? 00:55:23.740 --> 00:55:23.980 Yeah. 00:55:23.980 --> 00:55:28.940 I mean, what we do is we scrape your local AWS credentials, generate a secure token, pass that 00:55:28.940 --> 00:55:29.800 to all the Coil workers. 00:55:29.800 --> 00:55:32.400 So those Coil workers look like you. 00:55:32.400 --> 00:55:36.780 You then can operate on that data as though you were sitting, again, you have the ergonomic 00:55:36.780 --> 00:55:40.420 experience of being on your laptop, but you have the computational experience of being on 00:55:40.420 --> 00:55:40.840 the cloud. 00:55:40.840 --> 00:55:42.540 And it feels much more proximate. 00:55:42.540 --> 00:55:42.820 Yeah. 00:55:42.820 --> 00:55:43.500 Okay. 00:55:43.500 --> 00:55:44.360 That's really neat. 00:55:44.360 --> 00:55:49.480 Where do these Kubernetes clusters that run my Dask stuff, where do they live? 00:55:49.480 --> 00:55:49.780 Yeah. 00:55:49.780 --> 00:55:51.200 So there's two answers to that. 00:55:51.200 --> 00:55:55.780 One is you might have your own set of machines on-prem, you want to run Coil there. 00:55:56.120 --> 00:55:58.200 This is actually really common today with GPU clusters. 00:55:58.200 --> 00:56:00.760 There's really people buying all the GPU clusters. 00:56:00.760 --> 00:56:01.880 They don't really know how to handle them. 00:56:01.880 --> 00:56:03.040 Coil is a great way for that. 00:56:03.040 --> 00:56:08.380 But if you're on AWS, so we currently support AWS and Kubernetes going to other clouds whenever 00:56:08.380 --> 00:56:09.280 that makes sense for us. 00:56:09.280 --> 00:56:12.260 If you're on AWS, they're running on the cloud. 00:56:12.260 --> 00:56:13.580 We actually don't use Kubernetes. 00:56:13.580 --> 00:56:19.880 We chose to use Elastic Container Service, ECS, which is like a older, slightly simpler system. 00:56:19.880 --> 00:56:23.720 They could be running on our account if you want to use our hosted thing, or we can deploy 00:56:23.720 --> 00:56:24.820 it inside of your AWS account. 00:56:24.820 --> 00:56:25.080 Okay. 00:56:25.080 --> 00:56:25.640 Yeah. 00:56:25.640 --> 00:56:29.660 And I guess if it deploys on your AWS account, you could say, I'd like this to run in the 00:56:29.660 --> 00:56:33.480 Bahrain data center or the Sydney one or wherever. 00:56:33.480 --> 00:56:33.920 Sure. 00:56:33.920 --> 00:56:35.320 I mean, we could also do that anywhere. 00:56:35.320 --> 00:56:38.740 You can run like our public version of Coil. 00:56:38.900 --> 00:56:43.380 So we host a public version, which is designed mostly for individuals to increase data access. 00:56:43.380 --> 00:56:45.000 And that can run on any region. 00:56:45.000 --> 00:56:49.820 So because we're using systems like ECS, we don't have to maintain Kubernetes clusters 00:56:49.820 --> 00:56:50.200 everywhere. 00:56:50.200 --> 00:56:52.820 We can run your code anywhere AWS is running. 00:56:52.820 --> 00:56:55.860 Can you also run against an arbitrary Kubernetes cluster? 00:56:55.860 --> 00:57:00.980 So like if I have set up a Kubernetes cluster at Ludnode, and I give you the connection and 00:57:00.980 --> 00:57:03.260 credentials to the Kubernetes cluster, could I say run it there? 00:57:03.260 --> 00:57:07.860 Yeah, you would want to run Coil very close to that system so that we have the right access 00:57:07.860 --> 00:57:08.200 everywhere. 00:57:08.720 --> 00:57:08.980 Oh, yes. 00:57:08.980 --> 00:57:09.240 Yeah. 00:57:09.240 --> 00:57:09.620 Okay. 00:57:09.620 --> 00:57:10.160 Yeah. 00:57:10.160 --> 00:57:11.200 That sounds quite neat. 00:57:11.200 --> 00:57:17.940 It's fairly close conceptually to an infrastructure as a service sort of model in terms of like 00:57:17.940 --> 00:57:22.280 what I'm buying, what I'm getting, what I'm paying for, but specifically focused on helping 00:57:22.280 --> 00:57:24.400 data scientists not actually care about the infrastructure. 00:57:24.400 --> 00:57:26.340 I'm asking those like, what do I pay for? 00:57:26.340 --> 00:57:26.500 Right. 00:57:26.500 --> 00:57:31.460 Do I pay for like the CPU cycle computation, sort of like a Lambda, like the function call? 00:57:31.460 --> 00:57:33.520 Do I, for the machines that are running? 00:57:33.520 --> 00:57:37.380 That is a super interesting question for which honestly, I don't have a good answer yet. 00:57:37.380 --> 00:57:40.480 We are like talking about this every week right now. 00:57:40.480 --> 00:57:45.340 Right now, you either pay, it's free for individuals currently, we're in beta, or like big companies 00:57:45.340 --> 00:57:48.060 are signing us some check with some other pricing involved. 00:57:48.060 --> 00:57:48.500 Yeah. 00:57:48.500 --> 00:57:51.880 I actually recently wrote a blog post on cloud pricing, which I think is broken in lots of 00:57:51.880 --> 00:57:52.160 ways. 00:57:52.160 --> 00:57:52.460 Yeah. 00:57:52.520 --> 00:57:56.820 The obvious thing to do is to charge you a surcharge on top of whatever your cloud provider 00:57:56.820 --> 00:57:57.320 charges you. 00:57:57.320 --> 00:58:00.440 If Amazon charges you a hundred bucks, we'll charge you a hundred bucks too. 00:58:00.440 --> 00:58:01.860 That's kind of like the Databricks model. 00:58:01.860 --> 00:58:03.880 Databricks charges about a hundred percent markup. 00:58:03.880 --> 00:58:06.640 AWS charges about like a 40 or 50 percent markup. 00:58:06.640 --> 00:58:10.920 I actually hate this model because it totally misaligns incentives. 00:58:11.580 --> 00:58:17.700 So if we do this, then I'm incentivized to make your task workload as inefficient as possible 00:58:17.700 --> 00:58:19.840 and use as many resources as possible. 00:58:19.840 --> 00:58:20.660 You know what? 00:58:20.660 --> 00:58:24.120 Reserved instances and spot instances, we don't need any of that. 00:58:24.120 --> 00:58:30.160 Give me the most

expensive flavor of a large three or whatever the machine name is, right? 00:58:30.160 --> 00:58:33.900 Like that would be the incentive, which is actually the exact opposite of what you would hope. 00:58:33.900 --> 00:58:38.580 It's like if Toyota or Ford were incentivized to make you burn more gas, right? 00:58:38.660 --> 00:58:42.160 It's as though like the tar company has got a kickback from all the oil companies for burning 00:58:42.160 --> 00:58:45.220 gas and you'd never get the Prius in that situation or the Tesla. 00:58:45.220 --> 00:58:47.480 And we want to make the Tesla, right? 00:58:47.480 --> 00:58:52.260 We want to make the super sexy, super ergonomic experience that's highly efficient and saves 00:58:52.260 --> 00:58:52.960 you money in the long run. 00:58:52.960 --> 00:58:54.280 So we're still thinking about that. 00:58:54.280 --> 00:58:58.280 If your listeners have thoughts on the right pricing model, we'd love to hear from you at 00:58:58.280 --> 00:59:00.540 hello at coil.io. 00:59:00.540 --> 00:59:01.140 It is tricky. 00:59:01.140 --> 00:59:01.700 I don't know. 00:59:01.700 --> 00:59:06.880 I mean, I was just, I'm doing this new project where the video players for my online courses, 00:59:07.320 --> 00:59:11.780 I would like, so this is a whole like custom player, just HTML5 and JavaScript, Python. 00:59:11.780 --> 00:59:15.900 And I want when you hover over the little scrubber to like change the time, it to show 00:59:15.900 --> 00:59:17.640 you a preview of each thing. 00:59:17.640 --> 00:59:22.440 And I, okay, well, what I can do is I can re-encode all the videos really small and then 00:59:22.440 --> 00:59:25.600 like put a little hidden player and like just set the time and move that around. 00:59:25.600 --> 00:59:31.420 It all works beautifully, but I've got 200 hours of high def video that I need to turn 00:59:31.420 --> 00:59:31.860 into that. 00:59:32.100 --> 00:59:35.380 And so I literally sat down and said, okay, well, how much is it going to cost me to feed 00:59:35.380 --> 00:59:40.500 that through elastic transcoder at AWS and in bandwidth and get it back and get it. 00:59:40.500 --> 00:59:44.200 So I actually have that as those pictures I could show like, all right, that's going to 00:59:44.200 --> 00:59:45.200 cost about five or $600. 00:59:45.200 --> 00:59:46.140 All right. 00:59:46.140 --> 00:59:50.240 This feature, having the data to back this feature is worth $600 to me. 00:59:50.240 --> 00:59:50.840 I'll do it. 00:59:50.840 --> 00:59:55.980 But I just think this, this thinking about how do I pay for a thing and the trade-offs is 00:59:55.980 --> 00:59:56.840 really interesting. 00:59:57.220 --> 01:00:00.000 Yeah, no, I'm putting on my sales hat for a second too. 01:00:00.000 --> 01:00:03.580 Like that is a very clear win where like you need video. 01:00:03.580 --> 01:00:05.980 It will give you this much money in sales. 01:00:05.980 --> 01:00:06.240 Great. 01:00:06.240 --> 01:00:06.840 It makes sense. 01:00:06.840 --> 01:00:11.540 When you're talking about accelerating data science in a company, like the value that you're 01:00:11.540 --> 01:00:13.180 providing is so far removed. 01:00:13.180 --> 01:00:13.620 Yeah. 01:00:13.620 --> 01:00:17.840 So like, you know, if I was going to sell you an efficient refrigerator, I would say, 01:00:17.840 --> 01:00:21.980 Hey, look, it's going to save you, you know, $200 in electricity bill over the year, 01:00:21.980 --> 01:00:23.380 over 10 years is $2,000. 01:00:23.380 --> 01:00:24.220 Great. 01:00:24.220 --> 01:00:25.280 Pay me a thousand dollars. 01:00:25.280 --> 01:00:26.620 That's a very clear calculus. 01:00:27.000 --> 01:00:27.360 Yeah. 01:00:27.360 --> 01:00:29.380 But when it's like, Oh yeah, I'm going to help you scale Python. 01:00:29.380 --> 01:00:31.660 That's like not quite as clearly valuable. 01:00:31.660 --> 01:00:34.620 We need to get into a longer conversation of like, well, why do you care about Python? 01:00:34.620 --> 01:00:35.820 Why do you care about data science? 01:00:35.820 --> 01:00:37.500 How much value does that bring to you? 01:00:37.500 --> 01:00:39.560 What actually are you trying to solve for? 01:00:39.560 --> 01:00:43.680 Oh, you're trying to increase, you know, retention in your customers. 01:00:43.680 --> 01:00:48.220 Like it's like, you have to get a lot more in depth with your customers to figure out what 01:00:48.220 --> 01:00:52.220 the value proposition is of like more ergonomic scaled Python. 01:00:52.220 --> 01:00:57.040 You get into things like, well, what if we could get your recommender system, the next 01:00:57.040 --> 01:00:59.600 version into production a month sooner? 01:00:59.600 --> 01:01:00.000 Yeah. 01:01:00.000 --> 01:01:01.320 How much money would that get you? 01:01:01.320 --> 01:01:06.240 And what if we could get you answers 100 milliseconds quicker? 01:01:06.240 --> 01:01:09.220 How many fewer bouncing users would you have leaving your site? 01:01:09.300 --> 01:01:13.640 Like these are, they're meaningful, but they're, they're not as clear cut. 01:01:13.640 --> 01:01:14.820 So they are interesting discussions. 01:01:14.820 --> 01:01:15.260 I think. 01:01:15.260 --> 01:01:17.400 I think you've raised several important concerns there. 01:01:17.400 --> 01:01:21.960 And one in particular is like, there are a whole bunch of companies who have hired lots 01:01:21.960 --> 01:01:22.580 of data scientists. 01:01:22.580 --> 01:01:23.880 Cause that's what you do these days. 01:01:23.880 --> 01:01:25.720 And they actually haven't seen the value delivered. 01:01:25.800 --> 01:01:29.960 They've got data scientists who are getting, you know, really serious, serious salaries. 01:01:29.960 --> 01:01:34.840 And the data scientists will recognize the value of code, but the economic buyer in an organization 01:01:34.840 --> 01:01:36.900 will be like, wait a second, we're paying you all this money. 01:01:36.900 --> 01:01:38.620 Like, why are we paying more for

tooling? 01:01:38.620 --> 01:01:40.420 We thought you used all this OSS stuff. 01:01:40.420 --> 01:01:45.040 So there's a slight mismatch there as well, which is something culturally that we're, we're 01:01:45.040 --> 01:01:46.260 working to figure out as well. 01:01:46.380 --> 01:01:52.100 Well, I think just put it on my, like I used to work at big company X or whatever thinking 01:01:52.100 --> 01:01:58.400 hat, you know, from the people who make the decisions, there's also avoids security leaks 01:01:58.400 --> 01:02:00.280 and downtime. 01:02:00.280 --> 01:02:05.160 And, you know, there's, there's not just always the positive selling point or the positive advantages. 01:02:05.160 --> 01:02:11.060 There's the, you know, you were avoiding these three really bad potential outcomes by productizing 01:02:11.060 --> 01:02:13.420 your data science and you can do it quicker. 01:02:13.420 --> 01:02:14.080 Right. 01:02:14.080 --> 01:02:15.660 So these are also valuable. 01:02:15.660 --> 01:02:15.920 Yeah. 01:02:16.040 --> 01:02:19.620 Now the things that we've seen that work really well so far are what you just said. 01:02:19.620 --> 01:02:23.480 I mean, actually the biggest one is Databricks is expensive and they like, they're expensive, 01:02:23.480 --> 01:02:24.180 not in a good way. 01:02:24.180 --> 01:02:26.260 They're expensive because we leave machines on all day. 01:02:26.260 --> 01:02:28.160 If it was a better experience, that won't happen. 01:02:28.160 --> 01:02:29.300 And so we want that. 01:02:29.300 --> 01:02:30.400 We want to reduce our costs. 01:02:30.400 --> 01:02:32.780 It must feel really bad to just waste money, right? 01:02:32.780 --> 01:02:37.000 To literally just leave it running and not actually even because I, it just took a while. 01:02:37.000 --> 01:02:37.260 Right. 01:02:37.260 --> 01:02:37.540 Yeah. 01:02:37.540 --> 01:02:42.180 There's what Hugo was talking about of like a common theme we hear about is like, I've got this 01:02:42.180 --> 01:02:43.000 data science team. 01:02:43.000 --> 01:02:45.980 They use pandas and Python and PyTorch on a single machine. 01:02:45.980 --> 01:02:50.380 I've got like the scalable data engineering team and there's a lot of crosstalk between 01:02:50.380 --> 01:02:50.620 them. 01:02:50.620 --> 01:02:51.620 That's really slow. 01:02:51.620 --> 01:02:56.400 And we want to be able to allow our data science team to go directly to scale without having 01:02:56.400 --> 01:02:57.520 to interface with a different team. 01:02:57.520 --> 01:03:01.580 Because that communication, just that iteration cycle is just killing our performance. 01:03:02.260 --> 01:03:05.620 Another common sort of theme is, oops, we bought a bunch of hardware. 01:03:05.620 --> 01:03:09.660 We bought like, you know, 50 GPU boxes and we have no idea how to use them. 01:03:09.660 --> 01:03:12.540 Well, like there's only so many TensorFlow Keras runs we can do. 01:03:12.540 --> 01:03:15.360 We've seen like the Dask Rapids XGBoost thing. 01:03:15.360 --> 01:03:16.660 We want that. 01:03:16.660 --> 01:03:19.740 But like, we just have like a bunch of hardware sitting in a rack somewhere. 01:03:19.980 --> 01:03:25.620 Can you help us sell a product to manage that rack for us and give us a lot of utilization, 01:03:25.620 --> 01:03:28.640 a lot of value out of our pre-existing purchase? 01:03:28.640 --> 01:03:30.980 And those are the things that really tend to hit home. 01:03:30.980 --> 01:03:36.580 Well, I do hope that you guys have a lot of success because I'm always excited to see some 01:03:36.580 --> 01:03:43.600 interesting company taking a really powerful open source project and just adding value to 01:03:43.600 --> 01:03:45.280 that space and being successful. 01:03:45.860 --> 01:03:47.740 So it seems like you're on a good track. 01:03:47.740 --> 01:03:48.020 Thanks. 01:03:48.020 --> 01:03:48.560 Yeah. 01:03:48.560 --> 01:03:49.860 We've been on the track for a while. 01:03:49.860 --> 01:03:52.980 I mean, Dask is sort of unique among open source projects in the Python space. 01:03:52.980 --> 01:03:55.440 We've always had really good funding. 01:03:55.440 --> 01:03:57.180 Like there are a lot of funded Dask developers. 01:03:57.180 --> 01:04:00.600 And it's because we've always been sort of scrambling a little bit to get money. 01:04:00.600 --> 01:04:02.540 And this is maybe just the next step in that process. 01:04:02.540 --> 01:04:02.840 Yeah. 01:04:02.840 --> 01:04:03.320 Cool. 01:04:03.320 --> 01:04:07.180 We're going a little long in time, but I did want to just ask you really quickly some meta 01:04:07.180 --> 01:04:07.720 questions. 01:04:07.720 --> 01:04:11.620 So are you guys using Coiled to build Coiled? 01:04:11.620 --> 01:04:15.720 Is there like, what's the Python inside story, the Coiled Dask inside 01:04:15.720 --> 01:04:16.120 story? 01:04:16.120 --> 01:04:18.400 Right now we're not. 01:04:18.400 --> 01:04:19.640 This actually did come up recently. 01:04:19.640 --> 01:04:23.840 We actually find ourselves building a lot of conda environments for our users. 01:04:23.840 --> 01:04:26.840 And that is actually something that we've got, you know, web backend. 01:04:26.840 --> 01:04:28.220 We've got the scalability need. 01:04:28.220 --> 01:04:31.100 Dask is like an obvious way to scale that out. 01:04:31.100 --> 01:04:32.520 Currently we're using a single machine for that. 01:04:32.520 --> 01:04:33.240 Our needs are that big. 01:04:33.240 --> 01:04:36.440 But yeah, that would be the next step as a build farm. 01:04:36.440 --> 01:04:37.560 And you guys are using Django. 01:04:37.560 --> 01:04:38.220 Is that right? 01:04:38.220 --> 01:04:39.800 Sort of the API layer? 01:04:39.800 --> 01:04:40.240 Yeah. 01:04:40.240 --> 01:04:44.500 So Coiled actually looks kind of like a normal vanilla Python web application. 01:04:45.280 --> 01:04:49.240 It's, you know, Django and Postgres and, you know, Amazon ECS, as we mentioned. 01:04:49.240 --> 01:04:50.140 And it's been

great. 01:04:50.140 --> 01:04:51.800 It's actually, it's my first time using Django. 01:04:51.800 --> 01:04:53.160 I'm like a decade or two late. 01:04:53.160 --> 01:04:57.800 But it's been fantastic to seeing how much is ready for us out of the box. 01:04:57.800 --> 01:04:59.120 So yeah, couldn't be happier. 01:04:59.120 --> 01:04:59.860 Yeah, that's great. 01:05:00.160 --> 01:05:04.000 And it sounds like there's a non-trivial team size. 01:05:04.000 --> 01:05:07.320 I know a lot of people are saying I'm possibly looking for remote work. 01:05:07.320 --> 01:05:09.000 Is that a thing that you guys are doing? 01:05:09.000 --> 01:05:10.040 Are you looking for more people? 01:05:10.040 --> 01:05:11.860 Are you kind of steady state until you get a little farther? 01:05:11.860 --> 01:05:13.400 What's the status there? 01:05:13.400 --> 01:05:13.680 Yeah. 01:05:13.680 --> 01:05:15.220 So Coiled is a fully remote company. 01:05:15.220 --> 01:05:18.020 We're actually born in the era of COVID, which is fun. 01:05:18.020 --> 01:05:19.380 We've always been remote. 01:05:19.380 --> 01:05:20.180 We've always worked remotely. 01:05:20.180 --> 01:05:20.740 Yeah. 01:05:20.740 --> 01:05:25.400 We're maybe like five or six full-time and two or three part-time right now. 01:05:25.400 --> 01:05:28.320 Right now, I think I actually kind of like staying small. 01:05:28.320 --> 01:05:31.940 We raised a bunch of money, but we're sort of not in a hurry to burn it. 01:05:31.940 --> 01:05:34.980 By the time this episode airs, yeah, I wouldn't be surprised if we're looking for more folks. 01:05:34.980 --> 01:05:40.560 If you go to coil.io slash jobs, I think, or just coil.io, we'll have active job listings. 01:05:40.560 --> 01:05:41.160 Yeah. 01:05:41.160 --> 01:05:45.580 And that's both data science folks and web folks and DevOps people. 01:05:45.580 --> 01:05:47.420 It's a nice group of needs. 01:05:47.420 --> 01:05:47.820 Excellent. 01:05:47.820 --> 01:05:49.700 Well, like I said, really interesting project. 01:05:49.700 --> 01:05:51.940 And I hope you guys make a lot of progress with it. 01:05:51.940 --> 01:05:54.540 Now, before you get out of here, let me ask you the final two questions. 01:05:54.540 --> 01:05:57.700 If you're going to write some Python code, what editor do you use? 01:05:57.700 --> 01:05:58.500 Do you want to go first? 01:05:58.500 --> 01:06:00.120 I use JupyterLab these days. 01:06:00.120 --> 01:06:00.600 Right on. 01:06:00.600 --> 01:06:01.100 Matthew? 01:06:01.100 --> 01:06:04.460 So if I write Python code, I'm actually going to give me a cheeky answer. 01:06:04.460 --> 01:06:05.300 I'll say GitHub. 01:06:05.300 --> 01:06:08.700 So most of the Python code that I write today is indirect. 01:06:08.700 --> 01:06:11.780 I mostly try to nerd snipe other people into writing code. 01:06:11.780 --> 01:06:14.220 So maybe this is like the community maintainer's answer. 01:06:14.220 --> 01:06:14.620 Yeah. 01:06:14.620 --> 01:06:17.980 GitHub and I'll say, I'll

put in a plug for whereby as well, 01:06:18.160 --> 01:06:20.560 which is my favorite video conferencing tool. 01:06:20.560 --> 01:06:21.020 Oh, super. 01:06:21.020 --> 01:06:21.380 Okay. 01:06:21.380 --> 01:06:23.720 And notable PyPI package. 01:06:23.720 --> 01:06:27.640 We've already got a pip install desk, pip install coiled. 01:06:27.640 --> 01:06:28.180 Right? 01:06:28.180 --> 01:06:28.800 Those are good ones. 01:06:28.800 --> 01:06:30.820 But, you know, something like, oh, this is really cool. 01:06:30.820 --> 01:06:32.380 It's not super well known. 01:06:32.380 --> 01:06:34.740 Maybe people should check out X. 01:06:34.880 --> 01:06:36.600 I'll point to Napari, which I mentioned earlier. 01:06:36.600 --> 01:06:36.940 Yeah. 01:06:36.940 --> 01:06:37.840 The search image viewer. 01:06:37.840 --> 01:06:38.320 It's really cool. 01:06:38.320 --> 01:06:41.600 It's actually fun seeing a rich client application written in Python. 01:06:41.600 --> 01:06:43.020 Didn't know that was cool anymore. 01:06:43.020 --> 01:06:47.540 I'll point to Snorkel, which I mentioned earlier, which is for weak supervision. 01:06:47.540 --> 01:06:49.120 And it helps you build training data. 01:06:49.380 --> 01:06:53.200 I think training data is one of the biggest challenges we have in the ML space. 01:06:53.200 --> 01:06:58.900 Like I think scale AI raised a hundred million dollars last year or something like that. 01:06:58.900 --> 01:07:01.040 And they still hand label data for people. 01:07:01.040 --> 01:07:05.160 Like they're called scale AI, but they hand label data. 01:07:05.160 --> 01:07:06.080 ML Turk. 01:07:06.080 --> 01:07:06.620 Yeah. 01:07:06.620 --> 01:07:08.980 Mechanical Turk is still like a huge thing. 01:07:08.980 --> 01:07:09.200 Right? 01:07:09.260 --> 01:07:13.140 So any ways we can figure out to do this programmatically is really cool. 01:07:13.140 --> 01:07:19.980 And the Snorkel API allows domain experts to come in and write using basic functions and decorators, 01:07:19.980 --> 01:07:22.800 heuristics for labeling data, which is super cool. 01:07:22.800 --> 01:07:23.720 That's what I'll shout out today. 01:07:23.720 --> 01:07:24.080 All right. 01:07:24.080 --> 01:07:24.500 Super cool. 01:07:24.500 --> 01:07:26.260 I guess I'll throw in as well. 01:07:26.260 --> 01:07:27.420 Missing no. 01:07:27.420 --> 01:07:33.860 Missing N-O, which is a quick visualization for missing data and like pandas and stuff like that. 01:07:33.860 --> 01:07:34.960 Ask what's missing. 01:07:34.960 --> 01:07:36.160 You'll get a cool little graph. 01:07:36.280 --> 01:07:40.500 Like usually if the state is missing, also, I don't know, phone is missing. 01:07:40.500 --> 01:07:41.400 Like all sorts of cool stuff. 01:07:41.400 --> 01:07:42.520 So people can check that out as well. 01:07:42.520 --> 01:07:43.180 All right. 01:07:43.180 --> 01:07:44.620 So final call to action. 01:07:44.620 --> 01:07:46.160 People are interested in Dask. 01:07:46.160 --> 01:07:47.300 They're interested in Coiled. 01:07:47.300 --> 01:07:49.760 They want to parallelize some things. 01:07:49.760 --> 01:07:50.640 What do you guys tell them? 01:07:50.640 --> 01:07:51.660 I would love. 01:07:51.660 --> 01:07:53.780 So I've got a call to action for Coiled. 01:07:53.780 --> 01:07:55.360 Matt may have one for Dask. 01:07:55.360 --> 01:07:58.640 All the features we've just described of Coiled in rapid development. 01:07:58.640 --> 01:08:03.540 And we'd love to involve anybody who likes to break things at scale in this development cycle. 01:08:03.540 --> 01:08:06.020 So go to Coiled.io, sign up for our beta. 01:08:06.020 --> 01:08:10.220 And take our product for a test drive, crash it, and let's figure out how to build it together. 01:08:10.220 --> 01:08:12.620 So I'd love to involve you in that conversation. 01:08:12.620 --> 01:08:12.920 Excellent. 01:08:12.920 --> 01:08:13.360 Matthew? 01:08:13.360 --> 01:08:14.520 I'll have you go with that one. 01:08:14.520 --> 01:08:15.320 I'm actually going to... 01:08:15.320 --> 01:08:16.780 Sorry, I'm going to go out of thinking this. 01:08:16.780 --> 01:08:17.780 Mention one thing on that. 01:08:17.780 --> 01:08:21.160 It is super fun to run a beta. 01:08:21.160 --> 01:08:25.340 As an open source software developer, I'm used to never seeing how people use my software. 01:08:26.020 --> 01:08:31.620 And running a beta on a public service, there's a bunch of creepy spyware I can put on our website, 01:08:31.620 --> 01:08:35.080 which tells me exactly what you've done, which we'll take off at some point. 01:08:35.080 --> 01:08:38.880 It is amazing getting this kind of immediate feedback. 01:08:38.880 --> 01:08:39.120 Yeah. 01:08:39.120 --> 01:08:41.000 A different level of visibility, right? 01:08:41.000 --> 01:08:44.140 Into what people use and telemetry and whatnot, right? 01:08:44.220 --> 01:08:45.660 So yeah, I'll just echo Hugo. 01:08:45.660 --> 01:08:47.160 Go plan with a beta. 01:08:47.160 --> 01:08:50.460 It's super fun, mostly for us, but also for you. 01:08:50.460 --> 01:08:52.720 And yeah, you can play with that on the cloud. 01:08:52.720 --> 01:08:53.420 It's cool. 01:08:53.420 --> 01:08:55.060 Is there a free thing I can do? 01:08:55.060 --> 01:08:56.080 Everything is free right now. 01:08:56.080 --> 01:08:56.320 Yeah. 01:08:56.320 --> 01:08:56.600 Yeah. 01:08:56.600 --> 01:08:59.740 We'll limit how much you can spend, because that's the features that we care about. 01:08:59.740 --> 01:09:03.680 But yeah, I think we just limit you to 100 cores at a time. 01:09:03.680 --> 01:09:05.980 So you can't go totally wild, but you can do lots of fun things. 01:09:05.980 --> 01:09:07.200 Just 100 cores, huh? 01:09:07.200 --> 01:09:07.720 That's awesome. 01:09:07.720 --> 01:09:08.860 Just 100 cores. 01:09:09.340 --> 01:09:10.280 Yeah, that's really cool. 01:09:10.280 --> 01:09:10.860 All right. 01:09:10.860 --> 01:09:12.980 Well, you guys, thanks again for being on the show. 01:09:12.980 --> 01:09:14.180 Good luck with Coiled. 01:09:14.180 --> 01:09:18.180 It's a nice, natural extension of Dask, I think. 01:09:18.180 --> 01:09:19.800 So excited to see you doing it. 01:09:19.800 --> 01:09:20.000 Yeah. 01:09:20.000 --> 01:09:20.500 Thanks, Michael. 01:09:20.500 --> 01:09:21.560 Go start for the beta. 01:09:21.560 --> 01:09:23.640 Thanks so much for having us on the show, Michael. 01:09:23.640 --> 01:09:24.520 It's always fun. 01:09:24.520 --> 01:09:25.000 Yeah, you bet. 01:09:25.000 --> 01:09:25.600 Bye, guys. 01:09:25.600 --> 01:09:29.540 This has been another episode of Talk Python To Me. 01:09:29.540 --> 01:09:33.540 Our guests on this episode were Matthew Rockland and Hugo Bowne Anderson. 01:09:33.540 --> 01:09:37.100 And it's been brought to you by Brilliant.org and Monday.com. 01:09:38.360 --> 01:09:41.980 Brilliant.org encourages you to level up your analytical skills and knowledge. 01:09:41.980 --> 01:09:48.020 Visit talkpython.fm/brilliant and get Brilliant Premium to learn something new every day. 01:09:48.020 --> 01:09:52.880 Build your idea for an app and get it in front of hundreds of thousands of users on day one. 01:09:52.880 --> 01:09:58.860 Start building today at the Monday.com marketplace by visiting monday.com slash python. 01:09:58.860 --> 01:10:00.900 Want to level up your Python? 01:10:00.900 --> 01:10:05.760 If you're just getting started, try my Python Jumpstart by Building 10 Apps course. 01:10:06.060 --> 01:10:13.900 Or if you're looking for something more advanced, check out our new async course that digs into all the different types of async programming you can do in Python. 01:10:13.900 --> 01:10:18.580 And of course, if you're interested in more than one of these, be sure to check out our Everything Bundle. 01:10:18.580 --> 01:10:20.460 It's like a subscription that never expires. 01:10:20.460 --> 01:10:22.620 Be sure to subscribe to the show. 01:10:22.620 --> 01:10:25.120 Open your favorite podcatcher and search for Python. 01:10:25.120 --> 01:10:26.260 We should be right at the top. 01:10:26.260 --> 01:10:35.240 You can also find the iTunes feed at /itunes, the Google Play feed at /play, and the direct RSS feed at /rss on talkpython.fm. 01:10:35.240 --> 01:10:37.320 This is your host, Michael Kennedy. 01:10:37.320 --> 01:10:38.820 Thanks so much for listening. 01:10:38.820 --> 01:10:39.860 I really appreciate it. 01:10:39.860 --> 01:10:41.620 Now get out there and write some Python code. 01:10:41.620 --> 01:11:02.460 I really appreciate it. 01:11:02.460 --> 01:11:32.440 Thank you.