UDSKRIFT 🔗
WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 7/6/2024 4:52:57 PM
Duration: 3127.446
Channels: 1
1
00:00:13.344 --> 00:00:26.610
Hello, and welcome to podcast dot in it, the podcast about Python and the people who make it great. When you're ready to launch your next app or want to try out a project you hear about on the show, you'll need somewhere to deploy it. So take a look at our friends over at Linode. With the launch of their managed Kubernetes
2
00:00:27.035 --> 00:00:31.305
platform, it's easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers,
3
00:00:35.760 --> 00:00:36.820
40 gigabit networking,
4
00:00:37.200 --> 00:00:42.580
dedicated CPU and GPU instances, and worldwide data centers. Go to python podcast.com
5
00:00:43.285 --> 00:01:00.060
/linode today. That's l I n o d e, and get a $60 credit to try out our Kubernetes cluster of your own. And don't forget to thank them for their continued support of this show. You listen to this show to learn and stay up to date with the ways that Python is being used, including the latest in machine learning and data analysis.
6
00:01:00.585 --> 00:01:10.200
For more opportunities to stay up to date, gain new skills, and learn from your peers, there are a growing number of virtual events that you can attend from the comfort and safety of your own home. Go to python podcast.com/conferences
7
00:01:11.860 --> 00:01:35.525
to check out the upcoming events being offered by our partners and get registered today. Your host as usual is Tobias Macy. And and today I'm interviewing Matthew Rocklin and Hugo Bone Anderson about their work building a business around the Dask ecosystem at Coiled. So, Matt, can you start by introducing yourself? Hey, Tobias. Yeah. My name is Matthew Rocklin. I've been a long time open source maintainer in sort of the Py data space. So the NumPy, Pandas, Scikit, or Jupyter space.
8
00:01:36.085 --> 00:01:41.385
Mostly I think about scalable computing, so paralyzing that ecosystem, mostly with the library Dask.
9
00:01:41.765 --> 00:01:50.530
I've been sort of 1 of the maintainers of Dask for the last 5 or 6 years. And you're also CEO of Coiled. And I'm also CEO of Coiled. That's right.
10
00:01:51.390 --> 00:01:54.405
And Hugo, how about yourself? Well, that's a wonderful segue
11
00:01:54.865 --> 00:02:04.180
into into me. I'm, I run, data science evangelism and marketing, wearing a few other hats at at COIL, which we've we've we've just founded. My background
12
00:02:04.560 --> 00:02:05.300
is in,
13
00:02:05.840 --> 00:02:06.340
math,
14
00:02:06.880 --> 00:02:08.660
research science in cell biology,
15
00:02:09.040 --> 00:02:11.415
and data science. And I've done a lot of work in,
16
00:02:12.375 --> 00:02:15.114
data science education. I I recently
17
00:02:15.495 --> 00:02:18.474
joined Code from DataCamp where I I built out,
18
00:02:18.960 --> 00:02:19.360
I suppose,
19
00:02:20.080 --> 00:02:20.580
foundational,
20
00:02:22.080 --> 00:02:23.380
high data ecosystem
21
00:02:24.240 --> 00:02:29.675
educational material. And that's actually how Matt and I met around 4 years ago. We're collaborating with with with Anaconda
22
00:02:30.055 --> 00:02:36.955
on a lot of, online educational content for the PI Data Stack. Podcast listeners will send a Hugo from his previous podcast, Dataframed,
23
00:02:37.300 --> 00:02:38.360
where he was the host.
24
00:02:38.819 --> 00:02:45.540
How is it like being on this side of the microphone, Hugo? I really I really like it. I I like both sides, but I,
25
00:02:46.305 --> 00:02:55.280
I like also finding a new audience and a new listenership and getting new feedback on the types of things we work on and, interested in. I gotta say, I do like asking the questions
26
00:02:55.659 --> 00:02:57.040
though, as well. So forgive me if that happens
27
00:02:57.500 --> 00:03:10.735
at at some point, if my curiosity gets the better of me. Yeah. Absolutely. As somebody who's been on both sides of the mic myself, I can definitely relate to. It's very different been on both sides of the mic myself, I can definitely relate to. It's very different experience being the person being asked the questions than being the person who's driving the conversation.
28
00:03:11.035 --> 00:03:11.535
Absolutely.
29
00:03:12.190 --> 00:03:24.605
And so, Matt, do you remember how you first got introduced to Python? I think so. I've actually been on your show a couple of times, so I'm going to skip that question and instead answer how the first time I contributed a patch to Python. Yeah, that sounds great. Yeah, that was I was,
30
00:03:25.325 --> 00:03:38.370
I was playing around with SymPy a little bit. It's a sort of Mathematica clone inside of Python. And I, was a Google Summer of Code project. So Google Summer of Code is a great, great program. They give, you know, small stipends or they give stipends to people to work on open source projects.
31
00:03:38.805 --> 00:03:58.795
And I did that. It was a great time and that they taught me how to sort of live in the open source world and that, I stayed ever since. And Hugo, do you remember how you first got introduced to Python? Oh, I definitely do. So I did my grad graduate work in pure math, and I've done a bunch of applied math, and I started my first post doc or my only post doc. It it it moved. But,
32
00:03:59.235 --> 00:04:08.380
in Germany, in a cell biology lab, ostensibly to do applied math and mathematical modeling, became clear that a lot of my colleagues were dealing with very large,
33
00:04:09.020 --> 00:04:09.520
datasets.
34
00:04:10.060 --> 00:04:16.885
And I've done a bunch of programming bits and pieces around in in the past, but I needed to, you know, ingest, analyze,
35
00:04:17.425 --> 00:04:31.510
and reproduce scientific results on really large datasets that my biological collaborators were producing.
Load full transcript
And then I started self teaching Python. And it was actually I I learned R and Python and worked in both at that time, but it was the advent
36
00:04:32.315 --> 00:05:17.160
of what were then item, iPython Notebooks that really helped me so much along the way and helped me to educate, you know, in the end, hundreds of other people in in workshops at this, at the Max Planck Institute as well. And so as we've already mentioned, we're here to talk about your work on the DASK project and the business that you're building on top of it. So for people who aren't familiar with it, we did do an interview on the data engineering podcast that went fairly in-depth about Dask itself. But for anybody who hasn't listened to that or who wasn't familiar with the project, can you just give a quick overview about what Dask is and some of the motivations that you had for building it? Sure. Yeah. So Dask is an open source Python library that comes out of the PyData space. It was designed to paralyze other libraries, like NumPy, Pandas, scikit learn,
37
00:05:17.780 --> 00:05:56.074
but it very quickly became far more general. And so today it's used by dozens of other Py PyData libraries for a variety of different fields, doing everything from, you know, advanced machine learning with PyTorch or XGBoost to, you know, GPU accelerated work with RAPIDS to, you know, backing Django websites in the same way you might use Celery or, you know, backing Airflow or Prefect. So DAOs is sort of now like a general purpose distributed computing platform that's very Python native. It's pure Python. I don't know if it's a Tornado application. So, yeah, it helps you run Python as And I might just build on that by saying, Matt recently wrote, a wonderful what I think is a wonderful, but blog post on our our blog, which is coil.ioforward/blog
38
00:05:57.655 --> 00:05:58.875
for those interested.
39
00:05:59.335 --> 00:06:13.975
But about the goals of DASK and something that I didn't quite recognize, which of course now makes so much sense is, Matt listed 2 technical goals. 1 was to harness the power of all the cause of workstations in parallel, and the other was to support larger than memory computation.
40
00:06:14.355 --> 00:06:20.870
But in addition to that, there was a social goal, which was to invent nothing. And I quote that Matt and,
41
00:06:21.190 --> 00:06:33.224
everyone who created Dask wanted to be as familiar as possible to what users already knew in the Py data stack. And I think that's really important to recognize that all those things we're talking about are technical challenges, but they're also sociocultural
42
00:06:33.604 --> 00:07:04.885
challenges. And so having an API, which not only mimics the APIs that we know and love, but also for the most part runs those I APIs in in and runs the code you're thinking of writing in the back end makes it Python data science native. And this is in contrast to distributed computing frameworks such as Spark, which has its own strengths, But it it it's a strength of Dask that it's it's native for people already working in the Pi data ecosystem. Hugo, I love that in an explanation, you also referred to coiled. Io, the website, and the blog. You're like you're doing the the cross marketing
43
00:07:05.585 --> 00:07:20.195
pitch. Absolutely. I I appreciate that. And I'll also second the fact that it's a notable accomplishment that you have so far been able to make Dask essentially transparent to people who are trying to build these types of applications and numerical analyses
44
00:07:20.575 --> 00:07:21.075
because
45
00:07:21.535 --> 00:07:39.395
it's definitely not easy to be able to keep up with the API changes of libraries that you are not primarily responsible for? Yeah. No. I think, I mean, an interesting point is that there is no Dask API. Right? We, we work with the other communities, you know, Py Pandas, you know, Tornado, Concurrent Futures.
46
00:07:40.015 --> 00:07:46.169
Every Dask API is a pre existing API people are already familiar with. So, you know, if you are in an async application,
47
00:07:46.470 --> 00:07:54.755
you can, you know, submit features in the same way you would with an executor, like a thread pool executor, and you can await them with async await, and you can, you know,
48
00:07:55.135 --> 00:07:58.755
do all the things you normally do. So Dask is very much like a movement
49
00:07:59.535 --> 00:08:02.675
to scale the existing code we all already use
50
00:08:03.040 --> 00:08:04.500
rather than a reinvention
51
00:08:04.960 --> 00:08:08.180
of the Python space. So it's very much part of,
52
00:08:08.720 --> 00:08:10.260
the Python Python community.
53
00:08:10.745 --> 00:08:25.325
And as I mentioned, you were 1 of the early guests on the data engineering podcast where we dug fairly deep into Dask itself, but that was about 3 and a half years ago now. So I'm curious how the project has changed or evolved in that time since we last talked about it.
54
00:08:25.805 --> 00:08:26.845
Yeah. I would say,
55
00:08:27.325 --> 00:08:34.065
technically it hasn't actually changed a ton. Like DASK has been relatively stable over the last few years. Where we've seen a lot of activity
56
00:08:34.589 --> 00:08:49.955
is in the growth of sort of DASK adjacent projects. So this might be in domain specific applications. So, like, all the earth scientists made Pangio, which builds on top of X-ray, which is a DAS powered library, to do things like, you know, satellite imagery with NASA, climate change, oceanography.
57
00:08:50.415 --> 00:08:54.980
And there's, you know, 100 or 1000 people over there who we've been supporting, who all use DASK.
58
00:08:55.680 --> 00:09:31.360
You know, similarly, there's the RAPIDS effort out of NVIDIA. There's a large effort inside of NVIDIA, which is all powered by DASK and CUDA code to provide GPU accelerated data science. There's other projects like Prefect, which is sort of like a reinvented airflow and lots
more. So I think most of what we've been seeing is a lot of growth of people using DaaS, and most of the core contributors to DaaS are mostly in sort of service mode. We are serving other communities who are now trying to scale. I think this week, for example, we've actually seen a lot of great growth in the life sciences. So we've seen, like, biomedical imaging pop up. We've seen population genomics pop up. We've seen, you know, single cell analysis.
59
00:09:31.820 --> 00:09:36.455
So this week, I've had, you know, 2 conversations with completely different groups who are both building
60
00:09:36.995 --> 00:09:40.775
infrastructure on top of DASK. 1 example that I love is is NASA.
61
00:09:41.235 --> 00:09:51.090
I think they're doing a lot of lot of incredible work. And on top of that, like, think about how DASK has changed or evolved in the past 3 and a half years is also a function of how
62
00:09:51.845 --> 00:10:12.875
Python has evolved and the increased adoption. I mean, Dave Robinson, when he was at Stack Overflow, wrote that wonderful post, the incredible growth of Python, looking at Stack Overflow trends. Right? And that was 3 years ago, but we've been seeing such incredible growth growth since. And I think, you know, scikit learn is probably a dominant example, but there are so many others across different domains. And once again, as Dask is part of the Pi data ecosystem,
63
00:10:13.255 --> 00:10:34.904
we've seen kind of all these the entire ecosystem be be lifted up together. The other thing I'll add is with my evangelist hat hat on, I'm just so excited to start servicing more and more of the DASK stories that that exist. And I know I'm it may seem like I'm coming hard and fast with with sales y pictures, but this actually comes from a a place of serious excitement.
64
00:10:35.890 --> 00:10:38.790
At the moment, it's it's, mid August 2020.
65
00:10:39.170 --> 00:10:48.275
We're running, weekly livestreams where we have people who use DAS come on. So check out our YouTube channel if you wanna wanna check these out. Last week, we had Alex Egg,
66
00:10:48.835 --> 00:10:54.860
a senior data scientist at Grubhub come and talk about how he uses Dask all throughout the productionization of,
67
00:10:56.760 --> 00:11:02.780
search queries and intent recognition using Dask, TensorFlow, snorkel for week week supervision,
68
00:11:03.225 --> 00:11:04.125
and, like, using
69
00:11:04.745 --> 00:11:29.910
DAS to parallelize so many steps throughout the process. And that's that's really what I'm super excited about at the moment. So if you wanna check out more of these stories, definitely do come and join us for for a Science Thursday. Yeah. That's definitely 1 of the things that I've seen that's notable about the past few years is that the majority of the growth and evolution has been in that surrounding ecosystem and the number of additional libraries that are using Dask as their method for distributing computation
70
00:11:30.290 --> 00:11:36.765
rather than Dask taking the forefront. So you mentioned Prefect. I know that Daxter is another project that supports Dask as the
71
00:11:41.885 --> 00:11:47.360
Celery code with the Dask dot distributed project for being able to scale out their asynchronous
72
00:11:47.740 --> 00:11:52.560
tasks. And also, I think that when we first spoke, the dasqdot distributed
73
00:11:53.100 --> 00:12:05.310
effort was still fairly nascent. So seeing that become a sort of full fledged project in its own right as just the execution and, distribution method of the code being something that you can use in
74
00:12:16.105 --> 00:12:24.490
that we don't have an have insight into. So it'd be great to hear from listeners who use Dask about their stories as well. To echo on the the Grubhub example, so we were talking to Alex Haig. Actually, during
75
00:12:25.350 --> 00:12:28.010
that that stream, Alex imported from snorkel.dask
76
00:12:29.110 --> 00:12:40.570
import something. And I was like, woah. Wait. Like, snorkel has Dask support? I had no I had no idea that existed. And so that's kind of, like, the level of penetration we are now, which is which is a fun place to be. After having worked on Dask for the past few years,
77
00:12:40.950 --> 00:12:48.395
you have ultimately decided to build a business around it. I'm wondering if you can talk a bit about some of the decision points that led you
78
00:12:48.955 --> 00:13:13.575
to building that business and some of the motivations and goals that you have for building a company on top of and around task. Yeah. No. Building a business today on open source software, there's a lot of, like, interesting variables at play. In terms of motivations, there are several. I mean, first, I wouldn't mind getting some money, but let's, you know, put that aside for a moment and pretend that I'm kind of a nice guy. Yeah. I would say mostly it was that everyone was asking for it. Dask has been really well adopted by individuals
79
00:13:14.180 --> 00:13:19.320
and by other open source library authors like we've talked about, but not particularly well adopted by large enterprises.
80
00:13:19.779 --> 00:13:56.485
So you've looked at a company like, like, let's say, Ford or any sort of large Fortune 50 company. Almost certainly, there's, you know, hundreds of people inside the company using Dask, but it's very unlikely that the company as a whole has adopted it. And it seemed that they would adopt something like Oracle or Spark. To really adopt something like a software at a large institutional level, you really need to buy it from someone. Right? So Ford or NASA don't just adopt open source. They sort of want to buy something. So we need to give them something they could buy from. So we need to have a company for that. Also, we also sort of wanted to lower the barrier for individual users. So they're sort of both serving the very large users, large companies, and also serving individuals.
81
00:13:57.204 -->
00:13:59.945
I care very deeply about making computation
82
00:14:00.325 --> 00:14:38.195
and data literacy accessible. And, you know, scalability is the the hammer that I use the hammer that I work on. What we found is that a lot of people could could use Dask for a company on their laptops. And if their workplace happened to have a really nicely set up cluster, maybe they're at a university, they have access to some HPC machine, maybe they have an IT department who understands the cloud and Kubernetes. They were able to use DaaS happily at scale. But most people are under that situation. Most people were happy using Dask on a laptop, but they didn't really understand the sort of DevOps skills necessary to launch that scale. Yeah. So I might just add add on top of that. I think you hit the nail on the head several times in particular with all the DevOps that's that's required of, of modern data scientists,
83
00:14:38.670 --> 00:14:54.065
particularly whether it be, you know, using Docker and Kubernetes and getting everything up and running on AWS and getting something running locally and moving seamlessly between your local computation and the cloud is something which is so challenging these days. Tobias, I I listened to a recent episode
84
00:14:54.445 --> 00:15:00.290
of this podcast with someone from Netflix talking about meta flow. I mean, we have organized large organizations
85
00:15:00.670 --> 00:15:02.290
are building things internally
86
00:15:02.645 --> 00:15:06.905
to solve these these challenges as well, but not necessarily focused on the Piedata
87
00:15:07.445 --> 00:15:08.985
ecosystem. So we're essentially
88
00:15:09.525 --> 00:15:15.680
making a a product that makes scaling, really easy. And that's what our first product is that, we're we're launching
89
00:15:16.060 --> 00:15:17.920
very soon, and so that is mid August
90
00:15:18.475 --> 00:15:22.335
2020, and it's a DASK in the cloud service that just tries to be dead simple.
91
00:15:22.875 --> 00:15:26.655
Our current users show that it's about 3 to 5 minutes from opening
92
00:15:27.009 --> 00:15:33.589
their beta invitation email to doing computing on the cloud at scale from your laptop and having both environments
93
00:15:34.024 --> 00:15:35.005
and data access
94
00:15:35.545 --> 00:15:42.750
exactly the same on on your laptop and in the cloud. I'll also add 1 more thing that we're solving for the end user
95
00:15:43.050 --> 00:15:46.750
of of the data scientists, but we're also solving for institutional,
96
00:15:47.370 --> 00:15:47.870
cultural,
97
00:15:48.650 --> 00:15:53.585
challenges. And I'll be more specific. What I mean by that, open source. The open source ecosystem
98
00:15:54.045 --> 00:16:02.810
solves very well for people wanting to do and needing to do science. Right? Data science, scientific research. But it's building out some more collaborative
99
00:16:03.110 --> 00:16:13.515
techniques for sure, but it doesn't necessarily, and nor should it solve for institutional culture cultural challenges. So allowing management to have insight into advanced telemetry,
100
00:16:14.055 --> 00:16:20.630
so that they can, you know, encourage collaboration more, and and view costs and see what's happening across their org. Similarly,
101
00:16:21.010 --> 00:16:27.975
serving IT's I IT's needs. So we check all the boxes from from IT, from network security or usage controls,
102
00:16:28.515 --> 00:16:31.575
everything you need in order to analyze your your your data,
103
00:16:31.955 --> 00:16:42.585
safely. And I think part of the value prop is that our team has been doing this for for for a long time as well. So productizing that and taking it into organizations that require all these boxes to be ticked,
104
00:16:43.145 --> 00:16:46.845
is what we're really excited about as well as meeting the end needs of,
105
00:16:47.305 --> 00:16:54.269
the needs of end user data scientists. And I think that the overall push of moving to distributed computation
106
00:16:54.649 --> 00:17:10.905
is definitely very evident with the work that you're doing with DASK and the community that's built up around that and some of the other movers of the ecosystem. I'm thinking in particular about the Ray project and their recent release of the AnyScale company. So I'm wondering if you can just talk a bit more about some of the
107
00:17:11.309 --> 00:17:14.049
broader ecosystem changes that are necessitating
108
00:17:14.429 --> 00:17:33.669
these distributed capabilities and the ability to run Python code and other computation across large clusters of instances beyond just what's capable of maybe, you know, a laptop or a decently sized server? Yeah. It's interesting. There's a few things going on. 1 thing is the growth of data. Right? You got all these data data sets, they're scaling up,
109
00:17:34.210 --> 00:17:35.110
and they're becoming,
110
00:17:35.650 --> 00:17:37.750
you know, arduous to work on a laptop.
111
00:17:38.665 --> 00:17:53.769
Additionally, there's the sort of rise of machine learning and the use of computation and the the clear value the company is getting out of computation. And then finally, there's just a lot of more corporate engagements in Python. Right? Previously, you would have done all of this stuff in Java or in Scala, and there's there's sort of a rich ecosystem
112
00:17:54.149 --> 00:18:12.240
of tooling in the JVM world. You know, Spark, Hive, Hadoop, etcetera. As machine learning rises, as technologies like Docker have come up and show that you can run Python in production pretty comfortably, All of that sort of shifting over to the Python space. And as a result, we need to grow an distributed framework system,
113
00:18:12.860 --> 00:18:23.660
very quickly. That's the kind of work we've been doing over the last 5 years. Right? DaaS maintainers maintain the s 3 connections and the GCS connections, and we maintain how to deploy on Yarn and all those sorts of things. So there's,
114
00:18:24.100 --> 00:18:40.410
yeah, there's a lot of business needs. There's a rise of data, and there's a great there's a great need today. And I'll add add add to that. I I think we, you know, we had the big data hype start a decade ago or whatever it is now. And value, I think, was delivered clearly in in tech, but not necessarily in a lot of other verticals.
115
00:18:40.870 --> 00:18:48.415
And so I I think there's been some sort of like, if you view it as the Gartner hype cycle in some industries, we may be in the trough of disillusionment,
116
00:18:48.795 --> 00:19:06.355
but I I do think there are huge gains to be made with the amount of data we we we have out there, and but we just don't have the tooling to deal with it. And for data scientists and machine learning engineers and people doing, productionizing machine learning to to deal with it. So I'll use, Matt and I are writing an opinionated and and somewhat provocative
117
00:19:06.735 --> 00:19:08.915
post currently that we truly believe in about
118
00:19:09.250 --> 00:19:11.990
the fact that our all in 1 data platforms
119
00:19:12.450 --> 00:19:21.565
serve very important purposes in infrastructure and and data engineering. But currently, they don't meet the meet the needs of end user data scientists who need nimbleness,
120
00:19:21.945 --> 00:20:01.750
flexibility, and ergonomic products that allow them to, you know, work with the with the bleeding edge packages and moving, as we've said before, seamlessly between the laptop and the cloud. And I think in terms of actually getting deriving the next level of value from all the data out there, we're gonna require these types of products to to be built for data scientists to use to extract that value. And that's what we're what we're really excited about as well. I'll also add, though, that big data or whatever we wanna call it isn't always the answer. And if people like, should I do distributed machine learning? I always say no unless you you have to. Like, plot your learning curves as you increase the size of your dataset. And if your accuracy or whatever your, you know, evaluation metric
121
00:20:02.130 --> 00:20:08.855
of interest plateaus well before you need to go distributed, maybe you don't need much more data. You know? Select your your features
122
00:20:09.155 --> 00:20:11.895
accordingly before you go to a a distributed setting.
123
00:20:12.195 --> 00:20:27.335
So I think it's key, but it isn't always the the solution. It's question and data specific. And to the point of the challenges of running things like Daskin production or getting machine learning and data science workloads into a distributed capacity.
124
00:20:27.715 --> 00:20:38.419
For people who are using DASK, what are some of the existing sharp edges that you're looking to solve for as a company and just as ongoing maintainers of the Dask library and its ecosystem?
125
00:20:38.720 --> 00:20:45.524
Yeah. I would say generally deployment is really hard, especially in an enterprise context and especially for the kind of people who use Dask, who are typically,
126
00:20:45.985 --> 00:20:48.325
you know, data scientists, analysts, machine learning researchers
127
00:20:48.760 --> 00:20:56.700
without a lot of DevOps experience. It's honestly mostly a cultural thing. Like, it's actually not there's no novel thing that needs to be built to solve this problem. It's just
128
00:20:57.025 --> 00:21:07.590
how we sort of productionize it. How do we how do we turn that into sort of a a very routine operation? Tivu, you have lots of things thoughts here. Do you have thoughts? 1 thing springs to mind in particular is that when people
129
00:21:07.970 --> 00:21:11.190
try to move between local and the cloud or move to larger datasets,
130
00:21:11.685 --> 00:21:18.425
their sense of flow is is broken. And I don't necessarily wanna get too too holistic here, but data scientists
131
00:21:19.039 --> 00:21:23.440
do their best work when they ins are in some sort of flow state with their with their,
132
00:21:23.840 --> 00:21:32.715
environment, with their data, with their stack, with their computation, with their ecosystem. That's a huge thing that, PyData has done. It's something that the tidyverse has done for the r ecosystem
133
00:21:33.255 --> 00:21:41.370
as well, I think. But in the countless conversations I've had with people doing distributed computing and needing to jump back and forth between different
134
00:21:41.745 --> 00:21:49.685
contexts. It's the context switching which breaks a flow state, which makes the process arduous and not ergonomic, and it also,
135
00:21:50.740 --> 00:21:53.320
I think means everything just takes a lot longer
136
00:21:53.620 --> 00:22:07.215
as well. So that's why I always come back to, you know, meeting data scientists where they are. If they wanna do the DevOps and they enjoy doing that, that's fine. But for the most part, they wanna do what they do best and stay in that flow state of doing data science
137
00:22:07.700 --> 00:22:36.149
and machine learning. And I do think that is a technical and and cultural challenge, as Matt has said. Yeah. So maybe let's, like, give it give it, like, maybe an anecdotal example. Let's say that you're using scikit learn, and you want you're doing some hyperparameter optimization on random forests, and you want to make it run faster because it's taking a while. So, you know, you you do do a web search for scikit learn fast. You find the scikit learn docs. You find they point to Dask. You try using Dask. You use it use it on local machine. You're pretty happy. The single core to multicore on single machine experience,
138
00:22:36.914 --> 00:22:46.030
that that transition is really smooth today. A ton of people do that, and they're very happy. But then you wanna scale to many machines. So you say, hey. Great. I got this cloud account. You go to our IT department.
139
00:22:46.490 --> 00:22:50.430
You get access to your AWS
credentials. You then, you know, go on the DaaS documentation.
140
00:22:50.970 --> 00:22:54.735
You don't know anything about the cloud, but the DaaS talk's pretty good. They point you at Kubernetes.
141
00:22:56.395 --> 00:23:04.070
You, you know, spend an hour, learn how Kubernetes works and well enough to stand something up on, you know, AKS or something, and then you launch modules at DaaS Kubernetes.
142
00:23:04.450 --> 00:23:19.840
If you're actually a sharp data science developer, you could probably do that today in about an hour, without really knowing much about how Kubernetes works, which is, you know, testing it to lots of open source projects. After that, you might get it running and you say, actually, you know, turns out the the Docker image are running remotely
143
00:23:20.220 --> 00:23:42.720
doesn't have scikit learn installed. Bullcrap. Like, now I need to go build the Docker image. I need to push that up somewhere. You try it again, And, you know, the Python versions are different because, you know, we're serializing data in some weird way. You fix that again. You push it up again. Magically, it works. It's been a couple of days, but you're you're working now. You're really happy. A month later, IT and Knox Universe says, hey. Wait a minute. You left that machine up for a long time. You racked up around $10, 000,
144
00:23:43.615 --> 00:23:59.720
and there's no security on that machine. And so, you know, fortunately, no 1 found the address you were using, but, you know, you actually had axe you had credentials. You copied on the machine that were totally open to the to everyone in the world. And so you've been sort of locked down by IT now. So those are the that's sort of a a brief narrative, the kinds of things
145
00:24:00.295 --> 00:24:02.315
that, you know, data scientists runs into
146
00:24:02.695 --> 00:24:06.315
if they're actually really sharp and are able to jump past all those hurdles.
147
00:24:06.775 --> 00:24:15.390
But it's it's just the pain. Right? And it's not something that most people do. I couldn't agree more. And I I think 1 thing you've done wonderfully in there, Matt, is also help us reason about,
148
00:24:15.950 --> 00:24:31.980
distributed data work and and machine learning. I'll I'll pull out 1 example. There are lots in in in there with respect to the processes that that are necessary. But, I think it was maybe Travis Olofan, who I heard this from the first time, that distinct even distinguishing scaling up to multiple cores in a single workstation
149
00:24:32.360 --> 00:24:38.315
as opposed to scaling out to a variety of clusters or on the cloud or on on prem or whatever you're doing. And thinking
150
00:24:38.695 --> 00:24:45.929
constructively and reasoning about these processes and what you need to do. Because maybe for the work you need to do, you only need to scale up, right, to,
151
00:24:46.490 --> 00:24:48.669
multiple calls on on your local works workstation.
152
00:24:49.049 --> 00:24:49.710
But then,
153
00:24:50.090 --> 00:24:57.315
what what are the next steps? And that that will be scaling out. So I I think that framework's very instructive. And so in order to overcome some of these challenges
154
00:24:57.775 --> 00:25:02.755
and the complexities of building these distributed computation platforms
155
00:25:03.399 --> 00:25:07.419
and managing all of the different security implications and the scaling implications
156
00:25:07.720 --> 00:25:16.034
and the operational aspects of running code in production. What are you working on building at Coiled to help alleviate some of those pain points?
157
00:25:16.495 --> 00:25:20.230
Essentially, the first iteration of of product, hosted and scalable
158
00:25:20.690 --> 00:25:31.955
DAS clusters so that you can use cloud to launch clusters on managed resources like the cloud with a single click. Certain things we handle really well are deploying containers, hooking up networking securely,
159
00:25:32.415 --> 00:25:39.220
and making it easy to connect. So you as a data scientist can get back to your real work and, and and do what you do best.
160
00:25:39.600 --> 00:25:44.500
As we stated earlier, part of the value proposition is we don't want you to have to wait on on on DevOps
161
00:25:44.985 --> 00:25:51.565
or having data scientists have to do all the DevOps themselves. So you can use clusters of machines, advanced libraries, and GPUs.
162
00:25:52.909 --> 00:25:53.809
Cloud works anywhere.
163
00:25:54.190 --> 00:26:03.385
So currently including cloud services like AWS SageMaker, open source solutions like JupyterHub, or even from the comfort of your own laptop. This includes,
164
00:26:03.845 --> 00:26:06.825
I think as we hinted at it earlier, managed software environments,
165
00:26:07.125 --> 00:26:10.169
with conda or pip and docker, user management,
166
00:26:10.789 --> 00:26:12.330
advanced telemetry, cost controls,
167
00:26:12.789 --> 00:26:33.650
these types of things, and everything that you'd that your IT team, would be would be interested in as well with respect to auth and security. Yeah. Maybe I'll add to that because this is a Python podcast and because people actually on the podcast know what we're or know what we're doing. They can sort of appreciate this both from a user perspective and from an internal dev perspective. I'll tell you a little bit about how Coiled is engineered internally. Awesome. So Coiled combines,
168
00:26:34.030 --> 00:26:37.794
it's a Django web app. It's Postgres. It's Amazon ECS
169
00:26:38.174 --> 00:27:05.350
and Conda. Those are mostly the the tools that we're throwing together, and Dask. And so Coiled is a Django app that you can authenticate into that will launch on demand DaaS clusters on Amazon ECS. ECS is like a container service. It's like Kubernetes. It's a bit simpler, but more widely available. So you can log into Coiled, tell Coiled, hey, Create a task cluster for me somewhere, then it'll do that. And that's
that's step 1. Right? You then wanna tell that task cluster, hey. You need to have these libraries installed.
170
00:27:05.655 --> 00:27:32.409
And so Coiled handles building software environments. Most data scientists don't like using Docker, so we solve condor PIP environments for you. We hold on to those. We store those. We build Docker images for you, and we deploy them. We handle things like authentication with the cloud. So you may, on your laptop, have credentials to Amazon. We'll take those credentials, generate some secure tokens, ship those up to those workers as well, so they can operate as you. All of that's done securely, so you can drive that cluster of machines on
171
00:27:32.789 --> 00:27:38.409
AWS from your laptop or from, you know, some CircleCI scripts or from, you know, some
172
00:27:39.034 --> 00:27:42.975
GUI application running Python. So COIL is designed to solve
173
00:27:43.275 --> 00:27:54.120
the DaaS deployment problem and the sort of like peripheral problems just around that problem. Software management, cost management, etcetera. Then we also because we're watching everything you do, because we're watching
174
00:27:54.500 --> 00:28:08.360
how big you're scaling that thing up, we're we're a good place to impose limits. Right? You know, maybe you manage a team, you are swapping a credit card so people can use that, and you can give people access to that up to a certain amount. You can gate their access. You can say, hey. This guy can or cannot use GPUs.
175
00:28:08.740 --> 00:28:09.960
This person's really good.
176
00:28:10.260 --> 00:28:35.955
She can use, you know, $100 an hour of clusters if she wants to. And so that's it's all that sort of basic functionality around managing DASK. And, yeah, it's built with a bunch of actually like really boring technologies that have been around for a decade, but it's really comfortable. Like, as someone who uses Dask a ton, I use Dask in every single possible situation you can imagine it. We built the way that I prefer using Dask, and I I love it. It's actually really, it's really comfortable. It's really pleasant. It's really easy.
177
00:28:36.335 --> 00:28:40.674
Yeah. That's Dask. That's Coiled. Yeah. There's definitely a lot to be said for boring technologies
178
00:28:40.975 --> 00:28:45.860
because it's definitely interesting and exciting to be able to use bleeding edge new platforms
179
00:28:46.240 --> 00:28:59.765
or projects. But when you actually go to production with something, you want it to be boring because you don't want it to wake you up in the middle of the night every day for the next 10 weeks. Yeah. It's actually I have actually never used Django before. I've been in Python for, you know, a decade plus,
180
00:29:00.160 --> 00:29:10.034
and I've always been on the data science side. It's actually really fun for me to interact more with the website of Python. The people we have who were employed at Coio, the engineers, people like Rami or Dan or Marcos,
181
00:29:10.414 --> 00:29:16.115
all, like, understand this stack super well. And it's like, hey. Can we like, how hard is it to add, you know, GitHub authentication?
182
00:29:16.510 --> 00:29:19.890
And, like, sure. It takes a day because there's, like, the Django all auth package.
183
00:29:20.270 --> 00:29:21.090
This is probably
184
00:29:21.470 --> 00:29:34.340
really boring to most of your listeners. But for me, it's like I just discovered this new set of superpowers that we have. Like, all the Django plugins are it's great. It's like finding, you know, visualization libraries in the data science side for the first time. Yeah. It's definitely
185
00:29:34.720 --> 00:29:35.220
interesting
186
00:29:35.600 --> 00:29:38.260
how the breadth of the Python ecosystem
187
00:29:38.720 --> 00:29:42.340
can lead you to be somebody who has worked in the overall
188
00:29:42.705 --> 00:29:55.260
language for, you know, a decade plus and then still have some area of the community that is completely new to you that has all of these exciting new shiny bells and whistles that everybody else has considered old hat for the past 5 years.
189
00:29:55.720 --> 00:29:59.580
It's also why building a product like this is super easy.
190
00:29:59.885 --> 00:30:03.505
Right? We happen to have this amazing distributed computing stack
191
00:30:03.885 --> 00:30:05.424
and this amazing web stack
192
00:30:05.725 --> 00:30:07.105
and connections to everything.
193
00:30:07.730 --> 00:30:10.550
The fact that everything is available in the same language
194
00:30:11.250 --> 00:30:14.710
makes, you know, cobbling these things together just really
195
00:30:15.010 --> 00:30:16.070
amazing. It's great.
196
00:30:16.455 --> 00:30:18.235
Continuing on the thread of Python,
197
00:30:18.775 --> 00:30:26.320
the main tagline that you have on your website right now is focused on the fact that you're scaling Python, which is
198
00:30:26.860 --> 00:30:34.405
in some sense for somebody who's using Python, oh, this is great. But for somebody who is concerned primarily with just, I have this data problem,
199
00:30:34.705 --> 00:30:35.605
it might not necessarily
200
00:30:35.985 --> 00:30:37.205
be a a trigger
201
00:30:37.519 --> 00:30:40.179
for them to say, oh, this is what I need. So I'm curious
202
00:30:40.639 --> 00:30:49.845
what your thoughts are on the pros and cons of orienting your messaging on that aspect of the scalability of Python as opposed to focusing on a particular problem domain or industry?
203
00:30:50.305 --> 00:30:51.685
Yeah. Super interesting question.
204
00:30:52.145 --> 00:30:55.285
First, actually, like scaling Python right now is actually,
205
00:30:55.800 --> 00:30:57.820
like, hitting pretty well in the market.
206
00:30:58.360 --> 00:31:11.365
A lot of companies are building up their Python stack or they're scaling their Python stack, and they're looking for solutions. So So it's actually a pretty good time to try to sell that
messaging. But more generally, I think you may have a good point that it's actually really tricky
207
00:31:11.909 --> 00:31:21.289
to sell something that's so broad. Right? So, you know, if we go to the other extreme, for example, if we were selling a product that, you know, identified tumors and images
208
00:31:21.669 --> 00:31:22.794
to save lives in
209
00:31:23.174 --> 00:31:26.554
hospitals or, you know, a biofuel that increase fuel efficiency by 20%,
210
00:31:26.934 --> 00:31:29.434
like, the the value of those propositions
211
00:31:29.735 --> 00:31:35.940
is much clearer. It's much more easy and direct to sell. It's like, yep. I'm gonna save 20%. Makes sense to buy. Here's 10%.
212
00:31:36.640 --> 00:31:41.785
Transaction done. Scaling Python is a lot more it's a lot more broad. It's a lot more indirect.
213
00:31:42.485 --> 00:32:04.385
Right? You don't have to, like, get into a conversation with a customer about, k. Well, like, why do you care about data science? Why is that useful to you? Do you care about reducing cloud costs? Like, how how is it that making your people more efficient will actually save you money in the future? And so it's a much longer conversation to, to get to that point where they say, yes. This makes sense. We're going to invest, you know, a large chunk of money,
214
00:32:04.845 --> 00:32:05.905
to purchase your product.
215
00:32:07.450 --> 00:32:19.105
But on the other hand, it's super broad, and that's really, I think, where open source companies really shine. We can sell to, you know, to NASA, who who wants to care about satellite imagery. And then we can turn around and sell to,
216
00:32:19.405 --> 00:32:27.950
you know, Pfizer or some pharma company who cares about biomedical imaging. And, presently, biomedical imaging and satellite imagery are actually kind of the same problem. So it's,
217
00:32:28.590 --> 00:32:47.340
it's tricky. You have to really start to understand your customers and get a lot more empathy with people, but there's a ton of opportunity, which is fun. It's just like a very big puzzle to solve. I I agree. I agree with all of that, and I take it I over slightly different perspective. I think it's a great question as well, and this is something that is not boring and does wake me up at night, for for clarity.
218
00:32:47.684 --> 00:32:48.184
But
219
00:32:48.804 --> 00:33:01.390
we need to think about what our job is at we've got lots of jobs at COIL currently, but is our job to sell the product product as much as possible or at this early stage to build the best product possible? And I think it's it's definitely
220
00:33:01.690 --> 00:33:04.910
the latter. And to to that end, we don't necessarily
221
00:33:05.850 --> 00:33:24.400
want to be positioning ourselves as as solving industry specific problems. We wanna get data science teams on board to use our product and to form kind of an iterate a rapid iterative feedback loop into into product development. Right? So it's essentially, in that sense, a bottom up motion and a grassroots
222
00:33:24.855 --> 00:33:26.715
movement of of data scientists
223
00:33:27.095 --> 00:33:28.875
built into the product development cycle
224
00:33:29.255 --> 00:33:32.235
and solving their pain points. And, of course, as you state,
225
00:33:32.660 --> 00:33:45.165
if it later on, it becomes industry specific, we may even move to, you know, a a top down sales motion. I love, you know, Databricks and this type of stuff. I'm not saying that we will we'll we'll do that, but these are kind of a wide range of questions we're
226
00:33:45.465 --> 00:34:03.715
we're thinking about. It's interesting coming out of our programming podcast and talking about sales motions and, and, you know, sales strategies. But it's actually it's kind of interesting in this perspective because it's 1 of those cases where I think the the profitable thing and the, like, community thing happened to be aligned. So we're gonna make most of our money selling to big companies.
227
00:34:04.015 --> 00:34:23.829
Right? You know, Ford, NASA, whatever, are gonna buy something for 1, 000, 000 of dollars. But those can take a long time. In the meantime, like, we have a bunch of users who we can learn a lot about and who we can who can iterate with very quickly. And so we're both selling, you know, these big things that take years to close with big companies. But in the meantime, we're we're making a product that's really designed for individuals to use,
228
00:34:24.530 --> 00:34:32.905
because they bring in a lot of information about how people want to scale their Python code. And so for the next year or so, we're really just focusing on making
229
00:34:33.285 --> 00:34:34.905
the sort of public access
230
00:34:35.605 --> 00:34:41.980
product as ergonomic and as friendly as possible. And that happens to align very well with, you know, what I care about altruistically
231
00:34:42.680 --> 00:34:45.420
of, you know, increasing access, increasing
232
00:34:46.035 --> 00:34:48.455
our society's ability to reason about,
233
00:34:48.835 --> 00:34:56.570
the world through data. And so it's, again, 1 of those things where I think making money and helping the world end up being kind of the same actions. And it's also aligned with increasing
234
00:34:57.350 --> 00:35:07.424
adoption of the tools that we believe do good in in the world as well and that we build to do good in in the world. So if we can build a product that grows the Dask and Py Datapie
235
00:35:07.724 --> 00:35:11.025
as large as possible for people who find it valuable,
236
00:35:12.560 --> 00:35:16.820
I I think that's amazing. And, of course, there are a lot of nuances here to make sure that these incentives
237
00:35:17.360 --> 00:35:18.100
are aligned,
238
00:35:18.400 --> 00:35:20.340
but that's that's the goal. And
239
00:35:20.795 --> 00:35:26.495
I know Matt is, and I'm definitely excited to be part of of this movement of garnering even more OSS,
240
00:35:27.035 --> 00:35:29.890
institutional adoption through commercial products as well.
241
00:35:32.930 --> 00:35:35.910
This portion of podcast dot in it is brought to you by Datadog.
242
00:35:36.369 --> 00:35:44.035
Do you have an app in production that is slower than you like? Is its performance all over the place, sometimes fast and sometimes slow? Do you know why?
243
00:35:44.495 --> 00:35:53.860
With Datadog, you will. You can troubleshoot your app's performance with Datadog's end to end tracing and, in 1 click, correlate those Python traces with related logs and metrics.
244
00:35:54.195 --> 00:35:55.015
Use their detailed flame
245
00:35:55.315 --> 00:35:58.615
graphs to identify bottlenecks and latency in that app of yours.
246
00:35:59.075 --> 00:36:02.210
Start tracking the performance of your apps with a free trial at pythonpodcast.com/datadog.
247
00:36:04.350 --> 00:36:10.395
If you sign up for a trial and install the agent, Datadog will send you a free t shirt to keep you comfortable while you keep track of your apps.
248
00:36:13.435 --> 00:36:28.160
And continuing on that topic of the dichotomy of open source and commercial product. I'm wondering what you have seen so far as the challenges and being able to manage those competing tensions between the open source work that you do
249
00:36:28.585 --> 00:36:40.100
and working as maintainers and core contributors of the DASK project, as well as this proprietary stack that you're building around it and working on selling to companies to help drive the overall
250
00:36:40.480 --> 00:36:48.365
business forward, as well as trying to help support the open source ecosystem around it? I have tons of thoughts on that topic. That's like a whole podcast.
251
00:36:48.825 --> 00:36:58.000
Before I go off into things, Hugo, do you have thoughts? I I do, but I I will defer to you because I I think in you're actually in a very different position
252
00:36:58.780 --> 00:37:05.095
to me with your with your role in terms of being a core contributor and a maintainer of a package as well as
253
00:37:05.395 --> 00:37:09.175
building a product around that package. I okay. Having said that, my position
254
00:37:09.670 --> 00:37:14.250
it's very I mean, I I run I'm I'm head of data science evangelism
255
00:37:14.630 --> 00:37:33.039
on 1 side and marketing on on another. So I joke that I sometimes feel like 2 face from from Batman. I'm a good guy, though, in in in terms of I have an enterprise face with respect to marketing, but an evangelism face with respect to community development. And I I I feel like the role of COIL and the role of myself in in in these positions
256
00:37:33.339 --> 00:37:35.119
is to form connective tissue
257
00:37:35.555 --> 00:37:45.710
between both that creates value that's more than the sum of it some of its parts. So I know I know this is very abstract, but ideally, the work the open source community puts in, they get more out of. And similarly,
258
00:37:46.329 --> 00:37:47.150
with with the enterprise.
259
00:37:47.450 --> 00:37:49.470
And I think the direction from
260
00:37:49.770 --> 00:38:00.980
OSS to the enterprise is what we've been talking about, but the other direction is is incredibly important as well. And I hinted at that with respect to garnering further adoption of of open source, but also,
261
00:38:01.280 --> 00:38:06.020
thinking about viable models for for funding open source and for employing open source developers.
262
00:38:06.400 --> 00:38:32.484
And, I mean, we're both very excited to be part of a a fascinating lineage here. And, you know, there have been companies for several decades now that have learned learned lessons lessons here that we're we're in close close connection with. And that, you know, Matt Matt has worked for for for example. So I think that's that's my take. But I'm even more interested in Matt's take, due to his unique position. Yeah. I think I like your 2 face example. I'll say maybe many hats is the word term that I would use.
263
00:38:32.785 --> 00:38:40.740
It doesn't turn you into a superhero villain, at least, which I think. I'm sure there's superhero villains somewhere with lots of hats. They're not better. I like that. How about that?
264
00:38:41.200 --> 00:39:03.915
But, yeah, I mean, we've actually been kind of threading this needle with DASK since its inception. Right? DASK came out of a for profit company. It came out of Anaconda. Right? Anaconda was amazing in its support for DASK early on, and it hired, you know, me and lots of other people to work on Dask over time. And we had to sort of be very aware of when we were being Anacondins and when we were being open source Python people. I then moved to NVIDIA, and I had the same challenges with GPUs.
265
00:39:04.215 --> 00:39:09.995
And so, yeah, we've always had those sorts of challenges. I think it's now maybe, like, up another level. Like, I now get,
266
00:39:10.380 --> 00:39:22.675
like, direct compensation based on how well certain things do with task. So we need to be careful. But it's it's a care it's something that we're we're used to doing. I think, you know, for example, as an anecdote, a very small anecdote, Hugo asked me, I think, just yesterday,
267
00:39:23.055 --> 00:39:29.450
you know, hey. We're doing all these community blog posts about people who use Dask. Can we get the Dask Twitter account to retweet,
268
00:39:30.150 --> 00:39:35.770
the coiled content? And I said, like, yeah. Sure. That sounds great, but, like, I can't do that. Right? That would be a conflict of interest.
269
00:39:36.085 --> 00:39:39.625
Go find someone else who who also manages the Zast Twitter account
270
00:39:40.005 --> 00:39:40.905
and have them,
271
00:39:41.605 --> 00:39:54.029
retweet that for you. You've gotta work that out with them independently of me. So that's been you know, it's something we always deal with. Go ahead. I I was just gonna say, with respect to that example, I was
fascinated that, the Dask developer community had already
272
00:39:54.455 --> 00:40:05.890
thought about this, and you linked me to a GitHub issue in which it had been discussed, and there was already a path somewhat laid out laid out for this. And I think that what that flags for me is that, as you mentioned earlier,
273
00:40:06.270 --> 00:40:16.935
a lot of these concerns have kind of almost been preempted in in in some sense, and it's about figuring out a a seamless way to get these flows flows happening. But the open source developed community is an incredibly,
274
00:40:17.335 --> 00:40:18.555
intelligent, thoughtful,
275
00:40:19.020 --> 00:40:21.920
empathetic bunch of people for for for the for the most part.
276
00:40:23.500 --> 00:40:31.755
That's not entirely true. Always. I I agree. And I'm just thinking about Stack Overflow responses now, actually. What I would say maybe in in Dask's favor,
277
00:40:32.135 --> 00:40:35.195
we have a lot of practice at this. And Dask is actually really well
278
00:40:35.655 --> 00:40:44.870
well positioned to handle this sort of challenge because we already have a bunch of companies that are involved in Dask. Right? We have, you know, weekly maintainer calls every Tuesday morning.
279
00:40:45.170 --> 00:40:49.165
There's gotta be people at 6 or 7 different companies who attend those calls.
280
00:40:49.545 --> 00:40:51.565
It's not a lot of open source companies,
281
00:40:51.865 --> 00:40:59.960
they're a mega company first. They release some open source software, and they maintain that software as part of their company. If the company fails, the software goes under,
282
00:41:00.260 --> 00:41:11.315
and probably everyone in that soft everyone has commit rights to that software probably works for that company. Super common model. That's definitely not the DASK model. Right? DASK operates much more like Pandas or NumPy or Jupyter.
283
00:41:11.780 --> 00:41:14.200
It's very much a multi institution endeavor.
284
00:41:14.500 --> 00:41:21.080
It's not attached to 1 particular company. And that gives Dask a lot of its strength, a lot of its perspective, and a lot of its resilience.
285
00:41:21.765 --> 00:41:30.265
Another interesting element of the fact that you are starting COIL ed at this particular point in time is that we're in the middle of a global pandemic,
286
00:41:30.645 --> 00:41:45.665
and I'm sure that has added some additional complicating factors to what is already a stressful time. So I'm wondering if you can maybe talk a bit about some of the ways that that has manifested in your current journey of launching this company and building a business at the particular point of the world?
287
00:41:46.285 --> 00:41:50.145
Yeah. No. It was, it was great great timing in various ways.
288
00:41:50.559 --> 00:41:54.099
I think the company was incorporated, I wanna say, like, February 11th.
289
00:41:54.720 --> 00:41:57.299
So, like, you know, a few weeks before the world ended.
290
00:41:58.525 --> 00:42:00.305
Mostly, it's actually been fine.
291
00:42:01.724 --> 00:42:03.265
I think it's, like, some of our, like,
292
00:42:03.724 --> 00:42:05.265
intended anchor clients,
293
00:42:05.885 --> 00:42:08.660
like, step back, said wait. We're gonna wait until things,
294
00:42:09.120 --> 00:42:13.605
you know, sell down a little bit. They're not coming back. So there's definitely early clients that left,
295
00:42:14.005 --> 00:42:17.865
but there's actually a lot of churn right now in lots of ways, which is really fascinating.
296
00:42:18.484 --> 00:42:25.130
Hiring is super interesting right now. Right? A lot of people came on the market. They were then scooped up very quickly. There's a lot of sort of activity.
297
00:42:26.630 --> 00:42:31.530
We're we're a remote first company. We DASK has always been a remote first, so Coil does too,
298
00:42:32.045 --> 00:42:35.025
which has been great in some ways, although then stressed in some other ways.
299
00:42:35.885 --> 00:42:38.474
I'll let Hugo maybe talk about that for a moment.
300
00:42:39.940 --> 00:42:44.360
Yeah. I do think that it's interesting that we were already intending to build
301
00:42:44.660 --> 00:42:46.075
a 100% remote
302
00:42:46.714 --> 00:42:49.775
company already, or as I like to joke because we're in the distributed,
303
00:42:50.234 --> 00:42:51.214
computing space,
304
00:42:51.675 --> 00:42:53.454
100% distributed company.
305
00:42:55.289 --> 00:43:02.109
All that I'm saying there essentially is that a lot of the challenges we're facing due to that aspect of everything that's happening at the moment,
306
00:43:02.410 --> 00:43:05.025
we were we were preparing to to to face anyway.
307
00:43:05.485 --> 00:43:08.145
I do think 1 of the biggest stresses
308
00:43:08.925 --> 00:43:13.089
is the fact that I'm currently in Australia. So for context, I'm from Australia,
309
00:43:13.630 --> 00:43:22.825
I've lived in the US for 6 years, and I was planning to to still work in in the US, but I'm currently waiting for US consulates to reopen
310
00:43:23.125 --> 00:43:23.625
for,
311
00:43:24.085 --> 00:43:27.810
visa visa processing for me to get back on American soil. So
312
00:43:28.370 --> 00:43:32.550
the biggest stress currently is trying to build a company from seed stage
313
00:43:32.930 --> 00:43:39.405
on opposite sides of the world. There are a few wins that that can happen with respect to these obscene time zone differences.
314
00:43:39.785 --> 00:43:55.185
I think the biggest 1 being that we can hand off, stuff to each other at the end of my day, and vice versa, and then when I get up, Matt or whoever else has has has worked a lot on it, and and I can move on that. But in terms of, syncing on on a variety of things and being in the same headspace,
315
00:43:56.285 --> 00:43:57.825
and at the same time of day,
316
00:43:58.125 --> 00:44:00.390
there are a lot of kind of, daily,
317
00:44:01.170 --> 00:44:02.390
challenges that arise,
318
00:44:03.089 --> 00:44:09.835
because of that, but it's something we're thinking about actively and working on actively, and we're excited to find solutions as well.
319
00:44:10.135 --> 00:44:13.755
Yeah. I'd say as as all of us have learned, I think COVID kind of killed
320
00:44:14.135 --> 00:44:17.670
working hours. They killed the workday. It's a it's a mishmash today.
321
00:44:18.150 --> 00:44:21.049
But there's also lots of great opportunities. Like, companies
322
00:44:21.670 --> 00:44:29.895
are kind of in a state of reinvention right now, and, you know, they're looking for new things. It's actually, like, I would say sales interest has not dropped off significantly.
323
00:44:31.075 --> 00:44:49.984
And on top of the challenges of building a business and doing it during pandemic times, you're also building it on top of open source, which we've touched on a little bit as far as some of the challenges there. But what are some of the most interesting or unexpected or challenging lessons that you've learned in the process of building and launching a company that is oriented around an open source platform?
324
00:44:51.349 --> 00:44:52.570
Yeah. I would say the biggest
325
00:44:53.030 --> 00:44:56.250
the biggest not really a surprise, but maybe reinforced preexisting,
326
00:44:56.869 --> 00:44:57.369
biases
327
00:44:57.829 --> 00:45:06.865
was that nobody has any idea how to do this well yet. There are a ton of models of successful companies. You know, there's the Red Hat model. There's, you know, Mongo and Confluence. There's Databricks,
328
00:45:07.244 --> 00:45:10.145
something totally different. There's Anaconda in the Python space.
329
00:45:10.460 --> 00:45:17.359
But they're all very different paths, but none of them were worn very well. It's all a challenge. You know, for example, we're currently working through,
330
00:45:17.775 --> 00:45:20.915
you know, contracts legal with, like, 5 giant companies,
331
00:45:21.695 --> 00:45:25.075
each of whom have their own legal teams. You know, and we have, like, our
332
00:45:25.510 --> 00:45:26.010
amazing
333
00:45:26.790 --> 00:45:29.130
legal rep part time that we that we engage.
334
00:45:29.670 --> 00:45:36.795
But none of those companies has any idea how to deal with open source software in any sort of reasonable manner. You know, IP issues, it's all it's all a pain. So
335
00:45:37.174 --> 00:45:45.810
step by step, it feels very much like an uphill battle through relatively rough terrain. On the grand scheme of it, though, it's also great. Like, there's an incredible amount of support. It's surprising
336
00:45:46.110 --> 00:45:47.010
how many people
337
00:45:47.550 --> 00:45:48.290
know us,
338
00:45:48.625 --> 00:45:59.740
know the DaaS project, and just genuinely want to see us succeed. It's very heartwarming to see the the support. I think you get that being in an open source company. Yeah. I I think Matt hit several nails on their respective heads,
339
00:46:00.840 --> 00:46:05.215
a bunch of times there. I'll just add that there are a lot of people
340
00:46:05.615 --> 00:46:08.355
who who we really enjoy working with, building similar,
341
00:46:08.735 --> 00:46:09.555
similar companies
342
00:46:10.095 --> 00:46:13.235
at the moment, and that we're really excited to be part of a community,
343
00:46:13.855 --> 00:46:19.420
of people figuring this out, and a handful that that come immediately to mind from from friends and family.
344
00:46:20.040 --> 00:46:29.675
Travis Oliphant and all the wonderful people he he works with at at Quanside, and then Wes McKinney at at UrsaLab, and then Peter Wang and the entire team there at at Anaconda
345
00:46:30.055 --> 00:46:35.069
as well. So there are a lot of challenges to to figure out, but there are a lot of wonderful,
346
00:46:35.549 --> 00:46:53.170
incredible people who we're excited to work with closely on on all of these things. And as you build out more of the Coiled business and continue on the path of bringing DASK to more users, what are some of the things that you have planned for the future of both the Coiled company and the DASK project?
347
00:46:53.630 --> 00:46:56.849
On on the DASK side, I would say just keep it expanding.
348
00:46:57.230 --> 00:47:02.265
Right? There's a ton of new users. There's a ton of use cases, and we're just gonna grow that to new scientific domains,
349
00:47:02.805 --> 00:47:03.945
new industry verticals,
350
00:47:04.325 --> 00:47:16.035
and that's that's the focus, I think that's where we're seeing the most activity, and it's super exciting right now. And on the coiled side, we are just super excited to be building a product that is dead simple to use,
351
00:47:16.735 --> 00:47:17.715
for data scientists,
352
00:47:18.255 --> 00:47:18.995
and anybody
353
00:47:19.295 --> 00:47:22.835
who who needs to do data science and more and or machine learning at scale.
354
00:47:23.375 --> 00:47:34.954
So as a call to action, if you're doing data science and or machine learning at scale and you love to break things, we'd really like for you to take Coiled for a test drive, by signing up for our beta. And you can do so, at bit.ly/coiledhyphenbeta.
355
00:47:38.935 --> 00:47:41.115
And we're looking for feedback and conversations
356
00:47:41.415 --> 00:47:43.560
and and all of these things to build,
357
00:47:44.339 --> 00:47:46.280
as as good a product as we can.
358
00:47:47.140 --> 00:47:58.910
Are there any other aspects of the work that you're doing on DASK or at Coiled or just the overall space of machine learning and data science and helping them scale that we didn't discuss yet that you'd like to cover before we close out the show?
359
00:47:59.290 --> 00:48:16.020
There are hundreds of such topics, so I'll just I'll just hold those in my pocket for next time, Tobias. Alright. Well, you're welcome back anytime you like. So for anybody who does want to follow along with either of you and keep in touch, I'll have you
add your preferred contact information to the show notes. And so with that, I'll move us into the picks.
360
00:48:16.400 --> 00:48:25.885
And this week, I'm going to choose the book, the Hobbit. I've read it a few times and just recently started revisiting that in audiobook form with my kids. So definitely recommend,
361
00:48:26.265 --> 00:48:33.830
giving that 1 another look or a first look if you've never read it before. Definitely just a great story and a lot of good fun and just a good adventure.
362
00:48:34.370 --> 00:48:38.630
So, with that, I'll pass it to you, Matt. Do you have any picks this week? Sure. 1 of the DAS maintainers,
363
00:48:38.930 --> 00:49:01.205
Jim Chris just moved to the the prefect company. So I'll maybe in honor of him. So let's let's say Prefect. I think you might've done a an an episode with the Prefect founder on your other podcast, Data Engineering Podcast. Yes. I did. I'll add a link to that. So that I mean, if you if you like Airflow, anything Airflow is a great tool, you might find the Prefectus is an even greater tool. It also happens to be DAS powered, so it's a little,
364
00:49:01.645 --> 00:49:13.330
little self serving. And Hugo, do you have any picks this week? Yeah, man. I'm actually I'm currently rereading a book that I read late last year that impacted me a a great deal. It's called Race After Technology by Ruha Benjamin,
365
00:49:13.870 --> 00:49:17.535
who has been since then a constant source of inspiration to me.
366
00:49:18.095 --> 00:49:22.195
Ruha is an associate professor of African American Studies at Princeton University,
367
00:49:22.735 --> 00:49:24.380
and Race After Technology,
368
00:49:25.240 --> 00:49:26.700
delves essentially into
369
00:49:27.319 --> 00:49:27.980
the coevolution
370
00:49:28.519 --> 00:49:37.345
of both race and technology. And not only have I impact 1 another, but how actually they're so coupled and and and they co evolve. And what Ruha does in this book is provides
371
00:49:37.805 --> 00:49:39.670
an in-depth critique and analysis
372
00:49:40.130 --> 00:49:41.670
of a lot of the at scale
373
00:49:42.050 --> 00:49:43.109
algorithms that
374
00:49:43.730 --> 00:49:46.069
are harming already disenfranchised people,
375
00:49:46.755 --> 00:50:11.744
in particular, the the racist structures that are emerging. But she also gives us a very serious language to start describing the things we're seeing. So 1 in particular is what she refers to as the new gym code. I'm gonna read, just a small part of of what she describes as the new gym code, which is the employment of new technologies that reflect and reproduce existing iniquities, but that are promoted and perceived as more
376
00:50:12.339 --> 00:50:13.400
objective or progressive
377
00:50:13.780 --> 00:50:15.000
than the discriminatory
378
00:50:15.300 --> 00:50:17.640
systems of a previous era. Okay?
379
00:50:18.099 --> 00:50:20.415
So Ruha does an incredible job,
380
00:50:20.875 --> 00:50:21.375
at
381
00:50:21.755 --> 00:50:32.060
allowing us to perceive what's happening by building a language and structure around it. She also had a fascinating keynote recently at ICLR, which I encourage everyone to check out,
382
00:50:32.680 --> 00:50:34.220
where she makes clear,
383
00:50:35.640 --> 00:50:38.145
such things as, you know, when we talk about,
384
00:50:38.684 --> 00:50:47.500
deep learning and think, you know, how how powerful it is, but she makes clear. She says, for example, that computational depth in deep learning, for example, without sociological
385
00:50:48.040 --> 00:50:53.795
sociological depth is actually superficial fish learning. So I encourage everyone to check out the book Race After Technology,
386
00:50:54.494 --> 00:50:55.394
and her talk,
387
00:50:55.775 --> 00:51:22.575
her keynote, which I'll link to in the show notes. Alright. Well, thank you both for taking the time to join me today and discuss the work that you've been doing with DASK and building Coiled as a business around that. DASK is a project that I've been keeping a close eye on for a number of years now, and so I appreciate all the time and energy that you each put into the open source work and the business that you're building on top of it. I look forward to seeing where it takes you, and I hope you enjoy the rest of your day. Yeah. You too, Spence. Thanks a lot.
388
00:51:25.454 --> 00:51:33.900
Thank you for listening. Don't forget to check out our other show, the Data Engineering Podcast at data engineering podcast dot com for the latest on modern data management.
389
00:51:34.760 --> 00:51:36.700
And visit the site of python podcast.com
390
00:51:37.240 --> 00:51:40.700
to subscribe to the show, sign up for the mailing list, and read the show notes.
391
00:51:41.000 --> 00:51:45.535
And if you've learned something or tried out a project from the show, then tell us about it. Email host@podcastinit.com
392
00:51:46.795 --> 00:51:47.695
with your story.
393
00:51:48.420 --> 00:51:52.599
To help other people find the show, please leave a review on Itunes and tell your friends and coworkers.