#439: Pixi, A Fast Package Manager

WEBVTT
00:00:00.001 --> 00:00:04.340
In this episode, we have Wolf Volprecht and Ruben Arch from the Pixie Project here to talk about
00:00:04.340 --> 00:00:10.320
Pixie, a high-performance package manager for Python and other languages that actually manages
00:00:10.320 --> 00:00:15.500
Python itself too. They have a lot of interesting ideas on where Python packaging should go,
00:00:15.500 --> 00:00:20.440
and they're putting their time and effort behind them. Will Pixie become your next package manager?
00:00:20.440 --> 00:00:27.600
Listen in to find out. This is Talk Python in May, episode 439, recorded October 19th, 2023.
00:00:27.600 --> 00:00:46.900
Welcome to Talk Python in May, a weekly podcast on Python. This is your host, Michael Kennedy.
00:00:46.900 --> 00:00:51.980
Follow me on Mastodon, where I'm @mkennedy, and follow the podcast using @talkpython,
00:00:51.980 --> 00:00:57.480
both on fosstodon.org. Keep up with the show and listen to over seven years of past
00:00:57.480 --> 00:01:04.140
episodes at talkpython.fm. We've started streaming most of our episodes live on YouTube. Subscribe to
00:01:04.140 --> 00:01:09.700
our YouTube channel over at talkpython.fm/youtube to get notified about upcoming shows and
00:01:09.700 --> 00:01:16.940
be part of that episode. This episode is sponsored by Posit Connect from the makers of Shiny. Publish,
00:01:16.940 --> 00:01:22.080
share, and deploy all of your data projects that you're creating using Python. Streamlit, Dash,
00:01:22.200 --> 00:01:28.980
Shiny, Bokeh, FastAPI, Flask, Reports, Dashboards, and APIs. Posit Connect supports all of them.
00:01:28.980 --> 00:01:34.660
Try Posit Connect for free by going to talkpython.fm/posit, P-O-S-I-T.
00:01:34.660 --> 00:01:42.280
And it's brought to you by Python Tutor. Visualize your Python code step-by-step to understand just
00:01:42.280 --> 00:01:49.020
what's happening with your code. Try it for free and anonymously at talkpython.fm/python-tutor.
00:01:50.360 --> 00:01:55.060
Wolf, Ruben, welcome to Talk Python To Me. Hello. Thanks for having us. Yeah, it's great to have
00:01:55.060 --> 00:02:01.920
you here. We're going to dive into packaging once again. And we've talked about packaging a couple
00:02:01.920 --> 00:02:07.100
times over the last few months. It's a super interesting topic. And there are these times
00:02:07.100 --> 00:02:12.880
where it seems like there's a fixed way and everyone kind of agrees like this is how you do things.
00:02:12.880 --> 00:02:17.680
For example, I think Flask and Django have kind of been web frameworks for a long time. Then all of a
00:02:17.680 --> 00:02:22.360
sudden, you know, a thousand flowers bloom and there's a bunch of new ideas in the web space.
00:02:22.360 --> 00:02:27.000
I think that was driven by async and the typing stuff. And a bunch of people said, well, let's try
00:02:27.000 --> 00:02:31.880
new things now that we have these new ideas. And the other frameworks were more stable, couldn't make
00:02:31.880 --> 00:02:36.160
those adjustments. And I think people are just, you know, we're kind of at one of these explosion
00:02:36.160 --> 00:02:40.860
points of different ideas and different experiments in packaging. What do you all think?
00:02:41.020 --> 00:02:45.320
Yeah, that's an interesting way to put it. I think we definitely see a lot of interest in package
00:02:45.320 --> 00:02:51.020
management these days and new ideas being explored. But I also think that we're definitely standing on
00:02:51.020 --> 00:02:54.540
the shoulders of giants. So kind of similar to what you just described with the web frameworks,
00:02:54.540 --> 00:03:00.440
where actually, I think we are taking a lot of inspiration from multiple different ecosystems that
00:03:00.440 --> 00:03:05.200
are out there and try to kind of synthesize the best ideas into our tools.
00:03:05.400 --> 00:03:07.820
Yeah, you got some interesting ideas for sure. Ruben?
00:03:07.820 --> 00:03:13.220
I cannot really add to that anymore. I'm standing on the shoulders of giants like, whoa.
00:03:13.220 --> 00:03:14.860
So yeah, absolutely.
00:03:14.860 --> 00:03:15.600
But I think we'll go into that.
00:03:15.600 --> 00:03:20.480
Yeah, we sure will. Now, before we get into the topics, let's just do a quick introduction for folks
00:03:20.480 --> 00:03:25.280
who don't know you. I feel like this is a really interesting coincidence because the very last
00:03:25.280 --> 00:03:30.860
previous show that I did was with Sylvan and Jeremy, a bunch of folks from QuantStack. And,
00:03:31.060 --> 00:03:35.520
you know, just out of coincidence, like I said, your colleagues, right? So,
00:03:35.520 --> 00:03:37.300
Wolf, let's start with you. A little background on you.
00:03:37.300 --> 00:03:42.300
I did work at QuantStack for quite a while. And it's also where my journey with package management
00:03:42.300 --> 00:03:49.720
began. But maybe just taking one more step back, I studied in Zurich and I actually graduated in
00:03:49.720 --> 00:03:51.780
robotics there with a master's degree.
00:03:51.780 --> 00:03:52.340
Yeah.
00:03:52.340 --> 00:03:53.300
That's awesome.
00:03:53.300 --> 00:03:58.820
I had some fun times. I was also working with Disney Research on like a little robot that was
00:03:58.820 --> 00:04:05.340
drawing images in the sand and these kind of fun things. But at QuantStack, we were doing a lot of
00:04:05.340 --> 00:04:11.660
scientific computing stuff, initially trying to like reimplement NumPy in C++, which is a library
00:04:11.660 --> 00:04:17.660
called Xtensor and always doing a lot of package management and mostly in the CondaForge and Conda
00:04:17.660 --> 00:04:24.560
ecosystem. And Conda at some point became really slow and CondaForge became really large. And that led me to
00:04:24.560 --> 00:04:31.460
kind of experiment with new things, which resulted in Mamba. And then I got really lucky and had the
00:04:31.460 --> 00:04:37.280
opportunity to create my own little startup around more of these package management ideas, which is the
00:04:37.280 --> 00:04:43.000
current company called Prefix. And we'll dive more into Pixie and all these new things that we're doing,
00:04:43.000 --> 00:04:43.840
I think, later on.
00:04:43.840 --> 00:04:49.320
Yeah, that's a lot of interesting stuff. What language do you program a robot that writes in the sand in?
00:04:49.400 --> 00:04:53.960
It's always a mix of Python and C++. So I think I stuck to that up until now.
00:04:53.960 --> 00:04:58.200
Yeah, yeah, it sounds like it. Ruben, what's your story? Tell people a bit about yourself.
00:04:58.200 --> 00:05:06.540
Yeah, so I also started in robotics. I did a Megatronics engineering degree. And while working in robotics,
00:05:06.540 --> 00:05:13.880
I started at my previous company, Smart Robotics. And there we were building the new modern AI-driven
00:05:13.880 --> 00:05:19.800
robots. So that also involves a lot of deep learning packages and stuff like that. And that is kind of
00:05:19.800 --> 00:05:26.760
how I got into these package management solutions. And we started using Comba to package our C++ and our
00:05:26.760 --> 00:05:33.240
Python stuff and to make it easy to use in these virtual environments where we combine those packages.
00:05:33.800 --> 00:05:39.480
And it all was made easier by Mamba, which was built by Wolf. So that's how we got in touch.
00:05:39.480 --> 00:05:44.860
And later on, I moved to Wolf's company. So that's why I'm here now.
00:05:44.860 --> 00:05:47.820
Excellent. So you're a prefix dev as well.
00:05:47.820 --> 00:05:48.260
Yes.
00:05:48.260 --> 00:05:48.680
Awesome.
00:05:48.680 --> 00:05:50.260
I'm an expert dev there.
00:05:50.260 --> 00:05:50.500
Yeah.
00:05:50.500 --> 00:05:56.740
Cool. Well, I guess let's start with a little bit of maybe setting the stage. So you all talked about
00:05:56.740 --> 00:06:03.220
Conda and Conda Forge and really relying on that for a while and then wanting better performance,
00:06:03.220 --> 00:06:08.040
some other features we're going to talk about as well. But give us a quick background for those
00:06:08.040 --> 00:06:13.220
maybe non-data scientists or people who are not super into it. What is Conda and what is Conda Forge
00:06:13.220 --> 00:06:15.920
and the relationship of those things? Who wants to take that?
00:06:15.920 --> 00:06:23.820
Conda is, generally speaking, a package manager. That's all it is. Actually has nothing specific to AI,
00:06:24.140 --> 00:06:29.760
ML, data science, et cetera. But most people associate it with Python and machine learning,
00:06:29.760 --> 00:06:35.780
let's say. And Conda is written in Python and it's like, I don't know, 10 or 15 years old.
00:06:35.780 --> 00:06:43.640
And it kind of comes out of an era where there were no wheel files on PyPI and people had to
00:06:43.640 --> 00:06:46.980
compile stuff on their own machines. There was no good Windows support.
00:06:46.980 --> 00:06:50.760
Right. You can't use this. Where's your Fortran compiler? Come on.
00:06:50.760 --> 00:06:51.640
Yeah, exactly.
00:06:51.640 --> 00:06:52.700
What year is this again?
00:06:53.720 --> 00:06:55.700
And you need your GCC, et cetera.
00:06:55.700 --> 00:06:56.040
Yeah.
00:06:56.040 --> 00:07:02.940
That's kind of when Conda was born. And I think it really was one of those early tools that
00:07:02.940 --> 00:07:06.800
tried something with binary package management cross-platform. So basically,
00:07:06.800 --> 00:07:13.540
Conda allowed you to install Python and a bunch of Python packages that needed compiled extensions
00:07:13.540 --> 00:07:18.700
like NumPy, SciPy, et cetera. And it kind of comes out of this Travis Olyphant universe
00:07:18.700 --> 00:07:21.360
of scientific Python tools.
00:07:21.360 --> 00:07:23.240
Yeah. He's made a huge impact for sure.
00:07:23.240 --> 00:07:30.280
Yeah. But for us, sort of the key feature is just that it's like a cross-platform generic package
00:07:30.280 --> 00:07:36.240
manager that you can actually use for any language. So you can also create Conda packages for R.
00:07:36.480 --> 00:07:53.860
And there are actually quite a few R packages on Conda for us, let's say. And you can also do Julia or Rust, et cetera. So there's a lot of possibility and potential. I think it also kind of hits a sweet spot where Conda is really not a language specific package manager at the same time cross-platform. Because usually what you have is you have either like some sort of like Windows package manager or Linux package manager like apt-get or DNF on Fedora. Or you have like a language specific package manager like pip.
00:07:53.860 --> 00:08:11.740
Or Julia has package.jl or I don't know, R has CRAN, et cetera. And so Conda kind of sits at the crossroads of those two where it's not language specific and also cross-platform. And I think that makes it like really interesting. And then maybe I can also talk a little bit about Conda for it because I think that's a lot of things.
00:08:11.740 --> 00:08:41.720
Conda for it because I think that's a lot of things.
00:08:41.720 --> 00:09:11.700
Conda for it because I think that's a lot of things.
00:09:11.700 --> 00:09:23.620
like an SDK and other like DLLs that we need from the operating system. But everything above is like managed by Conda or Mamba or Pixie. So all of these tools work with on the base of the same packages.
00:09:23.620 --> 00:09:48.820
And that starts at like Bzip2 or Zlib, like these low-level compression libraries, OpenSSL and then up to Python. And then you can also get Qt, which is a graphical user interface library, which is written in C++. And applications that are building on top of Qt. So like, for example, physics simulation engines and stuff like this. So and you also get Qt and lots of libraries like this.
00:09:48.820 --> 00:10:04.660
All is not bound to like a specific operating system in that sense. And that makes it pretty, pretty nice. For example, also in CI, when you want to test your own software and stuff like this, you can use the same commands to set up basically the same packages across different platforms.
00:10:04.740 --> 00:10:17.080
Yeah, nice. So kind of like what Wheels did for pip and PyPI. Conda was way ahead of that game, right? But with a harder challenge, because it wasn't just Python packages. It was all these different ones, right?
00:10:17.080 --> 00:10:29.240
Yeah, including Python itself. So that's also one of the things that people sometimes maybe not realize, but Python itself is actually properly packaged on CondaForge and installable via Conda or Mamba or Pixie.
00:10:29.240 --> 00:10:33.520
Ruben, any thing you want to add to that before we start talking about what you all are creating?
00:10:33.520 --> 00:10:49.520
Yeah, so from my history, it's like this multi-platform stuff is less used in robotics, because a lot of the stuff is still running in Linux, but it moved it from the ability to run it only on Ubuntu to any version you want.
00:10:49.520 --> 00:11:14.280
And you could install any version of the robotic software you're running on, like any version of Ubuntu. So where we were locked, not just to Linux, but locked to a distribution of Linux, we were now like completely unbound and the developers can set up their own environments, which is just really powerful for the user itself. That brought it back into our company in a much better way.
00:11:14.280 --> 00:11:21.980
That's excellent. I'm always blown away at how much traffic these package managers have, how much bandwidth they use and things like that.
00:11:21.980 --> 00:11:22.240
Yeah.
00:11:22.240 --> 00:11:27.400
Who's hosting CondaForge and where you get that stuff from?
00:11:27.400 --> 00:11:39.720
Currently, CondaForge is entirely hosted by Anaconda.org. We do have a couple of mirrors available, but they are not really used. But one of the more exciting mirrors that we have is on GitHub itself.
00:11:39.720 --> 00:12:00.620
GitHub has this GitHub packages feature and we are using an OCI registry where you would usually put your like Docker containers and stuff like that. We upload all the packages there just as a backup. And we're planning to make it usable as well. So that would be nice for like your own GitHub actions and stuff because they could just like take the packet from sort of GitHub internally.
00:12:00.620 --> 00:12:03.720
Just write down the server rack in the data center.
00:12:03.720 --> 00:12:04.640
Yeah, exactly.
00:12:04.640 --> 00:12:06.920
Keep it local. Always good to be local.
00:12:06.920 --> 00:12:07.300
Yep.
00:12:07.400 --> 00:12:31.200
Okay. I want to focus mostly on Pixie for a conversation because I think that's got a lot of excitement. Maybe we'll get some time to talk about Mamba and other things as well. But yeah, let's you all wrote this interesting announcement entitled Let's Stop Dependency Hell talking about Pixie here. I think we can sort of talk through some of the ideas you laid out there and that'll give people a good idea of what this is all about.
00:12:31.280 --> 00:12:31.420
Yeah.
00:12:31.420 --> 00:12:31.460
Yeah.
00:12:31.460 --> 00:12:31.760
Yeah.
00:12:31.760 --> 00:13:01.260
Yeah. So first of all, let's start with some of the problems you're trying to solve here. So say we've all experienced issues with reproducibility and dependency management. I will tell you just yesterday. And if it was later in the day for me, it would probably be today. I'm running into a problem where with my courses website, where I try to install both the developer dependencies and the production dependencies. And it it's like this one requires greater than this dependency. And this one requires less than that dependency.
00:13:01.260 --> 00:13:21.960
You can't install it. You can't install it. I'm like, well, how am I supposed to do this? Like, I'd rather have it shaky than impossible. So, you know, it's dependency challenges are all too present for me. But yeah, let's let's maybe you can lay out some of the some of the ideas like what you had in mind when you're talking about reproducibility and challenges here.
00:13:21.960 --> 00:13:38.080
Yeah, I think you're not alone, first of all. So a lot of people have these kind of problems. And it's also not only in the Python world, let's say, but I think it's maybe a bit more pronounced in the Python world, just because there are so many packages and the way that package management in the Python world works.
00:13:38.200 --> 00:13:43.400
Yeah, I feel like we can always look over the JavaScript. People feel a little bit better, but it's still a challenge for us.
00:13:43.400 --> 00:14:13.380
That's true.
00:14:13.380 --> 00:14:18.380
Yeah, I feel like we can always look over the way that we can always look over the Python world.
00:14:18.380 --> 00:14:19.380
We can always look over the Python world, but we can always look over the Python world.
00:14:19.380 --> 00:14:20.380
We can always look over the Python world, but we can always look over the Python world.
00:14:20.380 --> 00:14:23.380
We can always look over the Python world, but we can always look over the Python world.
00:14:23.380 --> 00:14:24.380
And we can always look over the Python world.
00:14:24.380 --> 00:14:26.380
We can always look over the Python world.
00:14:26.380 --> 00:14:27.380
We can always look over the Python world.
00:14:27.380 --> 00:14:29.380
And we can always look over the Python world.
00:14:29.380 --> 00:14:31.380
We can always look over the Python world.
00:14:31.380 --> 00:14:33.380
We can always look over the Python world.
00:14:33.380 --> 00:14:35.380
We can always look over the Python world.
00:14:35.380 --> 00:14:37.380
And we can always look over the Python world.
00:14:37.380 --> 00:14:39.380
We can always look over the Python world.
00:14:39.380 --> 00:14:40.380
We can always look over the Python world.
00:14:40.380 --> 00:14:41.380
We can always look over the Python world.
00:14:41.380 --> 00:14:42.380
We can always look over the Python world.
00:14:42.380 --> 00:14:43.380
We can always look over the Python world.
00:14:43.380 --> 00:14:44.380
We can always look over the Python world.
00:14:44.380 --> 00:14:45.380
We can always look over the Python world.
00:14:45.380 --> 00:14:46.380
So it's not only Pixie.
00:14:46.380 --> 00:14:50.380
There's also one thing called Rattler build, which is actually building the conna packages.
00:14:50.380 --> 00:14:51.380
And there is another.
00:14:51.380 --> 00:14:57.380
And then we have the back end of our website prefix.dev, which is also written in Rust and also uses Rattler underneath.
00:14:57.380 --> 00:14:59.380
So that's really nice for us.
00:14:59.380 --> 00:15:00.380
And the big win.
00:15:02.380 --> 00:15:11.380
This portion of Talk Python To Me is brought to you by Posit, the makers of Shiny, formerly RStudio, and especially Shiny for Python.
00:15:11.380 --> 00:15:12.380
Let me ask you a question.
00:15:12.380 --> 00:15:14.380
Are you building awesome things?
00:15:14.380 --> 00:15:15.380
Of course you are.
00:15:15.380 --> 00:15:16.380
You're a developer or data scientist.
00:15:16.380 --> 00:15:17.380
That's what we do.
00:15:17.380 --> 00:15:19.380
And you should check out Posit Connect.
00:15:19.380 --> 00:15:26.380
Posit Connect is a way for you to publish, share, and deploy all the data products that you're building using Python.
00:15:26.380 --> 00:15:29.380
People ask me the same question all the time.
00:15:29.380 --> 00:15:32.380
Michael, I have some cool data science project or notebook that I built.
00:15:32.380 --> 00:15:35.380
How do I share it with my users, stakeholders, teammates?
00:15:35.380 --> 00:15:40.380
Do I need to learn FastAPI or Flask or maybe Vue or React.js?
00:15:40.380 --> 00:15:41.380
Hold on now.
00:15:41.380 --> 00:15:46.380
Those are cool technologies and I'm sure you'd benefit from them, but maybe stay focused on the data project.
00:15:46.380 --> 00:15:49.380
Let Posit Connect handle that side of things.
00:15:49.380 --> 00:15:53.380
With Posit Connect, you can rapidly and securely deploy the things you build in Python.
00:15:53.380 --> 00:16:00.380
Streamlit, Dash, Shiny, Bokeh, FastAPI, Flask, Quadro, Reports, Dashboards, and APIs.
00:16:00.380 --> 00:16:02.380
Posit Connect supports all of them.
00:16:02.380 --> 00:16:08.380
And Posit Connect comes with all the bells and whistles to satisfy IT and other enterprise requirements.
00:16:08.380 --> 00:16:12.380
Make deployment the easiest step in your workflow with Posit Connect.
00:16:12.380 --> 00:16:18.380
For a limited time, you can try Posit Connect for free for three months by going to talkpython.fm/posit.
00:16:18.380 --> 00:16:23.380
That's talkpython.fm/posit.
00:16:23.380 --> 00:16:25.380
The link is in your podcast player show notes.
00:16:25.380 --> 00:16:29.380
Thank you to the team at Posit for supporting Talk Python.
00:16:29.380 --> 00:16:37.380
If I wanted to stick with, say, Conda, could I still use Rattler build and then somehow upload that to CondaForge, something along those lines?
00:16:37.380 --> 00:16:38.380
Okay.
00:16:38.380 --> 00:16:48.380
So, you can totally, like, that's kind of the baseline sort of commonality between all of these tools is that we are sharing the same sort of Conda packages and the same metadata.
00:16:48.380 --> 00:16:52.380
So, we definitely want to be 100% compatible package-wise with Conda for now.
00:16:52.380 --> 00:16:53.380
Excellent.
00:16:53.380 --> 00:17:01.380
We might have features later on, but we, like, we don't, like, we want to go through, like, Conda as a project has also, like, become much more community oriented.
00:17:01.380 --> 00:17:04.380
And there is, like, a process called Conda Enhancement Proposals.
00:17:04.380 --> 00:17:07.380
And we have already written a few of those.
00:17:07.380 --> 00:17:10.380
So, there are many ideas, but we can talk about that later.
00:17:10.380 --> 00:17:13.380
Trying to improve the overall system instead of overthrowing it.
00:17:13.380 --> 00:17:14.380
Yes, yes.
00:17:14.380 --> 00:17:15.380
Yeah.
00:17:15.380 --> 00:17:20.380
Like, we would love to, like, improve the entirety of, like, Conda packages, CondaForge, and all of this.
00:17:20.380 --> 00:17:22.380
Like, that's our main dream.
00:17:22.380 --> 00:17:28.380
So, and then with some of the low-level tools in Rattler and with Pixie, we're kind of combining a bunch of tools that already existed.
00:17:28.380 --> 00:17:32.380
And one thing essential for reproducibility is that you have log files.
00:17:32.380 --> 00:17:38.380
So, at the point where you are sort of resolving your dependencies, we are also writing them into a log file.
00:17:38.380 --> 00:17:42.380
And that's, like, something known from Poetry, from NPM, Yarn.
00:17:42.380 --> 00:17:44.380
Cargo also has it.
00:17:44.380 --> 00:17:48.380
And there's also a Conda project that's called CondaLock that writes log files.
00:17:48.380 --> 00:17:56.380
And so, we have adopted the same format that CondaLock uses, which is a YAML-based log file format, and implemented in Rattler.
00:17:56.380 --> 00:17:58.380
And we are exposing it and using it in Pixie.
00:17:58.380 --> 00:18:03.380
So, anytime you, like, add a new dependency to your project, we write it in a log file.
00:18:03.380 --> 00:18:13.380
And we make sure that, like, you can install the same packages, the same set of packages, the same versions, and SHA hashes in, like, the future.
00:18:13.380 --> 00:18:19.380
And the other part about reproducibility, and that's more on the repository side, is that CondaForge never deletes old packages.
00:18:19.380 --> 00:18:25.380
So, that's similar to PyPI, but not really this, like, it's different in a lot of, like, Linux distributions.
00:18:25.380 --> 00:18:29.380
But, with PyPI, it's also the case that, you know, old versions are just kept around.
00:18:29.380 --> 00:18:33.380
Do you ever worry that that might not be sustainable?
00:18:33.380 --> 00:18:44.380
Like, it's fine now, in 20 years, like, we cannot pay for the thing from 20 years, like, it's, we just can't get enough donations to support Flask 0.1.
00:18:44.380 --> 00:18:45.380
We just can't.
00:18:45.380 --> 00:18:46.380
It's out.
00:18:46.380 --> 00:18:49.380
That's the problem of the person that uses Flask 0.1, right?
00:18:49.380 --> 00:18:51.380
Like, that's not the problem of the repository.
00:18:51.380 --> 00:18:53.380
I think we're just making sure that...
00:18:53.380 --> 00:18:54.380
No, no, sure.
00:18:54.380 --> 00:18:55.380
For sure.
00:18:55.380 --> 00:19:02.380
You could still run it, and you should probably sandbox it like crazy so that there are no, like, zero days that could affect your system.
00:19:02.380 --> 00:19:09.380
You do have some things that are, like, self-hosted Conda capabilities that maybe we'll get a chance to talk about.
00:19:09.380 --> 00:19:16.380
Like, theoretically, you could download these and save as a company or an organization or a researcher.
00:19:16.380 --> 00:19:18.380
You could get the ones that actually count for you, right?
00:19:18.380 --> 00:19:19.380
Yeah.
00:19:19.380 --> 00:19:21.380
Like, you mean only have a subset of the packages that you need?
00:19:21.380 --> 00:19:25.380
Yeah, say I'm using 50 packages with the transitive closure of everything I'm using.
00:19:25.380 --> 00:19:32.380
And so I'm just going to make sure I have every version of those on Dropbox or on a hard drive I put away somewhere.
00:19:32.380 --> 00:19:39.380
It's actually pretty funny, because what you create on your local system is a cache of all the packages that you ever used.
00:19:39.380 --> 00:19:43.380
And you could activate that cache as a channel, like what Condor Forge is.
00:19:43.380 --> 00:19:46.380
You could make your own channel of all your packages locally.
00:19:46.380 --> 00:19:50.380
This is something we used when the internet went down in our company.
00:19:50.380 --> 00:19:54.380
And we still needed to share packages with each other and needed to make our environments.
00:19:54.380 --> 00:19:58.380
And just some people would spin up their own channel and you could use it from there.
00:19:58.380 --> 00:19:59.380
It's just a different URL.
00:19:59.380 --> 00:20:00.380
Yeah, that's awesome.
00:20:00.380 --> 00:20:01.380
Cool.
00:20:01.380 --> 00:20:03.380
I derailed your...
00:20:03.380 --> 00:20:04.380
There, Wolf.
00:20:04.380 --> 00:20:12.380
No, but yeah, I think like log files are the basis for reproducibility and then the fact that packages are never deleted.
00:20:12.380 --> 00:20:19.380
I think that's something that like log files make a little bit like a Docker container sort of.
00:20:19.380 --> 00:20:19.380
Yeah.
00:20:19.380 --> 00:20:20.380
Yeah.
00:20:20.380 --> 00:20:22.380
Because you know exactly what's in your software environment.
00:20:22.380 --> 00:20:26.380
We don't control the outside and we don't do sandboxing as of now.
00:20:26.380 --> 00:20:28.380
But that's kind of the way we think about log files.
00:20:28.380 --> 00:20:37.380
And it just makes it very convenient also to ship basically that log file plus the Pixie Tommel and stuff to your coworker and they can just run it.
00:20:37.380 --> 00:20:41.380
And we also resolve for multiple operating systems at the same time.
00:20:41.380 --> 00:20:51.380
So you can say, you can specify in your Pixie Tommel if you want Linux, macOS and Windows and we resolve everything at the same time in parallel with async Rust code and stuff like this.
00:20:51.380 --> 00:20:52.380
Nice.
00:20:52.380 --> 00:20:53.380
So it's pretty fast and nice.
00:20:53.380 --> 00:20:56.380
And yeah, the idea is that you can send it to your coworker.
00:20:56.380 --> 00:21:01.380
They can just do a Pixie run start, which will just give them everything they need and have them up and going.
00:21:01.380 --> 00:21:02.380
Really cool.
00:21:02.380 --> 00:21:11.380
In your announcement for Pixie, one of the things you said is you're looking for the convenience of modern package managers such as Cargo.
00:21:11.380 --> 00:21:12.380
Yeah.
00:21:12.380 --> 00:21:16.380
What's different than say pip and PyPI versus Cargo?
00:21:16.380 --> 00:21:18.380
Like when you say that, what are these new features you're like?
00:21:18.380 --> 00:21:20.380
I wish we had this already.
00:21:20.380 --> 00:21:21.380
We don't.
00:21:21.380 --> 00:21:22.380
So I'm going to build it.
00:21:22.380 --> 00:21:30.380
I think one thing that's just really nice with Cargo is, and that also attracts so many contributors to Rust projects, at least like that's the way I feel about it.
00:21:30.380 --> 00:21:34.380
It's that it's so easy to just say Cargo run whatever.
00:21:34.380 --> 00:21:39.380
And it's most of the time works and you just do Cargo build and like it builds.
00:21:39.380 --> 00:21:42.380
And that's the experience that we want to recreate with Pixie.
00:21:42.380 --> 00:21:47.380
And Cargo also comes with log files and Cargo just does this pretty nicely.
00:21:47.380 --> 00:21:57.380
I mean, there are some peculiarities about how Rust builds packages or thinks about dependencies where the result is pretty different, let's say from like Python ecosystem and stuff like this.
00:21:57.380 --> 00:22:01.380
But the baseline experience is definitely what we're also striving for.
00:22:01.380 --> 00:22:05.380
And part of the problem is maybe also that pip is not managing Python.
00:22:05.380 --> 00:22:11.380
So you always have that a little bit of a chicken and egg problem where you need to get Python first to be able to run pip.
00:22:11.380 --> 00:22:14.380
And with Pixie, you don't have that problem because we also manage Python.
00:22:14.380 --> 00:22:18.380
So you can specify in your Pixie what version of Python you want.
00:22:18.380 --> 00:22:20.380
You get it on Windows, macOS and Linux in the same way.
00:22:20.380 --> 00:22:22.380
And everything is just one command.
00:22:22.380 --> 00:22:26.380
And it's everything is also locked in your log file, etc.
00:22:26.380 --> 00:22:30.380
So that's kind of, yeah, we just control a bit more than pip.
00:22:30.380 --> 00:22:33.380
And I think that's what's giving us some power.
00:22:33.380 --> 00:22:41.380
And then pip also, like as far as I'm aware, and we recently had discussions with Python or Python package management developers.
00:22:41.380 --> 00:22:45.380
They haven't come up with a log file format that works for everyone yet.
00:22:45.380 --> 00:22:50.380
So Poetry has their own implementation and a bunch of other tools maybe have their own implementations as well.
00:22:50.380 --> 00:22:53.380
Right. There's the pip lock from pipenv and others.
00:22:53.380 --> 00:22:53.380
Yeah.
00:22:53.380 --> 00:22:55.380
We're also kind of working on that.
00:22:55.380 --> 00:23:02.380
I don't know if you saw that, but we just announced another tool that's also low level, sort of on the same level as Rattler, but it's called RIP.
00:23:02.380 --> 00:23:05.380
And it deals with Python resolving and the wheel files.
00:23:05.380 --> 00:23:06.380
Yeah.
00:23:06.380 --> 00:23:14.380
And so we want to kind of cross over those two worlds where we resolve the counter packages first, then we resolve the Python packages after.
00:23:14.380 --> 00:23:22.380
And we stick everything into the same log file that will for now be similar to the, yeah, basically based on the counter log format, which is a YAML file.
00:23:22.380 --> 00:23:23.380
Interesting.
00:23:23.380 --> 00:23:30.380
So this rip, I'm familiar that I didn't necessarily in my mind, tie it back to Pixie, but would that allow you to, could you mix and match?
00:23:30.380 --> 00:23:37.380
Like some stuff comes off Conda Forge and some stuff comes off of PIPI, but you express that in your dependency file?
00:23:37.380 --> 00:23:38.380
Yeah.
00:23:38.380 --> 00:23:42.380
Like there are parts of the semantics aren't yet figured out, let's say.
00:23:42.380 --> 00:23:43.380
Yeah.
00:23:43.380 --> 00:23:47.380
But the idea is definitely that you can install Python and NumPy, for example, from Conda Forge.
00:23:47.380 --> 00:23:49.380
And then, I don't know, scikit-learn from PIPI.
00:23:49.380 --> 00:23:52.380
Like that's maybe not the example of how you would use it, but.
00:23:52.380 --> 00:23:53.380
Yeah, of course.
00:23:53.380 --> 00:23:53.380
Right.
00:23:53.380 --> 00:23:55.380
Maybe you do one of the web frameworks, right?
00:23:55.380 --> 00:23:59.380
Like FastAPI versus some of the scientific stuff from Conda.
00:23:59.380 --> 00:23:59.380
Yeah.
00:23:59.380 --> 00:23:59.380
Yeah.
00:23:59.380 --> 00:24:00.380
Yeah.
00:24:00.380 --> 00:24:02.380
I find the, at least the official Conda stuff.
00:24:02.380 --> 00:24:06.380
Sometimes the framework, certain frameworks are a little bit behind and there are situations
00:24:06.380 --> 00:24:11.380
where having the latest one within an hour matters a lot.
00:24:11.380 --> 00:24:14.380
You know, for example, hey, it turns out, theoretically, it's not real.
00:24:14.380 --> 00:24:19.380
It turns out that say Flask has a super bad remote code execution problem.
00:24:19.380 --> 00:24:24.380
We just found out that if you send like a cat emoji as part of the URL, it's all over.
00:24:24.380 --> 00:24:26.380
So patch it now, right?
00:24:26.380 --> 00:24:30.380
Like you don't want to wait for that to like slowly get through some, you need that now, right?
00:24:30.380 --> 00:24:35.380
And PyPI, I find it's kind of the tip of the latest in that regard.
00:24:35.380 --> 00:24:37.380
I do agree to some extent.
00:24:37.380 --> 00:24:43.380
So it's like, we also found that a lot of, there are these no arch packages, like pure
00:24:43.380 --> 00:24:44.380
Python packages.
00:24:44.380 --> 00:24:48.380
And I think, and there's just way more packages on PyPI.
00:24:48.380 --> 00:24:51.380
And the turn of managing that on Conda Forge is a bit high.
00:24:51.380 --> 00:24:54.380
So that's also like, we have lots of reasons.
00:24:54.380 --> 00:24:58.380
And also in real world examples, we often find people mixing PyPI, pip and Conda.
00:24:58.380 --> 00:25:03.380
So that's why we're thinking like, we need proper sort of support for PyPI in our tool to
00:25:03.380 --> 00:25:05.380
make it really nice for Python developers.
00:25:05.380 --> 00:25:07.380
It would take it to another level for sure.
00:25:07.380 --> 00:25:12.380
And it would certainly make it stand out from what Conda does or what pip does, honestly.
00:25:12.380 --> 00:25:17.380
Conda, for example, they, there is a way to kind of like add some Python dependency or pip dependencies,
00:25:17.380 --> 00:25:22.380
but it's really just invoking pip as like a sub process and then installing some additional stuff into your environment.
00:25:22.380 --> 00:25:25.380
And it's not really nice, not really tightly integrated.
00:25:25.380 --> 00:25:31.380
And so we actually kind of took, did the work and wrote a resolver and rust.
00:25:31.380 --> 00:25:32.380
So SIT solver.
00:25:32.380 --> 00:25:38.380
And we've just extended it to also deal with Python or pip metadata, which is kind of what rip is.
00:25:38.380 --> 00:25:39.380
Yeah.
00:25:39.380 --> 00:25:45.380
So that's going to be very interesting just to figure out how to integrate those things and like really make them work nicely together.
00:25:45.380 --> 00:25:47.380
I want to talk about the ergonomics using Pixie.
00:25:47.380 --> 00:25:50.380
But first, maybe Ruben, you could address this first.
00:25:50.380 --> 00:25:58.380
But I opened this whole conversation with a thousand flowers blooming around the package management story.
00:25:58.380 --> 00:26:05.380
And I think for a long time, what people had seen was they're going to try to innovate within Python.
00:26:05.380 --> 00:26:11.380
So you install Python, you create your environment, and then like you have a different workflow with different tools.
00:26:11.380 --> 00:26:14.380
But some of the new ideas are starting to move to the outside.
00:26:14.380 --> 00:26:17.380
Like we'll also manage Python itself.
00:26:17.380 --> 00:26:21.380
If you say you want Python 3.10 and you only have 3.11 installed, we'll take care of that.
00:26:21.380 --> 00:26:27.380
And something built on Python has a real hard time installing Python because there's this chicken and egg probably needs it first, right?
00:26:27.380 --> 00:26:38.380
And it sounds like you all are taking that approach of we're going to be outside of Python, you know, built in Rust or any binary that just runs on its own would work to have a greater control, right?
00:26:38.380 --> 00:26:39.380
So yeah, I know.
00:26:39.380 --> 00:26:40.380
Just what are your thoughts on that?
00:26:40.380 --> 00:26:41.380
Yeah.
00:26:41.380 --> 00:26:46.380
So the one of the strong points is Pixie that you can install it as a standalone binary.
00:26:46.380 --> 00:26:53.380
So you have a simple script or you can even just download it and put it in your machine and then you can install whatever you want.
00:26:53.380 --> 00:26:55.380
So you're not limited to Python alone.
00:26:55.380 --> 00:26:59.380
And in a lot of cases, you want to mix a lot of stuff.
00:26:59.380 --> 00:27:07.380
So sometimes you need a specific version of SSH or sometime you need a specific version of open SSL or whatever that needs your package.
00:27:07.380 --> 00:27:21.380
And you would have these long lists of getting started to like, oh, you need to install this with APT or you need to install this with name anything, any other package manager, and then you can run pip install and then it should all work.
00:27:21.380 --> 00:27:35.380
And Pixie kind of moves it back to you have to have Pixie and you have to have the source code of the package that you're running or you're directly like using Pixie to install something.
00:27:35.380 --> 00:27:42.380
And then you're running the actual code that you're trying to run instead of going to read some kind of read me from a person on the internet.
00:27:42.380 --> 00:27:45.380
Yeah. And it's also pretty challenging for newcomers.
00:27:45.380 --> 00:27:46.380
The programming.
00:27:46.380 --> 00:27:48.380
This is really focused on making it easy.
00:27:48.380 --> 00:27:49.380
Yeah, exactly.
00:27:49.380 --> 00:27:50.380
I just want to run this.
00:27:50.380 --> 00:27:52.380
You're like, but what am I doing all this terminal stuff?
00:27:52.380 --> 00:27:53.380
Like, I just want to run.
00:27:53.380 --> 00:27:54.380
I wrote the program.
00:27:54.380 --> 00:27:55.380
I want it to go.
00:27:55.380 --> 00:27:56.380
Yeah.
00:27:56.380 --> 00:28:04.380
I feel like maybe that's part of why notebooks and that whole notebooks, Jupyter side of things is so popular because assuming somebody has created a server and got it started for you.
00:28:04.380 --> 00:28:06.380
Like you don't worry about those things, right?
00:28:06.380 --> 00:28:07.380
Yeah, exactly.
00:28:07.380 --> 00:28:07.380
Yeah.
00:28:07.380 --> 00:28:10.380
Let's talk about kind of the that beginner experience.
00:28:10.380 --> 00:28:24.380
If you have an example on your website somewhere where it just shows if you just check out a repository that's already been configured to use Pixie, it's just clone Pixie run to run start or something like that.
00:28:24.380 --> 00:28:25.380
Right.
00:28:25.380 --> 00:28:30.380
You don't have to create the environments and even that could potentially happen without Python, even on the machine initially.
00:28:30.380 --> 00:28:30.380
Right.
00:28:30.380 --> 00:28:31.380
Totally.
00:28:31.380 --> 00:28:31.380
Yeah.
00:28:31.380 --> 00:28:35.380
So funny part of Pixie is we Pixie itself is a Pixie project.
00:28:35.380 --> 00:28:43.380
So if we want to build Pixie, it is a Rust project, but we run Pixie run built in this case or Pixie install Pixie run install.
00:28:43.380 --> 00:28:52.380
So you kind of move everything back into the tasks in Pixie and you can you can run it using Pixie and Pixie will take care of your environment.
00:28:52.380 --> 00:28:53.380
Nice.
00:28:53.380 --> 00:28:53.380
Yeah.
00:28:53.380 --> 00:28:57.380
So basically, as I also said before, we're learning a lot, for example, from cargo.
00:28:57.380 --> 00:29:06.380
So we also have a single Pixie tommel file that kind of defines all of your dependencies, a bit of metadata about your project, and then you can define these tasks.
00:29:06.380 --> 00:29:12.380
And so like what we see on the screen is that we have a task that's called start and that just runs Python main dot PI.
00:29:12.380 --> 00:29:15.380
So that's pretty straightforward.
00:29:15.380 --> 00:29:21.380
But obviously, like you can go further, like you can have tasks that depend on other tasks and that we're learning a lot from.
00:29:21.380 --> 00:29:24.380
So there's a project called task file dot def.
00:29:24.380 --> 00:29:33.380
And we also want to integrate caching into these tasks so that if you like one task might download something on your system, like some assets that you need, like images and stuff.
00:29:33.380 --> 00:29:37.380
And if you already have them cached, then you don't need to re download them and these kind of things.
00:29:37.380 --> 00:29:41.380
So we're really like wanting to build a simple but powerful task system in there.
00:29:41.380 --> 00:29:44.380
And that benefits greatly from having these dependencies available.
00:29:44.380 --> 00:29:50.380
Because like in this case, what we see on the screen, we have two dependencies and one of those is Python 3.11.
00:29:50.380 --> 00:29:59.380
And that means the moment you run pixie run start, it will actually look at the log file and look at what you have in your local environment installed.
00:29:59.380 --> 00:30:03.380
And the environments are always local to the project, which is also a difference to call that number.
00:30:03.380 --> 00:30:07.380
So it will look into that environment and check if Python 3.11 is there.
00:30:07.380 --> 00:30:12.380
And if the version that you have in your environment corresponds to the one that's listed in the log file.
00:30:12.380 --> 00:30:21.380
If not, it will download the version and install it into your environment and like make sure that you have all the stuff that's necessary or listed to run your to run what you need.
00:30:21.380 --> 00:30:22.380
Nice.
00:30:22.380 --> 00:30:26.380
This portion of Talk Python To Me is brought to you by Python Tutor.
00:30:26.380 --> 00:30:31.380
Are you learning Python or another language like JavaScript, Java, C or C++?
00:30:31.380 --> 00:30:33.380
If so, check out Python Tutor.
00:30:33.380 --> 00:30:40.380
This free website lets you write code, run it and visualize what happens line by line as your code executes.
00:30:40.380 --> 00:30:45.380
No more messy print statements or fighting with the debugger to understand what code is doing.
00:30:45.380 --> 00:30:50.380
Python Tutor automatically shows you exactly what's going on step by step in an intuitive visual way.
00:30:50.380 --> 00:30:59.380
You'll see all the objects as they are represented in Python memory and how they are connected and potentially shared across variables over time.
00:30:59.380 --> 00:31:07.380
It's a great free tool to compliment what you're learning from books, YouTube videos and even online courses like the ones right here at Talk Python Training.
00:31:07.380 --> 00:31:12.380
In fact, I even used Python Tutor when creating our Python Memory Management and Tips course.
00:31:12.380 --> 00:31:17.380
It was excellent for showing just what's happening with references and containers in memory.
00:31:17.380 --> 00:31:19.380
Python Tutor is super easy to check out.
00:31:19.380 --> 00:31:24.380
Just visit talkpython.fm/python-tutor and click visualize code.
00:31:24.380 --> 00:31:29.380
It comes preloaded with an example and you don't even need an account to use it.
00:31:29.380 --> 00:31:33.380
Again, that's talkpython.fm/python-tutor to visualize your code for free.
00:31:33.380 --> 00:31:35.380
The link is in your podcast player show notes.
00:31:35.380 --> 00:31:38.380
Thank you to Python Tutor for sponsoring this episode.
00:31:38.380 --> 00:31:47.380
So for example, you got in your example Python 3.11 for some flexibility there on the very, very end.
00:31:47.380 --> 00:31:48.380
Yeah.
00:31:48.380 --> 00:31:53.380
Does that download a binary version or does it build from source or what happens when it needs that?
00:31:53.380 --> 00:31:54.380
Yeah.
00:31:54.380 --> 00:31:57.380
So typically, like Gonda is a binary package manager.
00:31:57.380 --> 00:31:59.380
So usually what you download is binary.
00:31:59.380 --> 00:32:00.380
Yeah.
00:32:00.380 --> 00:32:05.380
We are working on the source dependency capabilities where also Rackler build, what I mentioned before, is going to play a big role.
00:32:05.380 --> 00:32:12.380
Because the idea is that you can also run Pixie build at some point soon and that will build your Gonda package out of your Pixie project.
00:32:12.380 --> 00:32:21.380
But we would use the same capabilities to basically also allow you to get local dependencies and then build them ad hoc and put them into your environment.
00:32:21.380 --> 00:32:21.380
Yeah.
00:32:21.380 --> 00:32:28.380
So that comes back to the example you gave before with the problem that there's a huge share of a bug or something.
00:32:28.380 --> 00:32:35.380
And you would want to use a non-support, yeah, a version that's not shared around the world yet.
00:32:35.380 --> 00:32:40.380
So you need this GitHub link and that package you need to install.
00:32:40.380 --> 00:32:45.380
And that's something we still want to support through this local or URL based dependency.
00:32:45.380 --> 00:32:47.380
But for that, we first need to be able to build it.
00:32:47.380 --> 00:32:48.380
Yeah.
00:32:48.380 --> 00:32:50.380
Kind of like the Git plus on pip install.
00:32:50.380 --> 00:32:51.380
Yeah.
00:32:51.380 --> 00:32:52.380
Yeah, exactly.
00:32:52.380 --> 00:32:57.380
I found out where this little section was here where this Pixie is made for collaboration on your announcement
00:32:57.380 --> 00:33:01.380
where it just says Git clone some repo, Pixie run, start build, whatever.
00:33:01.380 --> 00:33:02.380
Yeah.
00:33:02.380 --> 00:33:04.380
Maybe just talk through like what happens there.
00:33:04.380 --> 00:33:08.380
Because if I don't even have Python, much less a virtual environment, much less the things installed.
00:33:08.380 --> 00:33:13.380
You know, if I try this at Python, if they just say clone this, go here, Python run.
00:33:13.380 --> 00:33:16.380
Like if you don't have Python, it'll just say Python, what is that?
00:33:16.380 --> 00:33:20.380
If you do have Python, it'll say, you know, FastAPI, what is that?
00:33:20.380 --> 00:33:21.380
Right?
00:33:21.380 --> 00:33:22.380
Like there's a lot of steps.
00:33:22.380 --> 00:33:23.380
Yeah.
00:33:23.380 --> 00:33:24.380
So that really simplifies.
00:33:24.380 --> 00:33:27.380
And that's kind of what I was talking about with the beginners as well.
00:33:27.380 --> 00:33:29.380
Like, you know, maybe speak to what's happening here.
00:33:29.380 --> 00:33:30.380
Yeah.
00:33:30.380 --> 00:33:36.380
So when you do Pixie run, it will create and you have nothing on your system, right?
00:33:36.380 --> 00:33:38.380
Except for Pixie and that repository.
00:33:38.380 --> 00:33:42.380
Then it's going to create a hidden folder inside of your project.
00:33:42.380 --> 00:33:43.380
That's called .pixie.
00:33:43.380 --> 00:33:47.380
Pixie and in there, it will install all of these tools that are dependencies of the project.
00:33:47.380 --> 00:33:50.380
So Python, NumPy, scikit-learn, whatever.
00:33:50.380 --> 00:33:54.380
And that like, and then when you do Pixie run, it will invoke.
00:33:54.380 --> 00:33:57.380
Actually, there's a thing called Dino task share, which we're using.
00:33:57.380 --> 00:34:02.380
And that's basically something like it looks like bash, but it also works in Windows, which
00:34:02.380 --> 00:34:03.380
is like the key feature here.
00:34:03.380 --> 00:34:04.380
Yeah.
00:34:04.380 --> 00:34:06.380
So that will sort of run the task.
00:34:06.380 --> 00:34:11.380
And in this case, like some task is probably defined inside of the Pixie Thomas and that
00:34:11.380 --> 00:34:12.380
might run something like Python.
00:34:12.380 --> 00:34:19.380
I don't know, start Flask or start Jupyter or, you know, whatever the developer desires
00:34:19.380 --> 00:34:20.380
to do.
00:34:20.380 --> 00:34:24.380
But the cool thing is that it will like in the background, activate the environment, like the
00:34:24.380 --> 00:34:27.380
virtual environment and use it to run your software.
00:34:27.380 --> 00:34:28.380
Yeah, that's really cool.
00:34:28.380 --> 00:34:31.380
And that, yeah, most of that kind of happens behind the scenes.
00:34:31.380 --> 00:34:36.380
So also with Conda, for example, or Mamba, it's usually multiple steps.
00:34:36.380 --> 00:34:42.380
So usually what you would do is you do like Mamba create my environment and then the environment
00:34:42.380 --> 00:34:45.380
will have some name and then you would need to do Mamba activate my environment.
00:34:45.380 --> 00:34:48.380
And then, then only you would be able to run stuff.
00:34:48.380 --> 00:34:53.380
And what you're running is also probably going to look more complicated than just typing Pixie
00:34:53.380 --> 00:34:55.380
run some task, which does all of that.
00:34:55.380 --> 00:34:56.380
Right.
00:34:56.380 --> 00:34:59.380
The some task is almost an alias for the actual run command, right?
00:34:59.380 --> 00:35:00.380
Yeah.
00:35:00.380 --> 00:35:01.380
Yeah.
00:35:01.380 --> 00:35:04.380
It could be something very complicated and it could also be multiple tasks that actually
00:35:04.380 --> 00:35:06.380
run in the background because they can depend on each other.
00:35:06.380 --> 00:35:07.380
Excellent.
00:35:07.380 --> 00:35:14.380
I really like that the virtual environment or all the binary configuration stuff is a sub
00:35:14.380 --> 00:35:16.380
directory of the project.
00:35:16.380 --> 00:35:18.380
That's always bothered me about Conda.
00:35:18.380 --> 00:35:25.380
If I go, I've got, I think I have about 260 GitHub repos on my, my GitHub profile and I check
00:35:25.380 --> 00:35:27.380
out other people's stuff and check it out.
00:35:27.380 --> 00:35:32.380
And so if I go just to my file system and I go in there, I'm like, I haven't messed with
00:35:32.380 --> 00:35:33.380
this for a year.
00:35:33.380 --> 00:35:34.380
Was that on the old computers on my laptop?
00:35:34.380 --> 00:35:35.380
Is on my mini?
00:35:35.380 --> 00:35:36.380
Like what, what was that on?
00:35:36.380 --> 00:35:37.380
I don't.
00:35:37.380 --> 00:35:39.380
So it, it could be, I haven't set it up or maybe I have.
00:35:39.380 --> 00:35:40.380
Right.
00:35:40.380 --> 00:35:44.380
And if I go there and I see there's a V and V folder or something along those lines, I'm
00:35:44.380 --> 00:35:47.380
like, Oh yeah, it might be out of date, but I definitely have done something with this
00:35:47.380 --> 00:35:48.380
here.
00:35:48.380 --> 00:35:49.380
I probably can run it.
00:35:49.380 --> 00:35:51.380
Whereas the Conda style, like you don't know.
00:35:51.380 --> 00:35:52.380
What did you name it?
00:35:52.380 --> 00:35:53.380
Yeah.
00:35:53.380 --> 00:35:55.380
If you have 200 of them, what is the right one?
00:35:55.380 --> 00:35:56.380
Do I activate it?
00:35:56.380 --> 00:36:02.380
And then also if something kind of goes haywire, it's like, you know, I'm just gonna RMRF that
00:36:02.380 --> 00:36:04.380
that folder and it's, it's out.
00:36:04.380 --> 00:36:07.380
Just, just recreate it on the new version of whatever.
00:36:07.380 --> 00:36:08.380
Right.
00:36:08.380 --> 00:36:10.380
But if it's somewhere else, you know, there's just like this, this disconnect.
00:36:10.380 --> 00:36:14.380
It's, I know there's like a command flagged override or something into like get Conda to
00:36:14.380 --> 00:36:16.380
put it locally, but defaults are powerful.
00:36:16.380 --> 00:36:17.380
Right.
00:36:17.380 --> 00:36:18.380
And I really like that.
00:36:18.380 --> 00:36:22.380
It's, it's like there and you can just blast away the dot pixie and, you know, start over
00:36:22.380 --> 00:36:23.380
if you need to.
00:36:23.380 --> 00:36:27.380
We also using the same tricks that Conda uses and a bunch of other package managers.
00:36:27.380 --> 00:36:32.380
So you can have these multiple environments, but they actually share the underlying files.
00:36:32.380 --> 00:36:37.380
So if you use the same Python 3.11 version and multiple environments, it's not like you
00:36:37.380 --> 00:36:38.380
don't duplicate those files.
00:36:38.380 --> 00:36:40.380
You don't lose a lot of storage, for example.
00:36:40.380 --> 00:36:41.380
That's nice.
00:36:41.380 --> 00:36:42.380
And the other thing that's really cool.
00:36:42.380 --> 00:36:45.380
I mean, Conda also gives you that, but you can have completely different Python versions
00:36:45.380 --> 00:36:47.380
in all of these environments.
00:36:47.380 --> 00:36:50.380
And it's, it's very like straightforward to use.
00:36:50.380 --> 00:36:53.380
Like you don't need to run it through containers or stuff like that.
00:36:53.380 --> 00:36:55.380
It's just like all in your system.
00:36:55.380 --> 00:36:57.380
And yeah, very nice and isolated.
00:36:57.380 --> 00:36:58.380
Yeah.
00:36:58.380 --> 00:37:03.380
So one thing that I ran across here that was pretty interesting while just researching this
00:37:03.380 --> 00:37:07.380
is you said Pixie and Conda like Nix are language agnostics.
00:37:07.380 --> 00:37:08.380
And I'm like, what is this Nix thing?
00:37:08.380 --> 00:37:09.380
Yeah.
00:37:09.380 --> 00:37:10.380
And that brought me over to Nix OS.
00:37:10.380 --> 00:37:11.380
What is, what is this?
00:37:11.380 --> 00:37:14.380
Nix basically is a functional package manager.
00:37:14.380 --> 00:37:15.380
Okay.
00:37:15.380 --> 00:37:19.380
It works with a functional programming language, which is kind of an interesting idea.
00:37:19.380 --> 00:37:23.380
And a lot of people that know Nix really love it.
00:37:23.380 --> 00:37:27.380
So we would like for Pixie to also be as loved as Nix is by Nix people.
00:37:27.380 --> 00:37:35.380
And basically what's nice about the functional programming language is that it kind of, you know, from the input, the output.
00:37:35.380 --> 00:37:39.380
So you can cache the function execution and you know, okay.
00:37:39.380 --> 00:37:43.380
Like if the function didn't change and the inputs didn't change, then the output is also not going to change.
00:37:43.380 --> 00:37:44.380
Right.
00:37:44.380 --> 00:37:45.380
You can cache the heck out of it.
00:37:45.380 --> 00:37:47.380
You can parallelize it so much and so on.
00:37:47.380 --> 00:37:48.380
Yeah.
00:37:48.380 --> 00:37:57.380
So that's kind of what, like, that's how I understand Nix is that basically you have a function that you execute to, let's say, get bash on your system or get Python on your system.
00:37:57.380 --> 00:38:06.380
And once you have executed that function for that specific Python version, you know, that you have, you know, Python with that hash in your system somewhere.
00:38:06.380 --> 00:38:21.380
And then Nix has some magic to kind of string things together so that you can also sort of do something like a Conda activate where it would put the right version of Python, NumPy and whatever you install through Nix onto your like system path and make it usable.
00:38:21.380 --> 00:38:22.380
Mm hmm.
00:38:22.380 --> 00:38:25.380
And so I think Nix and Pixie are competitors.
00:38:25.380 --> 00:38:32.380
Anyway, the thing about the functional language is that it also makes it like way less beginner friendly, at least of my opinion.
00:38:32.380 --> 00:38:33.380
Yeah, I agree.
00:38:33.380 --> 00:38:42.380
The way Pixie kind of works is like really straightforward in a way that you just define your dependencies and ranges and stuff and you get the binaries with Nix.
00:38:42.380 --> 00:38:46.380
Sometimes you need to like, usually you build things from source.
00:38:46.380 --> 00:38:47.380
So that's also a difference.
00:38:47.380 --> 00:38:51.380
I think they have like distributed caches that you could use and things like that.
00:38:51.380 --> 00:38:57.380
But honestly, I'm not a user of Nix, so I'm not sure how widely these caches, like widely used these caches are.
00:38:57.380 --> 00:39:02.380
But we definitely look at Nix as like also another source of like inspiration.
00:39:02.380 --> 00:39:09.380
And I think they have something really good going for them because people that use Nix, they are like super evangelical about it.
00:39:09.380 --> 00:39:12.380
Well, it also probably helps its functional programming, right?
00:39:12.380 --> 00:39:15.380
People who do functional programming, like they love functional programming.
00:39:15.380 --> 00:39:16.380
That's for sure.
00:39:16.380 --> 00:39:18.380
The pureness of it is pretty nice.
00:39:18.380 --> 00:39:26.380
And then Nix also goes like a step further where you can sort of manage your entire like configuration and everything through the same system.
00:39:26.380 --> 00:39:27.380
And that's also pretty powerful.
00:39:27.380 --> 00:39:31.380
And maybe maybe we can find some interesting ways of like supporting something similar.
00:39:31.380 --> 00:39:40.380
But in a way, like if you look at Pixie, I think we are trying to we don't actually care so much about Conda in a way or like maybe that's also the wrong way to put it.
00:39:40.380 --> 00:39:45.380
But but basically what we're looking is also like, how does Docker do things and how does Nix do things?
00:39:45.380 --> 00:39:48.380
And like, how can we kind of like learn from those tools?
00:39:48.380 --> 00:39:52.380
And yeah, we have a pretty well defined vision for ourselves.
00:39:52.380 --> 00:39:55.380
And the main part is that we just want to make it easy to get started.
00:39:55.380 --> 00:39:59.380
So you shouldn't have the hassle of learning a new thing to get started.
00:39:59.380 --> 00:40:04.380
You should just know like the bare minimum of information on how to run something.
00:40:04.380 --> 00:40:10.380
And Pixie is there to help you instead of we do something like with a complete vision that's making it perfect.
00:40:10.380 --> 00:40:14.380
And we're even doing it in the specific OS that you need to install.
00:40:14.380 --> 00:40:18.380
We want this to be used on every OS and we want this to be used by everyone.
00:40:18.380 --> 00:40:21.380
So you can share your code with anyone anywhere.
00:40:21.380 --> 00:40:23.380
That's something we really focus on.
00:40:23.380 --> 00:40:25.380
Sure. The clone and then just Pixie run.
00:40:25.380 --> 00:40:26.380
That's pretty easy.
00:40:26.380 --> 00:40:28.380
It's pretty easy for people to do, right?
00:40:28.380 --> 00:40:29.380
I would say so.
00:40:29.380 --> 00:40:34.380
So that's the experience of someone's set up a project for you on your announcement post.
00:40:34.380 --> 00:40:42.380
You'll have a nice little example of not a terribly complicated example of an app that you might or a project you set up.
00:40:42.380 --> 00:40:49.380
But maybe just talk through like if I want to start with just maybe I have a GitHub repo already, but I haven't set it like what's the process there?
00:40:49.380 --> 00:40:54.380
If you already have a GitHub repository, for example, you would just do Pixie in it and then give it.
00:40:54.380 --> 00:40:58.380
Yeah, basically you would just say dot because that's your current folder.
00:40:58.380 --> 00:41:01.380
Or if you don't have anything, you would just do something like Pixie to any my project.
00:41:01.380 --> 00:41:05.380
And that will create the my project folder for you with a Pixie.toml file inside.
00:41:05.380 --> 00:41:12.380
And then once you have that, you can do Pixie at Python and you can use like the specifiers from Gondar.
00:41:12.380 --> 00:41:19.380
So you could do something like Python equals 3.11 and that would get you Python 3.11 into the dependencies of that project.
00:41:19.380 --> 00:41:22.380
And then when you and also install it at that point.
00:41:22.380 --> 00:41:32.380
And after it installs, it creates that log file that you can also like should check into your repository so that you know what the latest versions were that were like working for your project.
00:41:32.380 --> 00:41:36.380
Okay, like the pinned, basically the pinned versions or constraints.
00:41:36.380 --> 00:41:37.380
Yeah.
00:41:37.380 --> 00:41:46.380
One other thing that happens when you do Pixie add is that it actually goes and tries to figure out like what's the latest version that's available for that package and then already puts a pin into your dependencies.
00:41:46.380 --> 00:41:52.380
So what we see on the screen is like we do Pixie add cowpy and then it adds cowpy 1.1.5.star.
00:41:52.380 --> 00:41:54.380
So that's a pretty specific version already.
00:41:54.380 --> 00:41:55.380
Nice.
00:41:55.380 --> 00:41:56.380
And you haven't done it here.
00:41:56.380 --> 00:42:03.380
But so example is Pixie run cowpy and then the parameters hello blog reader and it like does the cow saying hello blog reader.
00:42:03.380 --> 00:42:12.380
But when you talked earlier about the tasks or whatever, you could just say create a task called cow and it is Python cowpy hello blog reader.
00:42:12.380 --> 00:42:12.380
Right.
00:42:12.380 --> 00:42:13.380
Right.
00:42:13.380 --> 00:42:15.380
And that you would just say Pixie run cow and the same thing would happen.
00:42:15.380 --> 00:42:17.380
Is that I got that all put together, right?
00:42:17.380 --> 00:42:18.380
That's absolutely the case.
00:42:18.380 --> 00:42:26.380
And but basically everything, any binary executable that you have in your environment, like in this case cowpy, you can also call with Pixie run whatever.
00:42:26.380 --> 00:42:32.380
Like so you could also do Pixie run Python and they would start Python 3.11 or whatever you have installed inside of that environment.
00:42:32.380 --> 00:42:34.380
Yeah. And that would actually do the REPL and everything.
00:42:34.380 --> 00:42:35.380
Yeah. Yeah.
00:42:35.380 --> 00:42:37.380
Just like having it globally installed.
00:42:37.380 --> 00:42:42.380
So one other feature of Pixie that we haven't mentioned before is that you can still do global install.
00:42:42.380 --> 00:42:45.380
So sometimes you have that comment line tool that you really love.
00:42:45.380 --> 00:42:50.380
One of the things we I usually install is bat, which is like cat with wings.
00:42:50.380 --> 00:42:56.380
What you can do with Pixie is you can do a Pixie global installed bat and that will install bat and make it globally available.
00:42:56.380 --> 00:43:01.380
So you can run it from wherever it's not tied to any like project environment.
00:43:01.380 --> 00:43:04.380
It's just on your system in your home folder, essentially.
00:43:04.380 --> 00:43:07.380
And you can just run that wherever you are.
00:43:07.380 --> 00:43:08.380
And it works.
00:43:08.380 --> 00:43:12.380
The one that comes to mind for me a lot is Pipex is one of them.
00:43:12.380 --> 00:43:16.380
That's exactly where we got this using similar mechanisms to that.
00:43:16.380 --> 00:43:20.380
So every tool that you installed this way is installed into its own virtual environment.
00:43:20.380 --> 00:43:22.380
So they don't have any overlap.
00:43:22.380 --> 00:43:25.380
You can install versions that are completely unrelated.
00:43:25.380 --> 00:43:27.380
Even the different pythons, right?
00:43:27.380 --> 00:43:37.380
One thing that I also like a lot about this and, you know, pour one out for poor old pep, something, something, something about the Dunder PI packages folder.
00:43:37.380 --> 00:43:52.380
I can't remember what the PEP number is, but basically the idea that if I'm just in the right place, the run command should grab whatever local environment is the one I've set up rather than explicitly go and find the environment, activating the environment, etc.
00:43:52.380 --> 00:43:58.380
So it looks like when you say Pixie run, there's no Pixie activate or any of those things, right?
00:43:58.380 --> 00:43:59.380
How does that work?
00:43:59.380 --> 00:44:10.380
The way condo environments work is that you need to have some sort of like little activation thing where basically the past variable environment variables changed and adjusted and some other activation scripts are run.
00:44:10.380 --> 00:44:19.380
And with Pixie what we're doing is we run those in the background and then we extract all the environment variables that are necessary for the activation basically to work.
00:44:19.380 --> 00:44:27.380
And then we just inject it right before we execute what you want to execute like CalPy in this case.
00:44:27.380 --> 00:44:31.380
Yeah. So there's like an implicit activate or you don't even have to say activate in Python.
00:44:31.380 --> 00:44:39.380
You can just, if you just use that Python, you say the path to the virtual environment Python run that like that's sufficient.
00:44:39.380 --> 00:44:40.380
Yeah.
00:44:40.380 --> 00:44:41.380
That's more or less what happens.
00:44:41.380 --> 00:44:45.380
Like, yeah, sometimes, you know, packages can have different requirements when it comes to activation.
00:44:45.380 --> 00:44:55.380
So like Python doesn't have many requirements when it comes to activation, but some other packages, they might need some other like environment variables that are specific to the environment location where they are installed, etc.
00:44:55.380 --> 00:45:04.380
Sure. Well, even Python virtual environments can get weird where like you can set environment variables that get set during the activation of the virtual environment, right?
00:45:04.380 --> 00:45:08.380
Like, I don't think many people do that because it's transient, but it could.
00:45:08.380 --> 00:45:20.380
We also have a pixie shell command. So if you want to have that experience of like an activated environment, you can use pixie shell and then it is like basically a shell that acts like an activated environment.
00:45:20.380 --> 00:45:22.380
Like poetry has the same and many others.
00:45:22.380 --> 00:45:31.380
The example here shows like I'm in the top level of the project and I say pixie run. What if I'm like three directories down and I say pixie run? What happens then?
00:45:31.380 --> 00:45:39.380
The exact same thing will happen because pixie runs from the root of the project and all your tasks are by default running from the root of the project.
00:45:39.380 --> 00:45:48.380
So you define them with the boss in your project as they are always and then where you are, you can run those tasks as they are.
00:45:48.380 --> 00:45:56.380
But if you want to run something in that directory, you can just use pixie run and then your own commands to do X on that directory.
00:45:56.380 --> 00:46:05.380
And then there's this other way of using it, like the pixie itself will run down the path that you're in and we'll find the first pixie project that it encounters.
00:46:05.380 --> 00:46:08.380
And for instance, pixie itself has some examples.
00:46:08.380 --> 00:46:14.380
So if you move into the example directory and then in one of the examples, those are their own pixie project.
00:46:14.380 --> 00:46:21.380
So if you run it there, pixie run start, it will start the example instead of the actual pixie project of pixie.
00:46:21.380 --> 00:46:27.380
Interesting. So you could have a nested one, like there's a main one, but then inside you could have a little sub pixie projects.
00:46:27.380 --> 00:46:30.380
Yeah, a little bit like node and npm in that regard.
00:46:30.380 --> 00:46:36.380
We have an issue that's open about mono repo support and cargo does a pretty nice job.
00:46:36.380 --> 00:46:37.380
Yeah.
00:46:37.380 --> 00:46:38.380
Yeah.
00:46:38.380 --> 00:46:40.380
That sounds like a really good idea for mono repo support.
00:46:40.380 --> 00:46:46.380
There's a different problem that you normally with mono repo is have some shared dependencies.
00:46:46.380 --> 00:46:59.380
So if you, for instance, have in your, your, your root of your repository, you have quite an dependency defined, then you want that shared between all the packages.
00:46:59.380 --> 00:47:00.380
Yeah.
00:47:00.380 --> 00:47:01.380
Down in your repo tree.
00:47:01.380 --> 00:47:03.380
So that's something we still have to support.
00:47:03.380 --> 00:47:10.380
So right now there are like two separate projects and the pixie tool will just find the first projects that encounters.
00:47:10.380 --> 00:47:15.380
But we need some kind of way to define a workspace or mono repo.
00:47:15.380 --> 00:47:16.380
If you would say it like that.
00:47:16.380 --> 00:47:19.380
And then you could like link those environments together.
00:47:19.380 --> 00:47:26.380
And if you start a lower level one, you would start the main one with it or something like that.
00:47:26.380 --> 00:47:27.380
That's still in the works.
00:47:27.380 --> 00:47:29.380
Look at the dependencies of the top one.
00:47:29.380 --> 00:47:33.380
And then you might add some more in your little sub project type of thing, something like that.
00:47:33.380 --> 00:47:34.380
Yeah.
00:47:34.380 --> 00:47:36.380
Well, even what you already have sounds pretty excellent for you.
00:47:36.380 --> 00:47:37.380
Yeah.
00:47:37.380 --> 00:47:47.380
So currently if you have like a system where you have a backend server that's completely Python or Rust or whatever, you could have that as a separate project and then have another project that is like the front end.
00:47:47.380 --> 00:47:50.380
So you do some, you install npm there or whatever.
00:47:50.380 --> 00:47:53.380
And those are completely separate within your repository.
00:47:53.380 --> 00:48:05.380
And the main repository is just some tooling to, for instance, lint everything or, or something like that, or install your base dependencies that you want to use in the, in the complete repository.
00:48:05.380 --> 00:48:08.380
But you could already set it up pretty nicely.
00:48:08.380 --> 00:48:19.380
I'm sure if you have a truly large organization with a monorepo, which for people that know, that just means like all the code or the whole organization is in one huge repository instead of a bunch of projects.
00:48:19.380 --> 00:48:23.380
But dependencies across projects, it's just within that kind of that file structure.
00:48:23.380 --> 00:48:24.380
Like it's a lot.
00:48:24.380 --> 00:48:32.380
I was complaining about having a dependency that had two things that wanted the same library, both lower than and greater than some version number.
00:48:32.380 --> 00:48:34.380
Like that's for one project, you know what I mean?
00:48:34.380 --> 00:48:35.380
You put it all together.
00:48:35.380 --> 00:48:36.380
It's only going to get more challenging.
00:48:36.380 --> 00:48:42.380
So tools like this, these sub projects and stuff, I think could help go like, all right, this part needs these things.
00:48:42.380 --> 00:48:43.380
Cause that's the data science part.
00:48:43.380 --> 00:48:45.380
This other part needs that thing.
00:48:45.380 --> 00:48:47.380
Cause that's the microservice part.
00:48:47.380 --> 00:48:54.380
So what else do people know about Pixie taking dependency, taking PR, PRs and contributions?
00:48:54.380 --> 00:48:55.380
Definitely.
00:48:55.380 --> 00:48:57.380
Like we also stood like pretty early.
00:48:57.380 --> 00:49:05.380
So we love people that test Pixie and tell us the feedback on like our discord channel or on like GitHub.
00:49:05.380 --> 00:49:09.380
I think we have discussions open as well and issues, any feedback is appreciated.
00:49:09.380 --> 00:49:12.380
I will really like trying to take package management to the next level.
00:49:12.380 --> 00:49:18.380
That includes like building packages that includes like package signing stuff like this, security, et cetera.
00:49:18.380 --> 00:49:21.380
There are so many things and issues to work on.
00:49:21.380 --> 00:49:23.380
And I think it's going to be very fun.
00:49:23.380 --> 00:49:24.380
Yeah.
00:49:24.380 --> 00:49:26.380
I'm also actually organizing packaging con.
00:49:26.380 --> 00:49:27.380
Okay.
00:49:27.380 --> 00:49:28.380
That's happening.
00:49:28.380 --> 00:49:31.380
And like a week from now, actually, and really looking forward to that.
00:49:31.380 --> 00:49:35.380
So that's going to be fun to chat with a lot of package manager developers.
00:49:35.380 --> 00:49:37.380
Does it have an online components?
00:49:37.380 --> 00:49:38.380
Yeah.
00:49:38.380 --> 00:49:40.380
So it's in Berlin, but it's also hybrid.
00:49:40.380 --> 00:49:42.380
So you can join virtually if you want.
00:49:42.380 --> 00:49:45.380
Will the videos be on something like YouTube later?
00:49:45.380 --> 00:49:46.380
Yep.
00:49:46.380 --> 00:49:46.380
Yep.
00:49:46.380 --> 00:49:47.380
Okay, cool.
00:49:47.380 --> 00:49:52.380
If the timing lines up, you'll have to give me the link to the videos and I'll put it into the show notes for people.
00:49:52.380 --> 00:49:56.380
I get, we might somehow miss like the conference runs, but the videos aren't yet up.
00:49:56.380 --> 00:49:59.380
But if they are, you know, send me a link and we'll make it part of the show.
00:49:59.380 --> 00:50:00.380
So people can check it out.
00:50:00.380 --> 00:50:07.380
And one of prefix bus will also talk about this, these rust crates that we've been building and how it all fits together.
00:50:07.380 --> 00:50:17.380
If you want to learn more about that and if you want to contribute, like also if you want to learn rust, like we're more than happy to kind of like help you, like guide you as time permits, obviously.
00:50:17.380 --> 00:50:20.380
Yeah, we're trying to be really active on our channels.
00:50:20.380 --> 00:50:23.380
So on GitHub, we have some good first issues.
00:50:23.380 --> 00:50:26.380
And if you have some questions, just ask around.
00:50:26.380 --> 00:50:32.380
And in our discord, we're very active and really try to react as fast as possible to anything.
00:50:32.380 --> 00:50:37.380
Right at the bottom of prefix.dev, you've got your little discord icon down there.
00:50:37.380 --> 00:50:40.380
So people can click on that to kind of be part of it.
00:50:40.380 --> 00:50:41.380
Right.
00:50:41.380 --> 00:50:42.380
I think it's also on the top.
00:50:42.380 --> 00:50:43.380
Yeah.
00:50:43.380 --> 00:50:44.380
Yeah.
00:50:44.380 --> 00:50:51.380
I see you all both are like me and have like, not accepted that, that X Twitter is called X.
00:50:51.380 --> 00:50:52.380
Yeah.
00:50:52.380 --> 00:50:53.380
I'm not changing mine.
00:50:53.380 --> 00:50:55.380
They should come out with the final logo, right?
00:50:55.380 --> 00:50:57.380
Like that's not, that can't be it.
00:50:57.380 --> 00:50:58.380
I can't be it.
00:50:58.380 --> 00:50:59.380
It's like a child.
00:50:59.380 --> 00:51:01.380
Like I'm just, this is what I got.
00:51:01.380 --> 00:51:02.380
And it's there.
00:51:02.380 --> 00:51:05.380
Maybe I need, I should probably put an EX Twitter in there just for.
00:51:05.380 --> 00:51:06.380
Yeah.
00:51:06.380 --> 00:51:07.380
Yeah.
00:51:07.380 --> 00:51:12.380
And then a quick question from Elliot's, any meaning behind the name Pixie?
00:51:12.380 --> 00:51:14.380
We thought very long about the name.
00:51:14.380 --> 00:51:16.380
We had a bunch of different versions.
00:51:16.380 --> 00:51:22.380
Like initially we thought PX, just P and X, but that was somehow like hard.
00:51:22.380 --> 00:51:23.380
Have you considered X?
00:51:23.380 --> 00:51:25.380
I can't even just use that for whatever.
00:51:25.380 --> 00:51:26.380
Just kidding.
00:51:26.380 --> 00:51:27.380
Sorry.
00:51:27.380 --> 00:51:28.380
Back to Twitter.
00:51:28.380 --> 00:51:29.380
Burnt.
00:51:29.380 --> 00:51:30.380
It is burnt.
00:51:30.380 --> 00:51:36.380
We also thought about packs like P A X, but that's currently executable that you already
00:51:36.380 --> 00:51:39.380
have on your system if you're using Linux or Mac.
00:51:39.380 --> 00:51:40.380
So that didn't work.
00:51:40.380 --> 00:51:41.380
That'd be tricky.
00:51:41.380 --> 00:51:42.380
Yeah.
00:51:42.380 --> 00:51:42.380
Yeah.
00:51:42.380 --> 00:51:42.380
Yeah.
00:51:42.380 --> 00:51:42.380
Yeah.
00:51:42.380 --> 00:51:43.380
Yeah.
00:51:43.380 --> 00:51:44.380
We thought about P E X.
00:51:44.380 --> 00:51:46.380
I don't know.
00:51:46.380 --> 00:51:50.380
We wanted to derive it a little bit from the name prefix because that's kind of the company
00:51:50.380 --> 00:51:55.380
name, but Pixie seemed really cool because it's a, apparently a magical fairy and we want to
00:51:55.380 --> 00:51:57.380
make a package management magic.
00:51:57.380 --> 00:51:58.380
Yeah, exactly.
00:51:58.380 --> 00:51:59.380
It's, I think the name is great.
00:51:59.380 --> 00:52:00.380
It's short enough to type.
00:52:00.380 --> 00:52:01.380
It's pretty unique.
00:52:01.380 --> 00:52:04.380
You can, it's somewhat Google-able, right?
00:52:04.380 --> 00:52:05.380
Yeah.
00:52:05.380 --> 00:52:06.380
And you can pronounce it.
00:52:06.380 --> 00:52:07.380
Yeah.
00:52:07.380 --> 00:52:08.380
That was also important to it.
00:52:08.380 --> 00:52:09.380
Yeah.
00:52:09.380 --> 00:52:10.380
You don't have to debate.
00:52:10.380 --> 00:52:12.380
Is it PI PI or is it PI PI?
00:52:12.380 --> 00:52:14.380
Like let's say, no, like, make it lowercase.
00:52:14.380 --> 00:52:15.380
Make it lowercase.
00:52:15.380 --> 00:52:16.380
It's not an acronym.
00:52:16.380 --> 00:52:17.380
You don't say the letters.
00:52:17.380 --> 00:52:21.380
We created this thing called MicroMamba, which I don't want to like go into too much detail,
00:52:21.380 --> 00:52:24.380
but a lot of people complained about MicroMamba being too long to type.
00:52:24.380 --> 00:52:27.380
So we had to stay under the five character limit.
00:52:27.380 --> 00:52:28.380
Yeah.
00:52:28.380 --> 00:52:30.380
I think there's, there's value in that.
00:52:30.380 --> 00:52:31.380
Yeah.
00:52:31.380 --> 00:52:32.380
There's definitely value in that.
00:52:32.380 --> 00:52:35.380
So let's close out our conversation with where you all are headed.
00:52:35.380 --> 00:52:36.380
What's next?
00:52:36.380 --> 00:52:37.380
Yeah.
00:52:37.380 --> 00:52:40.380
Like we are super excited about a bunch of upcoming features.
00:52:40.380 --> 00:52:44.380
One is definitely what I already mentioned, Pixie builds so that you can build a package
00:52:44.380 --> 00:52:45.380
right away from Pixie.
00:52:45.380 --> 00:52:47.380
To prepare them for Conda Forge, right?
00:52:47.380 --> 00:52:52.380
Well, for Conda Forge or like maybe you also have some internal stuff or your own private
00:52:52.380 --> 00:52:53.380
things and stuff.
00:52:53.380 --> 00:52:58.380
We just want to make that easy because that is currently way too hard to like make a Conda
00:52:58.380 --> 00:53:00.380
packages like a bunch of steps.
00:53:00.380 --> 00:53:06.380
And that also kind of precludes that you could use source and get dependencies for like other
00:53:06.380 --> 00:53:07.380
Pixie projects.
00:53:07.380 --> 00:53:12.380
Because basically what we do, what we will do in the background is like, if you depend on
00:53:12.380 --> 00:53:16.380
a source dependency for another Pixie project, we will build it into a like package on the
00:53:16.380 --> 00:53:17.380
fly and then put it into your environment.
00:53:17.380 --> 00:53:20.380
And then like integrating with the PyPI ecosystem.
00:53:20.380 --> 00:53:22.380
That's what we're actually working on the most right now.
00:53:22.380 --> 00:53:25.380
And that is the rip thing that I told you about.
00:53:25.380 --> 00:53:26.380
Yeah, that's awesome.
00:53:26.380 --> 00:53:29.380
Because we just see a lot of need in the community to have this.
00:53:29.380 --> 00:53:32.380
A lot of projects in the wild are kind of mixing it.
00:53:32.380 --> 00:53:33.380
Yeah.
00:53:33.380 --> 00:53:37.380
If you're working with PyPI, I will switch my stuff over and give it a try and see how
00:53:37.380 --> 00:53:38.380
it works.
00:53:38.380 --> 00:53:39.380
That would be great.
00:53:39.380 --> 00:53:40.380
Until then I can't, right?
00:53:40.380 --> 00:53:44.380
I've just got, I've got like hundreds of packages and a lot of them I'm sure are just unique to
00:53:44.380 --> 00:53:45.380
PyPI.
00:53:45.380 --> 00:53:47.380
We are not far away.
00:53:47.380 --> 00:53:51.380
Like I think the hard bits are solved and that was like resolving because it works quite
00:53:51.380 --> 00:53:53.380
different from Conda.
00:53:53.380 --> 00:53:57.380
You need to like get the individual wheel files to get the metadata, etc.
00:53:57.380 --> 00:54:02.380
And like that doesn't scale if you need all the metadata upfront, which is actually the case
00:54:02.380 --> 00:54:03.380
in Conda.
00:54:03.380 --> 00:54:05.380
You have all the metadata upfront, but with PyPI you don't.
00:54:05.380 --> 00:54:07.380
And so we had to make the solver lazy.
00:54:07.380 --> 00:54:10.380
We had to make the solver generic and we are through that process now.
00:54:10.380 --> 00:54:14.380
And now it's basically just engineering work in that sense to integrate it with Pixie.
00:54:14.380 --> 00:54:16.380
But it's going to happen and it's going to be nice.
00:54:16.380 --> 00:54:17.380
I'm sure.
00:54:17.380 --> 00:54:18.380
Yeah.
00:54:18.380 --> 00:54:23.380
We also have some ideas of like, can we somehow merge Pixie Tomil into PyProject Tomil so that
00:54:23.380 --> 00:54:28.380
it's like more natural to like Python developers and you only need to manage one file.
00:54:28.380 --> 00:54:32.380
And I think PyProject Tomil gives us the flexibility that we would need to do that.
00:54:32.380 --> 00:54:33.380
I think it does.
00:54:33.380 --> 00:54:38.380
You've got things like Hatch and others that kind of got a way to go in there.
00:54:38.380 --> 00:54:39.380
Yeah.
00:54:39.380 --> 00:54:43.380
And then we have some other ideas that are a bit more out there maybe, but, or not really,
00:54:43.380 --> 00:54:46.380
but like we already have a set up Pixie action for GitHub.
00:54:46.380 --> 00:54:47.380
That's really nice.
00:54:47.380 --> 00:54:53.380
And then another idea is like, how can you go from a virtual environment to a Docker image
00:54:53.380 --> 00:54:54.380
easily?
00:54:54.380 --> 00:54:56.380
So that's also something that we're thinking about.
00:54:56.380 --> 00:54:57.380
Okay.
00:54:57.380 --> 00:54:58.380
These kind of things.
00:54:58.380 --> 00:54:59.380
All very exciting.
00:54:59.380 --> 00:55:00.380
Awesome.
00:55:00.380 --> 00:55:01.380
How long has this been around?
00:55:01.380 --> 00:55:02.380
I'm your blog post is two months old.
00:55:02.380 --> 00:55:03.380
It's announcing this stuff.
00:55:03.380 --> 00:55:04.380
So.
00:55:04.380 --> 00:55:05.380
Yeah.
00:55:05.380 --> 00:55:09.380
I mean, I think we maybe made the repository public months earlier than the blog post or
00:55:09.380 --> 00:55:16.380
so, but it like prefix as a company is like just very little over a year old.
00:55:16.380 --> 00:55:22.380
And that's when we like really started to build the website, the platform, Pixie, Rattler,
00:55:22.380 --> 00:55:23.380
and all of these things.
00:55:23.380 --> 00:55:26.380
So I think Pixie, we saw that maybe like five months ago.
00:55:26.380 --> 00:55:29.380
So not too old, still very fresh.
00:55:29.380 --> 00:55:30.380
Yeah.
00:55:30.380 --> 00:55:31.380
Yeah.
00:55:31.380 --> 00:55:32.380
I think it's a new software smell.
00:55:32.380 --> 00:55:33.380
Yeah.
00:55:33.380 --> 00:55:34.380
Exactly.
00:55:34.380 --> 00:55:35.380
Definitely.
00:55:35.380 --> 00:55:40.380
I hope you don't get the, yeah, like, but we also know how to, yeah.
00:55:40.380 --> 00:55:41.380
You don't want that.
00:55:41.380 --> 00:55:44.380
Personally, I'm very surprised how stable it is already.
00:55:44.380 --> 00:55:51.380
And I think that's partly due to the use of rust and the fact that we can very heavily
00:55:51.380 --> 00:55:55.380
check some of the, you know, workings of the tool before we ship it.
00:55:55.380 --> 00:55:57.380
Well, it looks like it's off to a really good start.
00:55:57.380 --> 00:55:58.380
I like a lot of the ideas here.
00:55:58.380 --> 00:56:00.380
So yeah, keep up with the good work.
00:56:00.380 --> 00:56:05.380
Before we wrap it up, we're basically out of time, but there's the, always the open source
00:56:05.380 --> 00:56:07.380
dream of I'm going to build a project.
00:56:07.380 --> 00:56:09.380
It's going to get super popular.
00:56:09.380 --> 00:56:13.380
The dream used to be, I'm going to do some consulting around it, right?
00:56:13.380 --> 00:56:17.380
I've created project X project X is popular so I can charge high consulting rates.
00:56:17.380 --> 00:56:19.380
That's the dream of the nineties.
00:56:19.380 --> 00:56:24.380
I think the new dream is I'm going to start a company around my project and, and have some
00:56:24.380 --> 00:56:27.380
kind of open core model and something interesting there.
00:56:27.380 --> 00:56:28.380
You guys have prefix.dev.
00:56:28.380 --> 00:56:29.380
What's the dream for you?
00:56:29.380 --> 00:56:31.380
Like how, what's your, how are you approaching this?
00:56:31.380 --> 00:56:36.380
I think a ton of people would be interested to just hear like, how did you make that happen?
00:56:36.380 --> 00:56:37.380
You know?
00:56:37.380 --> 00:56:40.380
So you saved the hardest question for last.
00:56:40.380 --> 00:56:43.380
You don't have to answer it, but I do think it's, it's interesting.
00:56:43.380 --> 00:56:44.380
Yeah.
00:56:44.380 --> 00:56:45.380
Package management is a hard problem.
00:56:45.380 --> 00:56:51.380
And there are, there are lots of sort of sub problems that I would say enterprise customers
00:56:51.380 --> 00:56:53.380
in a way are willing to pay for.
00:56:53.380 --> 00:56:59.380
That includes like security managed repositories, let's say like basically red heads and like more
00:56:59.380 --> 00:57:04.380
or less red heads project product is that they have this like, I know five or 10 years or something
00:57:04.380 --> 00:57:09.380
like of support for like old versions of packages for enterprise customers.
00:57:09.380 --> 00:57:15.380
And I think we have a pretty interesting approach to package management that is pretty easy to kind of grasp.
00:57:15.380 --> 00:57:32.380
And like part of why we want to make Pixie build a thing is also because we want people to make more packages and then upload them to our website and kind of grow this entire thing in popularity and make it super useful so that we hopefully end up with customers that are supporting our work.
00:57:32.380 --> 00:57:33.380
Awesome.
00:57:33.380 --> 00:57:34.380
Well, good luck to both of you.
00:57:34.380 --> 00:57:37.380
And thanks for being on the show to share what you're up to.
00:57:37.380 --> 00:57:38.380
Sure.
00:57:38.380 --> 00:57:39.380
Thank you.
00:57:39.380 --> 00:57:42.380
This has been another episode of Talk Python To Me.
00:57:42.380 --> 00:57:43.380
Thank you to our sponsors.
00:57:43.380 --> 00:57:45.380
Be sure to check out what they're offering.
00:57:45.380 --> 00:57:48.380
It really helps support the show.
00:57:48.380 --> 00:57:49.380
Python Tutor.
00:57:49.380 --> 00:57:54.380
Visualize your Python code step by step to understand just what's happening with your code.
00:57:54.380 --> 00:58:00.380
Try it for free and anonymously at talkpython.fm/python-tutor.
00:58:00.380 --> 00:58:03.380
Want to level up your Python?
00:58:03.380 --> 00:58:07.380
We have one of the largest catalogs of Python video courses over at Talk Python.
00:58:07.380 --> 00:58:12.380
Our content ranges from true beginners to deeply advanced topics like memory and async.
00:58:12.380 --> 00:58:15.380
And best of all, there's not a subscription in sight.
00:58:15.380 --> 00:58:17.380
Check it out for yourself at training.talkpython.fm.
00:58:17.380 --> 00:58:22.380
Be sure to subscribe to the show, open your favorite podcast app, and search for Python.
00:58:22.380 --> 00:58:24.380
We should be right at the top.
00:58:24.380 --> 00:58:30.380
You can also find the iTunes feed at /itunes, the Google Play feed at /play, and the direct
00:58:30.380 --> 00:58:33.380
RSS feed at /rss on talkpython.fm.
00:58:33.380 --> 00:58:36.380
We're live streaming most of our recordings these days.
00:58:36.380 --> 00:58:40.380
If you want to be part of the show and have your comments featured on the air, be sure to
00:58:40.380 --> 00:58:44.380
subscribe to our YouTube channel at talkpython.fm/youtube.
00:58:44.380 --> 00:58:46.380
This is your host, Michael Kennedy.
00:58:46.380 --> 00:58:47.380
Thanks so much for listening.
00:58:47.380 --> 00:58:48.380
I really appreciate it.
00:58:48.380 --> 00:58:50.380
Now get out there and write some Python code.
00:58:50.380 --> 00:58:51.380
I'll see you next time.
00:58:51.380 --> 00:58:51.380
I'll see you next time.
00:58:51.380 --> 00:58:52.380
Bye.
00:58:52.380 --> 00:58:53.380
Bye.
00:58:53.380 --> 00:58:54.380
Bye.
00:58:54.380 --> 00:58:54.380
Bye.
00:58:54.380 --> 00:58:55.380
Bye.
00:58:55.380 --> 00:58:56.380
Bye.
00:58:56.380 --> 00:58:57.380
Bye.
00:58:57.380 --> 00:58:58.380
Bye.
00:58:58.380 --> 00:58:59.380
Bye.
00:58:59.380 --> 00:59:00.380
Bye.
00:59:00.380 --> 00:59:01.380
Bye.
00:59:01.380 --> 00:59:02.380
Bye.
00:59:02.380 --> 00:59:03.380
Bye.
00:59:03.380 --> 00:59:03.380
Bye.
00:59:03.380 --> 00:59:04.380
Bye.
00:59:04.380 --> 00:59:05.380
Bye.
00:59:05.380 --> 00:59:06.380
Bye.
00:59:06.380 --> 00:59:06.380
Bye.
00:59:06.380 --> 00:59:07.380
Bye.
00:59:07.380 --> 00:59:07.380
Bye.
00:59:07.380 --> 00:59:07.880
you
00:59:07.880 --> 00:59:08.380
you
00:59:08.380 --> 00:59:08.880
you
00:59:08.880 --> 00:59:10.880
Thank you.
00:59:10.880 --> 00:59:40.860
Thank you.