#496 A lake house in Seattle

WEBVTT
00:00:00.020 --> 00:00:05.140
Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds.
00:00:05.340 --> 00:00:07.040
This is episode 496,
00:00:07.520 --> 00:00:08.280
recorded Tuesday,
00:00:09.000 --> 00:00:09.840
September 15th.
00:00:09.860 --> 00:00:10.400
I'm Michael Kennedy.
00:00:10.800 --> 00:00:11.840
And I'm Calvin Hendryx-Parker.
00:00:12.380 --> 00:00:15.380
This episode is brought to you by Logfire from Pydantic.
00:00:15.920 --> 00:00:19.200
If you want observability for your apps and your AI agents,
00:00:19.460 --> 00:00:21.280
Logfire is the business.
00:00:21.680 --> 00:00:23.140
I will be telling you more about them later.
00:00:23.260 --> 00:00:25.020
Find the link at the top of the show notes.
00:00:25.280 --> 00:00:26.700
Follow us on the socials.
00:00:27.000 --> 00:00:32.099
All the various things you can think of are there on the episode page as well.
00:00:32.480 --> 00:00:33.320
And sign up for the newsletter.
00:00:33.660 --> 00:00:37.180
I just sent out the most recent one a couple days ago.
00:00:37.300 --> 00:00:37.920
It was a little bit late.
00:00:38.060 --> 00:00:38.140
Sorry,
00:00:38.300 --> 00:00:38.380
folks.
00:00:38.580 --> 00:00:45.380
But really cool stuff that we add, like extra information that doesn't even appear in the show that helps you get a little more out of the show.
00:00:45.680 --> 00:00:45.820
Yep.
00:00:46.040 --> 00:00:47.320
I love all the context it adds.
00:00:47.620 --> 00:00:48.040
I do, too.
00:00:48.320 --> 00:00:48.660
I do, too.
00:00:48.740 --> 00:00:50.120
I'm like, well, that's pretty good.
00:00:50.200 --> 00:00:51.000
We found some good stuff here.
00:00:51.800 --> 00:00:56.360
I would say that this newsletter that we're writing here, it shouldn't go extinct.
00:00:57.160 --> 00:01:00.560
But some things might need to go extinct.
00:01:01.240 --> 00:01:02.120
What's going on here?
00:01:02.560 --> 00:01:07.040
So I found, so this is a blog post from,
00:01:08.020 --> 00:01:09.440
what's Eddie's last name?
00:01:10.060 --> 00:01:11.280
Hold on, it's down here at the bottom of his copyright.
00:01:11.550 --> 00:01:12.300
Eddie Atkinson.
00:01:13.020 --> 00:01:17.420
He gave a talk at the most recent latency conference.
00:01:18.180 --> 00:01:21.460
So it's actually a talk from last year, but I think he kind of brought it back a little,
00:01:21.600 --> 00:01:26.480
evergreened it into a blog post last week about pandas that should go extinct.
00:01:26.580 --> 00:01:31.580
and we're not talking about the cute little fluffy things that are used for international diplomacy,
00:01:32.160 --> 00:01:34.860
but the Python data frame library.
00:01:36.040 --> 00:01:38.540
Michael, how many times have you thought you had big data,
00:01:39.040 --> 00:01:43.220
only to find out you were ready to defunstrate your laptop because pandas was the problem?
00:01:43.500 --> 00:01:44.760
You know what? It's happened.
00:01:45.080 --> 00:01:45.400
It's happened.
00:01:45.760 --> 00:01:48.760
I'm going to get my dictionary real quick, and then I'm going to know that that happened.
00:01:50.100 --> 00:01:54.520
I think the issue is a lot of folks really don't have truly big data problems.
00:01:54.720 --> 00:01:58.760
I mean, we've done some big data projects in the past, which were 10,000 tables,
00:01:59.360 --> 00:01:59.940
petabytes of data.
00:02:00.160 --> 00:02:01.020
That's truly big data.
00:02:01.640 --> 00:02:04.760
Most folks probably lie in the medium-sized data,
00:02:05.280 --> 00:02:06.620
but Pandas definitely tops out.
00:02:06.730 --> 00:02:10.960
I mean, he does some interesting benchmarks in here, gives a couple good code examples,
00:02:11.640 --> 00:02:15.840
actually shows a really interesting post from Amazon Redshift team,
00:02:16.340 --> 00:02:21.540
where they were looking at the composition of many of the tables that are out there in the Redshift environment.
00:02:21.600 --> 00:02:25.920
If anybody's going to have a good view on what the size of data is and what big data could be,
00:02:26.460 --> 00:02:28.320
they're probably the ones to look at that.
00:02:28.380 --> 00:02:29.400
But if you look at this chart,
00:02:30.220 --> 00:02:33.480
they basically say on a continuum of data size,
00:02:33.920 --> 00:02:35.700
most folks start over here in Excel.
00:02:36.220 --> 00:02:39.020
You've got like under a gigabyte, around a gigabyte of data.
00:02:39.320 --> 00:02:41.560
About that point in time, Excel falls over.
00:02:41.980 --> 00:02:44.520
It's probably time to pick up another tool to handle that.
00:02:44.680 --> 00:02:47.359
And a lot of people reach for pandas because I think there's just a lot of
00:02:47.400 --> 00:02:52.760
built up inertia or momentum in the community around the pandas and data frames and it's an easy
00:02:53.040 --> 00:02:58.520
ui it's been taught in a lot of universities so there's just not a lot of like need to kind of
00:02:58.540 --> 00:03:01.600
move out of that space because there's a lot of good code examples a lot of blog posts have been
00:03:01.680 --> 00:03:06.560
produced a lot of data science is based on pandas but they're really based on data frames and there's
00:03:06.700 --> 00:03:11.040
more than one library out there to handle data frames and probably do it more efficient so if
00:03:11.040 --> 00:03:15.839
you actually looked at the chart here they're basically saying when you get up into like the 10
00:03:15.860 --> 00:03:19.040
10 gigabyte range for data sizes.
00:03:19.380 --> 00:03:20.360
Pandas is probably still pretty good,
00:03:20.940 --> 00:03:21.580
but then there's a gap.
00:03:21.740 --> 00:03:25.480
It falls off somewhere between 10 and 100 gigabytes of data.
00:03:25.820 --> 00:03:28.620
And 100 gigabytes of data these days is not unfathomable.
00:03:28.820 --> 00:03:30.760
You can easily go find sample data sets
00:03:30.760 --> 00:03:32.580
that are in that realm, in that range, in that size.
00:03:33.200 --> 00:03:35.000
And so the next thing they reach for
00:03:35.320 --> 00:03:37.640
is typically a commercial tool like Databricks,
00:03:37.920 --> 00:03:38.400
Snowflake,
00:03:39.020 --> 00:03:40.100
Dask, or some of these other things
00:03:40.220 --> 00:03:41.040
that are like Spark.
00:03:41.280 --> 00:03:43.819
So you're distributing the memory
00:03:43.840 --> 00:03:45.560
of that data set across many machines
00:03:45.960 --> 00:03:48.340
or maybe even across one very large machine,
00:03:48.540 --> 00:03:49.580
but doing it in a distributed manner.
00:03:50.160 --> 00:03:52.160
Most people probably don't need to go that far.
00:03:52.400 --> 00:03:54.520
Like most people probably are still sitting in the range
00:03:54.560 --> 00:03:56.500
where you can see on this chart that Polars handles.
00:03:56.840 --> 00:04:01.040
Polars can handle straight up into 100 gigabytes of data
00:04:01.340 --> 00:04:02.260
easily on a single machine.
00:04:03.340 --> 00:04:05.460
And then DuckDB takes it to the next step,
00:04:05.660 --> 00:04:08.360
which actually kind of fitting for this episode.
00:04:08.420 --> 00:04:09.980
I think this will be an interesting episode
00:04:10.060 --> 00:04:11.259
because there's a lot of information here
00:04:11.280 --> 00:04:13.420
about DuckDB later on in the show.
00:04:13.940 --> 00:04:17.140
But he kind of goes on again and shows that basically
00:04:17.390 --> 00:04:20.220
the average size of a row in Redshift is about a kilobyte.
00:04:20.500 --> 00:04:22.680
Every Redshift cluster has like 10 machines in it.
00:04:22.940 --> 00:04:25.820
They're capable of guzzling 8 gigabytes per second
00:04:25.960 --> 00:04:26.360
from S3,
00:04:26.770 --> 00:04:27.940
but really in actuality,
00:04:28.520 --> 00:04:31.280
almost 95% of the tables in Redshift
00:04:31.560 --> 00:04:33.500
contain fewer than 100 gigabytes of data.
00:04:34.120 --> 00:04:36.040
Most people are still in the range
00:04:36.040 --> 00:04:37.900
of just using a single machine with DuckDB
00:04:38.220 --> 00:04:39.260
or even just Polars,
00:04:39.490 --> 00:04:40.260
which is probably similar,
00:04:40.820 --> 00:04:41.940
simpler to maintain and manage,
00:04:42.320 --> 00:04:43.860
but it's above the reach of pandas.
00:04:44.040 --> 00:04:45.300
That's why the post is kind of
00:04:45.360 --> 00:04:47.120
going on about pandas needing to go extinct.
00:04:47.780 --> 00:04:48.580
Another interesting bit,
00:04:48.920 --> 00:04:49.040
again,
00:04:49.360 --> 00:04:50.000
kind of good code
00:04:50.160 --> 00:04:50.720
examples in here.
00:04:51.220 --> 00:04:55.720
When we get down into some of the tables for the performance,
00:04:56.100 --> 00:04:57.260
what strikes you
00:04:57.380 --> 00:04:57.460
here,
00:04:57.620 --> 00:04:57.740
Michael,
00:04:57.940 --> 00:04:58.820
on their memory usage?
00:04:59.300 --> 00:05:00.500
The pandas library,
00:05:00.800 --> 00:05:02.480
we're talking about 30, this is one of
00:05:02.520 --> 00:05:03.800
those examples, I can't remember which one,
00:05:04.300 --> 00:05:06.520
but four minutes basically for the duration of the
00:05:06.480 --> 00:05:06.760
processing,
00:05:07.380 --> 00:05:08.320
38 gigabytes of RAM.
00:05:08.680 --> 00:05:11.580
If you get into pullers, that gets halved, 18 gig of RAM.
00:05:11.890 --> 00:05:14.360
And if you go into DuckDB to do the same operation,
00:05:14.920 --> 00:05:18.160
five seconds at 1.93 gigabytes of RAM.
00:05:18.920 --> 00:05:20.700
So even the-
00:05:20.700 --> 00:05:23.780
It's like nine to 20 times as much,
00:05:23.980 --> 00:05:24.120
yeah.
00:05:24.400 --> 00:05:24.940
Yeah, I mean,
00:05:25.150 --> 00:05:27.040
an iPhone could do this operation
00:05:27.500 --> 00:05:28.880
against 100 gigabytes of data.
00:05:29.680 --> 00:05:30.420
It's fine though,
00:05:30.450 --> 00:05:31.340
'cause you can just get more memory,
00:05:31.460 --> 00:05:32.260
memory's cheap these days.
00:05:32.820 --> 00:05:33.120
Memory's cheap.
00:05:33.280 --> 00:05:34.680
Totally cheap, totally, totally cheap.
00:05:35.880 --> 00:05:38.280
So I just think folks need to put this one to bed.
00:05:39.220 --> 00:05:40.740
Pandas was probably a good way to start,
00:05:41.180 --> 00:05:42.520
but you'll notice in here,
00:05:42.600 --> 00:05:45.180
like the Polars notation or syntax,
00:05:45.600 --> 00:05:46.200
really, really similar.
00:05:46.620 --> 00:05:48.420
I like the DuckDB syntax.
00:05:49.280 --> 00:05:50.560
I think he's got some examples in here
00:05:50.700 --> 00:05:53.120
where he reads in and does some more operations.
00:05:53.540 --> 00:05:55.420
He does a couple against some larger machines,
00:05:55.800 --> 00:05:57.480
then falls back into an older laptop,
00:05:57.740 --> 00:05:58.640
like a framework 13,
00:05:59.240 --> 00:06:02.260
to show that this is still useful as a developer tool.
00:06:03.340 --> 00:06:06.140
So I think here, Pollard versus DuckDB kind of comes down
00:06:06.280 --> 00:06:07.140
to your workload,
00:06:07.300 --> 00:06:08.380
your experience, and your preference.
00:06:09.040 --> 00:06:09.540
It's a good post.
00:06:09.620 --> 00:06:12.200
I really liked all the code snippets he links over
00:06:13.280 --> 00:06:15.380
into the GitHub where you can actually try it yourself.
00:06:16.160 --> 00:06:19.320
It goes against the New York City taxi data set.
00:06:19.640 --> 00:06:20.700
It's got a ton of data in there,
00:06:21.020 --> 00:06:21.980
so it's fun to play with.
00:06:22.300 --> 00:06:24.120
And you can see that we want to wait minutes
00:06:24.300 --> 00:06:25.140
or do you want to wait seconds?
00:06:25.640 --> 00:06:27.360
And would you want to use all your memory for this?
00:06:28.200 --> 00:06:28.560
And actually,
00:06:28.760 --> 00:06:30.200
another thing that Pollard
00:06:30.220 --> 00:06:32.920
and DuckDB did much better was utilizing the CPU.
00:06:33.300 --> 00:06:36.480
PurePandas in this case was using like a multi-core machine,
00:06:37.020 --> 00:06:38.200
146% CPU,
00:06:38.720 --> 00:06:39.960
where if you go to PureDuckDB,
00:06:40.400 --> 00:06:42.380
there were over 800% CPU.
00:06:42.580 --> 00:06:47.020
So obviously eight to 10 cores are being fully utilized as opposed to
00:06:47.280 --> 00:06:48.900
basically one and one and a half cores.
00:06:49.180 --> 00:06:49.460
Yeah,
00:06:49.660 --> 00:06:49.940
that's awesome.
00:06:50.520 --> 00:06:51.720
I feel like this is a pretty
00:06:52.440 --> 00:06:57.400
data heavy episode for the data science crew out there. It wasn't on purpose, but we ended up that
00:06:57.560 --> 00:06:57.620
way.
00:06:57.840 --> 00:07:00.500
I have a little bit of a real-time follow-up for you, Calvin,
00:07:00.920 --> 00:07:02.059
for people who...
00:07:02.140 --> 00:07:02.160
Yeah,
00:07:02.860 --> 00:07:04.160
I was gonna ask the one last bit in here.
00:07:04.240 --> 00:07:05.960
He does mention Apache Arrow,
00:07:06.070 --> 00:07:06.980
and if you've not played with it,
00:07:07.270 --> 00:07:08.840
it allowed him to switch back and forth
00:07:08.990 --> 00:07:09.800
between pandas,
00:07:09.980 --> 00:07:10.360
Polars,
00:07:10.580 --> 00:07:11.420
and DuckDB
00:07:12.350 --> 00:07:14.540
without having to reload or copy the data into RAM.
00:07:14.860 --> 00:07:16.660
So you could actually do the same operation
00:07:16.940 --> 00:07:17.660
with each of the libraries
00:07:17.870 --> 00:07:19.620
without actually having to take the data back out of RAM.
00:07:19.800 --> 00:07:22.060
So check out Arrow,
00:07:22.580 --> 00:07:23.120
which is really cool.
00:07:23.700 --> 00:07:25.540
It's kind of a little bonus side bit
00:07:25.550 --> 00:07:27.000
that was in the blog post.
00:07:27.800 --> 00:07:30.320
So data folks who got medium-sized data,
00:07:31.020 --> 00:07:32.800
This is going to be a godsend for you.
00:07:33.200 --> 00:07:33.280
Yeah.
00:07:33.490 --> 00:07:35.480
I think Arrow is the foundation of pandas too,
00:07:35.660 --> 00:07:36.340
if I remember correctly,
00:07:36.580 --> 00:07:37.720
and also a polar.
00:07:37.960 --> 00:07:38.700
So that's pretty sweet.
00:07:39.140 --> 00:07:39.300
Yeah.
00:07:39.580 --> 00:07:40.660
My real-time follow-up here.
00:07:40.680 --> 00:07:41.680
It was a real-time follow-up here.
00:07:42.070 --> 00:07:42.180
Oh.
00:07:43.240 --> 00:07:47.720
If you were working with one of these and you want to switch to the other,
00:07:47.980 --> 00:07:48.680
I had Marco
00:07:48.820 --> 00:07:49.320
Garale on,
00:07:49.680 --> 00:07:53.020
really on Talk Python a while ago to talk about narwhals.
00:07:53.440 --> 00:07:58.540
And narwhals is a facade adaptive layer that speaks native polars,
00:07:58.810 --> 00:08:00.920
but also talks pandas.
00:08:01.340 --> 00:08:03.060
So if you want to try like,
00:08:03.280 --> 00:08:03.380
oh,
00:08:03.540 --> 00:08:04.260
let's see what we're doing.
00:08:04.290 --> 00:08:07.420
You know, you could use this as a intermediate layer to kind of swap that out a little more
00:08:07.700 --> 00:08:08.760
easily than rewrite and everything.
00:08:09.060 --> 00:08:09.260
Yeah.
00:08:09.530 --> 00:08:11.760
I think people just need to drop pandas.
00:08:11.810 --> 00:08:12.100
I mean,
00:08:12.470 --> 00:08:13.300
it was great.
00:08:13.370 --> 00:08:14.620
It was great 10 years ago.
00:08:15.760 --> 00:08:16.120
Yeah.
00:08:17.460 --> 00:08:17.560
Yeah.
00:08:18.100 --> 00:08:19.380
I have some funny jokes,
00:08:19.540 --> 00:08:20.860
but let's carry it.
00:08:20.960 --> 00:08:21.720
Let's move on.
00:08:22.040 --> 00:08:24.240
Let's move on to Pydantic Pint.
00:08:24.680 --> 00:08:27.180
So Pydantic Pint is really interesting.
00:08:27.640 --> 00:08:28.340
Do you know Pint?
00:08:28.720 --> 00:08:29.500
Are you familiar with Pint?
00:08:29.520 --> 00:08:34.780
actually i've never i've never used pint so pint we've covered that on the show back in the day
00:08:34.780 --> 00:08:40.599
and pint is interesting because if you're i mean all you got to do is say mars lander sample return
00:08:40.880 --> 00:08:46.200
whatever and it's like the 100 million dollar plus fail because somebody used feet and somebody used
00:08:46.580 --> 00:08:52.040
neighbors or something like that right and so pint lets you do math in python with units attached
00:08:52.480 --> 00:08:58.540
which is pretty cool right so i can say instead of just having a distance i have 42 i have 42
00:08:58.560 --> 00:09:04.280
kilometers and you can say like two miles to whatever and so on so it it basically mean forces
00:09:04.390 --> 00:09:09.880
you to work in units right a lot of times we don't do this as regular programmers but if you do
00:09:10.040 --> 00:09:15.280
anything scientific well there you go right so that's the background on pint but what i want to
00:09:15.280 --> 00:09:21.520
talk about is actually not pint it's called pidantic pint because pidantic is an awesome
00:09:21.720 --> 00:09:27.640
library that lets you validate the inputs and parse them and everything whenever you read some sort of
00:09:27.520 --> 00:09:31.340
JSON, right? Like if it's FastAPI or just a JSON file or whatever,
00:09:31.740 --> 00:09:32.340
you need database,
00:09:33.260 --> 00:09:34.440
SQL model,
00:09:34.680 --> 00:09:35.320
all those things. Yeah.
00:09:35.550 --> 00:09:41.800
So Pydantic Pint takes this idea and adds units to your data validation
00:09:41.980 --> 00:09:42.160
libraries.
00:09:42.760 --> 00:09:45.800
So instead of saying I have a box that has a length and a width,
00:09:46.050 --> 00:09:46.960
I could say I have a box
00:09:46.960 --> 00:09:53.360
that has a length and a width that is a Pint quantity and the validation is to convert it to
00:09:53.300 --> 00:09:53.420
meters.
00:09:53.870 --> 00:09:57.000
So even if you parse something that says feet or centimeters or whatever,
00:09:57.260 --> 00:09:58.200
it will show up
00:09:58.500 --> 00:09:58.660
correctly.
00:09:59.020 --> 00:09:59.620
What do you think?
00:10:00.380 --> 00:10:01.300
That's definitely handy.
00:10:01.460 --> 00:10:01.720
Yeah.
00:10:02.040 --> 00:10:03.200
It's the kind of thing that's like,
00:10:03.370 --> 00:10:03.860
you don't have to,
00:10:04.240 --> 00:10:05.860
you're not going to use it a lot unless you're really in,
00:10:06.120 --> 00:10:06.260
you know,
00:10:06.310 --> 00:10:07.180
some kind of engineering or something.
00:10:07.980 --> 00:10:09.560
Oh my God, this is so good.
00:10:09.720 --> 00:10:10.540
It's like so perfect.
00:10:10.730 --> 00:10:12.940
This has got to solve so many, like you said,
00:10:13.140 --> 00:10:15.560
small mistakes that end up in huge damage.
00:10:16.120 --> 00:10:17.300
A hundred percent. So yeah.
00:10:17.540 --> 00:10:18.780
And to be able to validate with it too.
00:10:19.130 --> 00:10:19.940
Yeah. Just automatically,
00:10:20.140 --> 00:10:23.260
right. Just all the Pydantic validations have a Pydantic base model
00:10:23.280 --> 00:10:25.260
And it parses over to whatever it is.
00:10:25.420 --> 00:10:26.660
And if you put, I don't know,
00:10:27.060 --> 00:10:28.880
liters into the length,
00:10:29.450 --> 00:10:30.180
well, liters,
00:10:30.390 --> 00:10:31.780
you can't convert liters to meters.
00:10:32.420 --> 00:10:33.780
So I don't know.
00:10:34.260 --> 00:10:37.360
And it kind of fits perfectly under the Pydantic scope
00:10:37.720 --> 00:10:40.120
of the data validation and serialization.
00:10:40.560 --> 00:10:41.400
Like it just,
00:10:41.660 --> 00:10:42.180
there's natural,
00:10:42.470 --> 00:10:44.820
like this should exist and they made it exist.
00:10:45.200 --> 00:10:45.700
Yeah, it's really cool.
00:10:45.700 --> 00:10:47.140
So you can have say a FastAPI endpoint
00:10:47.290 --> 00:10:49.040
that just automatically just takes units
00:10:49.240 --> 00:10:50.440
and automatically converts units.
00:10:50.550 --> 00:10:52.380
And yeah, it's a really nice one there.
00:10:52.720 --> 00:10:53.720
Speaking of really nice,
00:10:53.920 --> 00:11:00.240
now this transition here has nothing to do with the sponsorship,
00:11:00.400 --> 00:11:01.000
the previous one.
00:11:01.140 --> 00:11:03.820
They just happen to do great stuff in open source code too.
00:11:04.480 --> 00:11:09.320
But Pydantic also happens to create Logfire, which I told you about at the beginning.
00:11:09.720 --> 00:11:13.020
So let me go ahead and tell you about our sponsorship offer,
00:11:13.280 --> 00:11:13.620
Logfire,
00:11:13.820 --> 00:11:14.560
not Pydantic
00:11:14.700 --> 00:11:16.780
Pint, which is not even from them, but it's based on Pydantic.
00:11:17.140 --> 00:11:18.000
So here's the deal.
00:11:18.600 --> 00:11:19.620
It's 2 a.m.
00:11:19.780 --> 00:11:20.940
Your AI agent failed.
00:11:21.280 --> 00:11:21.900
Was it the model?
00:11:22.040 --> 00:11:22.620
A tool call?
00:11:22.660 --> 00:11:23.120
the database,
00:11:23.570 --> 00:11:25.960
just the general unreliability of,
00:11:26.320 --> 00:11:28.180
hey, I think a new model is coming. So the current
00:11:28.190 --> 00:11:29.560
one starts breaking periodically.
00:11:29.890 --> 00:11:31.420
So most observability tools,
00:11:31.650 --> 00:11:32.700
they can't tell you because
00:11:33.070 --> 00:11:34.200
they only see part of your stack.
00:11:34.560 --> 00:11:35.760
PyDentaglogfire sees all of it.
00:11:36.080 --> 00:11:37.160
One trace across your agents,
00:11:37.380 --> 00:11:39.700
LMs, APIs, and databases down to the infrastructure,
00:11:40.160 --> 00:11:40.420
services,
00:11:40.760 --> 00:11:41.100
Kubernetes,
00:11:41.300 --> 00:11:41.520
hosts.
00:11:41.980 --> 00:11:42.540
It's built on open
00:11:42.760 --> 00:11:44.820
telemetry with SDKs for Python,
00:11:45.380 --> 00:11:45.760
TypeScript,
00:11:45.960 --> 00:11:46.440
and Rust.
00:11:47.160 --> 00:11:49.359
And it works with any OTEL compatible
00:11:49.380 --> 00:11:49.640
language.
00:11:50.140 --> 00:11:50.540
Every prompt,
00:11:50.880 --> 00:11:51.300
token count,
00:11:51.500 --> 00:11:54.620
and cost right next to your vector searches and API calls.
00:11:54.940 --> 00:11:59.420
You query everything with Postgres compatible SQL to understand what your app is doing,
00:12:00.400 --> 00:12:02.420
and so can your coding agent.
00:12:02.610 --> 00:12:05.100
It can use the same way because it talks SQL really well.
00:12:05.260 --> 00:12:05.500
If you
00:12:05.720 --> 00:12:06.720
connect it to the MCP server,
00:12:07.050 --> 00:12:09.940
your agent can also figure out what is going on. So stop guessing,
00:12:10.250 --> 00:12:10.720
read the trace,
00:12:10.940 --> 00:12:11.660
Pydenic, Logfire.
00:12:11.960 --> 00:12:12.400
AI,
00:12:12.670 --> 00:12:14.140
it is still just engineering,
00:12:14.750 --> 00:12:16.039
even if it's weird engineering
00:12:16.060 --> 00:12:16.400
these days.
00:12:16.690 --> 00:12:20.260
So visit pythonbytes.fm/logfire today and sign up.
00:12:20.540 --> 00:12:21.540
Get 10 million records free
00:12:21.810 --> 00:12:22.220
every month,
00:12:22.440 --> 00:12:23.340
no credit card required.
00:12:23.710 --> 00:12:24.700
You can even click,
00:12:24.870 --> 00:12:26.260
and I really like this, there's a little
00:12:26.480 --> 00:12:27.340
copy of this text,
00:12:27.560 --> 00:12:29.860
onboard you with your agent.
00:12:30.340 --> 00:12:32.180
Click that and it gives you a prompt you can drop
00:12:32.320 --> 00:12:36.460
in to Claude Code or Codex or whatever and it automatically knows what to do to set up Logfire
00:12:36.520 --> 00:12:38.660
in your app. So thank you to Pydantic for supporting the show.
00:12:39.000 --> 00:12:41.359
Calvin, I know you're a big fan of the
00:12:41.420 --> 00:12:42.580
visibility into the token.
00:12:43.080 --> 00:12:43.180
Yeah.
00:12:43.820 --> 00:12:47.280
I was just curious now if copy the setup prompt is the new
00:12:47.740 --> 00:12:48.720
pipe to bash,
00:12:49.130 --> 00:12:50.920
like pipe some curl to bash.
00:12:51.380 --> 00:12:51.540
Yes.
00:12:51.790 --> 00:12:53.860
This is the replacing that. I think it is.
00:12:54.260 --> 00:12:54.960
That's why I was getting it. Yeah. No,
00:12:55.120 --> 00:12:56.780
no. I think it is. And it's amazing.
00:12:57.780 --> 00:12:58.860
I have some stuff that
00:12:58.870 --> 00:13:00.120
I'm working on. I'm like, oh, this is like,
00:13:01.020 --> 00:13:02.000
this idea is perfect.
00:13:02.150 --> 00:13:02.920
I love it so much.
00:13:03.210 --> 00:13:03.740
So yeah,
00:13:04.210 --> 00:13:05.980
pretty cool. Thanks to Pydantic for sponsoring the show.
00:13:06.370 --> 00:13:06.680
Thank you.
00:13:07.000 --> 00:13:09.399
And let's jump over to
00:13:09.820 --> 00:13:13.460
your topic next. You've got in the wrong order in the other. So what's next?
00:13:13.620 --> 00:13:13.960
Well,
00:13:14.980 --> 00:13:15.440
speaking of
00:13:15.800 --> 00:13:16.120
Pydantic,
00:13:16.920 --> 00:13:22.000
this comes from Bob Builderboss, a friend of the show. I know you've had him on numerous
00:13:22.240 --> 00:13:24.780
times for other events and things,
00:13:25.240 --> 00:13:27.520
but this one is about how to run,
00:13:27.980 --> 00:13:29.240
how Rust code becomes something
00:13:29.280 --> 00:13:29.920
you can import.
00:13:30.340 --> 00:13:32.840
I think it's interesting that we can,
00:13:33.080 --> 00:13:34.980
if people are complaining about performance,
00:13:35.260 --> 00:13:41.440
the first news article I had about getting rid of pandas and bringing back in with polars and
00:13:41.660 --> 00:13:43.300
DuckTV was about performance.
00:13:43.560 --> 00:13:45.080
This is similarly veined.
00:13:45.360 --> 00:13:48.400
Like if I've got a very computationally
00:13:48.900 --> 00:13:53.660
intense data structure or function that's happening in my program,
00:13:53.900 --> 00:13:55.360
it'd be sure be nice if I could
00:13:55.440 --> 00:14:00.240
maybe replace it out with a Rust version of that, but have it act natively inside of my Python code.
00:14:00.580 --> 00:14:06.780
So this blog post from Bob goes over basically what Pydantic V2 does, which is a data validation
00:14:06.940 --> 00:14:09.480
library that most Python apps are using these days.
00:14:09.660 --> 00:14:11.480
It's actually a Rust extension under the
00:14:11.560 --> 00:14:12.360
covers that does the work.
00:14:12.620 --> 00:14:13.180
Its core,
00:14:13.360 --> 00:14:15.860
Pydantic core, is all built with Py03,
00:14:16.080 --> 00:14:17.340
the same tool chain we're
00:14:17.340 --> 00:14:18.340
going to use here in this example.
00:14:18.800 --> 00:14:23.120
So it goes over some examples of basically you write a normal Rust
00:14:23.240 --> 00:14:23.520
module,
00:14:23.820 --> 00:14:26.200
you annotate it with some specific macros,
00:14:26.560 --> 00:14:28.719
and then you'll actually be able to import that
00:14:28.780 --> 00:14:32.360
into your Python code fairly naturally.
00:14:32.770 --> 00:14:37.040
I think there's basically the Rust parser is incredibly fast.
00:14:37.640 --> 00:14:41.060
Python never touches anything until the boundary crossing.
00:14:41.130 --> 00:14:42.680
So if you call for some data,
00:14:43.210 --> 00:14:44.440
that all happens over in Rust.
00:14:44.860 --> 00:14:48.220
The thing that I think the article covers that's really important is that
00:14:48.520 --> 00:14:51.100
if you pull back that data across the boundary,
00:14:51.760 --> 00:14:55.380
those Rust results get turned into Python objects.
00:14:55.780 --> 00:14:59.640
And so you're going to want to think carefully about how you bring back parts of that, because
00:14:59.800 --> 00:15:03.480
maybe you're only interested in a small piece of what is coming back and you don't need to
00:15:03.720 --> 00:15:03.900
populate.
00:15:04.100 --> 00:15:04.580
Say, for example,
00:15:05.220 --> 00:15:09.120
100,000 JSON values means that you're going to get 100,000 Python dictionary
00:15:09.320 --> 00:15:12.280
objects after the parsing is all done and everything gets passed back.
00:15:12.500 --> 00:15:16.360
You don't incur the penalty until you cross that threshold back into Python land.
00:15:16.880 --> 00:15:18.580
Maybe you don't need all 100,000,
00:15:18.800 --> 00:15:21.340
but there's some other operation you can do to get down
00:15:21.500 --> 00:15:22.860
to just the pieces you need.
00:15:22.920 --> 00:15:25.880
So what's nice is errors cross a boundary too.
00:15:26.140 --> 00:15:28.040
So if Rust runs into errors,
00:15:28.440 --> 00:15:30.040
those come back as Python exceptions.
00:15:30.500 --> 00:15:33.360
So it makes it easy to debug and figure out what's going on.
00:15:33.370 --> 00:15:37.360
So you get clean Python semantics while still leveraging Rust.
00:15:38.140 --> 00:15:41.080
So basically for anyone porting Rust, if you're returning a scalar,
00:15:41.240 --> 00:15:41.660
don't sweat it.
00:15:42.080 --> 00:15:43.260
If you're returning a big structure,
00:15:43.460 --> 00:15:44.400
profile the boundary.
00:15:44.870 --> 00:15:51.540
And if that's a real cost, you want to switch over and maybe do more of the algorithm on the Rust side.
00:15:51.860 --> 00:15:56.660
So much like you can use C or other languages in,
00:15:57.000 --> 00:15:59.200
I don't know if you can, can you use Ruby to do this kind of thing?
00:15:59.620 --> 00:16:00.320
I don't know if you can or not,
00:16:00.560 --> 00:16:03.000
but you definitely use Rust.
00:16:03.270 --> 00:16:04.380
And I'm kind of excited about that.
00:16:04.450 --> 00:16:05.920
I know you've been doing some coursework on it.
00:16:06.020 --> 00:16:10.480
And it sounds like Bob has also made some learning materials to lead folks through.
00:16:10.910 --> 00:16:16.519
It just feels like there's a really nice friendship between the Rust communities and the Python communities
00:16:16.980 --> 00:16:19.520
and all the niceties that have been put in place
00:16:19.700 --> 00:16:22.200
to allow us to use Rust almost natively
00:16:22.430 --> 00:16:23.080
over in the Python world.
00:16:23.400 --> 00:16:24.000
So thanks,
00:16:24.200 --> 00:16:26.380
Bob, for the awesome post about that.
00:16:26.680 --> 00:16:26.860
He's,
00:16:27.040 --> 00:16:28.240
again, code examples in here,
00:16:28.740 --> 00:16:30.720
kind of explains to the Python folks
00:16:30.860 --> 00:16:31.880
who have never touched Rust
00:16:32.080 --> 00:16:33.760
what the function signatures look like,
00:16:33.790 --> 00:16:36.060
which I appreciate because breaking it down
00:16:36.150 --> 00:16:37.660
and telling me what each of those pieces means
00:16:38.080 --> 00:16:40.760
that I can now pretty easily read some Rust code
00:16:40.810 --> 00:16:41.720
and understand what's going on
00:16:41.790 --> 00:16:44.640
because it doesn't look terribly foreign to me,
00:16:44.780 --> 00:16:45.960
but it's just different enough,
00:16:46.080 --> 00:16:50.160
but this goes over a good usage of what each of those pieces mean for you.
00:16:50.360 --> 00:16:52.180
Yeah, it's pretty surprisingly similar to Python,
00:16:52.490 --> 00:16:52.580
honestly.
00:16:52.840 --> 00:16:52.960
Yeah,
00:16:53.250 --> 00:16:53.360
yeah,
00:16:53.520 --> 00:16:54.980
and incredibly fast,
00:16:55.140 --> 00:16:56.780
but you get to think about things a little differently
00:16:56.920 --> 00:16:58.540
because of the way it manages memory,
00:16:59.000 --> 00:17:01.480
and I think that's the big differentiator.
00:17:01.660 --> 00:17:03.440
What was that cartoon with the guys?
00:17:03.570 --> 00:17:05.459
Like, I'd gladly pay you on Tuesday.
00:17:05.839 --> 00:17:06.380
It was Popeye.
00:17:07.339 --> 00:17:08.300
Popeye, that was Popeye,
00:17:08.480 --> 00:17:08.540
right?
00:17:08.699 --> 00:17:08.939
Yeah, it was.
00:17:08.980 --> 00:17:10.319
I mean, I was thinking borrow checkers.
00:17:10.520 --> 00:17:15.339
It was Wimpy, who will glad you pay you Tuesday for a hamburger today.
00:17:15.540 --> 00:17:16.020
Yeah,
00:17:16.240 --> 00:17:19.280
that's the difference of Rust is you've got the borrow checker.
00:17:20.130 --> 00:17:20.600
Always checking.
00:17:20.910 --> 00:17:21.160
Yep, yep.
00:17:21.500 --> 00:17:21.839
Always checking.
00:17:22.620 --> 00:17:22.860
So yeah,
00:17:23.089 --> 00:17:23.400
I don't know.
00:17:23.900 --> 00:17:28.199
You've been doing a little more with Rust and Python and teaching some folks these things.
00:17:28.459 --> 00:17:28.680
Yeah,
00:17:28.880 --> 00:17:29.020
yeah,
00:17:29.220 --> 00:17:30.220
a little bit, a little bit.
00:17:30.480 --> 00:17:33.180
I have two follow-ups here.
00:17:33.530 --> 00:17:36.080
So you talked about DuckDB,
00:17:36.530 --> 00:17:39.020
but the question is, do you have a lake house?
00:17:39.400 --> 00:17:39.940
Do you have a lake?
00:17:40.070 --> 00:17:40.360
I mean,
00:17:40.570 --> 00:17:41.020
a lake house.
00:17:41.100 --> 00:17:43.360
So we've heard of data lakes,
00:17:44.320 --> 00:17:47.720
which is a place you just kind of dump a ton of like an insane.
00:17:47.860 --> 00:17:49.940
This is like back to your big data thing, DuckDB thing.
00:17:50.410 --> 00:17:53.280
You just dump a bunch of data into this data lake and you figure it out.
00:17:53.540 --> 00:17:53.640
Well,
00:17:53.960 --> 00:17:55.320
that's grown up a little bit.
00:17:55.680 --> 00:17:58.920
And now there's this thing called DuckDB,
00:17:59.240 --> 00:18:00.860
but it's an implementation,
00:18:01.190 --> 00:18:03.820
an example of what's called an open lake format.
00:18:04.380 --> 00:18:04.600
Who knew?
00:18:04.730 --> 00:18:05.060
Do you know?
00:18:05.380 --> 00:18:06.980
I didn't know that.
00:18:07.080 --> 00:18:08.840
We've done lake house implementations.
00:18:08.870 --> 00:18:10.820
I didn't know there was an open lake format now.
00:18:10.920 --> 00:18:12.260
So the story is,
00:18:12.920 --> 00:18:16.860
what if we could use S3 to scale our data access?
00:18:17.780 --> 00:18:19.180
S3 scale is pretty large.
00:18:19.540 --> 00:18:21.680
If you can read stuff off the file system instead of out of memory,
00:18:21.920 --> 00:18:23.640
you can scale that tremendously large.
00:18:23.940 --> 00:18:29.540
In the open lake story is, well, if you put file formats in S3
00:18:29.920 --> 00:18:32.100
that everything could read, like maybe JSON files
00:18:32.260 --> 00:18:34.720
that tell you what the files mean, you could read them first.
00:18:35.080 --> 00:18:36.360
Here's where the data lives in each piece.
00:18:36.720 --> 00:18:37.600
And then parquet files.
00:18:38.040 --> 00:18:38.120
Yeah.
00:18:38.320 --> 00:18:40.020
Or zipped CSV.
00:18:40.400 --> 00:18:45.440
don't know take your pick right it could be whatever so i just had the folks from duck lake on which is
00:18:45.540 --> 00:18:54.200
a duck db implementation story of uh this open lake format well then i get this message here saying
00:18:54.480 --> 00:18:59.620
yep guess what aws i'm sure you know this is a i did i did see this one come by yeah yeah and
00:18:59.920 --> 00:19:06.599
in duck labs just offered basically the h1 is bad let me read let me read the first sentence today
00:19:06.620 --> 00:19:11.480
we are announcing that Amazon has signed an agreement to acquire Duck Labs,
00:19:11.890 --> 00:19:12.520
the Amsterdam
00:19:12.810 --> 00:19:15.960
based company behind the open source analytical database DuckDB.
00:19:16.240 --> 00:19:18.160
I'll put a link to the announcement
00:19:18.210 --> 00:19:18.840
and I thought, well,
00:19:19.240 --> 00:19:19.320
hmm,
00:19:19.760 --> 00:19:20.480
what does this even mean?
00:19:21.180 --> 00:19:22.700
And I didn't, I wasn't entirely sure.
00:19:22.870 --> 00:19:23.480
So I,
00:19:23.890 --> 00:19:24.520
I like,
00:19:24.770 --> 00:19:26.380
I went and did some looking here. I'm like,
00:19:26.540 --> 00:19:26.720
there's,
00:19:27.100 --> 00:19:28.920
there's actually a lot of pieces in play.
00:19:29.680 --> 00:19:32.400
So let me lay it out and I'll tell you what part AWS acquired,
00:19:32.820 --> 00:19:34.220
what part didn't. Okay. So first of
00:19:34.200 --> 00:19:34.240
Well,
00:19:34.400 --> 00:19:35.920
thanks to Dylan McConnell who sent this in.
00:19:36.200 --> 00:19:38.260
What does this mean for the DuckDB ecosystem?
00:19:38.540 --> 00:19:39.400
So first of all,
00:19:39.560 --> 00:19:39.860
DuckDB,
00:19:40.160 --> 00:19:41.280
which you gave a shout out before,
00:19:41.660 --> 00:19:42.020
twice really,
00:19:43.340 --> 00:19:46.660
is the open source in-pros analytical SQL engine,
00:19:47.300 --> 00:19:48.020
MIT licensed.
00:19:48.400 --> 00:19:49.440
It's like SQLite,
00:19:49.840 --> 00:19:50.960
but for columnar data,
00:19:51.410 --> 00:19:52.240
which if you're And
00:19:52.240 --> 00:19:53.360
way more, and way, way more.
00:19:53.940 --> 00:19:54.180
Yeah.
00:19:54.420 --> 00:19:54.860
Anything you
00:19:54.860 --> 00:19:56.740
pointed at becomes SQL queryable. It's
00:19:56.740 --> 00:19:57.020
amazing.
00:19:57.520 --> 00:19:57.620
Yeah,
00:19:57.750 --> 00:19:58.340
yeah, yeah. So you
00:19:58.340 --> 00:20:00.680
can say pointed at a pandas data frame and then
00:20:00.680 --> 00:20:02.800
do SQL queries against your Panda. Like there's
00:20:02.800 --> 00:20:03.700
a bunch of plugins.
00:20:03.940 --> 00:20:05.940
It's far beyond just a database,
00:20:06.580 --> 00:20:09.700
but it's an in-process sort of data processing engine,
00:20:10.180 --> 00:20:11.620
much like that.
00:20:11.920 --> 00:20:14.300
So the IP of this is not owned by any company.
00:20:14.500 --> 00:20:16.860
It's held in a DuckDB foundation.
00:20:17.240 --> 00:20:17.360
Oh,
00:20:17.520 --> 00:20:18.140
good. Thank goodness.
00:20:18.180 --> 00:20:18.820
That sounds good.
00:20:19.100 --> 00:20:19.600
That is good.
00:20:19.600 --> 00:20:21.580
It was spun out of CWI Amsterdam
00:20:21.810 --> 00:20:24.500
from the folks who mentioned that article.
00:20:25.000 --> 00:20:25.100
Good,
00:20:25.540 --> 00:20:26.940
but we'll come back to it.
00:20:27.180 --> 00:20:28.280
Then there's Duck Labs.
00:20:28.720 --> 00:20:32.960
And the story is Amazon AWS has acquired Duck Labs.
00:20:33.360 --> 00:20:36.480
This is the company formerly branded DuckDB Labs,
00:20:36.700 --> 00:20:39.460
founded over five years ago by Hannes
00:20:39.880 --> 00:20:44.280
Mullison and Mark Roosevelt to give DuckDB a stable home,
00:20:44.720 --> 00:20:45.260
bootstrapped,
00:20:45.500 --> 00:20:46.220
grew to 30 people.
00:20:46.760 --> 00:20:48.300
Now they can go chill on their island,
00:20:48.480 --> 00:20:49.300
which congrats to them.
00:20:49.400 --> 00:20:49.620
That's awesome.
00:20:50.360 --> 00:20:52.000
Because DuckDB really has taken over, right?
00:20:52.600 --> 00:20:53.920
Then there's DuckLake,
00:20:54.280 --> 00:20:55.920
which I mentioned earlier.
00:20:56.560 --> 00:20:58.980
It's one of three projects by DuckLab.
00:20:59.380 --> 00:21:01.200
And there's DuckDB,
00:21:01.460 --> 00:21:01.960
there's DuckLake.
00:21:01.990 --> 00:21:05.440
And then there's an API for working with this called Quack.
00:21:05.780 --> 00:21:06.960
Check out the Talk Python episode.
00:21:07.390 --> 00:21:10.420
But it's an open lake, lake house format.
00:21:10.450 --> 00:21:13.200
All the waterfowl puns are great.
00:21:13.480 --> 00:21:13.880
It is.
00:21:14.020 --> 00:21:16.820
And we actually on the podcast had a fun conversation about like,
00:21:17.140 --> 00:21:18.380
do you need a more serious name?
00:21:18.810 --> 00:21:20.980
Like RPC for your data lake.
00:21:21.140 --> 00:21:21.220
Like,
00:21:21.380 --> 00:21:23.000
no, they're like, we're calling it Quack.
00:21:23.220 --> 00:21:23.580
Come on now.
00:21:23.900 --> 00:21:26.000
And then also we have MotherDuck,
00:21:26.320 --> 00:21:31.340
which I think I thought Mother Duck was online version of DuckDB.
00:21:31.730 --> 00:21:32.180
But no,
00:21:32.480 --> 00:21:36.300
that is a separate Seattle company selling serverless hosted DuckDB.
00:21:36.860 --> 00:21:39.400
And originally it was started in partnership with DuckDB Labs
00:21:39.410 --> 00:21:42.080
and they worked closely with Hannes and Mark for years
00:21:42.680 --> 00:21:45.540
and even contracted Duck Labs for some of the engineering.
00:21:45.980 --> 00:21:46.640
So now,
00:21:46.820 --> 00:21:48.060
what does this all mean?
00:21:48.310 --> 00:21:50.740
So the foundation owning DuckDB is awesome,
00:21:51.220 --> 00:21:52.200
but it has three directors,
00:21:52.710 --> 00:21:53.940
the people who own Duck Labs.
00:21:55.720 --> 00:22:00.180
So there's a bit of a, how much independence is it really going to have?
00:22:00.480 --> 00:22:03.020
There are some other folks,
00:22:03.360 --> 00:22:05.580
other governance and so on there.
00:22:05.880 --> 00:22:06.680
But, you know, it's cool.
00:22:06.760 --> 00:22:07.220
There's a foundation.
00:22:07.680 --> 00:22:09.780
It's not super independent of DuckDB at the moment.
00:22:09.980 --> 00:22:10.780
Maybe it will be, though,
00:22:11.020 --> 00:22:11.460
after this.
00:22:11.800 --> 00:22:15.040
Mother Duck immediately moved into the business that Duck Labs vacated.
00:22:15.320 --> 00:22:18.220
They now sell enterprise support for DuckDB and so on.
00:22:19.200 --> 00:22:22.040
Everyone expects an AWS DuckDB service,
00:22:22.400 --> 00:22:24.060
probably a Duck Lake as well.
00:22:24.180 --> 00:22:30.180
already used s3 right but maybe just a little more formal yeah there's s3 query and some adjacent
00:22:30.620 --> 00:22:35.520
like technologies that sound like this may maybe this will augment or replace what's really nice
00:22:35.570 --> 00:22:42.400
about ducklapse is it runs a local duckdb or a local postgres server and a lot of the chatty api
00:22:42.400 --> 00:22:46.600
that would come from an open table format and the metadata now all happen in the database and then
00:22:46.600 --> 00:22:51.500
it just fetches and reads the files yeah that's pretty cool yeah we can't beat physics if we can
00:22:51.400 --> 00:22:55.680
keep the data, the bits where they're at physically and bring the compute to it.
00:22:56.020 --> 00:22:56.440
That's the win.
00:22:56.660 --> 00:22:56.740
Yeah.
00:22:57.040 --> 00:23:02.060
So the duck lake angle actually is probably the most interesting one because AWS heavily
00:23:02.440 --> 00:23:04.180
committed to iceberg through S3 tables,
00:23:04.500 --> 00:23:06.780
which is a competitor,
00:23:07.220 --> 00:23:08.820
at least a competitor competing
00:23:09.040 --> 00:23:10.960
concept to a duck lake.
00:23:11.390 --> 00:23:12.500
So yeah, check it out.
00:23:12.770 --> 00:23:15.140
I think AWS just got better.
00:23:15.360 --> 00:23:15.760
We'll see what that
00:23:15.880 --> 00:23:16.600
means for the rest of the world.
00:23:16.890 --> 00:23:18.860
What do you think? I mean, you're on the inside of this a little bit.
00:23:19.180 --> 00:23:19.240
Yeah,
00:23:19.420 --> 00:23:20.760
a little bit, but you know, they've,
00:23:21.020 --> 00:23:23.240
we've had mixed reviews on their handling of open source,
00:23:23.620 --> 00:23:30.720
but luckily they don't have any control over the open source other than they've just bought out the founders who are on the board of the open source foundation.
00:23:31.220 --> 00:23:33.060
It's still separate enough of an entity.
00:23:33.170 --> 00:23:34.880
I don't think there's a conflict here.
00:23:35.210 --> 00:23:36.200
I mean, it's going to be good for the project.
00:23:36.390 --> 00:23:37.380
The project is already incredible.
00:23:37.540 --> 00:23:38.680
Like the DuckDB stuff is,
00:23:38.960 --> 00:23:39.420
it's just like,
00:23:39.740 --> 00:23:48.860
if someone had thought about SQLite and said, I need to grow all these other features that handle all kinds of crazy data and do give me native like JSON access and functions.
00:23:49.000 --> 00:23:54.640
And it's a really great platform for building cool little utilities or talking to giant chunks of data,
00:23:55.040 --> 00:23:56.600
as we saw in the first article.
00:23:56.700 --> 00:23:56.920
Yeah,
00:23:57.120 --> 00:23:57.700
yeah, yeah, absolutely.
00:23:57.760 --> 00:23:58.740
And use barely any memory.
00:23:59.220 --> 00:23:59.360
I mean,
00:23:59.560 --> 00:24:00.960
that's why this will run and work.
00:24:01.280 --> 00:24:03.340
Yeah, the DuckDB part is really interesting on that.
00:24:03.360 --> 00:24:04.999
And then the Duck Lake is like insane.
00:24:05.840 --> 00:24:09.240
You know, you could have terabytes of Parquet files all broken in little bits.
00:24:09.240 --> 00:24:09.740
No big deal.
00:24:10.180 --> 00:24:10.920
No big deal.
00:24:11.140 --> 00:24:11.620
NBD,
00:24:12.100 --> 00:24:12.460
NBD.
00:24:12.800 --> 00:24:13.040
Yeah,
00:24:13.240 --> 00:24:13.640
I love it.
00:24:13.640 --> 00:24:14.900
Well, how about some extras?
00:24:15.320 --> 00:24:20.620
Well, I will continue on my beating the uv drum.
00:24:21.090 --> 00:24:22.240
The latest release,
00:24:22.480 --> 00:24:25.120
every week we got something new, the latest release from the uv folks,
00:24:25.490 --> 00:24:28.700
we get code signing on Mac and Windows.
00:24:29.490 --> 00:24:32.080
So the binaries are now officially signed,
00:24:32.480 --> 00:24:32.940
code signed.
00:24:33.700 --> 00:24:33.820
Again,
00:24:34.070 --> 00:24:36.160
I think this is all coming together,
00:24:37.080 --> 00:24:39.500
ensuring we can secure the software supply
00:24:39.780 --> 00:24:40.500
chain part of this.
00:24:40.530 --> 00:24:42.660
So I'm excited to see that release too.
00:24:42.740 --> 00:24:46.920
I was hoping I wouldn't see a uv thing this week, but sure enough, it popped up in my feed.
00:24:47.290 --> 00:24:52.580
And I was like, I have to mention it because they just keep making everything better and better and better.
00:24:52.800 --> 00:24:54.640
So uv is now code signed.
00:24:55.040 --> 00:25:00.600
So you can trust that it came from the right source on your own machine if you're on Mac and Linux.
00:25:01.040 --> 00:25:02.160
Yeah, that's excellent.
00:25:02.620 --> 00:25:02.900
Excellent, excellent.
00:25:03.060 --> 00:25:03.240
Yeah.
00:25:03.620 --> 00:25:04.460
I see it's still going.
00:25:04.650 --> 00:25:07.300
Oh my gosh, the code signing is such a pain these days.
00:25:07.520 --> 00:25:11.920
It used to be you could just build an EXE or a.app and you could just go here, try my app.
00:25:12.540 --> 00:25:13.540
How dangerous is that?
00:25:15.140 --> 00:25:15.500
I know.
00:25:15.640 --> 00:25:16.820
That's how the world used to be, though.
00:25:17.440 --> 00:25:18.240
Well, we used to have, what,
00:25:18.400 --> 00:25:22.040
R login with no password to remote machines?
00:25:22.540 --> 00:25:22.980
What could go wrong?
00:25:23.420 --> 00:25:23.740
It's fine.
00:25:24.020 --> 00:25:24.340
It's fine.
00:25:24.520 --> 00:25:24.780
Trust.
00:25:25.200 --> 00:25:26.200
You've got to have a lot of trust.
00:25:27.040 --> 00:25:28.740
Why would somebody do something mean to computers?
00:25:29.140 --> 00:25:29.540
I don't know.
00:25:29.660 --> 00:25:29.980
I don't know.
00:25:30.220 --> 00:25:32.040
I remember in Windows 95,
00:25:32.640 --> 00:25:35.460
we had a bunch of them at a university I worked at.
00:25:35.860 --> 00:25:38.300
Plugged straight in to Ethernet.
00:25:38.640 --> 00:25:40.800
And the Ethernet, everyone got its own IP address.
00:25:41.080 --> 00:25:41.640
And guess what?
00:25:41.680 --> 00:25:43.580
That thing got taken over pretty quickly.
00:25:44.160 --> 00:25:45.380
The university I was at,
00:25:45.630 --> 00:25:47.720
they all had public IP addresses in the labs too.
00:25:48.080 --> 00:25:48.800
Yeah, it didn't go well.
00:25:49.080 --> 00:25:49.240
No.
00:25:49.740 --> 00:25:50.600
It did not go well.
00:25:51.040 --> 00:25:53.620
Speaking of things that need to be patched and updated, check this out.
00:25:54.040 --> 00:25:56.460
So there's two things that involve restarts.
00:25:56.840 --> 00:25:58.100
But I have a MacBook Pro,
00:25:58.600 --> 00:25:59.180
M5 Pro.
00:25:59.500 --> 00:26:00.000
Very nice.
00:26:00.120 --> 00:26:00.360
Love it.
00:26:00.360 --> 00:26:02.640
I got it this summer or earlier, maybe spring.
00:26:02.690 --> 00:26:03.480
I don't know, whenever I got it.
00:26:03.800 --> 00:26:05.620
And it came with a power supply.
00:26:05.960 --> 00:26:06.500
One of those power bricks.
00:26:06.940 --> 00:26:12.960
the power brick had to reboot the other day to to update itself i was sitting there working and i
00:26:12.960 --> 00:26:19.000
saw this article come up come by say apple releases a firmware update for the 140 watt usbc power
00:26:19.240 --> 00:26:23.620
adapter which is the one that runs the macbook and it's almost that that's almost actual size
00:26:23.690 --> 00:26:29.180
right there it's so big yeah it's it's a beefy boy it is uh yeah i think this is even a little
00:26:29.310 --> 00:26:34.040
small this big old picture of it it's it's it's heavy but i was i read this article and i was
00:26:33.960 --> 00:26:35.820
sitting there working and my Mac,
00:26:36.070 --> 00:26:37.320
when it comes off of power,
00:26:37.620 --> 00:26:38.780
it dims the monitor.
00:26:39.070 --> 00:26:39.200
Yeah.
00:26:39.550 --> 00:26:40.080
So I'm,
00:26:40.200 --> 00:26:43.820
I'm just, you know, and I was an hour or two later, I was sitting there and everything goes dim for a
00:26:43.950 --> 00:26:44.040
second.
00:26:44.350 --> 00:26:46.560
The little power just disconnects.
00:26:46.810 --> 00:26:47.240
And then two,
00:26:47.570 --> 00:26:47.660
two,
00:26:47.820 --> 00:26:47.900
three,
00:26:48.280 --> 00:26:49.140
five seconds later, something
00:26:49.240 --> 00:26:49.500
like that,
00:26:49.780 --> 00:26:50.280
power comes back,
00:26:50.500 --> 00:26:52.860
brightness comes back. I'm like, I just,
00:26:53.380 --> 00:26:54.920
my power break just rebooted.
00:26:55.080 --> 00:26:56.300
What in the world is going on here?
00:26:56.480 --> 00:26:57.080
That's crazy.
00:26:57.680 --> 00:27:00.560
I've, I've, I've replaced all my Mac power books.
00:27:00.940 --> 00:27:01.480
I've got,
00:27:01.950 --> 00:27:03.580
not that I'm trying to be an ad for Anchor
00:27:03.580 --> 00:27:04.980
or anything like that, but the Anchor Prime
00:27:05.540 --> 00:27:09.000
has 160 watt, like itty bitty little power brick
00:27:09.160 --> 00:27:09.280
that,
00:27:09.520 --> 00:27:10.440
because I travel quite a bit,
00:27:10.780 --> 00:27:13.500
it has four USB-C ports on it
00:27:13.690 --> 00:27:18.220
and they can all deliver a combined sum of 160 watts.
00:27:18.610 --> 00:27:22.220
So I can full bore charge my MacBook Pro M4 Max
00:27:22.820 --> 00:27:25.440
and my iPad and my phone all at the same time.
00:27:25.680 --> 00:27:26.180
That's beautiful.
00:27:26.640 --> 00:27:28.340
And it's smaller than that brick.
00:27:28.980 --> 00:27:30.680
Yeah, I'm also a fan of the Anchor stuff.
00:27:30.960 --> 00:27:33.780
This one, since I had it anyway, I just plugged it into the wall part of the house.
00:27:33.880 --> 00:27:36.780
And just if I'm in that part of the house, I just grabbed that cord.
00:27:37.120 --> 00:27:41.320
But yeah, normally if I travel, I have an Anker that's actually a power brick, a little battery.
00:27:41.760 --> 00:27:45.000
And it has two USB things and it'll do not quite as high as yours,
00:27:45.280 --> 00:27:46.020
but pretty high.
00:27:46.360 --> 00:27:49.000
And it's super nice because it's also a power brick,
00:27:49.380 --> 00:27:49.460
right?
00:27:49.640 --> 00:27:50.540
So if I need to charge up,
00:27:50.700 --> 00:27:52.900
it'll even charge the MacBook,
00:27:53.180 --> 00:27:54.900
but then just get it and just plug it into the wall.
00:27:54.980 --> 00:27:56.420
And then it just becomes a power thing.
00:27:56.740 --> 00:27:58.160
I'm also a fan of this Anker stuff.
00:27:58.500 --> 00:28:00.280
Okay, a couple more extras really quick here.
00:28:00.880 --> 00:28:07.700
we've got do not play yeah so the story of vs code the official documentary is out have you
00:28:07.800 --> 00:28:12.360
watched this no i have not watched this it's an hour and 38 minutes and i'm here for it okay
00:28:12.880 --> 00:28:17.720
all right it's got a lot of people that uh maybe you didn't see coming like Eric gamma for example
00:28:17.900 --> 00:28:21.800
you know thinking back to the gang of four patterns and all that kind of stuff because
00:28:22.000 --> 00:28:27.839
he was apparently involved in the early days um so yeah cool we had uh cult repo do the python
00:28:27.860 --> 00:28:33.500
documentary we had them do the jet brains document or intellij documentary and here's the vs code one
00:28:33.660 --> 00:28:37.680
i'm really loving these like high quality production i mean these are these are nice
00:28:38.120 --> 00:28:43.360
little nice videos they yeah a lot of people behind it um i mean there's such there's an
00:28:43.440 --> 00:28:48.920
audience for all these things i'm here for it too i love the fact that like the underdogs can feel
00:28:49.660 --> 00:28:54.700
they are important for us and we can now hear more of the story about how some of these things came
00:28:54.660 --> 00:29:00.120
about yeah it's really interesting i mean vs code has taken over so much and then yeah anyway and the
00:29:00.160 --> 00:29:05.940
origins are way way more uh less ambitious let's say it's cool to check out so it also has over
00:29:06.020 --> 00:29:09.580
half a million views so there is an audience for this apparently there's absolutely audience okay
00:29:09.720 --> 00:29:14.320
think speaking of things that make you reboot yesterday last night yesterday mac os golden
00:29:14.500 --> 00:29:21.279
gate ios golden gate watch os 27 golden gate all those things came out so uh-oh did you upgrade i
00:29:21.300 --> 00:29:22.300
I did. Why wouldn't I mean,
00:29:22.460 --> 00:29:23.440
I'm like, oh, let's go.
00:29:23.980 --> 00:29:25.040
I'm not yet upgraded on this.
00:29:25.080 --> 00:29:26.040
I'm usually of that opinion,
00:29:26.400 --> 00:29:31.020
but lately I may wait a month or until a.1 or.1 to come out.
00:29:32.460 --> 00:29:34.840
I spent one day working with it and so far it's okay.
00:29:35.280 --> 00:29:35.480
Okay.
00:29:35.760 --> 00:29:37.960
I'm going to upgrade then on your full recommendation.
00:29:38.420 --> 00:29:38.780
Well,
00:29:39.260 --> 00:29:41.820
I've not upgraded my MacBook or my streaming computer.
00:29:41.870 --> 00:29:43.600
I only recorded my main desktop.
00:29:44.380 --> 00:29:45.280
So we'll see.
00:29:45.640 --> 00:29:47.300
Ask me next week. Ask me how I feel about it.
00:29:47.360 --> 00:29:47.460
You're right.
00:29:48.360 --> 00:29:48.580
Honestly,
00:29:48.770 --> 00:29:52.040
the one thing to be a little careful about as developers is the Rosetta 2.
00:29:52.580 --> 00:29:52.660
Yeah.
00:29:53.080 --> 00:29:53.720
That's going away.
00:29:54.260 --> 00:29:56.920
So your ability to run Intel compiled stuff,
00:29:57.270 --> 00:29:59.280
you might think, Michael, why would I run Intel
00:29:59.600 --> 00:29:59.900
compiled stuff?
00:30:00.070 --> 00:30:00.360
Like, ooh,
00:30:00.560 --> 00:30:01.000
Docker.
00:30:01.500 --> 00:30:03.480
Certain Docker things only have Intel versions.
00:30:03.900 --> 00:30:04.620
So that's going to be a mega hassle.
00:30:04.620 --> 00:30:05.380
I mean, that's pretty rare.
00:30:06.280 --> 00:30:07.940
People have cross-compiled most of this stuff.
00:30:07.990 --> 00:30:09.840
Because when was the last time an Intel Mac was released?
00:30:10.560 --> 00:30:14.960
Yeah, but let's suppose I'm deploying to an x86 server.
00:30:15.380 --> 00:30:15.500
Yeah.
00:30:15.940 --> 00:30:16.880
And I want to test something.
00:30:17.440 --> 00:30:18.720
I think it's gotten a lot better.
00:30:19.240 --> 00:30:20.120
It definitely has gotten better.
00:30:20.600 --> 00:30:21.260
But it used to be,
00:30:21.480 --> 00:30:22.460
the thing is, I don't know.
00:30:22.520 --> 00:30:26.420
It used to be certain stuff would only work in an x86 version of Linux.
00:30:26.420 --> 00:30:27.040
Oh, I remember this.
00:30:27.200 --> 00:30:27.260
Yeah.
00:30:27.380 --> 00:30:30.760
But that was like three, four, five years ago when I was really dealing with that.
00:30:31.000 --> 00:30:35.780
Actually, it was when I was dealing with like Databricks and trying to coordinate that stuff.
00:30:36.120 --> 00:30:36.320
Nice.
00:30:36.660 --> 00:30:38.400
So this one still has it, but the one after it,
00:30:38.560 --> 00:30:39.040
whatever that's called,
00:30:39.200 --> 00:30:39.480
won't.
00:30:39.720 --> 00:30:43.260
So this is like your last safe upgrade if you're worried about the other thing.
00:30:43.560 --> 00:30:46.020
So did Apple actually deliver some AI features this time?
00:30:46.360 --> 00:30:46.980
Well, I'll tell you what.
00:30:47.060 --> 00:30:48.680
the new Siri caught me off guard.
00:30:48.700 --> 00:30:49.300
I'm like, oh yeah,
00:30:49.460 --> 00:30:49.860
I did.
00:30:50.180 --> 00:30:51.360
I did actually upgrade the phone.
00:30:51.480 --> 00:30:52.560
And I guess it has the new Siri
00:30:52.640 --> 00:30:53.200
because it sounded,
00:30:53.740 --> 00:30:55.080
I asked it something like set a timer
00:30:55.120 --> 00:30:56.400
and it said something completely different
00:30:56.460 --> 00:30:56.920
that I'm used to.
00:30:57.020 --> 00:30:57.920
And it sounded better.
00:30:58.000 --> 00:30:58.180
I'm like,
00:30:58.400 --> 00:30:58.800
oh wait,
00:30:59.360 --> 00:31:00.600
I haven't had a chance to test it though.
00:31:01.080 --> 00:31:01.380
All right.
00:31:01.540 --> 00:31:03.040
It did set the timer like a champ.
00:31:03.160 --> 00:31:03.620
Let me tell you.
00:31:04.260 --> 00:31:04.620
Well done.
00:31:05.280 --> 00:31:05.600
Way to go.
00:31:07.100 --> 00:31:07.380
All right.
00:31:07.580 --> 00:31:08.660
Let's talk a joke.
00:31:09.040 --> 00:31:09.200
Okay.
00:31:09.380 --> 00:31:09.760
Speaking of,
00:31:10.500 --> 00:31:10.680
you know,
00:31:10.780 --> 00:31:12.360
the new Siri is supposed to be agentic.
00:31:12.660 --> 00:31:14.760
So the joke is we're agentic now.
00:31:14.940 --> 00:31:15.780
We're an agentic startup.
00:31:15.860 --> 00:31:16.080
You ready?
00:31:16.460 --> 00:31:17.120
This is how you,
00:31:17.680 --> 00:31:19.720
there's certain things you've got to position yourself.
00:31:19.890 --> 00:31:23.420
I was just watching an ad because I started watching football yesterday.
00:31:24.160 --> 00:31:28.220
And normally ads are excluded from my life, but apparently not in football,
00:31:28.420 --> 00:31:29.100
American football.
00:31:29.440 --> 00:31:32.940
And there's some ad for Zoom that Zoom is an AI company.
00:31:33.100 --> 00:31:34.160
They're not about meetings anymore.
00:31:34.520 --> 00:31:34.800
Nope.
00:31:35.060 --> 00:31:39.160
They can book the thing so that your dry cleaning gets picked up.
00:31:39.320 --> 00:31:40.840
They can do a slideshow.
00:31:40.910 --> 00:31:41.200
Like what?
00:31:41.660 --> 00:31:41.840
Okay.
00:31:42.300 --> 00:31:44.760
So everyone's got to be some kind of AI thing now.
00:31:44.960 --> 00:31:45.740
So here's the joke.
00:31:48.960 --> 00:31:54.880
i changed all of our loading dot dot dot states to thinking dot dot dot we're an agentic startup now
00:31:55.420 --> 00:31:59.600
perfect i'm gonna get right on that yeah get right in there like you can there's so much vc
00:31:59.730 --> 00:32:06.620
money to be had from this go for it discombobulating i'm thinking oh oh i hate that so much about
00:32:06.730 --> 00:32:10.740
Claude Code it drives me crazy that it's got all these random little words yeah the reason i don't
00:32:10.760 --> 00:32:11.340
like it is I don't,
00:32:12.000 --> 00:32:16.720
it feels like it's made for someone with ADHD who just can't possibly let it
00:32:16.860 --> 00:32:18.120
just be for like five seconds.
00:32:18.640 --> 00:32:21.240
And so if I'm like doing something else, I'd look over and like word
00:32:21.340 --> 00:32:24.040
starts. I'm like, Oh, maybe it's no, it's not done. It's like, Oh, maybe it's done.
00:32:24.380 --> 00:32:24.680
Oh no,
00:32:24.780 --> 00:32:26.040
no. It's just still like randomly.
00:32:26.800 --> 00:32:29.080
Like, could it just have the little icon go?
00:32:29.380 --> 00:32:29.440
No,
00:32:29.920 --> 00:32:30.160
no, no,
00:32:30.240 --> 00:32:30.280
no.
00:32:30.520 --> 00:32:31.780
It's thinking is combobulating.
00:32:32.080 --> 00:32:33.020
It's wording.
00:32:33.360 --> 00:32:34.180
I don't know. What is it doing?
00:32:34.350 --> 00:32:35.840
That's why you need her to exactly.
00:32:36.440 --> 00:32:36.620
That's,
00:32:36.960 --> 00:32:38.440
that's, that's the story for another episode.
00:32:38.820 --> 00:32:39.520
All right. Sounds good.
00:32:39.940 --> 00:32:40.200
All right. Well,
00:32:40.860 --> 00:32:42.100
thanks as always for being here, Calvin.
00:32:42.270 --> 00:32:43.100
And thank you everyone for listening.
00:32:43.360 --> 00:32:43.980
We'll talk to you soon.
00:32:44.360 --> 00:32:44.560
Yeah. Bye.