TRANSKRIP 🔗
WEBVTT
00:00:00.001 --> 00:00:03.760
If you want to get better at something, oftentimes the path is pretty clear.
00:00:03.760 --> 00:00:09.160
If you want to get better at swimming, you go to the pool and practice your strokes and put in time doing the laps.
00:00:09.160 --> 00:00:14.520
Want to get better at mountain biking? Hit the trails and work on drills focusing on different aspects of riding.
00:00:14.520 --> 00:00:16.460
You can do the same for programming.
00:00:16.460 --> 00:00:21.660
Reuven Lerner is back on the podcast to talk about his book, Pandas Workout.
00:00:21.660 --> 00:00:27.400
We dive into strategies for learning pandas in Python, as well as some of his workout exercises.
00:00:27.900 --> 00:00:33.160
This is Talk Python To Me, episode 471, recorded July 7th, 2024.
00:00:33.160 --> 00:00:36.380
Are you ready for your host? Here he is.
00:00:36.380 --> 00:00:39.820
You're listening to Michael Kennedy on Talk Python To Me.
00:00:39.820 --> 00:00:43.500
Live from Portland, Oregon, and this segment was made with Python.
00:00:43.500 --> 00:00:50.180
Welcome to Talk Python To Me, a weekly podcast on Python.
00:00:50.180 --> 00:00:52.420
This is your host, Michael Kennedy.
00:00:52.420 --> 00:00:57.760
Follow me on Mastodon, where I'm @mkennedy, and follow the podcast using @talkpython.
00:00:57.760 --> 00:00:58.380
Talk Python.com.
00:00:58.380 --> 00:01:00.760
Both accounts over at fosstodon.org.
00:01:00.760 --> 00:01:05.680
And keep up with the show and listen to over nine years of episodes at talkpython.fm.
00:01:05.680 --> 00:01:10.260
If you want to be part of our live episodes, you can find the live streams over on YouTube.
00:01:10.260 --> 00:01:16.580
Subscribe to our YouTube channel over at talkpython.fm/youtube and get notified about upcoming shows.
00:01:16.820 --> 00:01:18.700
This episode is brought to you by Sentry.
00:01:18.700 --> 00:01:20.480
Don't let those errors go unnoticed.
00:01:20.480 --> 00:01:22.320
Use Sentry like we do here at Talk Python.
00:01:22.320 --> 00:01:25.700
Sign up at talkpython.fm/sentry.
00:01:25.700 --> 00:01:28.500
And it's also brought to you by Scalable Path.
00:01:28.500 --> 00:01:35.020
If you're a founder or engineering leader, you know how hard it is to find top-tier developers while keeping costs low.
00:01:35.020 --> 00:01:40.720
Scalable Path is a software staffing company that helps you build remote dev teams that just fit.
00:01:40.720 --> 00:01:44.900
Build your team at talkpython.fm/Scalable Path.
00:01:45.240 --> 00:01:51.160
Before we jump into the interview, I want to let you know that we still have some spots left in my Code in a Castle event.
00:01:51.160 --> 00:01:55.440
If you're looking to learn some of the premier frameworks and techniques in Python,
00:01:55.440 --> 00:01:58.980
and you'd like to have a bucket list type of experience while doing so,
00:01:58.980 --> 00:02:02.000
then check out talkpython.fm/Castle.
00:02:02.000 --> 00:02:09.160
In October, I'll
Load full transcript
be running a six-day Python course for an intimate audience in a villa in Tuscany.
00:02:09.160 --> 00:02:14.320
Half the time we'll be learning Python, and the other half will be exploring the best of what Italy has to offer.
00:02:14.660 --> 00:02:20.300
Check out the course outline, the excursions, and all the details at talkpython.fm/Castle.
00:02:20.300 --> 00:02:27.140
Or if you'd like to just shoot me an email, Michael at talkpython.fm, or find me on the socials, and I'm happy to talk about it.
00:02:27.140 --> 00:02:28.080
Hope to see you there.
00:02:28.080 --> 00:02:31.280
Reuven, welcome back to Talk Python To Me.
00:02:31.280 --> 00:02:31.920
How are you doing?
00:02:31.920 --> 00:02:33.080
I'm doing great.
00:02:33.080 --> 00:02:34.100
Great to be back here.
00:02:34.100 --> 00:02:34.860
Nice to see you.
00:02:35.080 --> 00:02:36.480
Yeah, it's great to see you as well.
00:02:36.480 --> 00:02:38.040
I'm a little concerned, though.
00:02:38.040 --> 00:02:42.720
There's some possibilities that maybe my Facebook ads are going to get messed up.
00:02:42.720 --> 00:02:49.380
We are talking about pandas and pandas internationally, and I heard that you're some kind of animal trafficker.
00:02:49.380 --> 00:02:51.220
Do you want to start the show with that story?
00:02:51.220 --> 00:02:51.960
It's out of control.
00:02:51.960 --> 00:02:53.480
It is the craziest story.
00:02:53.480 --> 00:02:59.440
So I occasionally advertise on Facebook, advertise my pandas and Python training.
00:02:59.440 --> 00:03:03.960
And I guess it was like two years ago, I tried doing a little bit more advertising.
00:03:03.960 --> 00:03:15.480
And basically, I didn't really pay much attention to it until about a year ago when I noticed that when I tried to do some more advertising and it said, you are not allowed to advertise on any meta properties.
00:03:15.480 --> 00:03:16.640
That's really weird.
00:03:16.740 --> 00:03:17.240
Like, what did I do?
00:03:17.240 --> 00:03:17.500
I looked.
00:03:17.500 --> 00:03:20.100
I could not find any indication of what I'd done wrong.
00:03:20.100 --> 00:03:22.420
So it says, if you want to appeal, click here.
00:03:22.420 --> 00:03:23.400
So I clicked here.
00:03:23.400 --> 00:03:28.080
And within 30 minutes or so, I get email back saying, your appeal has been checked and denied.
00:03:28.080 --> 00:03:31.740
You will never be allowed to advertise on any meta property again.
00:03:31.740 --> 00:03:33.980
And this was like, what have I done?
00:03:33.980 --> 00:03:35.640
Like, what are you teaching?
00:03:35.640 --> 00:03:35.700
I'm pretty innocent.
00:03:35.700 --> 00:03:37.560
Some courses and some books.
00:03:37.560 --> 00:03:38.220
Come on, man.
00:03:38.220 --> 00:03:41.160
I figured also, I appealed.
00:03:41.160 --> 00:03:42.660
Someone must have looked through this.
00:03:42.840 --> 00:03:47.620
Anyway, in poking around online, someone said to me, oh, I was caught by the same thing.
00:03:47.620 --> 00:03:51.640
It is
illegal to sell rare and endangered animals.
00:03:51.640 --> 00:03:57.620
And they believe that since you were selling Python and Pandas training, then you must have been training in live and endangered animals.
00:03:57.620 --> 00:03:59.020
And thus, you are banned for life.
00:03:59.020 --> 00:04:00.160
Well, I said, you know what?
00:04:00.160 --> 00:04:02.180
I got some, first of all, like nuts.
00:04:02.180 --> 00:04:05.040
And yet, a great story to tell to my machine learning classes.
00:04:05.040 --> 00:04:06.380
So good on that front.
00:04:06.380 --> 00:04:08.200
But I was like, this is easy to resolve, right?
00:04:08.200 --> 00:04:09.700
I'll just contact someone at Facebook.
00:04:09.700 --> 00:04:10.940
No one's available.
00:04:11.280 --> 00:04:14.200
I got to text on my connections, like people I know who work there.
00:04:14.200 --> 00:04:20.660
And I finally get the answer back saying that for legal purposes, they delete all data that's more than six months old.
00:04:20.660 --> 00:04:21.580
Something like that.
00:04:21.580 --> 00:04:25.600
So they could not go back and check to see why I was banned.
00:04:25.600 --> 00:04:27.580
And thus, this ban really would be for life.
00:04:27.580 --> 00:04:31.960
So because I waited a while, because it had been like a year between the ban and me noticing.
00:04:31.960 --> 00:04:33.280
Oh, my gosh.
00:04:33.280 --> 00:04:34.520
So they couldn't do anything about it.
00:04:34.640 --> 00:04:39.300
So I was like half laughing and half like, you got to be kidding me about this.
00:04:39.300 --> 00:04:42.220
And I posted on my blog about this.
00:04:42.220 --> 00:04:45.400
And you guys picked it up on Python Bytes.
00:04:45.400 --> 00:04:47.840
It was picked up in a bunch of other places like Hacker News.
00:04:48.340 --> 00:04:51.720
And about a month later, I checked and it was back.
00:04:51.720 --> 00:04:53.000
No one told me anything.
00:04:53.000 --> 00:04:54.280
No one said anything.
00:04:54.280 --> 00:04:58.040
Just magically and quietly, my account was restored.
00:04:58.040 --> 00:04:59.720
Running to the press never helps.
00:04:59.720 --> 00:05:00.980
It never helps.
00:05:00.980 --> 00:05:01.620
That's right.
00:05:01.620 --> 00:05:02.080
That's right.
00:05:02.260 --> 00:05:03.780
But it was so absurd.
00:05:03.780 --> 00:05:08.780
And I think it was at least like four senior engineers at Facebook who tried to help me with this.
00:05:08.780 --> 00:05:09.960
And they were all like, we tried.
00:05:09.960 --> 00:05:10.880
There's nothing we can do.
00:05:10.880 --> 00:05:11.460
Crazy.
00:05:11.460 --> 00:05:24.280
It reminds me, honestly, it reminds me so much of my App Store review experience, getting the new Talk Python training mobile apps in iOS, in the apps, in the iOS App Store, in the Google App Store.
00:05:24.280 --> 00:05:30.140
They were tragically incompetent in their own special way, right?
00:05:30.140 --> 00:05:32.540
Somewhat malicious, somewhat not malicious.
00:05:32.540 --> 00:05:34.660
Just like, for example, Apple.
00:05:34.660 --> 00:05:35.540
I think it was Apple.
00:05:35.540 --> 00:05:35.960
It could have been.
00:05:35.960 --> 00:05:36.340
I don't know.
00:05:36.340 --> 00:05:37.100
Apple or Google.
00:05:37.100 --> 00:05:37.780
It has to be one of them.
00:05:37.780 --> 00:05:44.320
Said, you know, we've denied your application to publish this application because you're trying to impersonate in an existing one.
00:05:44.320 --> 00:05:47.320
I said, what app am I trying to impersonate?
00:05:47.320 --> 00:05:50.380
Like, it's hundreds of hours of stuff that we've created.
00:05:50.380 --> 00:05:52.340
Like, they're like, well, first you might be hijacked.
00:05:52.340 --> 00:05:58.580
So, well, in the description, it says, if you want to learn Python, you can take our courses.
00:05:58.580 --> 00:06:02.380
Well, there's already an app called LearnPython, and you're trying to impersonate.
00:06:02.380 --> 00:06:03.180
I'm like, what?
00:06:03.180 --> 00:06:07.880
Like, it said if you want to learn Python, but there's an app called LearnPython.
00:06:07.880 --> 00:06:13.580
I'm like, I just, I don't even under, like, my mind is, how do you read that description and think this is a trademark?
00:06:13.580 --> 00:06:20.100
So, if you come up with an app called, like, Eat at a Restaurant, then no food order delivery place.
00:06:20.100 --> 00:06:22.480
Yeah, you're going to be acquired by Uber Eats straight away.
00:06:22.480 --> 00:06:23.180
Exactly.
00:06:23.180 --> 00:06:26.320
And so, we went, but we, you would think, okay, fair mistake.
00:06:26.320 --> 00:06:27.100
Like, yeah, yeah, okay.
00:06:27.100 --> 00:06:27.860
Something caught it.
00:06:27.860 --> 00:06:31.460
Just like you said, a simple request and a human will look and it'll be fine.
00:06:31.460 --> 00:06:33.680
No, they're like, well, look, obviously you're doing this.
00:06:33.680 --> 00:06:34.520
I'm like, obviously.
00:06:34.520 --> 00:06:35.980
Let's try another scenario.
00:06:35.980 --> 00:06:38.580
What if I said I wanted to learn to play the guitar?
00:06:38.940 --> 00:06:41.600
And there was an app called Learn Guitar, but I don't want to learn.
00:06:41.600 --> 00:06:43.880
It's not, it's not, the title is not Learn Guitar.
00:06:43.880 --> 00:06:45.800
It's just the act of learning a guitar.
00:06:45.800 --> 00:06:46.920
How else would you explain it?
00:06:46.920 --> 00:06:47.880
Like, oh, okay.
00:06:47.880 --> 00:06:50.000
So, I guess we see, we understand now.
00:06:50.000 --> 00:06:52.060
You can have that sentence.
00:06:52.060 --> 00:06:56.800
And it's just, these things are really, and you're at the complete mercy of them.
00:06:56.800 --> 00:07:01.680
It's really, it's both like comically funny, but it's also like painful because we spent
00:07:01.680 --> 00:07:02.880
four months building that app.
00:07:02.880 --> 00:07:04.980
They wouldn't accept for this stupid reason, you know?
00:07:04.980 -->
00:07:09.180
And they're taking a fair amount of money off the top for whatever people are earning from
00:07:09.180 --> 00:07:09.500
these apps.
00:07:09.500 --> 00:07:11.840
It's not like this is a charity or something.
00:07:11.840 --> 00:07:17.980
What I found amazing with Facebook was there was literally no way to contact a human being.
00:07:17.980 --> 00:07:21.920
Like, I tried all sorts of searches and forums and on, on, and on.
00:07:22.240 --> 00:07:27.080
And like, and, and I found a lot of other people who had been caught up in this sort
00:07:27.080 --> 00:07:33.440
of ridiculous situation, but like, there's no form even to say, hey, I think you made a
00:07:33.440 --> 00:07:33.880
mistake.
00:07:33.880 --> 00:07:35.560
Why don't you have a human look at this?
00:07:35.560 --> 00:07:38.340
Because that would cost them too much, you know, someone's salary.
00:07:38.340 --> 00:07:38.700
Yeah, absolutely.
00:07:38.700 --> 00:07:42.740
It's, anyway, I don't want to spend too much time on it, but boy, is that a crazy story?
00:07:42.740 --> 00:07:46.740
I mean, we're going to be trading in some, we'll be trading in some pandas today.
00:07:46.740 --> 00:07:49.480
And so I just, I'm going to bleep that part.
00:07:49.480 --> 00:07:52.220
Every time when the word pandas is said, we're bleeping it out on the YouTube.
00:07:52.220 --> 00:07:52.620
version.
00:07:52.620 --> 00:07:54.520
Well, this could be entertaining then.
00:07:54.520 --> 00:07:55.480
Wow.
00:07:55.480 --> 00:07:57.780
Ruben was really testy.
00:07:57.780 --> 00:07:59.460
Like all those bleep, bleep, bleep.
00:07:59.460 --> 00:08:01.660
No, seriously though.
00:08:01.660 --> 00:08:04.060
You know, let's, let's catch up.
00:08:04.060 --> 00:08:05.260
We'll talk about your book.
00:08:05.260 --> 00:08:08.020
What have you been up to since the last time we spoke?
00:08:08.020 --> 00:08:09.420
I can't remember the last time I had you on.
00:08:09.420 --> 00:08:10.600
It's been a couple of years, I think.
00:08:10.600 --> 00:08:11.880
It's been, it's been some time.
00:08:11.880 --> 00:08:12.120
Yeah.
00:08:12.120 --> 00:08:16.780
So I'm continuing to do Python pandas training at companies.
00:08:17.680 --> 00:08:22.180
Since the pandemic, much more of it is online, just because companies are now used to doing
00:08:22.180 --> 00:08:26.640
stuff on Zoom, team, WebEx, rather than bringing me there in person.
00:08:26.640 --> 00:08:27.280
That's fine.
00:08:27.280 --> 00:08:29.100
So I traveled to conferences rather than to clients.
00:08:29.100 --> 00:08:31.680
And, you know, it sort of extended my flexibility on that.
00:08:31.900 --> 00:08:36.680
And I've also been building up my online training stuff, both a whole lot of courses.
00:08:36.680 --> 00:08:39.920
And I've got an online bootcamp that I run for Python and pandas twice a year.
00:08:39.920 --> 00:08:43.600
The big thing, which is actually connected to what we're doing here also, is I have a new
00:08:43.600 --> 00:08:47.060
newsletter called Bamboo Weekly, where I have pandas exercises.
00:08:47.060 --> 00:08:51.140
It's sort of like the same style as the book, but every week I take a topic and I do it, I
00:08:51.140 --> 00:08:52.300
do it based on current events.
00:08:52.300 --> 00:08:56.920
So if there's something going on in the news, I try to find a data set, a real world data set
00:08:56.920 --> 00:08:57.980
that has something to do with that.
00:08:58.240 --> 00:09:00.100
And then we try to experiment with that.
00:09:00.100 --> 00:09:02.720
Yeah, that sounds really fun to see like a real world example every week.
00:09:02.720 --> 00:09:03.560
Yeah, yeah.
00:09:03.560 --> 00:09:05.220
So people are like, how do you come up with that?
00:09:05.220 --> 00:09:09.700
And I say, well, I listen to a lot of podcasts and I read a lot of newspapers and something
00:09:09.700 --> 00:09:15.320
somewhere, like I read The Economist, for example, and they had this short, cute article about
00:09:15.320 --> 00:09:18.440
the number of animals that go through Heathrow Airport every year.
00:09:18.440 --> 00:09:21.320
I was like, wait, there's got to be a data set for that.
00:09:21.320 --> 00:09:26.500
And sure enough, the Heathrow Airport Authority publishes a data set in CSV of how many animals
00:09:26.500 --> 00:09:26.980
go through.
00:09:27.340 --> 00:09:32.140
And it was like 200 horses and one and a half billion butterflies and everything in
00:09:32.140 --> 00:09:32.500
between.
00:09:32.500 --> 00:09:39.800
And so it was great fun to sort of play with that data and ask questions about it and give
00:09:39.800 --> 00:09:42.140
people practice with something that's dirty and messy.
00:09:42.140 --> 00:09:43.500
I don't mean the animals being dirty, right?
00:09:43.500 --> 00:09:44.240
Like very messy.
00:09:44.240 --> 00:09:47.160
Like you need to like really, you know, wrestle with it because that's the only way you're
00:09:47.160 --> 00:09:47.740
going to improve.
00:09:47.740 --> 00:09:50.560
So I'm having a lot of fun on all the training front.
00:09:50.560 --> 00:09:54.900
I definitely see more and more use of, I'm sure this won't surprise you, more and more use
00:09:54.900 --> 00:09:58.900
of Python in the data space as it just like catches fire there.
00:09:58.900 --> 00:09:58.940
Yeah.
00:09:58.940 --> 00:10:01.540
It's just going downhill, picking up speed, isn't it?
00:10:01.540 --> 00:10:02.360
It's extraordinary.
00:10:02.360 --> 00:10:07.860
Like I still remember asking people in my intro Python courses, so what are you here for?
00:10:07.860 --> 00:10:11.620
They're like, yeah, my company's thinking of doing some stuff with data analysis and pandas
00:10:11.620 --> 00:10:16.040
or at least as data analysis, I knew that even in NumPy before pandas got you out.
00:10:16.040 --> 00:10:19.400
And me thinking, hmm, I should really learn this stuff because it sounds like it's going
00:10:19.400 --> 00:10:19.920
to be popular.
00:10:19.920 -->
00:10:22.500
And holy cow, it's like it's everywhere.
00:10:22.500 --> 00:10:25.400
And the whole ecosystem is just growing, growing, growing.
00:10:25.400 --> 00:10:29.880
It's like people are sort of seeing pandas as the like underlying infrastructure on which
00:10:29.880 --> 00:10:32.180
they build their software tools and their companies.
00:10:32.180 --> 00:10:36.540
Or even the thing that defines the API on which they can innovate, right?
00:10:36.540 --> 00:10:40.720
Think DASK or a whole bunch of other things in that space, right?
00:10:40.720 --> 00:10:45.120
That if you know pandas, chances are, if you're not too crazy, you can like do grid computing
00:10:45.120 --> 00:10:46.300
by changing the import.
00:10:46.300 --> 00:10:47.740
It's extraordinary.
00:10:47.740 --> 00:10:48.640
Just extraordinary.
00:10:48.640 --> 00:10:49.160
Yeah.
00:10:49.160 --> 00:10:50.560
Yeah, it absolutely is.
00:10:50.660 --> 00:10:54.140
And so back to your newsletter, pandas eat bamboo.
00:10:54.140 --> 00:10:54.700
Is that right?
00:10:54.700 --> 00:10:55.320
That's so-
00:10:55.320 --> 00:10:56.620
I understand why you're banned from Facebook.
00:10:56.620 --> 00:10:56.900
Okay.
00:10:56.900 --> 00:10:57.360
All right.
00:10:57.360 --> 00:10:59.900
It was actually my father's idea.
00:10:59.900 --> 00:11:02.460
I was like, I need a like a catchy name for something.
00:11:02.460 --> 00:11:05.600
And so he was like, well, what was something with bamboo?
00:11:05.600 --> 00:11:06.880
I was like, right, right.
00:11:06.880 --> 00:11:09.460
It's like, you know, it's food for thought and food for pandas.
00:11:09.460 --> 00:11:09.800
Yeah.
00:11:09.800 --> 00:11:10.500
Yeah.
00:11:10.500 --> 00:11:11.400
That's actually really clever.
00:11:11.400 --> 00:11:12.040
I like it a lot.
00:11:12.040 --> 00:11:12.640
I like it a lot.
00:11:12.640 --> 00:11:14.840
So you've been on a journey.
00:11:14.840 --> 00:11:16.740
You've been on a journey to write a book, right?
00:11:16.740 --> 00:11:17.520
Yes.
00:11:17.520 --> 00:11:19.200
I was convinced to do a second one.
00:11:19.200 --> 00:11:20.180
Right.
00:11:20.240 --> 00:11:24.680
So I did a book with Manning called Python Workout, which was exercise in Python.
00:11:24.680 --> 00:11:29.020
And when that finished, they said, so what other topics do you think would be useful?
00:11:29.020 --> 00:11:32.800
I said, well, I'm doing a lot of pandas and people definitely need a lot of exercise and
00:11:32.800 --> 00:11:33.680
a lot of practice in that.
00:11:33.680 --> 00:11:39.640
And so I got to work both collecting the exercises I do with my corporate training and also like
00:11:39.640 --> 00:11:41.520
coming up with some new ones as I learn new things.
00:11:41.520 --> 00:11:46.900
Because pandas is so, so, so huge that it's very easy to get lost in there and not even
00:11:46.900 --> 00:11:49.240
know what are the important topics to learn.
00:11:49.740 --> 00:11:53.200
And so thanks to like working with people all the time, I sort of see also where they
00:11:53.200 --> 00:11:56.020
get stuck and where they have problems and where it's really confusing to them.
00:11:56.020 --> 00:11:57.060
So yeah.
00:11:57.060 --> 00:11:59.640
So 200 and some odd exercises later.
00:11:59.640 --> 00:12:02.640
By the way, I'll just tell you why did it come out when it came out?
00:12:02.640 --> 00:12:04.560
Because I'm terribly bad at deadlines.
00:12:04.560 --> 00:12:07.700
I said to the Manning people, I really want to have it at my booth at PyCon.
00:12:07.700 --> 00:12:09.600
They said, okay, big talker.
00:12:09.600 --> 00:12:10.960
You want it mid-May, huh?
00:12:11.080 --> 00:12:12.480
So you better have everything done.
00:12:12.480 --> 00:12:15.260
And they sort of backtracked on the calendar and said, okay, so you better have the whole
00:12:15.260 --> 00:12:16.040
thing done by December.
00:12:16.040 --> 00:12:21.160
And then finally I had some like, you know, fire under me and got it done.
00:12:21.160 --> 00:12:21.360
Awesome.
00:12:21.360 --> 00:12:22.980
As opposed to, I should get more done.
00:12:22.980 --> 00:12:23.120
Yeah.
00:12:23.120 --> 00:12:28.720
The version I actually have in my Apple Books here I pulled up is the Manning Early Access
00:12:28.720 --> 00:12:29.760
Preview version.
00:12:29.860 --> 00:12:31.320
But it's out for real now.
00:12:31.320 --> 00:12:31.500
Yeah.
00:12:31.500 --> 00:12:31.940
Here.
00:12:31.940 --> 00:12:33.820
I've even got the paper copy here.
00:12:33.820 --> 00:12:34.420
Woohoo.
00:12:34.420 --> 00:12:34.940
Look at that.
00:12:34.940 --> 00:12:35.560
I know.
00:12:35.560 --> 00:12:36.880
That's a pretty hefty book, honestly.
00:12:36.880 --> 00:12:37.680
I still keep looking at it.
00:12:37.680 --> 00:12:38.720
That's a proper book.
00:12:38.720 --> 00:12:39.400
It's like, wow.
00:12:39.400 --> 00:12:39.920
I know.
00:12:39.920 --> 00:12:40.220
I know.
00:12:40.220 --> 00:12:42.620
It feels like it's really quite some feeling there to see.
00:12:42.620 --> 00:12:44.260
I guess I really finished it.
00:12:44.260 --> 00:12:44.740
Congratulations.
00:12:44.740 --> 00:12:50.680
So I wanted to talk to you about maybe just kind of falling on with your bamboo idea.
00:12:50.680 --> 00:12:52.100
Like give us some examples.
00:12:52.100 --> 00:12:54.480
Give us some problems that people are solving with pandas.
00:12:54.480 --> 00:12:59.180
And I mean, we're not going to talk through the code super detailed, but you could say like this
00:12:59.180 --> 00:13:03.840
aspect or this feature of pandas like dot look or whatever is how you access and solve these
00:13:03.840 --> 00:13:04.300
problems, right?
00:13:04.300 --> 00:13:06.120
Like, so just kind of exploring that space.
00:13:06.120 --> 00:13:09.680
I had Wes McKinney on before five episodes ago, something like that.
00:13:09.680 --> 00:13:11.440
And I was just like, how do you learn pandas?
00:13:11.440 --> 00:13:13.140
It's like so big, you know?
00:13:13.140 --> 00:13:15.080
So I've actually changed.
00:13:15.080 --> 00:13:20.380
Well, let me first say I'm definitely one of those people who thinks you should learn Python
00:13:20.380 --> 00:13:21.300
before pandas.
00:13:21.300 --> 00:13:28.000
Like I definitely think that knowing the language well will serve you very, very well in all sorts
00:13:28.000 --> 00:13:29.800
of weird, small ways necessarily.
00:13:29.800 --> 00:13:33.960
But at the same time, when you learn pandas, you have to learn that some of the paradigms
00:13:33.960 --> 00:13:37.320
you learned, like some of the idioms from Python are not appropriate.
00:13:37.320 --> 00:13:42.220
So I was giving a class in like optimizing pandas, like a short class, we'll call it micro
00:13:42.220 --> 00:13:44.980
class, like 90 minutes long, about a year or so ago.
00:13:44.980 --> 00:13:48.940
And at the end, I was like, oh, and by the way, obviously just never do for loops.
00:13:48.940 --> 00:13:50.160
And everyone's like, wait, wait, wait, what?
00:13:50.160 --> 00:13:51.480
I said, what do you mean what?
00:13:51.820 --> 00:13:56.680
And they were like, we were taught in our intro pandas class that you should do a for loop
00:13:56.680 --> 00:13:58.120
to do anything across the data frame.
00:13:58.120 --> 00:13:59.720
And I was like, okay, what?
00:13:59.720 --> 00:14:00.720
How can this be?
00:14:00.720 --> 00:14:06.400
And so people think that because it's in Python, you should do it the same way.
00:14:06.440 --> 00:14:11.960
But there are all these different idioms, especially whole vectorization that you need to internalize.
00:14:11.960 --> 00:14:17.540
Otherwise, as I like to say to people, you should hope to be paid by the hour because like these
00:14:17.540 --> 00:14:19.320
things are just going to take forever to run.
00:14:19.880 --> 00:14:24.260
And so people don't necessarily understand sort of how to approach pandas stuff.
00:14:24.260 --> 00:14:28.880
And then they don't understand sort of, let's see, I mean, I'll give like outline and then
00:14:28.880 --> 00:14:29.920
we can sort of go into it.
00:14:29.920 --> 00:14:32.220
Certainly how to access things with .lock.
00:14:32.220 --> 00:14:34.420
Certainly how multi-indexes work.
00:14:34.420 --> 00:14:37.020
How to work with the different d types.
00:14:37.020 --> 00:14:41.660
Because these are things that we don't think about in the standard pandas world each day.
00:14:41.660 --> 00:14:46.240
The closest analogy would maybe be that where you're working with a dictionary, you want
00:14:46.240 --> 00:14:47.640
to think about what your keys are going to be.
00:14:47.640 --> 00:14:49.460
But even then, it doesn't come close to it.
00:14:49.460 --> 00:14:52.520
So like .lock, people are like, so fine.
00:14:52.520 --> 00:14:57.200
So .lock is used for retrieving from a series or retrieving from a data frame.
00:14:57.200 --> 00:15:01.760
But it's so much more than that because you have two parts to it, or potentially two parts.
00:15:01.760 --> 00:15:04.960
You have the row selector and you have the column selector.
00:15:04.960 --> 00:15:11.260
And each of those can be an individual name, a list of names, a Boolean series, or even a
00:15:11.260 --> 00:15:11.560
Lambda.
00:15:11.560 --> 00:15:13.940
And you can mix and match those in so many different ways.
00:15:13.940 --> 00:15:19.120
And once you see those options, like your head sort of explodes with, oh, wait, I never thought
00:15:19.120 --> 00:15:21.100
that I could access my data like that.
00:15:21.100 --> 00:15:23.880
And then you're like, wait, you can also assign your data like that.
00:15:23.880 --> 00:15:25.700
And then they're just sort of astonished.
00:15:25.700 --> 00:15:27.200
Yeah, it's really wild.
00:15:27.200 --> 00:15:32.740
You know, I think pandas has its own idiomatic style that is different than what you would
00:15:32.740 --> 00:15:34.100
call pythonic, right?
00:15:34.100 --> 00:15:35.280
Like it's pandonic.
00:15:35.400 --> 00:15:38.800
I don't know what the name is, but idiomatic pandas, right?
00:15:38.800 --> 00:15:38.800
I like that.
00:15:38.800 --> 00:15:44.940
Where there's things that are specific to pandas, like this vectorization stuff, right?
00:15:44.940 --> 00:15:46.880
Instead of looping over, right?
00:15:46.880 --> 00:15:51.180
You know, you think about the Python performance angles or the data science performance angles,
00:15:51.180 --> 00:15:51.480
right?
00:15:51.480 --> 00:15:56.800
A lot of the speed that we get out of tools like pandas and numpy and polars and others is,
00:15:56.800 --> 00:16:01.020
as you take the data, you push it down into some native layer and you just leave it there.
00:16:01.020 --> 00:16:06.840
And you kind of speak to the native layer from Python and you say, deep down in your insides,
00:16:06.840 --> 00:16:10.520
you got this thing, multiply all 1 million of them by two or whatever, right?
00:16:10.520 --> 00:16:17.540
But if you loop over it, you're like converting it out of C into Python objects, then you're
00:16:17.540 --> 00:16:20.700
operating it, then you're putting it back just 2 million times.
00:16:20.700 --> 00:16:22.480
All of a sudden, all those benefits are gone.
00:16:22.480 --> 00:16:25.120
And so certainly learning those types of things.
00:16:25.120 --> 00:16:29.740
I mean, the vectorization, I think most people get pretty soon, although it sounds like not
00:16:29.740 --> 00:16:30.540
that's really everyone.
00:16:30.540 --> 00:16:34.440
I was really like flabbergasted.
00:16:34.440 --> 00:16:35.600
But there's way more than that, right?
00:16:35.600 --> 00:16:38.120
There's a whole, that's just probably the most obvious thing.
00:16:38.120 --> 00:16:41.620
This portion of Talk Python To Me is brought to you by Sentry.
00:16:41.620 --> 00:16:42.740
Code breaks.
00:16:42.740 --> 00:16:44.040
It's a fact of life.
00:16:44.040 --> 00:16:46.180
With Sentry, you can fix it faster.
00:16:46.180 -->
00:16:51.900
As I've told you all before, we use Sentry on many of our apps and APIs here at Talk Python.
00:16:51.900 --> 00:16:57.120
I recently used Sentry to help me track down one of the weirdest bugs I've run into in a long
00:16:57.120 --> 00:16:57.620
time.
00:16:57.620 --> 00:16:58.560
Here's what happened.
00:16:58.960 --> 00:17:04.580
When signing up for our mailing list, it would crash under a non-common execution pass, like
00:17:04.580 --> 00:17:09.840
situations where someone was already subscribed or entered an invalid email address or something
00:17:09.840 --> 00:17:10.320
like this.
00:17:10.320 --> 00:17:16.320
The bizarre part was that our logging of that unusual condition itself was crashing.
00:17:16.320 --> 00:17:19.520
How is it possible for our log to crash?
00:17:19.520 --> 00:17:22.100
It's basically a glorified print statement.
00:17:22.100 --> 00:17:23.800
Well, Sentry to the rescue.
00:17:24.240 --> 00:17:28.620
I'm looking at the crash report right now, and I see way more information than you'd expect
00:17:28.620 --> 00:17:29.860
to find in any log statement.
00:17:29.860 --> 00:17:33.300
And because it's production, debuggers are out of the question.
00:17:33.800 --> 00:17:40.740
I see the traceback of course, but also the browser version, client OS, server OS, server OS version,
00:17:40.740 --> 00:17:45.180
whether it's production or Q and A, the email and name of the person signing up.
00:17:45.180 --> 00:17:47.240
That's the person who actually experienced the crash.
00:17:47.420 --> 00:17:50.080
Dictionaries of data on the call stack and so much more.
00:17:50.080 --> 00:17:51.040
What was the problem?
00:17:51.040 --> 00:17:58.280
I initialized the logger with the string info for the level rather than the enumeration dot info,
00:17:58.280 --> 00:18:00.720
which was an integer based enum.
00:18:00.780 --> 00:18:07.300
So the logging statement would crash saying that I could not use less than or equal to between strings and ints.
00:18:07.300 --> 00:18:08.700
Crazy town.
00:18:08.700 --> 00:18:14.880
But with Sentry, I captured it, fixed it, and I even helped the user who experienced that crash.
00:18:14.880 --> 00:18:16.320
Don't fly blind.
00:18:16.320 --> 00:18:18.000
Fix code faster with Sentry.
00:18:18.000 --> 00:18:22.020
Create your Sentry account now at talkpython.fm/sentry.
00:18:22.260 --> 00:18:30.500
And if you sign up with the code TALKPYTHON, all capital, no spaces, it's good for two free months of Sentry's business plan,
00:18:30.500 --> 00:18:34.380
which will give you up to 20 times as many monthly events as well as other features.
00:18:34.380 --> 00:18:36.620
Accessing things in that way.
00:18:36.620 --> 00:18:40.260
By the way, you asked before, like, how should people even approach learning pandas?
00:18:40.260 --> 00:18:44.600
And so I've started thinking about it a little differently based on feedback from people that I would sort of walk through it.
00:18:44.600 --> 00:18:46.160
Okay, here's a series.
00:18:46.160 --> 00:18:47.520
Here's a data frame.
00:18:47.540 --> 00:18:52.380
And after creating some, you know, fake ones with, you know, made up numbers or random numbers,
00:18:52.380 --> 00:18:53.980
then we'll start reading in files.
00:18:53.980 --> 00:18:55.140
And then we'll do this.
00:18:55.140 --> 00:18:56.040
And then we'll get to visualization.
00:18:56.040 --> 00:19:02.680
And a student of mine said, you know, you're sort of missing the big, like, people miss the big picture doing that.
00:19:02.680 --> 00:19:03.940
And they want to get the excitement.
00:19:03.940 --> 00:19:08.920
Why don't you start with read a CSV file, visualize it right there inside of Jupyter,
00:19:08.920 --> 00:19:11.400
and then people will be so impressed and amazed.
00:19:11.400 --> 00:19:12.800
And then you fill in the gaps.
00:19:12.800 --> 00:19:14.560
And so I've started doing that a bit.
00:19:14.560 --> 00:19:20.540
And I think that has not been a bad approach to catch their attention, give them a sense of what the possibilities are.
00:19:20.540 --> 00:19:25.620
And like, okay, let's now walk through each of these little pieces and build up to what we saw that first day.
00:19:25.620 --> 00:19:27.040
And that's been fun.
00:19:27.040 --> 00:19:28.480
Yeah, I totally agree.
00:19:28.480 --> 00:19:31.240
That's also something that I really strive for.
00:19:31.240 --> 00:19:33.320
I don't always do whenever I'm doing presentations.
00:19:33.320 --> 00:19:43.280
But, you know, just because someone is chosen to sit in a seat for a day-long course or an hour-long presentation doesn't mean that they couldn't use a little inspiration, right?
00:19:43.280 --> 00:19:45.180
And if you're like, wow, you did those three lines.
00:19:45.180 --> 00:19:46.080
And now we have this picture.
00:19:46.080 --> 00:19:47.100
And I understand all that.
00:19:47.100 --> 00:19:48.020
Like, tell me more.
00:19:48.020 --> 00:19:48.240
Right?
00:19:48.240 --> 00:19:48.960
That right there.
00:19:48.960 --> 00:19:50.060
That's right.
00:19:50.060 --> 00:19:50.460
Right.
00:19:50.460 --> 00:19:52.980
Now you have their actual attention the whole time.
00:19:53.060 --> 00:19:54.020
And they're enjoying it.
00:19:54.020 --> 00:20:00.200
And it's, yeah, it's so often it's like, well, in order to show you the nice stuff, I got to give you every level of detail.
00:20:00.200 --> 00:20:01.260
It's like, no, you don't.
00:20:01.260 --> 00:20:03.100
You're going to make a leap.
00:20:03.100 --> 00:20:04.160
Don't do it that way.
00:20:05.560 --> 00:20:06.100
That's right.
00:20:06.100 --> 00:20:06.840
That's right.
00:20:06.840 --> 00:20:08.420
And you mentioned, like, three lines of code.
00:20:08.420 --> 00:20:12.380
One of the amazing things about Pandas is how often you can write very, very little code.
00:20:12.380 --> 00:20:17.220
But it's, like, getting to that code that takes a while, really taking advantage of it.
00:20:17.220 --> 00:20:17.580
Yeah.
00:20:17.580 --> 00:20:19.620
We've been knowing that it's possible, right?
00:20:19.620 --> 00:20:20.480
Right.
00:20:20.480 --> 00:20:21.080
Right, right, right.
00:20:21.080 --> 00:20:21.920
People don't even know.
00:20:21.920 --> 00:20:29.220
So a lot of times, like, I've been using it now for long enough that I sort of intuit, oh, there's got to be a method that does this.
00:20:29.220 --> 00:20:31.760
Like, someone has encountered this problem before.
00:20:31.760 --> 00:20:38.200
And so either there's a method or there's an option or there's an add-on, like, something somewhere that just makes it trivially easy.
00:20:38.200 --> 00:20:43.260
And that's part of the exploration that I try to do both in the book and the weekly and, like, in my training general.
00:20:43.260 --> 00:20:45.800
And also, truth be told, I'm constantly learning stuff, right?
00:20:45.800 --> 00:20:52.520
Like, it's rare for me to teach and not discover some option or some method that I did not know about because it's just so incredibly vast.
00:20:52.520 --> 00:20:52.760
Yeah.
00:20:52.760 --> 00:20:54.960
Or Pandas 2 comes out or something like that.
00:20:54.960 --> 00:20:55.500
Yes.
00:20:55.500 --> 00:20:56.440
Yes, indeed.
00:20:56.440 --> 00:21:01.400
I mean, I've been exploring, I mean, tomorrow, tomorrow I head off to Prague for EuroPython,
00:21:01.400 --> 00:21:04.360
where I'm giving a talk on PyArrow in Pandas.
00:21:04.360 --> 00:21:06.060
And so I've been looking into that a lot.
00:21:06.060 --> 00:21:07.940
And, oh, boy, right?
00:21:07.940 --> 00:21:10.180
I mean, I've been using it for, say, a year or so.
00:21:10.180 --> 00:21:11.360
But it's amazing.
00:21:11.360 --> 00:21:19.200
And yet there are all these subtle changes that are happening in Pandas as a result that you need to know about or you need to expect when you use it.
00:21:19.200 --> 00:21:30.160
But it's another, like, tool in our toolbox that we can pull out to make Pandas more effective, more efficient, deal with larger data, and also interact with other and interoperate with other systems.
00:21:30.160 --> 00:21:30.920
Yeah, for sure.
00:21:31.040 --> 00:21:33.620
It's just like, as I said, the whole ecosystem is just exploding.
00:21:33.620 --> 00:21:34.600
It's really quite something.
00:21:34.600 --> 00:21:35.340
Yeah, it really is.
00:21:35.340 --> 00:21:41.740
And I think the Pandas 2 stuff is going to make a pretty big difference, changing, like, the internals away from just tables of numbers, basically.
00:21:41.740 --> 00:21:42.440
Okay.
00:21:42.440 --> 00:21:53.260
Let's talk about, so the way I thought we could maybe explore this Pandas workout book is let's just pick some fun exercises that you put together and talk about them.
00:21:53.380 --> 00:21:57.780
Give us a quick overview of what the workout aspect of this book means anyway.
00:21:57.780 --> 00:21:58.780
Sure.
00:21:58.780 --> 00:21:59.340
Sure.
00:21:59.340 --> 00:22:05.160
So the idea is that you can't learn everything all at once or quickly.
00:22:05.340 --> 00:22:07.920
That's sort of like working out physically.
00:22:07.920 --> 00:22:09.400
It's a long haul.
00:22:09.400 --> 00:22:10.240
It's a long haul.
00:22:10.240 --> 00:22:14.160
And every day you get a little better, a little stronger, a little more flexible.
00:22:14.160 --> 00:22:22.480
And so if you see, you know, your Pandas learning journey as it's going to take me several months, not going to take me a day.
00:22:22.700 --> 00:22:34.000
Then if every day you do a little bit of practice, you learn something new in some new direction, at the end of that journey, you're going to be able to solve many, many more problems in better, more idiomatic and more efficient ways.
00:22:34.000 --> 00:22:38.120
And you'll be able to put these pieces together in ways that you didn't even expect.
00:22:38.120 --> 00:22:39.840
So that's the basic idea.
00:22:39.840 --> 00:22:42.820
And so the book is divided into, I want to say, 12 chapters.
00:22:42.820 --> 00:22:47.940
I know we sort of rejiggered it at some point where each chapter focuses on a different aspect of Pandas.
00:22:48.600 --> 00:22:53.500
But it's really sort of the total experience of going through it.
00:22:53.500 --> 00:22:59.060
And so we have 200 exercises, plus there's like a, for lack of a better term, a midterm and a final.
00:22:59.060 --> 00:23:04.580
Like a meteor projects that people can go through that people ask for after the Python book.
00:23:04.580 --> 00:23:11.980
And each exercise then has not only the main exercise where I pose a problem, I give an explanation, I give an answer, note the order.
00:23:11.980 --> 00:23:14.620
The answer comes at the end, so you won't peek as easily.
00:23:14.620 --> 00:23:18.300
We talked about putting it in the back of the book and like that just didn't work out so well.
00:23:18.560 --> 00:23:21.260
So at least you have to wade through the explanation a little bit or turn the page.
00:23:21.260 --> 00:23:22.460
And then after the-
00:23:22.460 --> 00:23:24.580
You can buy the one book with the questions.
00:23:24.580 --> 00:23:27.120
You know, there was talk about that too.
00:23:27.120 --> 00:23:27.560
Oh, you're serious.
00:23:27.560 --> 00:23:27.780
Okay.
00:23:27.780 --> 00:23:29.040
You can buy the answers later.
00:23:29.040 --> 00:23:30.020
There was definitely-
00:23:30.020 --> 00:23:31.480
I was joking.
00:23:31.480 --> 00:23:32.620
Ten times the price.
00:23:32.620 --> 00:23:34.440
I didn't mean to give anyone ideas.
00:23:34.440 --> 00:23:34.880
I'm joking.
00:23:34.880 --> 00:23:36.060
No, no, it's fine.
00:23:36.060 --> 00:23:41.480
And then like after the answer, we have three, what we call beyond the exercise, which are,
00:23:41.480 --> 00:23:50.480
okay, now that you've kind of gotten the basics, let's push either on the same data set or even like sort of go farther.
00:23:50.480 --> 00:23:56.140
And so it's like, that's why we say it's 200 exercises because
it's 50 official exercises and other three for each one.
00:23:56.140 --> 00:23:57.980
And those tend to be much harder.
00:23:58.100 --> 00:24:01.920
And I don't give a full explanation, although I do give the solution online in a Jupyter notebook.
00:24:01.920 --> 00:24:04.880
You can download Jupyter for all these things and see how I solved it.
00:24:04.880 --> 00:24:05.100
Nice.
00:24:05.240 --> 00:24:08.700
And for many of them, there's a link to the Pandas Tutor.
00:24:08.700 --> 00:24:10.700
I think, yeah, you've definitely spoken to Philip well in the past.
00:24:10.700 --> 00:24:13.860
So he is not only a Python Tutor, but Pandas Tutor.
00:24:13.860 --> 00:24:14.500
He and his team.
00:24:14.500 --> 00:24:15.680
It's amazing.
00:24:15.680 --> 00:24:22.820
And so you can just click on a link and it will take you to usually a miniature version of the data set because it's too big for Pandas Tutor.
00:24:22.820 --> 00:24:26.620
And then you can sort of see the visualization, how these things work.
00:24:26.620 --> 00:24:26.880
Yeah.
00:24:26.880 --> 00:24:28.820
I don't think I talked about Pandas Tutor.
00:24:28.820 --> 00:24:33.560
Maybe I have talked to Philip about it, but I've certainly used it in some of my courses.
00:24:34.100 --> 00:24:36.140
But it's worth bringing it up just to talk.
00:24:36.140 --> 00:24:37.800
Like, see how that thing works?
00:24:37.800 --> 00:24:39.980
It's something special, this thing.
00:24:39.980 --> 00:24:48.680
It really gives you some deep understanding into like, okay, if I run, you know, this group by command, then here's how all the pieces like flow back together.
00:24:48.680 --> 00:24:50.280
And so you're using this during your book?
00:24:50.280 --> 00:24:50.940
So, yeah.
00:24:50.940 --> 00:24:54.080
So, I mean, every exercise has a link in the solution.
00:24:54.080 --> 00:24:56.800
You click on it and it brings you to the code that I use.
00:24:56.800 --> 00:25:04.180
It's usually, again, a sort of miniature version of the code, miniature version of the data set that you can then see the visualization there in your browser.
00:25:04.180 --> 00:25:04.660
Nice.
00:25:04.660 --> 00:25:07.460
Yeah, because you've got to basically, I think, encode the data in the URL.
00:25:07.460 --> 00:25:09.680
So it can't be too much.
00:25:09.680 --> 00:25:10.520
Right.
00:25:10.520 --> 00:25:11.160
Right.
00:25:11.280 --> 00:25:21.540
So I would like basically take the data in the data frame, turn it into a Python dictionary, and then like set it up because it won't work with files for security reasons.
00:25:21.540 --> 00:25:21.720
Yeah.
00:25:21.940 --> 00:25:28.540
So assign that dictionary to a variable or no, no, I'm sorry, I turned to CSV in a string and then assign that string to a variable and read it in.
00:25:28.540 --> 00:25:35.000
And then I would see if it overflowed the pandas work, now pandas tutor limit.
00:25:35.000 --> 00:25:40.800
And if it did, I sort of iterated until I got it small enough to fit in there and big enough to be useful and interesting.
00:25:40.800 --> 00:25:42.180
That took a few iterations.
00:25:42.180 --> 00:25:42.740
That's cool.
00:25:42.740 --> 00:25:44.880
No, but it's a super, super good resource.
00:25:44.880 --> 00:25:48.020
For learning pandas, I also think for exploring, right?
00:25:48.120 --> 00:25:52.520
Like you end up with some code that like, I'm not really sure what this does, or this is actually new to me.
00:25:52.520 --> 00:25:54.660
Like a lot of things you might encounter in this book.
00:25:54.660 --> 00:25:56.000
You're like, let me visualize that.
00:25:56.000 --> 00:25:56.280
Right.
00:25:56.280 --> 00:25:57.280
Because, yeah.
00:25:57.280 --> 00:25:57.720
Right.
00:25:57.720 --> 00:25:57.980
Right.
00:25:57.980 --> 00:26:06.280
I mean, as I said, like I do a lot of training in pandas and very often people have already played with it a bit, used it, even used it for a year or two.
00:26:06.280 --> 00:26:13.200
And because it's so large, like it's not unusual for someone to say, oh, I had no idea that this functionality existed.
00:26:13.200 --> 00:26:16.580
Why have I been wasting my time doing X and Y and Z?
00:26:17.320 --> 00:26:20.140
I'll give you one example that I've been using more and more.
00:26:20.140 --> 00:26:24.020
So, you know, Matt Harrison's a big fan of the method chaining approach.
00:26:24.020 --> 00:26:26.420
And at first I was like, yeah, yeah, yeah, Matt, whatever.
00:26:26.420 --> 00:26:28.860
Like, yo, stop pushing out on everyone.
00:26:28.860 --> 00:26:30.080
You and your fluent interfaces.
00:26:30.080 --> 00:26:31.260
Keep them to yourself.
00:26:31.260 --> 00:26:31.620
Right.
00:26:31.620 --> 00:26:38.220
And then I was like, actually, this is a great way to build things up little by little, line by line.
00:26:38.220 --> 00:26:39.060
And I can use this.
00:26:39.060 --> 00:26:40.580
It's pedagogically very useful.
00:26:40.580 --> 00:26:43.360
Because I say, okay, let's think about how we want to break down this problem.
00:26:43.360 --> 00:26:44.080
We'll do this.
00:26:44.080 --> 00:26:44.780
Then we'll do this.
00:26:44.860 --> 00:26:50.120
Then we'll do this and you can see it sort of going line by line until voila, we have the analysis that we want.
00:26:50.120 --> 00:26:53.800
And so I inserted that into a lot of places in the book.
00:26:53.800 --> 00:26:58.820
Sort of like one of the last edits that I did was to go back and change it to be more method chaining.
00:26:58.820 --> 00:27:02.700
And I use it now all the time in my training and in Bamboo Weekly.
00:27:02.700 --> 00:27:06.400
And so I bow to Matt on that.
00:27:06.400 --> 00:27:07.320
He was right.
00:27:07.320 --> 00:27:10.100
And I was stubbornly resistant for no good reason.
00:27:10.100 --> 00:27:10.400
Yeah.
00:27:10.400 --> 00:27:12.100
I really like that style as well.
00:27:12.100 --> 00:27:14.440
You see, it's officially night here where I am.
00:27:14.440 --> 00:27:15.780
It's just everything switched dark.
00:27:16.260 --> 00:27:22.960
I just said that yesterday when I was doing office hours and all of a sudden I was sharing my screen and it changed the color.
00:27:22.960 --> 00:27:25.480
I'm glad I'm not the only one that happens to you.
00:27:25.480 --> 00:27:29.680
So, yeah, I'm a big fan of the method chaining fluent interfaces.
00:27:29.680 --> 00:27:34.640
I mean, I would love to even see like Python itself in the standard library adopt that more.
00:27:34.640 --> 00:27:35.100
Right.
00:27:35.100 --> 00:27:39.980
Like there's so many things you operate on that will change something, but then it will return.
00:27:39.980 --> 00:27:41.940
It will not return anything.
00:27:41.940 --> 00:27:43.680
It's like a void method as much as we have those.
00:27:43.680 --> 00:27:43.840
Right.
00:27:43.840 --> 00:27:44.820
It returns none effectively.
00:27:45.100 --> 00:27:48.480
So you can't say, you know, dot sort, dot this, dot that.
00:27:48.480 --> 00:27:50.340
You have to like multi-step it.
00:27:50.340 --> 00:27:51.460
And I would just love to see more of it.
00:27:51.460 --> 00:27:52.180
But let's talk.
00:27:52.180 --> 00:27:55.320
Well, I'll just say there on that front.
00:27:55.320 --> 00:28:02.780
So Candace like does have the option to either get back a new data frame or to say in place equals true.
00:28:02.780 --> 00:28:06.380
And then it does it locally, like it does it on that data structure and then returns none.
00:28:06.380 --> 00:28:10.980
And people are consistently convinced that this is faster, more efficient, better.
00:28:11.380 --> 00:28:17.900
And so I've been like trying to tell people, no, the Python core developers keep saying, do not think that is true.
00:28:17.900 --> 00:28:18.840
It is not true.
00:28:18.840 --> 00:28:21.500
And we are getting rid of in place equals true at some point.
00:28:21.500 --> 00:28:24.080
Stop using it so that you can do method chaining.
00:28:24.080 --> 00:28:28.960
And so no small number of people, again, in my course, are like, oh, really?
00:28:28.960 --> 00:28:29.900
I had no idea.
00:28:29.900 --> 00:28:34.120
I feel like, you know, I'm spreading the gospel as a word.
00:28:34.120 --> 00:28:37.740
Throw that whole expression in some parentheses and dot yourself away.
00:28:37.740 --> 00:28:38.180
Let's go.
00:28:38.180 --> 00:28:38.680
All right.
00:28:38.680 --> 00:28:39.780
That's right.
00:28:39.980 --> 00:28:44.220
Let's dot our way on over to exercise seven of the many.
00:28:44.220 --> 00:28:48.360
And let's just talk about long, medium and short taxi rides.
00:28:48.360 --> 00:28:54.340
Tell us like kind of like we can only talk so much about the code, but like let's let's talk a bit about it and get people.
00:28:54.340 --> 00:29:00.480
Like I said, I want to expose people to like what are some of the problems and aspects of pandas that you can use to solve them.
00:29:00.480 --> 00:29:00.820
Sure.
00:29:00.900 --> 00:29:06.420
So one of my favorite data sets to work with is New York City taxi information.
00:29:06.420 --> 00:29:09.300
It's like everyone can identify with it.
00:29:09.300 --> 00:29:10.040
You understand it.
00:29:10.040 --> 00:29:13.720
And so this exercise uses a very small subset of that.
00:29:13.720 --> 00:29:16.720
Maybe we'll talk next about the pandemic taxis, which is a much larger one.
00:29:16.880 --> 00:29:21.740
But this is just 100,000, no, 10,000, 10,000 taxi rides from like five years ago, six years ago.
00:29:21.740 --> 00:29:31.780
And the question is, well, how can we divide up this data set, which tells us how long, how far, how much people paid, when they were picked up, all that information.
00:29:31.780 --> 00:29:36.940
How can we find out like the distance that they went and categorize that?
00:29:37.160 --> 00:29:42.980
And the reason that this can be useful is we so often have numeric data that we need to put into categories, right?
00:29:42.980 --> 00:29:44.020
What are best sellers?
00:29:44.020 --> 00:29:45.380
What are poor sellers?
00:29:45.380 --> 00:29:49.180
Who are, you know, the most, you know, employee of the year, that sort of thing.
00:29:49.180 --> 00:29:52.840
There's so many places where turning something into a category will be useful.
00:29:52.840 --> 00:29:58.840
And so it's very tempting to think, okay, I'll like do some if statements or I'll do some for statements.
00:29:58.840 --> 00:30:03.880
But actually, pandas provides us with pd.cut, which just does it for us.
00:30:04.140 --> 00:30:09.700
And this is one of those examples of once you learn it, you're like, oh, wow, I get it.
00:30:09.700 --> 00:30:11.240
I don't have to have, oh, yeah, I even have here.
00:30:11.240 --> 00:30:16.040
Like you might think you would want to say, let's set all the categories, you know, to be medium.
00:30:16.040 --> 00:30:19.480
And then where it's less than two miles, we'll call it short.
00:30:19.480 --> 00:30:21.780
And where it's greater than 10 miles, we'll call it long.
00:30:21.780 --> 00:30:24.620
But you can just say pd.cut, we're going to cut it two.
00:30:24.620 --> 00:30:25.660
We're going to cut it 10.
00:30:25.660 --> 00:30:27.100
Anything less than two is short.
00:30:27.100 --> 00:30:28.660
Anything greater than 10 is long.
00:30:28.660 --> 00:30:29.820
Anything in the middle is medium.
00:30:29.820 --> 00:30:30.440
Done.
00:30:30.440 --> 00:30:31.000
Oh, that's interesting.
00:30:31.000 --> 00:30:32.660
And pd.cut gives you back a new series.
00:30:32.660 --> 00:30:37.120
This portion of Talk Python To Me is brought to you by Scalable Path.
00:30:37.120 --> 00:30:44.180
If you're a founder or engineering leader, you know how hard it is to find top tier developers while keeping costs low.
00:30:44.180 --> 00:30:46.360
That's where Scalable Path comes in.
00:30:46.360 --> 00:30:51.020
They're a software staffing company that helps you build remote dev teams that just fit.
00:30:51.020 --> 00:30:57.440
If you're
wondering what sets this staffing company apart, well, one big differentiator is their approach.
00:30:57.840 --> 00:30:59.960
They're founded and run by developers.
00:30:59.960 --> 00:31:05.660
Scalable Path understands that finding the right developer is not just about technical skills.
00:31:05.660 --> 00:31:10.240
It's about personality, work ethic, and how well they mesh with your team.
00:31:10.240 --> 00:31:18.340
Their software architects will take the time to understand your vision and needs and then develop technical challenges for the roles you're looking to hire.
00:31:18.740 --> 00:31:25.620
And these technical tests are conducted live on video by senior software developers so there's no gaming the system.
00:31:25.620 --> 00:31:27.980
And Scalable Path takes it one step further.
00:31:27.980 --> 00:31:36.440
They evaluate each developer's soft skills like communication, attitude, and work style before presenting best suited candidates to you.
00:31:36.440 --> 00:31:41.600
Scalable Path has built a network of over 35,000 remote developers.
00:31:42.140 --> 00:31:45.340
No more endless searches or sleepless nights worrying about the right hire.
00:31:45.340 --> 00:31:48.720
And here's a special offer for Talk Python listeners.
00:31:48.720 --> 00:31:51.820
You'll get 20% off of your first month.
00:31:51.820 --> 00:31:54.760
So are you ready to scale your dev team and your business?
00:31:54.760 --> 00:31:59.160
Get started by visiting talkpython.fm/scalablepath.
00:31:59.160 --> 00:32:01.580
The link is in your podcast player's show notes.
00:32:01.580 --> 00:32:04.540
Thank you to Scalable Path for supporting the show.
00:32:05.860 --> 00:32:13.240
So the way you're doing it is you're setting everything to medium and then you're defining short and you're defining long instead of defining the three categories.
00:32:13.240 --> 00:32:17.920
So that's like the sort of you might think this would be a good way to do it.
00:32:17.920 --> 00:32:19.100
And it works, right?
00:32:19.100 --> 00:32:21.980
As I like to say in my courses, unfortunately, this works.
00:32:21.980 --> 00:32:23.620
Right?
00:32:23.620 --> 00:32:25.840
Like so you will get the right answer this way.
00:32:26.120 --> 00:32:30.540
But if you then like go to the next page here, you'll see that you can just use PD Cut.
00:32:30.540 --> 00:32:31.840
I see.
00:32:31.840 --> 00:32:32.500
Yeah.
00:32:32.500 --> 00:32:34.920
And then you just say, here are the bins.
00:32:34.920 --> 00:32:36.320
Here are the labels.
00:32:36.320 --> 00:32:36.940
Go.
00:32:36.940 --> 00:32:43.740
Well, this goes back to that thing we talked about, like looping over stuff versus just going, this is what I want you to do.
00:32:43.740 --> 00:32:47.480
Do it 100% deep down inside the best you can.
00:32:47.480 --> 00:32:48.340
Don't bother me.
00:32:48.340 --> 00:32:49.520
Just figure it out for me.
00:32:49.520 --> 00:32:49.800
Right?
00:32:49.800 --> 00:32:50.820
That's exactly right.
00:32:50.820 --> 00:33:00.220
And this means that like a whole lot of people have worked on PD Cut and have made it work efficiently, way more efficiently than you or I could do in our code.
00:33:00.220 --> 00:33:01.800
Presumably.
00:33:01.800 --> 00:33:02.240
Right?
00:33:02.240 --> 00:33:03.400
It's not going to be any worse.
00:33:03.400 --> 00:33:05.120
And it'll probably be a lot better.
00:33:05.120 --> 00:33:11.660
The other thing is you get back and you can see it sorted at the end, the output of the series you get here.
00:33:11.660 --> 00:33:14.400
So you feed it a series and you get back a series.
00:33:14.400 --> 00:33:18.360
And the series looks at first glance like it's a bunch of strings.
00:33:18.360 --> 00:33:21.700
And so you're going to have short, short, medium, medium, long, long, long, but it's actually not strings.
00:33:21.700 --> 00:33:25.160
It's actually a category, which is a pandas version of an enum.
00:33:25.160 --> 00:33:29.080
So really it's very small because it's just integers being stored there.
00:33:29.080 --> 00:33:31.480
And then those integers are associated with strings.
00:33:31.480 --> 00:33:34.400
So that's an example of like where they have thought through it.
00:33:34.400 --> 00:33:38.660
And they've basically said, yeah, we're going to make this more efficient than you would probably think to do on your own.
00:33:38.660 --> 00:33:39.080
Right.
00:33:39.080 --> 00:33:39.620
Awesome.
00:33:39.620 --> 00:33:43.280
Well, I would, you know, to be honest, I've been happy if I came up with that first solution.
00:33:44.720 --> 00:33:46.320
Just using lock instead of there.
00:33:46.320 --> 00:33:46.920
That's pretty cool.
00:33:46.920 --> 00:33:48.460
But this, this cuts is super nice.
00:33:48.460 --> 00:33:48.920
Yeah.
00:33:48.920 --> 00:33:52.400
Just to find the boundaries with as bins and then off it goes.
00:33:52.400 --> 00:33:52.680
Right.
00:33:52.680 --> 00:33:53.100
Right.
00:33:53.100 --> 00:33:55.740
And there's an option there include lowest.
00:33:55.740 --> 00:33:58.240
And so I actually didn't know about this for a while.
00:33:58.240 --> 00:34:03.960
So I'd be like, okay, so, so think about the bins has to be from some, some, you know, some small number to some large number.
00:34:03.960 --> 00:34:05.580
And the question is then, well, wait a second.
00:34:05.580 --> 00:34:07.160
What about that leftmost bin?
00:34:07.260 --> 00:34:07.700
What about that?
00:34:07.700 --> 00:34:12.740
Like if it's up to and not including, it's like less than, but not less than equal, then how do I do it?
00:34:12.740 --> 00:34:15.320
So I would always be like, well, I'll take the min.
00:34:15.320 --> 00:34:18.040
I'll do like the series dot min minus one.
00:34:18.040 --> 00:34:19.780
And then it's guaranteed to be lower than that.
00:34:19.780 --> 00:34:23.320
But no, it turns out the pandas developers thought about this long before I did.
00:34:23.320 --> 00:34:26.720
And there's an option, a
keyword argument you can pass, include lowest equals true.
00:34:26.720 --> 00:34:27.260
Done.
00:34:27.260 --> 00:34:29.800
Now it's less than an equal as opposed to just less than.
00:34:29.800 --> 00:34:30.140
Right.
00:34:30.140 --> 00:34:35.540
Because all these kind of things, the boundary conditions are always tricky, especially on floating points, right?
00:34:35.540 --> 00:34:36.300
Oh, yes.
00:34:36.300 --> 00:34:37.020
Oh, yes.
00:34:37.180 --> 00:34:41.160
There's probably at least two spacecrafts that have crashed because of this.
00:34:41.160 --> 00:34:41.640
All right.
00:34:41.640 --> 00:34:47.320
The next one we want to go to is number 12, Finding Outliers.
00:34:47.320 --> 00:34:47.960
Yeah.
00:34:47.960 --> 00:34:58.920
So, you know, it's this is not a book about statistics and I'm not an expert in statistics, but there are a whole lot of statistical ideas that permeate working with pandas.
00:34:58.920 --> 00:35:05.120
And so it's like, you know, mean, standard deviation, median, understanding sort of how they can differ from one another.
00:35:05.120 --> 00:35:05.540
Right.
00:35:05.580 --> 00:35:14.500
And especially the whole mean versus median thing where it's so easy to have a few outliers pull your data up or pull your data down and then sort of fool you.
00:35:14.500 --> 00:35:14.880
Right.
00:35:15.220 --> 00:35:17.600
And so often people want to find the opposite.
00:35:17.600 --> 00:35:18.100
And so often people want to find the opposite.
00:35:18.100 --> 00:35:24.320
And I've certainly found this, especially with corporate training with like cybersecurity people where they're like, we are always looking for outliers.
00:35:24.320 --> 00:35:27.600
Like, you know, who was the user who was logging in at unusual times?
00:35:27.600 --> 00:35:27.700
Right.
00:35:27.700 --> 00:35:31.600
Who logged in for 20 IP addresses within one minute.
00:35:31.600 --> 00:35:31.860
Right.
00:35:31.860 --> 00:35:33.360
That's right.
00:35:33.660 --> 00:35:34.660
That's not a good person.
00:35:34.660 --> 00:35:34.680
That's not a good person.
00:35:34.680 --> 00:35:35.340
They're not good.
00:35:35.340 --> 00:35:37.040
I'm to tell you just that right now.
00:35:37.040 --> 00:35:39.580
They just try to be very, very effective.
00:35:39.580 --> 00:35:42.920
And so and so.
00:35:43.040 --> 00:35:50.420
So this finding outliers is, OK, let's find out then not who was sort of normal, but like who was abnormal.
00:35:50.420 --> 00:35:50.860
Right.
00:35:50.860 --> 00:35:54.260
Who who is exhibiting, you know, unusual behavior.
00:35:54.260 --> 00:36:01.700
And so here it's like, OK, let's take our set of numbers and let's find out who was more than one standard deviation.
00:36:01.700 --> 00:36:04.820
I can't remember if it's one or two above the mean or below the mean.
00:36:04.820 --> 00:36:08.980
And let's just find those values and ignore the sort of normal values.
00:36:08.980 --> 00:36:17.340
And then we could talk a little about, you know, IQR, the inter-portile range, which is a standard statistical idea that we don't talk about enough, even though it's super useful.
00:36:17.340 --> 00:36:17.780
Right.
00:36:17.780 --> 00:36:19.540
People came up with these ideas a long time ago.
00:36:19.540 --> 00:36:20.820
Oh, here's a fun fact.
00:36:21.040 --> 00:36:26.420
John Tukey, who came up with a lot of these ideas, he also invented the word bit in computers.
00:36:26.420 --> 00:36:26.860
Oh, wow.
00:36:26.860 --> 00:36:27.340
OK.
00:36:27.340 --> 00:36:29.800
So, yeah, I was like, wow, what a guy.
00:36:29.800 --> 00:36:34.600
So you can see here then that like I'm going to take the, you know, trip distance.
00:36:34.600 --> 00:36:36.080
Let's find, you know, trip distance.
00:36:36.080 --> 00:36:45.200
Let's find where the trip distance is less than the trip distance at the first quarter of the first quarter minus one half times the IQR.
00:36:45.200 --> 00:36:45.600
Right.
00:36:45.600 --> 00:36:49.760
Meaning let's take the distance between the first quarter and the third quarter.
00:36:50.040 --> 00:36:51.660
That's our inter-portile range.
00:36:51.660 --> 00:36:55.200
That gives us a sense of like where is the bulk, where are the bulk of the numbers?
00:36:55.200 --> 00:36:58.620
And let's find out who is below that or who is above that.
00:36:58.620 --> 00:37:08.300
And so it's not surprising if we're looking for taxi distances, you're not going to have a lot of outliers that are very low because you can't go below zero miles in your taxi.
00:37:08.300 --> 00:37:10.680
But you can go very high.
00:37:10.680 --> 00:37:11.920
You can go very large.
00:37:11.920 --> 00:37:24.040
And so looking for those trips that are greater than the 75th percentile plus one and a half times the IQR, we actually find, you know, I see here in front of me, you know, about 1800 taxi rides out of the 10,000.
00:37:24.260 --> 00:37:28.340
So about 19 percent of the taxi rides are much, much longer than the mean ride.
00:37:28.340 --> 00:37:28.620
Interesting.
00:37:29.060 --> 00:37:39.180
And you can use this and you can imagine people in like the New York City Taxi Limousine Commission saying, oh, we can use this to plan, to charge, to, you know, send taxis to different places.
00:37:39.180 --> 00:37:42.960
Or have a special program for long distance stuff or whatever.
00:37:42.960 --> 00:37:43.220
Yeah.
00:37:43.220 --> 00:37:43.540
Right.
00:37:43.540 --> 00:37:43.880
Yeah.
00:37:44.080 --> 00:37:44.260
Right.
00:37:44.260 --> 00:37:45.580
Or if you're Uber, you know where to play.
00:37:45.580 --> 00:37:50.520
They actually used to have the longitude and latitude of where people were picked up and dropped off.
00:37:50.520 --> 00:37:51.720
And they got rid of that.
00:37:51.720 --> 00:37:58.680
And I'm sure both for privacy reasons and because Uber and Lyft can look at that data and say, oh, well, we know now when to send cars where.
00:37:58.680 --> 00:38:07.840
Although while true, I
imagine that that data has been downloaded and archived with geolocation for certain people.
00:38:07.840 --> 00:38:10.080
You can still get all that old data.
00:38:10.080 --> 00:38:11.440
It's just the newer stuff that they don't do.
00:38:11.440 --> 00:38:12.560
And they still do it by neighborhood.
00:38:12.560 --> 00:38:13.140
I see.
00:38:13.160 --> 00:38:16.220
I also wonder, like, how precise that longitude and latitude was.
00:38:16.220 --> 00:38:20.340
Like, you could probably identify which home, like, was going to which other home.
00:38:20.340 --> 00:38:21.260
Yeah.
00:38:21.260 --> 00:38:22.700
Now you're getting into a problem.
00:38:22.700 --> 00:38:23.320
How bad that would look.
00:38:23.320 --> 00:38:23.580
Yeah.
00:38:23.580 --> 00:38:34.400
Now you're getting into a shady spot or certain types of establishments, you know, that could be fraught with consequences for the people whose home you've already identified where they were picked up at or all sorts of stuff.
00:38:34.400 --> 00:38:35.240
Exactly.
00:38:35.240 --> 00:38:35.720
All right.
00:38:35.720 --> 00:38:35.920
Fair.
00:38:35.920 --> 00:38:36.320
Exactly.
00:38:36.320 --> 00:38:36.820
Fair.
00:38:36.820 --> 00:38:41.220
That's way more important than whether Uber can lease their vehicles strategically, you know?
00:38:41.220 --> 00:38:41.940
Right.
00:38:41.940 --> 00:38:42.240
Right.
00:38:42.240 --> 00:38:44.340
And they might have some data on their own, too, you know?
00:38:44.340 --> 00:38:44.740
Nice.
00:38:44.740 --> 00:38:45.420
Okay.
00:38:45.420 --> 00:38:51.280
So you can use this IQR under quartile range feature to pull that out here, right?
00:38:51.280 --> 00:38:51.760
Right.
00:38:51.760 --> 00:38:52.160
Right.
00:38:52.160 --> 00:38:55.140
And, you know, we can pull that out and then do simple multiplication.
00:38:55.140 --> 00:39:02.580
Right at the end of the day, what we're doing is we're pulling out this data and then just doing very, very simple statistical analysis of it.
00:39:02.840 --> 00:39:05.000
Just to sort of say, how many outliers do we have?
00:39:05.000 --> 00:39:05.880
How many low outliers?
00:39:05.880 --> 00:39:06.820
How many high outliers?
00:39:06.820 --> 00:39:11.200
And you can see this, like, at the end of the day, it's not that complicated in terms of math.
00:39:11.200 --> 00:39:14.460
Sometimes people are like, well, how much math do I have to know to learn this stuff?
00:39:14.460 --> 00:39:15.800
I'm like, I can get through it.
00:39:15.800 --> 00:39:16.560
I promise you.
00:39:16.560 --> 00:39:17.480
Not that much.
00:39:17.480 --> 00:39:18.260
It's not that hard.
00:39:18.260 --> 00:39:20.680
A few basic ideas and you're basically set.
00:39:20.680 --> 00:39:21.040
Yeah.
00:39:21.040 --> 00:39:21.520
Very cool.
00:39:21.740 --> 00:39:21.940
All right.
00:39:21.940 --> 00:39:25.360
On to the next one, which is endemic taxis.
00:39:25.360 --> 00:39:26.780
That's a different kind of taxi.
00:39:26.780 --> 00:39:36.980
So this was actually, so the two studies we looked at so far were with my tiny little 10,000 taxi ride sample from a few years ago.
00:39:36.980 --> 00:39:42.360
I then took taxi rides from January and July of 2019 and 2020.
00:39:42.360 --> 00:39:43.640
So four months there.
00:39:43.640 --> 00:39:48.100
And the question was, I think I actually only looked at July in this exercise.
00:39:48.100 --> 00:39:48.400
Yeah.
00:39:48.400 --> 00:39:51.380
July 2019, July 2020, comparing taxi rides.
00:39:51.500 --> 00:39:58.340
Now, just to remind you, July 2020, not a great time for tourism in New York City or anywhere else.
00:39:58.340 --> 00:40:04.980
And so the question was, what differences do we see between July 2019 and July 2020?
00:40:04.980 --> 00:40:09.180
How much did it go down in terms of taxi use?
00:40:09.180 --> 00:40:10.420
How much less did people pay?
00:40:10.420 --> 00:40:17.600
And then my favorite part of this was, did people use cash more in 2020 or less in 2020?
00:40:18.080 --> 00:40:24.820
And so you see, first of all, it's like a decline of something like 80% in terms of taxi rides from 2019 to 2020.
00:40:24.920 --> 00:40:29.180
Again, not a huge surprise to anyone who lived through the pandemic and like saw what was going on there.
00:40:29.180 --> 00:40:35.020
And so my gut feeling was, well, no one wanted to touch anyone else or touch anything that anyone else had touched.
00:40:35.020 --> 00:40:38.340
So clearly people would have used credit cards much more.
00:40:38.340 --> 00:40:39.480
And no, it turns out.
00:40:39.480 --> 00:40:41.160
There's that screen, Ruben.
00:40:41.160 --> 00:40:42.640
You got to touch the OK.
00:40:42.640 --> 00:40:45.160
That drove me crazy during the pandemic.
00:40:45.160 --> 00:40:46.780
It's like, I'm going to just do touch lists.
00:40:46.780 --> 00:40:48.400
And then you like sign it.
00:40:48.400 --> 00:40:48.880
Are you OK?
00:40:48.880 --> 00:40:50.180
You're like, mm-hmm.
00:40:50.180 --> 00:40:52.760
I never thought of that.
00:40:52.760 --> 00:40:57.260
I guess in Israel, like you just, you just like, you know, do the tapping of your car.
00:40:57.260 --> 00:40:58.300
You didn't have to sign anything.
00:40:58.300 --> 00:41:01.460
But yeah, the US, they're like, oh, hit OK to confirm.
00:41:01.520 --> 00:41:05.600
It's like, OK, I thought we had just escaped me touching this disease-ridden thing.
00:41:05.600 --> 00:41:06.360
But no.
00:41:06.360 --> 00:41:09.480
Also, you have to like indicate how much of a tip you want to leave.
00:41:09.480 --> 00:41:10.700
But that's a whole separate thing.
00:41:10.700 --> 00:41:13.740
We're at the, I don't know, the car wash.
00:41:13.740 --> 00:41:14.860
Don't you want to tip the car?
00:41:14.860 --> 00:41:16.000
No, I don't want to tip the car wash.
00:41:16.000 --> 00:41:18.680
That's the other thing is you should have looked at tips.
00:41:18.680 --> 00:41:21.300
Like that could have been interesting, actually, as well.
00:41:21.300 --> 00:41:22.120
I think, oh, I did.
00:41:22.120 --> 00:41:23.580
I think, no, I think I did.
00:41:23.580 --> 00:41:26.080
I definitely looked at tips, but I can't remember what it was like.
00:41:26.080 --> 00:41:27.020
Around the pandemic, yeah.
00:41:27.020 --> 00:41:29.320
Because I know a lot of people gave tips to like kind of say thanks.
00:41:29.320 --> 00:41:31.040
And I was wondering if that would show up, you know, like,
00:41:31.100 --> 00:41:34.480
thanks for being out here amongst the diseased ones, you know.
00:41:34.480 --> 00:41:39.200
So it turns out that people used cash more during the pandemic than credit.
00:41:39.200 --> 00:41:41.600
And I like raised this with my family.
00:41:41.600 --> 00:41:42.940
I was like, what the heck?
00:41:42.940 --> 00:41:46.700
I think it was my sister who said, well, everyone who would use,
00:41:46.700 --> 00:41:50.020
like the people who are sort of higher income earners who use their credit cards more,
00:41:50.020 --> 00:41:51.140
they were staying home.
00:41:51.140 --> 00:41:55.300
It's the people who like were forced to go to work who use cash more,
00:41:55.300 --> 00:41:57.340
who were taking the taxis because they didn't want to take the subway.
00:41:57.340 --> 00:41:59.460
Like there was like sort of an in-between sort of thing.
00:41:59.460 --> 00:42:00.420
OK, interesting.
00:42:00.680 --> 00:42:04.780
You had all these like wild pieces of data and analysis that you can do.
00:42:04.780 --> 00:42:08.180
And here it was not just like pulling all this data.
00:42:08.180 --> 00:42:12.100
But one of the big things was also I want to read in a lot of data.
00:42:12.100 --> 00:42:14.180
I want to read in from two different CSV files.
00:42:14.180 --> 00:42:15.340
Now what do I do?
00:42:15.340 --> 00:42:15.660
Right?
00:42:15.660 --> 00:42:17.100
Because it's very obvious.
00:42:17.100 --> 00:42:18.860
You read in a CSV file, you get a data frame.
00:42:18.860 --> 00:42:20.340
You read another CSV file, you get a data frame.
00:42:20.340 --> 00:42:22.160
But I want to treat those as one.
00:42:22.160 --> 00:42:24.200
And I want to then be able to distinguish between them.
00:42:24.200 --> 00:42:25.020
How do I do that?
00:42:25.020 --> 00:42:28.240
And so sure enough, Pandas has a concat method.
00:42:28.440 --> 00:42:30.460
And I use concat now all the time.
00:42:30.460 --> 00:42:35.300
I read in multiple CSV files into multiple data frames.
00:42:35.300 --> 00:42:36.420
I concat them together.
00:42:36.420 --> 00:42:40.420
And you concat them either horizontally or vertically, depending on what you want to do.
00:42:40.420 --> 00:42:42.260
And this comes back to where I said-
00:42:42.260 --> 00:42:44.440
Their columns are the same, then you might do it vertically.
00:42:44.440 --> 00:42:49.720
But if you want to augment it potentially, it's like, oh, now we have the sale percentage data,
00:42:49.720 --> 00:42:52.780
but it goes along with the columns exactly or the rows exactly.
00:42:52.780 --> 00:42:53.420
Precisely.
00:42:53.420 --> 00:42:53.600
Okay.
00:42:53.600 --> 00:42:54.140
Precisely.
00:42:54.140 --> 00:43:00.820
And so another nice way to do this also is not just read this one and read that one,
00:43:00.820 --> 00:43:03.880
but you can use a list comprehension with something like glob.
00:43:03.880 --> 00:43:04.280
Right?
00:43:04.280 --> 00:43:06.520
So glob.glob on star.csv.
00:43:06.520 --> 00:43:10.960
Get back a list of data frames and then just hand that to pd.concat.
00:43:10.960 --> 00:43:15.360
And so that's where knowing Python and be able to pull that out and use those techniques
00:43:15.360 --> 00:43:16.660
can really, really come in handy.
00:43:16.660 --> 00:43:17.040
Yeah.
00:43:17.040 --> 00:43:19.180
Earlier you mentioned, I want to go back to it real quick.
00:43:19.180 --> 00:43:22.040
You mentioned that learn Python first and then Pandas.
00:43:22.040 --> 00:43:27.540
A lot of times when I think learn Python, I think, okay, well, learn the language and then
00:43:27.540 --> 00:43:30.160
learn the standard library to a good degree.
00:43:30.160 --> 00:43:35.100
And then begin to chop away at the half million things on PyPI that are interesting.
00:43:35.100 --> 00:43:37.860
You know, and it's like a never ending sort of thing.
00:43:37.860 --> 00:43:40.460
And then there's, of course, this joke, like I learned Python.
00:43:40.600 --> 00:43:42.460
It was a good weekend or something like that.
00:43:42.460 --> 00:43:42.720
Right?
00:43:42.720 --> 00:43:44.780
Like, how do you, how do you square these two things?
00:43:44.780 --> 00:43:49.180
But I feel like the amount of Python that you need to learn is mostly centered around the
00:43:49.180 --> 00:43:51.780
language and really is actually not that much.
00:43:51.780 --> 00:43:54.400
And then you kind of learn the past, the pandas way.
00:43:54.400 --> 00:43:54.740
Right.
00:43:54.740 --> 00:43:56.100
Theoretically, if you wanted to.
00:43:56.100 --> 00:43:56.700
Right.
00:43:56.700 --> 00:43:56.900
Right.
00:43:56.900 --> 00:43:57.240
For sure.
00:43:57.320 --> 00:44:01.180
Look, I am a big fan of objects and classes and all that stuff.
00:44:01.180 --> 00:44:08.320
But when I'm talking to people who specifically want to use Python for data analytics with pandas,
00:44:08.320 --> 00:44:11.840
I say to them, objects will help you, but they're not going to be crucial.
00:44:12.180 --> 00:44:18.140
Like if there's a part of the course that you want to sort of drop, save some money, save some time, then that's a place where we can save.
00:44:18.140 --> 00:44:23.460
Because the odds of an analyst using pandas writing their own classes are pretty slim.
00:44:23.460 --> 00:44:26.680
I think it'll give them some perspective on how these classes work.
00:44:26.680 --> 00:44:27.000
Sure.
00:44:27.280 --> 00:44:29.280
But I don't think that like they need to learn that.
00:44:29.280 --> 00:44:32.880
And like a lot of the standard library there, it's hard to say.
00:44:32.880 --> 00:44:33.220
Right.
00:44:33.220 --> 00:44:34.780
So as I said, I love glob.
00:44:34.780 --> 00:44:35.100
Right.
00:44:35.100 --> 00:44:36.100
Globbing is fantastic.
00:44:36.100 --> 00:44:40.000
But that's definitely not in like my intro class.
00:44:40.000 --> 00:44:41.180
I'm going to say, oh, by the way.
00:44:41.180 --> 00:44:41.400
Yeah.
00:44:41.400 --> 00:44:47.180
I would bet there's probably 10 modules that if you knew, you might not need to learn more for six months.
00:44:47.180 --> 00:44:50.420
You know, doing Jupyter pandas type of work.
00:44:50.420 --> 00:44:50.620
Right.
00:44:50.620 --> 00:44:53.100
Like path lib and a couple of things like that.
00:44:53.100 --> 00:44:53.360
Right.
00:44:53.360 --> 00:44:53.880
That's right.
00:44:53.880 --> 00:44:54.420
That's right.
00:44:54.420 --> 00:44:55.920
I mean, I just made up that number.
00:44:55.920 --> 00:44:56.220
Yeah.
00:44:56.500 --> 00:45:02.400
It's mostly like just being able to sort of work with the core data structures, understanding the syntax and how it works.
00:45:02.400 --> 00:45:04.900
And even like defining some simple functions.
00:45:04.900 --> 00:45:05.340
Right.
00:45:05.340 --> 00:45:09.060
I think most people using pandas are not at the end going to define functions.
00:45:09.060 --> 00:45:15.000
Although as I've gone on with my use of pandas, I see, you know, lambda, this is where it's at.
00:45:15.000 --> 00:45:17.000
Like knowing how to use it really, really helps.
00:45:17.000 --> 00:45:19.540
So there I do sort of go back with the pandas people.
00:45:19.540 --> 00:45:21.620
I'm like, okay, this is going to be super weird.
00:45:21.620 --> 00:45:23.600
We're going to talk about anonymous functions now.
00:45:24.160 --> 00:45:30.220
And then, and then like, I typically do that with the more advanced pandas groups, not with the like introductory ones.
00:45:30.220 --> 00:45:34.360
But if you can sort of wrap your mind around that, then it does help quite a bit.
00:45:34.360 --> 00:45:34.720
It does.
00:45:34.720 --> 00:45:40.280
So it takes what would be multiple step things and it lets you turn it into one of those changed expressions.
00:45:40.280 --> 00:45:45.720
Because instead of defining a function somewhere else, you can just put it in line as part of a lambda and just keep on going, you know?
00:45:45.720 --> 00:45:46.040
Great.
00:45:46.040 --> 00:45:46.520
That's right.
00:45:46.520 --> 00:45:49.680
And the fact that, so, so it's especially useful to find.
00:45:49.680 --> 00:45:51.160
So we mentioned dot lock earlier.
00:45:51.160 --> 00:45:53.420
So dot lock allows you to choose rows.
00:45:53.420 --> 00:45:54.260
Let's start with that.
00:45:54.260 --> 00:46:01.800
And so I can pass it a lambda and that lambda then gets the data frame you're working on as an argument.
00:46:02.020 --> 00:46:07.200
And then whatever returns, if it returns a Boolean series, then it allows you to filter.
00:46:07.200 -->
00:46:13.100
Well, you can then have multiple dot locks and multiple lambdas in a row to do successive filtering.
00:46:13.100 --> 00:46:16.760
And yes, that is less efficient than doing it all in one fell swoop.
00:46:16.760 --> 00:46:19.880
But boy, oh boy, it's easier to think about and it's easier to use.
00:46:20.160 --> 00:46:24.020
So you just like whittle it down, whittle it down, whittle it down each line with its own lambda.
00:46:24.020 --> 00:46:36.020
And so understanding how to do that, understanding why it's important that the lambda, inside the lambda, you're using the temporary parameter from the lambda as opposed to the overall variable for your data frame.
00:46:36.020 --> 00:46:38.160
Because you're chaining it, that's useful as well.
00:46:38.160 --> 00:46:39.580
Well, 100%.
00:46:39.580 --> 00:46:44.160
And then you can also like comment out certain lines at the end and see what the intermediate values are.
00:46:44.160 --> 00:46:45.800
And so you say it's less efficient.
00:46:45.800 --> 00:46:48.040
It's less computationally efficient.
00:46:48.160 --> 00:46:51.900
It might be more efficient as a human being trying to understand what the heck is happening, right?
00:46:51.900 --> 00:46:53.900
That is spot on.
00:46:53.900 --> 00:47:01.680
I often tell people that I think Python is a language for an age in which computers are cheap and people are expensive.
00:47:01.680 --> 00:47:02.120
Yes.
00:47:02.120 --> 00:47:08.740
Because like, right, our efficiency is the big bottleneck in terms of time, in terms of money.
00:47:08.740 --> 00:47:12.040
So, right, if it takes my computer a few seconds more, who the heck cares?
00:47:12.040 --> 00:47:12.260
Right.
00:47:12.260 --> 00:47:16.200
My M2 Pro doesn't care if it's like a tiny bit more.
00:47:16.200 --> 00:47:16.680
I don't care.
00:47:16.680 --> 00:47:17.780
That's right.
00:47:17.960 --> 00:47:18.340
That's right.
00:47:18.340 --> 00:47:21.020
Or my cluster of GPUs, take your pick.
00:47:21.020 --> 00:47:24.120
Although at that point, it starts to cost real money to burn those things, you know?
00:47:24.120 --> 00:47:24.480
Right.
00:47:24.480 --> 00:47:26.180
But still, I mean, depending on that.
00:47:26.180 --> 00:47:29.680
Look, if you see that your computation is taking a long time, okay.
00:47:29.680 --> 00:47:32.200
So then you sort of find that and you improve that.
00:47:32.200 --> 00:47:32.540
Yeah.
00:47:32.540 --> 00:47:33.560
That's a really good point.
00:47:33.560 --> 00:47:36.860
Like a lot of things are, don't worry about that until it's actually going to become a problem.
00:47:36.860 --> 00:47:37.660
It looks slow.
00:47:37.660 --> 00:47:39.460
It turns out it's probably a blink of an eye.
00:47:39.460 --> 00:47:39.960
All right.
00:47:39.960 --> 00:47:40.860
Let's keep moving on.
00:47:41.000 --> 00:47:45.420
We're getting short on time and we have a plethora of things to work with.
00:47:45.420 --> 00:47:48.720
I kind of want to go to wine words.
00:47:48.720 --> 00:47:49.760
Wine words.
00:47:49.760 --> 00:47:50.280
Wine words.
00:47:50.280 --> 00:47:50.640
All right.
00:47:50.640 --> 00:47:52.400
Let me remember what exercise that is.
00:47:52.400 --> 00:47:53.640
That is 37.
00:47:53.640 --> 00:47:54.160
All right.
00:47:54.160 --> 00:47:54.560
Yeah.
00:47:54.800 --> 00:48:00.300
So people think of pandas, not wrongly, as being great at working with numbers.
00:48:00.300 --> 00:48:05.900
But it turns out that it's fantastic at working with two other kinds of data types.
00:48:05.900 --> 00:48:08.940
One is strings and one is dates and time.
00:48:08.940 --> 00:48:14.440
And you can get a ton, a ton out of analysis working with these if you know how to work with it.
00:48:14.440 --> 00:48:17.560
So there is, I forget where it is.
00:48:17.560 --> 00:48:23.780
There's a machine learning archive of data sets.
00:48:23.780 --> 00:48:28.560
And one of them is 150,000 wine reviews from Wine Magazine.
00:48:28.560 --> 00:48:29.180
Oh, wow.
00:48:29.180 --> 00:48:30.280
And so I said, okay.
00:48:30.280 --> 00:48:31.560
Is it Kaggle maybe or is it somewhere else?
00:48:31.560 --> 00:48:35.560
I think Kaggle has a version of it, but I think it's elsewhere.
00:48:35.560 --> 00:48:36.900
Wine Mag.
00:48:36.900 --> 00:48:37.940
150,000 reviews.
00:48:37.940 --> 00:48:38.240
Okay.
00:48:38.240 --> 00:48:39.040
Beautiful.
00:48:39.040 --> 00:48:39.840
I don't know.
00:48:39.840 --> 00:48:40.660
I'll find out where it is.
00:48:40.660 --> 00:48:41.160
Anyway.
00:48:41.160 --> 00:48:41.580
No worries.
00:48:41.580 --> 00:48:44.900
So I said, okay, let's find out.
00:48:44.900 --> 00:48:45.200
Right?
00:48:45.260 --> 00:48:49.100
Because you drink a bottle of wine and you read the back and you sort of like, you know,
00:48:49.100 --> 00:48:50.440
roll your eyes at what they've written.
00:48:50.440 --> 00:48:52.000
Although, although I'll put in a plug.
00:48:52.000 --> 00:48:57.400
I read this fantastic book a few years ago called Cork Dork by this journalist who decided
00:48:57.400 --> 00:48:58.220
to become a sommelier.
00:48:58.220 --> 00:49:00.820
And she took the exam and her journey toward there.
00:49:00.820 --> 00:49:04.660
She was like, and she convinced me these words actually have real meaning and people are very
00:49:04.660 --> 00:49:05.400
serious about it.
00:49:05.400 --> 00:49:07.900
So I will not roll my eyes quite as much anymore.
00:49:07.900 --> 00:49:10.240
Getting hints of nutmeg, but they're troubling.
00:49:10.240 --> 00:49:11.380
Yes.
00:49:11.380 --> 00:49:13.400
They don't belong.
00:49:13.660 --> 00:49:17.480
So the question was, okay, what about these reviews of wine?
00:49:17.480 --> 00:49:19.320
What words are people using?
00:49:19.320 --> 00:49:24.940
And are they using certain words more with California wines and certain words more with French wines
00:49:24.940 --> 00:49:27.940
or certain words more with red or with white or rosé?
00:49:27.940 --> 00:49:33.320
And
so we can then take this text, break it apart and search for it.
00:49:33.320 --> 00:49:35.340
We can search for it using plain old Panda stuff.
00:49:35.340 --> 00:49:39.360
We can search for it using regular expressions, which it has built in and works very well.
00:49:39.960 --> 00:49:42.540
So here, so let's see.
00:49:42.540 --> 00:49:43.660
So analysis on the words.
00:49:43.660 --> 00:49:47.800
Here, so one of the 10 most common words for red wine, right?
00:49:47.800 --> 00:49:48.800
So how do we do that?
00:49:48.800 --> 00:49:51.460
Well, we have to take the description and break it apart.
00:49:51.460 --> 00:49:55.260
And if you're used to using just plain old Python, you're like, oh, well, I guess I'll
00:49:55.260 --> 00:49:56.420
break that into a list.
00:49:56.420 --> 00:49:57.340
But now what?
00:49:57.340 --> 00:49:58.860
Now I have a series of lists.
00:49:58.860 --> 00:49:59.820
Now what do I do?
00:50:00.040 --> 00:50:03.640
And so one of the key methods to know here is something called explode.
00:50:03.640 --> 00:50:10.940
And explode is let's take a series of lists and turn that into a very, very, very long series.
00:50:10.940 --> 00:50:15.980
And so basically each element in the list becomes an element in the series and they all share
00:50:15.980 --> 00:50:16.520
an index.
00:50:16.520 --> 00:50:18.220
So you know where they're originally from.
00:50:18.220 --> 00:50:20.500
And then, you know, the world's your oyster.
00:50:20.500 --> 00:50:23.140
So you can get rid of punctuation, right?
00:50:23.140 --> 00:50:25.120
So I have a S dot stir lower.
00:50:25.120 --> 00:50:25.400
Okay.
00:50:25.400 --> 00:50:26.440
So we lowercase everything.
00:50:26.440 --> 00:50:27.620
Then dot stir split.
00:50:27.780 --> 00:50:28.700
And we split into a list.
00:50:28.700 --> 00:50:30.720
Then we say explode, get into a long series.
00:50:30.720 --> 00:50:33.600
Then we once again run strip, angry with punctuation.
00:50:33.600 --> 00:50:40.340
And now we can use is in, which is yet another fantastic pandas method to say, are these words
00:50:40.340 --> 00:50:43.840
in, like find the lines where these words are located.
00:50:43.840 --> 00:50:45.640
And then we can just do a value count.
00:50:45.640 --> 00:50:46.900
Definitely my favorite method.
00:50:46.900 --> 00:50:48.660
How often does this thing show up?
00:50:48.660 --> 00:50:49.620
And then we use head.
00:50:49.620 --> 00:50:50.320
Ta-da.
00:50:50.320 --> 00:50:53.720
We've got the 10 most common words there.
00:50:53.720 --> 00:50:55.620
And when you break it down like that.
00:50:55.620 --> 00:50:59.260
The value count sorts probably most common to least common, right?
00:50:59.260 --> 00:51:00.040
Yeah, yeah, exactly.
00:51:00.040 --> 00:51:04.400
So value counts, not only does it count how often something shows up, but exactly.
00:51:04.400 --> 00:51:06.600
It sorts it from most common to least common.
00:51:06.600 --> 00:51:07.860
So then head is just head.
00:51:07.860 --> 00:51:08.080
Yeah.
00:51:08.080 --> 00:51:10.120
However much you take there.
00:51:10.120 --> 00:51:11.300
And then that's the ones.
00:51:11.300 --> 00:51:12.160
Those are the popular ones.
00:51:12.160 --> 00:51:12.620
That's right.
00:51:12.620 --> 00:51:13.160
That's right.
00:51:13.160 --> 00:51:15.840
So here I have like, you know, the page we're looking at.
00:51:15.840 --> 00:51:17.980
So we get find where the country is France.
00:51:17.980 --> 00:51:19.380
So that's the row selector.
00:51:19.380 --> 00:51:21.160
And we want the column selector to be description.
00:51:21.160 --> 00:51:22.320
So we only want the description.
00:51:22.320 --> 00:51:26.200
And then we have our function top 10 words, which did what I just described.
00:51:26.200 --> 00:51:29.300
We're going to take these common wine words, lowercase them, get rid of punctuation, so
00:51:29.300 --> 00:51:30.280
forth, pass it in.
00:51:30.280 --> 00:51:31.760
And we get back our top words.
00:51:31.760 --> 00:51:36.360
And we find out what words are used to, you know, are associated with French wines as opposed
00:51:36.360 --> 00:51:38.580
to California wines and so on and so forth.
00:51:38.580 --> 00:51:39.040
Excellent.
00:51:39.040 --> 00:51:39.460
Yeah.
00:51:39.460 --> 00:51:40.660
Very, very cool example.
00:51:40.660 --> 00:51:41.180
All right.
00:51:41.180 --> 00:51:44.220
We, I think we got time for two more to go through.
00:51:44.220 --> 00:51:46.900
You want to pick two that are popular or you want to leave?
00:51:47.160 --> 00:51:48.500
let's see.
00:51:48.500 --> 00:51:49.080
Let's see.
00:51:49.080 --> 00:51:54.220
Let's do, maybe 32 multi-state.
00:51:54.220 --> 00:51:54.760
Actually.
00:51:54.760 --> 00:51:55.440
Hmm.
00:51:55.440 --> 00:51:55.920
Yeah.
00:51:55.920 --> 00:51:56.540
Multi.
00:51:56.540 --> 00:51:58.120
Oh, no, let's do best tippers.
00:51:58.120 --> 00:51:58.960
Best tippers.
00:51:58.960 --> 00:51:59.740
Let's do best tippers.
00:51:59.740 --> 00:52:00.300
Yeah.
00:52:00.300 --> 00:52:01.080
You mentioned tippers.
00:52:01.080 --> 00:52:03.760
Oh, you're going to open up a whole, this is going to be a whole thing.
00:52:03.760 --> 00:52:04.520
Best tippers.
00:52:04.520 --> 00:52:05.440
Number 42.
00:52:05.440 --> 00:52:06.180
Let's go.
00:52:06.180 --> 00:52:08.020
No, no, no judgment here, folks.
00:52:08.020 --> 00:52:08.580
All right.
00:52:08.840 --> 00:52:13.020
So the question is like, so we'll try to understand as everybody here when people tip their taxi
00:52:13.020 --> 00:52:14.180
drivers more generously.
00:52:14.180 --> 00:52:14.800
All right.
00:52:14.800 --> 00:52:17.540
So the question was, did they tip better?
00:52:17.540 --> 00:52:21.080
And we looked at 2019 before the pandemic in January and July.
00:52:21.080 --> 00:52:24.520
So do they tip more in the winter or do they tip more in the summer?
00:52:24.520 --> 00:52:26.580
And so this involves several things.
00:52:26.580 --> 00:52:29.180
First
of all, it involved using dates and time.
00:52:29.180 --> 00:52:34.360
Again, one of these things that pandas is just amazing at that people just are not aware of all
00:52:34.360 --> 00:52:35.480
the flexibility you have there.
00:52:35.760 --> 00:52:39.320
Another thing is how easy it is to create a new column, right?
00:52:39.320 --> 00:52:42.640
So we spoke before a bit how you can use broadcasting to just multiply.
00:52:42.640 --> 00:52:46.680
I have a column multiplied by something or a column adding to another column.
00:52:46.680 --> 00:52:48.860
But you can create a new column just by assigning to it.
00:52:48.860 --> 00:52:50.460
Again, it's sort of like assigning to a dictionary.
00:52:50.460 --> 00:52:51.700
It just then is there.
00:52:51.700 --> 00:52:56.420
And so you can calculate the percentage that people tip, put that in a new column and then
00:52:56.420 --> 00:53:01.160
say, well, let's now group by the month and let's find out.
00:53:01.160 --> 00:53:03.980
Take the mean or the max or some sort of deviation.
00:53:03.980 --> 00:53:04.140
Exactly.
00:53:04.140 --> 00:53:04.900
Exactly.
00:53:05.720 --> 00:53:11.320
And we can find out whether people tip more on average in January or in July.
00:53:11.320 --> 00:53:18.120
I honestly don't remember what the answer is, which will hopefully not tick off too many
00:53:18.120 --> 00:53:19.220
readers.
00:53:19.220 --> 00:53:19.880
Here we go.
00:53:19.880 --> 00:53:20.340
Let's see.
00:53:20.340 --> 00:53:21.340
Oh, here we go.
00:53:21.340 --> 00:53:22.640
Go back one page there.
00:53:22.640 --> 00:53:25.800
So 32% of taxi riders in New York don't tip at all.
00:53:25.800 --> 00:53:26.360
That's right.
00:53:26.360 --> 00:53:28.240
That was a surprising thing to me.
00:53:28.240 --> 00:53:31.420
And then I don't remember exactly what it was in a different month.
00:53:31.420 --> 00:53:33.680
And you're talking summer versus winter.
00:53:33.840 --> 00:53:37.860
There's probably a tourism angle versus non locals versus tourists.
00:53:37.860 --> 00:53:41.780
I mean, I know people go to New York and not in the summer, but not as much, I imagine.
00:53:41.780 --> 00:53:42.360
That's right.
00:53:42.360 --> 00:53:42.940
That's right.
00:53:42.940 --> 00:53:47.080
There are all these different factors that, that like, you know, come into it.
00:53:47.080 --> 00:53:47.340
Yeah.
00:53:47.340 --> 00:53:50.500
One of the things kind of takeaways I'm feeling as we were talking about all these is like,
00:53:50.660 --> 00:53:54.820
there's a lot of interesting questions that can be asked and answered really quickly for
00:53:54.820 --> 00:54:00.960
like sociology and urban planning and all kinds of interesting questions that don't feel like
00:54:00.960 --> 00:54:01.820
programmer questions.
00:54:01.820 --> 00:54:02.400
Yes.
00:54:02.400 --> 00:54:06.660
So, so, you know, I mean, I talked to a lot of people, you probably do too, who are
00:54:06.660 --> 00:54:10.500
like interested in advancing their careers with Python.
00:54:10.500 --> 00:54:13.560
And they're like, well, you know, I don't have a computer science background.
00:54:13.560 --> 00:54:15.540
Can I got a job as a programmer?
00:54:15.540 --> 00:54:20.260
And their vision of a programmer is either someone working at a startup or at like one
00:54:20.260 --> 00:54:22.560
of the big companies, Google, Amazon, Facebook, and so forth.
00:54:22.560 --> 00:54:28.800
I say to them, look, there are an awful lot of people who have great jobs working at supermarket
00:54:28.800 --> 00:54:33.080
chains and insurance companies in the backroom analyzing data.
00:54:33.080 --> 00:54:38.060
And they are crucial, but we don't think of them as programmers and in governments and in
00:54:38.060 --> 00:54:39.980
cities and weather forecasting places.
00:54:39.980 --> 00:54:45.400
Like everyone nowadays is using data and collecting and analyzing it and having these skills either
00:54:45.400 --> 00:54:51.020
gives you abilities to do your job better or gives you the ability to move into a new job
00:54:51.020 --> 00:54:55.060
that you couldn't have done before that is that these places are desperately looking to fill.
00:54:55.060 --> 00:54:55.740
Yeah, I can't.
00:54:55.740 --> 00:54:56.780
Yeah, I totally agree.
00:54:56.780 --> 00:54:58.320
I can't remember which episode it was.
00:54:58.320 --> 00:54:59.680
I didn't name it right.
00:54:59.680 --> 00:55:04.940
So there's an episode I did quite a while ago about our programmers and our data scientists
00:55:04.940 --> 00:55:06.820
and Python data scientists working together.
00:55:06.820 --> 00:55:12.700
And I think it was the research arm of Kroger, maybe that had like 200 data scientists.
00:55:12.700 --> 00:55:16.520
Like that's a wow, that's a proper group of data scientists.
00:55:16.520 --> 00:55:20.100
I mean, a lot of times data scientists, I feel like there's a couple of them for a company
00:55:20.100 --> 00:55:21.820
compared to a software team or something.
00:55:21.820 --> 00:55:27.240
No, and I tell people also, you don't like it's worth learning to write and it's worth learning
00:55:27.240 --> 00:55:27.800
to speak.
00:55:27.800 --> 00:55:31.780
Not because you're going to be like, you know, a Pulitzer Prize winning writer, not because
00:55:31.780 --> 00:55:34.720
you're going to be like, you know, whatever prize you would get for like speaking well,
00:55:34.720 --> 00:55:38.200
just because having these skills makes you more effective at your job.
00:55:38.200 --> 00:55:42.640
And people who can write a program to suck up some data, analyze it and come back with
00:55:42.640 --> 00:55:47.280
a result, especially if it's like a public data set that has something to do with what
00:55:47.280 --> 00:55:47.960
they're working on.
00:55:47.960 --> 00:55:52.000
They are so much more valuable to their company than they would have been otherwise.
00:55:52.000 --> 00:55:53.240
Oh yeah, absolutely.
00:55:53.240 -->
00:55:58.340
So I found it because I have a search engine on Talk Python that searches the transcripts
00:55:58.340 --> 00:55:59.560
and everything else.
00:55:59.560 --> 00:56:03.400
If you want to know something people about historical shows, when you hit search, it's not just like
00:56:03.400 --> 00:56:03.880
the show notes.
00:56:03.880 --> 00:56:12.740
So scaling data science across Python and R episode 236 with Ethan Swan, Bradley, Bokme,
00:56:12.740 --> 00:56:16.000
and the company is 84.51 degrees.
00:56:16.000 --> 00:56:17.200
So anyway, that's...
00:56:17.200 --> 00:56:18.700
Oh, I've seen that name before.
00:56:18.700 --> 00:56:19.400
Yeah, yeah, yeah.
00:56:19.400 --> 00:56:19.880
That's what it was.
00:56:19.880 --> 00:56:20.860
So people can check that out.
00:56:20.860 --> 00:56:22.180
They're interested, but...
00:56:22.180 --> 00:56:22.860
Very, very cool.
00:56:22.860 --> 00:56:23.220
Yeah.
00:56:23.220 --> 00:56:23.780
Super cool.
00:56:23.780 --> 00:56:24.460
Okay.
00:56:24.460 --> 00:56:25.220
Last one.
00:56:25.220 --> 00:56:25.880
What are we doing?
00:56:25.880 --> 00:56:27.720
Let's do cities.
00:56:27.720 --> 00:56:28.500
Cities.
00:56:28.500 --> 00:56:29.520
What number is it?
00:56:29.520 --> 00:56:30.560
43.
00:56:30.820 --> 00:56:30.940
Okay.
00:56:30.940 --> 00:56:31.400
Oh, that's right.
00:56:31.400 --> 00:56:39.720
So I found a few years ago this JSON file containing the thousand largest cities in the United States.
00:56:39.720 --> 00:56:44.360
So first of all, like good to work with JSON because people need to know how to work with
00:56:44.360 --> 00:56:44.920
that kind of data.
00:56:44.920 --> 00:56:45.180
Yeah.
00:56:45.180 --> 00:56:51.060
Second of all, okay, let's not look at the numbers so much as let's do some plotting.
00:56:51.060 --> 00:56:54.200
And so I'm going to like make enemies now.
00:56:54.560 --> 00:56:56.780
I really cannot handle Matplotlib.
00:56:56.780 --> 00:57:00.400
I find it, it's like so incredibly powerful.
00:57:00.400 --> 00:57:04.120
And anytime I want to do even the tiniest thing with it, I have to look up the documentation
00:57:04.120 --> 00:57:05.800
and remind myself how it works.
00:57:05.800 --> 00:57:07.640
Maybe it's because I don't use it enough.
00:57:07.640 --> 00:57:12.400
And so I just use the pandas interface for plotting 95% of the time.
00:57:12.400 --> 00:57:13.520
Let's call it 90% of the time.
00:57:13.520 --> 00:57:14.780
The rest of the time I usually use Seaborn.
00:57:15.540 --> 00:57:20.000
So, so I said, okay, let's see if we can do some plotting here of these largest cities.
00:57:20.000 --> 00:57:25.640
So for example, let's see growth in Pennsylvania cities, like which cities in Pennsylvania?
00:57:25.640 --> 00:57:26.500
Oh, I'm sorry.
00:57:26.500 --> 00:57:27.940
Like, let's see a dual bar plot.
00:57:27.940 --> 00:57:29.820
How many large cities are in each state?
00:57:29.820 --> 00:57:30.280
Okay.
00:57:30.280 --> 00:57:31.660
Well, that's
a group by, right?
00:57:31.660 --> 00:57:32.920
So you have to know how to do grouping.
00:57:32.920 --> 00:57:36.500
You have to know what you're grouping on and what, how to use count as opposed to mean,
00:57:36.500 --> 00:57:36.760
right?
00:57:36.760 --> 00:57:37.720
That count even exists.
00:57:37.720 --> 00:57:42.060
But then if I want to see a bar plot where it's sorted from smallest to largest, you got
00:57:42.060 --> 00:57:42.720
to know how to do sorting.
00:57:42.720 --> 00:57:43.940
And then we can do a bar plot.
00:57:44.080 --> 00:57:47.900
So sure enough, you do a dot plot dot bar on your series.
00:57:47.900 --> 00:57:48.540
Kablam.
00:57:48.540 --> 00:57:50.200
You have a bar plot right there.
00:57:50.200 --> 00:57:50.780
Yeah, that's awesome.
00:57:50.780 --> 00:57:52.960
And we just get it and like, you know, it's nicely sorted there.
00:57:52.960 --> 00:57:54.100
Quick takeaways.
00:57:54.100 --> 00:57:55.080
Holy cow.
00:57:55.080 --> 00:57:57.400
Does California have more big cities than I realized?
00:57:57.400 --> 00:58:00.060
And, you know, where's New York and New Jersey?
00:58:00.060 --> 00:58:01.880
Like it's way down the line.
00:58:01.880 --> 00:58:04.880
You think of those as having like pretty megatropolis type places.
00:58:04.880 --> 00:58:06.300
Massachusetts.
00:58:06.300 --> 00:58:06.940
That's right.
00:58:06.940 --> 00:58:07.600
That's right.
00:58:07.600 --> 00:58:09.340
But it's how many cities.
00:58:09.340 --> 00:58:09.720
Yeah.
00:58:09.720 --> 00:58:10.100
Right.
00:58:10.100 --> 00:58:11.400
So, so, so, so right.
00:58:11.400 --> 00:58:11.760
Right.
00:58:11.760 --> 00:58:13.640
That's like the, so keep going.
00:58:13.640 --> 00:58:16.520
So like do a bar plot of a growth in Pennsylvania.
00:58:16.520 --> 00:58:17.700
So how are we going to do that?
00:58:17.700 --> 00:58:18.740
Well, wait a second.
00:58:18.740 --> 00:58:23.920
If we want to do growth, it's a percentage, but it's written in the JSON as number and then
00:58:23.920 --> 00:58:24.580
percent sign.
00:58:24.580 --> 00:58:25.380
That's a string.
00:58:25.380 --> 00:58:27.740
So we're going to have to get rid of the percent sign.
00:58:27.860 --> 00:58:31.760
And then we're going to have to change the D type from string to a floating point.
00:58:31.760 --> 00:58:34.760
And then we can sort it and then we can graph it.
00:58:34.760 --> 00:58:41.120
And so the ads type changing the D type basically is a deep down inside of pandas pars as an
00:58:41.120 --> 00:58:42.580
integer operation, right?
00:58:42.580 --> 00:58:45.140
Rather than looping over and parsing or whatever.
00:58:45.500 --> 00:58:45.780
Yeah.
00:58:45.780 --> 00:58:46.140
Yeah.
00:58:46.140 --> 00:58:46.180
Yeah.
00:58:46.180 --> 00:58:46.340
Yeah.
00:58:46.340 --> 00:58:48.720
Like basically, I mean, it has to do the looping.
00:58:48.720 --> 00:58:51.540
It's doing the looping there at, you know, below sea level.
00:58:51.540 --> 00:58:51.780
Yeah.
00:58:51.780 -->
00:58:52.660
Below sea level.
00:58:52.660 --> 00:58:53.100
Yeah.
00:58:53.100 --> 00:58:53.900
Down by the dead sea.
00:58:53.900 --> 00:58:54.200
I see.
00:58:54.200 --> 00:58:54.500
Okay.
00:58:54.500 --> 00:58:55.060
Got it.
00:58:57.460 --> 00:58:59.440
And so you see here like this bar plot.
00:58:59.440 --> 00:59:02.900
Now we have a whole bunch of cities that have gone down in size and a bunch of things gone
00:59:02.900 --> 00:59:03.460
up in size.
00:59:03.460 --> 00:59:07.360
And pandas is very happy to, you know, show, show the bar plot there.
00:59:07.360 --> 00:59:13.380
You know, again, sorted so that we see it from the greatest sort of shrinkage to the greatest
00:59:13.380 --> 00:59:13.780
growth.
00:59:13.780 --> 00:59:14.040
Right.
00:59:14.040 --> 00:59:18.080
With Pittsburgh having the most shrinkage and Allentown having the most growth.
00:59:18.080 --> 00:59:19.080
I would not guess that.
00:59:19.080 --> 00:59:22.680
And I would not have either given that like my sister-in-law and brother-in-law live in
00:59:22.680 --> 00:59:24.520
Allentown and they're like, Oh, Allentown.
00:59:24.860 --> 00:59:29.360
Actually, but I think that's a common thing that people think there, even though it seems
00:59:29.360 --> 00:59:30.840
like a perfectly great place to be.
00:59:30.840 --> 00:59:31.500
Yeah, sure.
00:59:31.500 --> 00:59:32.980
Funny.
00:59:32.980 --> 00:59:33.560
Awesome.
00:59:33.560 --> 00:59:35.160
Well, there's a lot of cool takeaways.
00:59:35.160 --> 00:59:36.200
One more.
00:59:36.200 --> 00:59:37.060
Let me show you one.
00:59:37.060 --> 00:59:39.560
Go to the last one here in this exercise.
00:59:39.560 --> 00:59:39.920
Okay.
00:59:39.920 --> 00:59:40.560
You got it.
00:59:40.560 --> 00:59:47.380
So one of the most important types of plots you can do in data analysis is a scatter plot.
00:59:47.380 --> 00:59:52.540
And you take your data frame and you say, give me the plot with based on your, you know,
00:59:52.580 --> 00:59:55.100
the X axis should be this, the Y axis should be that.
00:59:55.100 --> 00:59:59.900
Well, if we do a scatter plot based on longitude and latitude of the thousand largest U.S.
00:59:59.900 --> 01:00:02.660
cities, you basically come up with a map of the U.S.
01:00:02.660 --> 01:00:03.140
Yeah.
01:00:03.140 --> 01:00:03.780
How interesting.
01:00:03.780 --> 01:00:08.180
The other thing that you realize is California is technically bigger than a lot of other states.
01:00:08.180 --> 01:00:09.620
So that also counts for Y.
01:00:09.620 --> 01:00:10.240
Oh, yes.
01:00:10.240 --> 01:00:10.560
Yeah.
01:00:10.560 --> 01:00:11.140
Interesting.
01:00:11.140 --> 01:00:12.340
But there's a lot of people in California.
01:00:12.340 --> 01:00:13.020
Good place to be.
01:00:13.020 --> 01:00:14.880
There are a lot of people there for sure.
01:00:14.880 --> 01:00:17.560
If they're willing to pay the sunshine tax, it's nice there.
01:00:17.560 --> 01:00:19.820
Cool.
01:00:19.820 --> 01:00:20.520
All right, Reuben.
01:00:20.520 --> 01:00:22.080
This is really, really excellent.
01:00:22.080 --> 01:00:26.560
I know you want to give a shout out to a couple of exercises from Bamboo Weekly.
01:00:26.560 --> 01:00:29.160
So I'll include those in the show notes as well.
01:00:29.160 --> 01:00:32.800
And people can like take those and jump over and look at them as well.
01:00:32.800 --> 01:00:33.400
That's fine.
01:00:33.400 --> 01:00:33.880
That's great.
01:00:33.880 --> 01:00:34.220
Yeah.
01:00:34.220 --> 01:00:36.360
No, we went through a lot here just during this time.
01:00:36.360 --> 01:00:37.040
Yeah, for sure.
01:00:37.220 --> 01:00:46.080
And as I've discovered, there's basically, I mean, there is an infinite number of these sorts of things that you can do and different permutations of various sorts.
01:00:46.080 --> 01:00:51.600
I mean, we didn't even talk about like, you know, multi-indexing, which opens up literally a whole new dimension of this stuff.
01:00:51.600 --> 01:00:51.940
Yeah.
01:00:51.940 --> 01:00:52.640
But it's great fun.
01:00:52.640 --> 01:00:53.220
It's great, great fun.
01:00:53.220 --> 01:00:58.440
And a lot of the Pandas 2 stuff coming on changes the foundation and opens up more possibilities still.
01:00:58.440 --> 01:00:58.960
Absolutely.
01:00:58.960 --> 01:00:59.460
All right.
01:00:59.460 --> 01:01:00.840
Well, final thoughts.
01:01:00.840 --> 01:01:02.000
Final call to action.
01:01:02.000 --> 01:01:03.320
What do we got here?
01:01:03.320 --> 01:01:03.740
What do you say?
01:01:03.740 --> 01:01:10.380
Well, call to action, I mean, you can take a look at like, I guess my overall courses and so forth at LearnerPython.com.
01:01:10.380 --> 01:01:15.560
Or if you just want to improve your Pandas knowledge, you can look at BambooWeekly.com, the newsletter there.
01:01:15.560 --> 01:01:17.960
Does it cost money or does it just cost an email?
01:01:17.960 --> 01:01:23.180
So it is a paid newsletter, but the first two questions and answers every week are free.
01:01:23.180 --> 01:01:28.640
So like it's usually between five and nine questions each week.
01:01:28.640 --> 01:01:36.260
But even if you don't pay, I still want people to get some Pandas learning improvement practice so they can totally do that.
01:01:36.260 --> 01:01:36.680
Excellent.
01:01:36.680 --> 01:01:37.280
All right.
01:01:37.280 --> 01:01:37.460
Yeah.
01:01:37.460 --> 01:01:38.720
And check out the book, obviously.
01:01:38.720 --> 01:01:39.420
Oh, yeah.
01:01:39.420 --> 01:01:39.840
That too.
01:01:39.840 --> 01:01:41.560
PandasWorkout.com.
01:01:41.560 --> 01:01:41.960
Cool.
01:01:41.960 --> 01:01:42.660
I'll pull this up.
01:01:42.660 --> 01:01:42.940
Yeah.
01:01:42.940 --> 01:01:43.860
Awesome.
01:01:43.860 --> 01:01:44.820
It's actually pretty thick.
01:01:44.820 --> 01:01:45.620
Yes, it is.
01:01:45.620 --> 01:01:50.740
So at the end, I was like, oh, that's why it took a long time.
01:01:51.540 --> 01:01:53.320
Also procrastination, but you
know.
01:01:53.320 --> 01:01:53.560
Yeah.
01:01:53.560 --> 01:01:56.180
I'd love to hear from people if they have.
01:01:56.180 --> 01:02:02.060
I always love to hear like, you know, sort of interesting problems, data sets, issues that people are experiencing,
01:02:02.060 --> 01:02:08.540
just so I can sort of figure out what are the next directions for me to explore that I can then try to help people with as well.
01:02:08.540 --> 01:02:09.020
Awesome.
01:02:09.020 --> 01:02:15.540
Well, I appreciate all your help coming on the show, sharing your knowledge, and just riffing on Pandas together.
01:02:15.540 --> 01:02:16.120
It was a lot of fun.
01:02:16.120 --> 01:02:16.660
Excellent.
01:02:16.660 --> 01:02:17.400
My great pleasure.
01:02:17.400 --> 01:02:17.840
Yeah.
01:02:17.840 --> 01:02:18.160
Bye.
01:02:18.160 --> 01:02:18.840
All right.
01:02:18.840 --> 01:02:19.180
Bye-bye.
01:02:20.460 --> 01:02:23.160
This has been another episode of Talk Python To Me.
01:02:23.160 --> 01:02:24.980
Thank you to our sponsors.
01:02:24.980 --> 01:02:26.580
Be sure to check out what they're offering.
01:02:26.580 --> 01:02:28.000
It really helps support the show.
01:02:28.000 --> 01:02:30.440
Take some stress out of your life.
01:02:30.440 --> 01:02:36.220
Get notified immediately about errors and performance issues in your web or mobile applications with Sentry.
01:02:36.220 --> 01:02:41.220
Just visit talkpython.fm/sentry and get started for free.
01:02:41.220 --> 01:02:44.800
And be sure to use the promo code TALKPYTHON, all one word.
01:02:44.800 --> 01:02:47.160
This episode is brought to you by Scalable Path.
01:02:47.160 --> 01:02:53.500
If you're a founder or engineering leader, you know how hard it is to find top-tier developers while keeping costs low.
01:02:53.500 --> 01:02:59.180
Scalable Path is a software staffing company that helps you build remote dev teams that just fit.
01:02:59.180 --> 01:03:03.380
Build your team at talkpython.fm/scalablepath.
01:03:03.720 --> 01:03:04.980
Want to level up your Python?
01:03:04.980 --> 01:03:09.040
We have one of the largest catalogs of Python video courses over at Talk Python.
01:03:09.040 --> 01:03:14.220
Our content ranges from true beginners to deeply advanced topics like memory and async.
01:03:14.220 --> 01:03:16.880
And best of all, there's not a subscription in sight.
01:03:16.880 --> 01:03:19.780
Check it out for yourself at training.talkpython.fm.
01:03:20.180 --> 01:03:21.900
Be sure to subscribe to the show.
01:03:21.900 --> 01:03:24.680
Open your favorite podcast app and search for Python.
01:03:24.680 --> 01:03:26.000
We should be right at the top.
01:03:26.000 --> 01:03:35.340
You can also find the iTunes feed at /itunes, the Google Play feed at /play, and the direct RSS feed at /rss on talkpython.fm.
01:03:35.340 --> 01:03:38.320
We're live streaming most of our recordings these days.
01:03:38.320 --> 01:03:46.100
If you want to be part of the show and have your comments featured on the air, be sure to subscribe to our YouTube channel at talkpython.fm/youtube.
01:03:46.100 --> 01:03:48.200
This is your host, Michael Kennedy.
01:03:48.200 --> 01:03:49.500
Thanks so much for listening.
01:03:49.500 --> 01:03:50.660
I really appreciate it.
01:03:50.660 --> 01:03:52.560
Now get out there and write some Python code.
01:03:52.560 --> 01:04:13.680
I'll see you next time.