TRANSKRIP 🔗
WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 7/6/2024 4:38:30 PM
Duration: 4801.998
Channels: 1
1
00:00:28.435 --> 00:00:29.894
Hello, and welcome to podcast.init,
2
00:00:30.435 --> 00:00:39.460
the podcast about Python and the people who make it great. You can subscribe to our show on Itunes, Stitcher, TuneIn Radio, or add our RSS feed to your pod catcher of choice.
3
00:00:40.000 --> 00:00:53.500
You can also follow us on Twitter or Google Plus, and please give us feedback. You can leave a review on Itunes to help other people find the show, send us a tweet or an email, leave us a message on Google Plus or on our show notes, and you can also join our new community. Visitdiscourse.pythonpodcast.com
4
00:00:55.559 --> 00:01:00.219
for your opportunity to find out about upcoming guests, suggest questions, and propose show ideas.
5
00:01:01.204 --> 00:01:05.865
I would like to thank everyone who has donated to the show. Your contributions help us make the show sustainable.
6
00:01:06.244 --> 00:01:09.625
For details on how to support the show, you can visit our site at python podcastdot
7
00:01:10.820 --> 00:01:14.200
com. Linode is sponsoring us this week. Check them out at linode.com/podcastinit
8
00:01:15.540 --> 00:01:20.245
and get a $20 credit to try out their fast and reliable Linux virtual servers for your next project.
9
00:01:21.025 --> 00:01:28.490
I would also like to thank Hired, a job marketplace for developers and designers, for sponsoring this episode of podcast dot in it. Use the link hired.com/podcastin
10
00:01:29.590 --> 00:01:31.130
it to double your signing bonus.
11
00:01:31.590 --> 00:01:36.565
Your host today is Tobias Macy, and today I'm interviewing Matthew Rocklin and Alexander Shepanovsky
12
00:01:36.865 --> 00:01:39.285
about their work on functional libraries for Python.
13
00:01:40.225 --> 00:01:43.445
So could you guys please introduce yourselves? How about you go first, Alexander?
14
00:01:44.880 --> 00:01:47.700
Hello, everyone. My name is Alexander Shpanovski,
15
00:01:48.960 --> 00:01:50.659
and I'm a Python programmer.
16
00:01:51.495 --> 00:01:54.075
And open source author and talks
17
00:01:55.415 --> 00:01:55.915
giver.
18
00:01:57.335 --> 00:01:57.835
K.
19
00:01:58.615 --> 00:02:00.315
Alright. And how about you, Matthew?
20
00:02:00.700 --> 00:02:12.245
Hi, everyone. First, thanks for having me on the show. My name is Matthew Rocklin. I'm a full time open source developer at Continuum Analytics. I usually develop into the numeric Python stack. So So libraries that run on top of NumPy and Pandas.
21
00:02:12.785 --> 00:02:15.205
Previously, I did research in numerical methods and usability.
22
00:02:16.224 --> 00:02:19.685
And how did you each get introduced to Python? How about you go again, Alexander?
23
00:02:20.530 --> 00:02:22.470
Yeah. We can see a difference
24
00:02:22.930 --> 00:02:23.430
between,
25
00:02:24.770 --> 00:02:25.830
me and Matthew.
26
00:02:26.130 --> 00:02:27.670
Load full transcript
So I came from
27
00:02:28.275 --> 00:02:29.095
web development
28
00:02:30.355 --> 00:02:31.315
and, Matthew from,
29
00:02:31.795 --> 00:02:32.935
numeric programming.
30
00:02:34.195 --> 00:02:35.815
And I think this will,
31
00:02:36.435 --> 00:02:36.935
influence
32
00:02:37.760 --> 00:02:38.980
the rest of our,
33
00:02:39.760 --> 00:02:40.260
podcast.
34
00:02:41.040 --> 00:02:41.540
So
35
00:02:42.160 --> 00:02:42.819
it was,
36
00:02:43.280 --> 00:02:43.940
I believe,
37
00:02:44.400 --> 00:02:44.900
2,007
38
00:02:47.355 --> 00:02:47.855
And
39
00:02:49.035 --> 00:02:50.395
we we were using,
40
00:02:50.875 --> 00:02:52.175
Pearl back when,
41
00:02:52.555 --> 00:02:54.575
and we were using our own
42
00:02:55.240 --> 00:02:55.900
kind of
43
00:02:56.840 --> 00:02:57.340
framework.
44
00:02:59.000 --> 00:02:59.400
And,
45
00:03:00.599 --> 00:03:01.739
I decided with,
46
00:03:02.280 --> 00:03:04.060
we need to move on and use
47
00:03:06.725 --> 00:03:07.465
some something
48
00:03:08.645 --> 00:03:11.465
some frame framework, some open source framework.
49
00:03:11.925 --> 00:03:13.305
And I looked around,
50
00:03:13.845 --> 00:03:14.345
and
51
00:03:14.950 --> 00:03:15.930
there were only
52
00:03:16.310 --> 00:03:17.450
old things in
53
00:03:17.990 --> 00:03:19.530
Pearl. So I looked
54
00:03:20.390 --> 00:03:21.530
more around and
55
00:03:22.005 --> 00:03:25.945
it was a choice at the at the time between Django and,
56
00:03:27.765 --> 00:03:28.745
Ruby on Rails.
57
00:03:30.410 --> 00:03:34.670
Not really a choice even between languages but between frameworks.
58
00:03:35.690 --> 00:03:36.190
And
59
00:03:37.345 --> 00:03:37.905
even though,
60
00:03:39.185 --> 00:03:40.405
Ruby seemed
61
00:03:41.265 --> 00:03:42.165
more interesting,
62
00:03:43.665 --> 00:03:44.165
rails
63
00:03:44.810 --> 00:03:46.430
seems too dirty. Like,
64
00:03:47.130 --> 00:03:49.230
the last thing was that
65
00:03:49.610 --> 00:03:51.230
we are passing all the local
66
00:03:52.025 --> 00:03:56.685
variables from this, what we call, controller to a t templates.
67
00:03:57.065 --> 00:03:57.885
And so,
68
00:03:58.265 --> 00:03:59.245
I went with
69
00:03:59.670 --> 00:04:02.250
Django and was introduced to Python.
70
00:04:02.950 --> 00:04:05.290
And how about you, Matthew? How did you find Python?
71
00:04:06.310 --> 00:04:10.715
Yeah. So I actually picked it up in graduate school. Previously, I just joined coursework.
72
00:04:11.255 --> 00:04:17.195
Previously, I used C Sharp for software engineering and MATLAB for more data analysis and science work.
73
00:04:17.650 --> 00:04:19.090
Particularly, I used MATLAB because
74
00:04:19.410 --> 00:04:23.030
sort of had all the data science things for me, and also because my science collaborators,
75
00:04:23.410 --> 00:04:25.225
you know, physicists, not computer scientists,
76
00:04:25.625 --> 00:04:26.685
could understand it.
77
00:04:27.145 --> 00:04:30.845
I sort of picked up Python just doing coursework as a fun side project.
78
00:04:31.145 --> 00:04:34.685
And I was sort of very happily surprised when that it could do both software engineering
79
00:04:35.380 --> 00:04:47.965
and have the API that a data science user could use. So I could build tools and share them with my science collaborators. And that sort of combination really, really stuck with me. So that was sort of my why I jumped into it, and I've just haven't left since.
80
00:04:48.345 --> 00:04:49.005
To reiterate,
81
00:04:49.545 --> 00:04:57.250
Alexander's point about 2 different communities, I think it's a very valid valid point that people come into Python from a lot of different a lot of different perspectives.
82
00:04:57.710 --> 00:05:22.625
And we have, you know, we make different software. Sometimes it doesn't work well together. And very recently, it's been very exciting, actually, to see the web community and the data science community start to come together. So, you know, most scientists that I know now use Flask as sort of this, like, accessible thing. And a lot of the web developers I know are now using some of the more visual, tools for visualization inside of this data science world. So, again, it's very cool to see these 2 communities coming together.
83
00:05:23.645 --> 00:05:27.004
Although in some cases, like with the case of Alexander and I, we've,
84
00:05:27.405 --> 00:05:28.865
we've duplicated some efforts.
85
00:05:30.360 --> 00:05:31.419
This is not necessarily
86
00:05:32.199 --> 00:05:35.020
a bad thing. Absolutely not. No. It's great.
87
00:05:35.639 --> 00:05:50.110
And can you guys first explain what functional programming is and how it differs from the procedural or object oriented programming that most Pythonistas are familiar with? And recognizing that functional programming is kind of an overloaded term with lot of subcomponents. So feel free to delve into that as well.
88
00:05:50.810 --> 00:05:53.710
I don't think you should oppose functional programming and,
89
00:05:54.170 --> 00:05:54.670
procedural
90
00:05:55.050 --> 00:05:59.745
or object oriented 1. You should oppose functional programming and imperative.
91
00:06:00.765 --> 00:06:09.919
So in imperative programming, you, and the difference is in how you think about your program and how you decompose it. In imperative programming, you,
92
00:06:10.540 --> 00:06:11.040
think
93
00:06:11.419 --> 00:06:14.725
of a program as a algorithm which is a series of
94
00:06:15.105 --> 00:06:18.805
steps. So you decompose it in this into steps
95
00:06:19.185 --> 00:06:19.585
which,
96
00:06:19.985 --> 00:06:22.165
change the state of a world.
97
00:06:22.500 --> 00:06:29.800
And in in functional programming, you think of a program as a function. So it's you have input and output and some
98
00:06:30.865 --> 00:06:31.365
transformation
99
00:06:31.745 --> 00:06:33.205
from 1 to to seconds.
100
00:06:33.585 --> 00:06:36.965
And, you decompose it into a simpler function by
101
00:06:37.360 --> 00:06:46.100
using function composition and things like that. So in imperative programming, if you get some steps and you unite them, you get
102
00:06:46.785 --> 00:06:48.005
procedural programming.
103
00:06:48.625 --> 00:06:54.720
And even if you add objects, you get a object oriented programming. But these are, like, add ons
104
00:06:55.600 --> 00:06:56.980
on imperative programming.
105
00:06:57.440 --> 00:06:57.760
And,
106
00:06:58.400 --> 00:06:59.460
in functional programming,
107
00:07:00.160 --> 00:07:04.915
it's usually associated with things like pure functions, small nodes, and
108
00:07:05.295 --> 00:07:07.875
higher order functions and things like that, but
109
00:07:08.175 --> 00:07:09.235
these are only,
110
00:07:10.940 --> 00:07:13.680
the consequences of, a way of
111
00:07:14.060 --> 00:07:17.440
thinking about the program and a way of decomposing it.
112
00:07:17.819 --> 00:07:18.139
Because,
113
00:07:18.620 --> 00:07:20.240
when you start to decompose
114
00:07:20.765 --> 00:07:24.285
your big function to smaller functions, you start to see that,
115
00:07:25.005 --> 00:07:25.745
not functions
116
00:07:26.605 --> 00:07:29.825
are equally easy to decompose to.
117
00:07:30.160 --> 00:07:33.380
So it's easier to decompose to viewer functions
118
00:07:34.800 --> 00:07:37.140
and things like that. And you want
119
00:07:37.520 --> 00:07:38.020
to
120
00:07:38.560 --> 00:07:39.060
make,
121
00:07:39.925 --> 00:07:42.745
to give special special attention to side effects
122
00:07:43.045 --> 00:07:43.705
and probably
123
00:07:44.245 --> 00:07:44.745
separate
124
00:07:45.125 --> 00:07:47.865
side effects from the rest of the population.
125
00:07:48.790 --> 00:07:54.650
And, so all these things that are usually associated with functional programming, these are just consequences
126
00:07:55.270 --> 00:07:57.525
of the idea of functional programming.
127
00:07:57.985 --> 00:07:59.205
So this is my understanding.
128
00:07:59.985 --> 00:08:03.125
Yeah. So I have a I think that's a great definition. I think that
129
00:08:03.425 --> 00:08:11.909
many developers will have different understanding of this of this term. I think a Haskell developer and a closure developer will will think about what functional programming is a little bit differently.
130
00:08:12.655 --> 00:08:23.050
But I think the definition provided by Alex is Alexander is great. How I would define it actually is more of a historical definition. I would say that functional programming is a set of language features
131
00:08:23.670 --> 00:08:26.570
that come from languages that descended from 2 root languages.
132
00:08:27.355 --> 00:08:31.295
Lisp that originated around the same time as Fortran and Algol, and ML,
133
00:08:31.755 --> 00:08:32.975
sort of an academic
134
00:08:33.515 --> 00:08:36.175
language to prove theorems about type systems
135
00:08:36.610 --> 00:08:37.910
that originated in the 19 seventies.
136
00:08:38.610 --> 00:08:55.480
Lisp begat languages like common Lisp and Clojure, and ML begat languages like Haskell. And I would say that these languages have a lot of language features, as all languages do. And the language features that appeared only in these languages, but not in the sort of C and Fortran like languages that we're more familiar with, All of those features
137
00:08:55.940 --> 00:08:57.400
sort of tend to be called
138
00:08:58.020 --> 00:09:02.040
functional. So this might be function composition, functions as data, higher order functions,
139
00:09:02.545 --> 00:09:05.205
but also things like macros or algebraic type systems,
140
00:09:05.905 --> 00:09:09.365
sort of more fancy things, or more pedestrian things, like streaming iterators
141
00:09:09.985 --> 00:09:11.845
and a strong reliance on core data structures.
142
00:09:12.390 --> 00:09:15.770
Today, I would say that people often use the term functional programming
143
00:09:16.150 --> 00:09:17.690
just as a reaction to
144
00:09:18.150 --> 00:09:20.490
highly imperative or object oriented programming.
145
00:09:21.445 --> 00:09:30.370
There's sort of this push towards simplicity. So when I read blog posts, I sort of I get a variety of there's a variety of terms that also can be lumped into this term of functional
146
00:09:30.910 --> 00:09:36.690
programming. And if it's a Haskell programmer, it might be talking about algebraic type systems. If it's a closure programmer, it might be talking about macros.
147
00:09:37.815 --> 00:09:40.954
And so there's a there's a large grab bag of of concepts here.
148
00:09:41.735 --> 00:09:44.394
And how did you each get started with functional programming?
149
00:09:45.654 --> 00:09:46.555
For me, it's
150
00:09:47.930 --> 00:09:51.550
not like you you start with you don't start with functional programming
151
00:09:51.930 --> 00:09:56.350
at first because you don't know what functional programming is. But, you start thinking
152
00:09:56.895 --> 00:09:59.395
in a functional way sometimes.
153
00:10:00.095 --> 00:10:01.635
And and, you probably
154
00:10:02.175 --> 00:10:04.675
got introduced to some concept like function
155
00:10:05.690 --> 00:10:07.710
or higher order functions. And
156
00:10:08.330 --> 00:10:09.470
you can see
157
00:10:10.650 --> 00:10:14.750
a practical advantage of, for example, higher order functions because you,
158
00:10:15.765 --> 00:10:24.380
can write functions which are more custom customizable this way. And it could be easier when writing objects. You have less parts or less code.
159
00:10:24.920 --> 00:10:28.060
Like, like this, you get introduced to these concepts or and
160
00:10:28.440 --> 00:10:29.980
not only concepts, but
161
00:10:31.165 --> 00:10:31.825
the specifics,
162
00:10:33.005 --> 00:10:33.745
the application.
163
00:10:34.205 --> 00:10:37.505
When I I also got introduced to Common Lisp,
164
00:10:38.600 --> 00:10:40.860
I never actually used it for anything,
165
00:10:41.640 --> 00:10:43.160
practical, but it was,
166
00:10:43.640 --> 00:10:46.515
just played with it and extracted some
167
00:10:46.975 --> 00:10:54.640
ideas from it. So this is my story with functional programming. I I later took a a look into ML and Haskell,
168
00:10:55.040 --> 00:10:56.560
and I don't think,
169
00:10:57.280 --> 00:10:58.340
that much practical,
170
00:10:58.800 --> 00:11:00.360
especially Haskell. It's,
171
00:11:00.800 --> 00:11:02.740
has too much hassle in it.
172
00:11:03.095 --> 00:11:11.930
I think there are a lot of Haskell developers that would disagree with that, but I I I also agree with you. Yep. Even if you have less code, it not always,
173
00:11:12.310 --> 00:11:13.129
means you
174
00:11:13.509 --> 00:11:13.990
spend,
175
00:11:14.550 --> 00:11:15.990
less time writing it and,
176
00:11:16.709 --> 00:11:28.465
anybody else will spend less time reading it and understanding it or modifying this code. It's, there are huge amount of hassle with many things in Haskell, like IO.
177
00:11:29.140 --> 00:11:29.640
Okay.
178
00:11:30.820 --> 00:11:33.080
Matthew, how'd you get involved in functional programming?
179
00:11:34.100 --> 00:11:35.240
Yeah. So I think,
180
00:11:35.620 --> 00:11:37.320
like Alexander, I I probably
181
00:11:37.745 --> 00:11:53.040
started learning functional concepts before I really identified them as being functional concepts. I think that was a really great point. When I sort of intentionally started learning functional programming, it was actually a social activity. A couple of my friends, Avi Newman, John Jacobson, and I wanted to get together and drink some beers. We wanted some activity to do. And so we decided to,
182
00:11:53.579 --> 00:11:57.365
implement the same problem. It was, like, a job interview problem in a variety of languages.
183
00:11:57.824 --> 00:12:03.125
So we did it in C and in Fortran and in Clojure and in Haskell and in languages like mod.
184
00:12:03.824 --> 00:12:05.845
And it was actually really interesting to see
185
00:12:06.690 --> 00:12:11.190
what what about our solutions was intrinsic to the problem that would stay the same across all these languages.
186
00:12:11.570 --> 00:12:14.605
And what about our solutions was sort of a side effect of the language?
187
00:12:15.084 --> 00:12:21.985
And it was just a really fascinating experience. I highly recommend grabbing a couple of programming friends and doing the same thing, especially if they're in different disciplines.
188
00:12:22.365 --> 00:12:25.370
Neither Avi, John, or I had a common background.
189
00:12:26.070 --> 00:12:40.815
John Jacobson, in particular, was quite keen on closure, and he sort of brought us in that direction. And as a result, I sort of screwed on the closure for the following year, and that had a really strong impact on my programming style even today. I don't think I've ever used functional programming in I've never used a
190
00:12:41.115 --> 00:12:42.495
strictly functional language,
191
00:12:43.100 --> 00:12:43.760
in practice,
192
00:12:44.380 --> 00:12:52.640
but I took a lot of this a lot of the ideas that I learned from delving into functional programming into my everyday Python code. And I think there's really a lot of value to functional programming
193
00:12:53.165 --> 00:12:57.585
even if you're not doing it explicitly. Or if you're not in Haskell or enclosure,
194
00:12:58.045 --> 00:13:00.545
you can still use a lot of the concepts
195
00:13:01.139 --> 00:13:02.920
in everyday everyday style.
196
00:13:03.459 --> 00:13:10.120
And we've touched on it a bit, but what are some of the benefits of functional programming, and when might somebody want to use functional paradigms in their projects?
197
00:13:10.885 --> 00:13:12.825
I may first add a small
198
00:13:13.205 --> 00:13:13.845
remark on,
199
00:13:14.405 --> 00:13:14.905
writing,
200
00:13:15.525 --> 00:13:19.180
writing the same thing in different languages. There is a resource,
201
00:13:19.639 --> 00:13:23.980
called, called Rosetta Code. It's a wiki with a
202
00:13:24.519 --> 00:13:27.660
solution in different languages for different problems
203
00:13:28.665 --> 00:13:36.105
and could be quite interesting to look sometimes. And about the benefits on functional programming. For me, it's,
204
00:13:36.930 --> 00:13:38.529
like, you you can,
205
00:13:39.250 --> 00:13:40.470
use this customization
206
00:13:41.490 --> 00:13:44.550
with, which high order functions give you
207
00:13:44.954 --> 00:13:45.454
without,
208
00:13:46.875 --> 00:13:49.214
the immediate layer of, like, classes.
209
00:13:49.995 --> 00:13:56.980
Classes and factories and things like that. You can achieve in object oriented paradigm, but it's usually
210
00:13:57.440 --> 00:13:58.180
more hassle.
211
00:13:58.960 --> 00:14:01.220
When also, if you think
212
00:14:01.865 --> 00:14:02.365
not
213
00:14:02.825 --> 00:14:03.705
about concrete,
214
00:14:04.185 --> 00:14:06.125
specific features, but about,
215
00:14:06.665 --> 00:14:07.165
thinking,
216
00:14:07.545 --> 00:14:11.670
it could also be beneficial because the alternative is usually
217
00:14:12.130 --> 00:14:15.190
imperative programming where you think of steps.
218
00:14:16.050 --> 00:14:24.904
And, in functional programming, you think of, operations. And you think of data and you think of operations which you apply to it. And,
219
00:14:25.445 --> 00:14:25.945
it
220
00:14:26.460 --> 00:14:31.440
could be a higher level of thinking. So you might compress bigger algorithm
221
00:14:31.980 --> 00:14:37.345
to fit into your mind. This could be pro this is probably the highest benefits.
222
00:14:38.125 --> 00:14:39.985
But different features
223
00:14:40.445 --> 00:14:41.665
could also be beneficial.
224
00:14:42.285 --> 00:14:43.980
Yeah. So I totally agree with,
225
00:14:44.620 --> 00:14:47.440
functional programming lets you decompose a problem into functions
226
00:14:47.820 --> 00:14:50.079
that are pure and have clean
inputs and outputs,
227
00:14:50.459 --> 00:14:51.279
and how
228
00:14:51.605 --> 00:14:54.105
we can solve most problems using just
229
00:14:54.485 --> 00:14:59.945
simple data structures and functions that call each other and have very dependable inputs and outputs.
230
00:15:00.290 --> 00:15:02.709
I would I generally classify this into the concept of modularity.
231
00:15:03.170 --> 00:15:09.670
And I would say that if we had a more object oriented person on on the phone, they might also say that object oriented programming
232
00:15:10.274 --> 00:15:15.334
also enables modularity. It also enables people to break down large problems into small problems.
233
00:15:15.954 --> 00:15:19.574
I think the difference comes in the style in which 1 achieves that.
234
00:15:20.180 --> 00:15:27.000
So I think that functional programming, there's maybe a tenant that you don't need an object, which is this marriage of state and functions.
235
00:15:27.625 --> 00:15:31.405
And often, state and functions can live separately. And that's often sufficient.
236
00:15:31.865 --> 00:15:35.885
So I think you can actually solve a large number of problems, very hard problems,
237
00:15:36.660 --> 00:15:41.000
using just core data structures like lists and dictionaries and strings and numbers
238
00:15:41.460 --> 00:15:42.760
and simple functions.
239
00:15:44.155 --> 00:15:48.655
I think a lot of functional programming a lot of the resurgence recently has come out of sort of
240
00:15:50.235 --> 00:15:51.055
a a backlash
241
00:15:51.550 --> 00:16:05.685
or a reaction to the sort of heavy object oriented programming that came out of the nineties. Would you agree? What are your thoughts, Alexander? I can add to it, that's when you model your domain with in object oriented way, usually, you create these,
242
00:16:06.165 --> 00:16:08.165
types which represent some,
243
00:16:09.010 --> 00:16:09.730
data. And,
244
00:16:10.370 --> 00:16:14.390
when this is, always a a new class, it's new types.
245
00:16:14.770 --> 00:16:33.300
So you need to implement lots of operations to handle them. Right. If you model your data with, some data structures like, dates, lists, and so on, You already have lots of operation in language for them. So usually, it's it's less code and it's faster to model things.
246
00:16:34.235 --> 00:16:40.495
Yeah. And in addition, not only is your code simpler, but your code can more easily interact with other code that you didn't anticipate.
247
00:16:41.120 --> 00:16:45.540
For example, you know, if I just use dictionaries and lists, I can use the JSON library
248
00:16:45.840 --> 00:16:52.355
immediately, trivially, without having to think about writing a toJson method. I think has a good example here of the,
249
00:16:53.295 --> 00:16:55.955
of the HTTP request object in those languages.
250
00:16:56.335 --> 00:16:58.595
It's usually some object that has lots of sub objects.
251
00:16:58.959 --> 00:17:03.060
And if you wanna interact with an HTTP request object, you sort of need to have the documents
252
00:17:03.439 --> 00:17:07.380
side by side as you're walking through this object. Well, instead, it could just be a nested
253
00:17:07.919 --> 00:17:08.419
dictionary.
254
00:17:08.835 --> 00:17:11.895
And then I could, you know, pull requests off of some socket.
255
00:17:12.275 --> 00:17:15.495
I could dump them into a document store like MongoDB,
256
00:17:15.795 --> 00:17:17.655
or I could serialize and put them to disk,
257
00:17:18.049 --> 00:17:21.190
all without having to write any extra code to handle those interfaces.
258
00:17:21.490 --> 00:17:23.110
So I think 1 major feature
259
00:17:23.490 --> 00:17:25.495
of functional programming is just
260
00:17:25.895 --> 00:17:29.835
a reliance on shared abstractions that are common to language core data structures.
261
00:17:30.535 --> 00:17:33.434
But I think that there are also some other features that are also
262
00:17:33.770 --> 00:17:40.270
associated to functional languages that are still also valuable. Like, algebraic types are something that, you know, came out of ML and help compilers ensure
263
00:17:40.785 --> 00:17:45.285
correctness and improve speed. And these, you know, might be sort of strange if you're looking for it from Haskell.
264
00:17:45.585 --> 00:17:51.630
But if a newer language like Julia, it's very accessible. They use algebraic types very effectively to improve performance.
265
00:17:52.169 --> 00:17:54.270
Something like macros, which came out of Lisp,
266
00:17:54.809 --> 00:17:57.549
can help library developers make more powerful user APIs.
267
00:17:58.164 --> 00:18:03.705
This is something that's been very helpful in, say, the language R, which uses them to make very powerful user APIs,
268
00:18:04.245 --> 00:18:06.184
even if they're a little bit strange for developers.
269
00:18:07.280 --> 00:18:10.179
Immutability is something that's often referred to as a functional
270
00:18:10.480 --> 00:18:10.980
concept,
271
00:18:11.760 --> 00:18:22.195
and software transactional memory. They really help with concurrency. And these are there's lots of immutable data structures inside of Python or inside of external libraries within Python. And software transactional memory has recently been added to PyPI.
272
00:18:22.815 --> 00:18:27.190
So there's lots of ideas that are come from these languages that are that are still useful,
273
00:18:27.890 --> 00:18:29.670
in a variety of different contexts.
274
00:18:30.610 --> 00:18:31.170
Yeah. And,
275
00:18:31.810 --> 00:18:33.190
we have progress in
276
00:18:33.625 --> 00:18:37.885
things like that, and, I can see little progress in, object oriented programming.
277
00:18:38.664 --> 00:18:41.920
You may have alluded to it a bit earlier, but what is it about functional programming
278
00:18:48.400 --> 00:18:55.195
that much intimidating if you don't tell people it's, It's not that much intimidating if you don't tell people it's it's fashion.
279
00:18:56.535 --> 00:18:58.890
Yeah. And so you don't need to,
280
00:18:59.450 --> 00:19:00.750
in, if
281
00:19:01.290 --> 00:19:04.110
I go a bit aside Sure. When you,
282
00:19:05.050 --> 00:19:06.910
market like a functional functional
283
00:19:07.565 --> 00:19:10.545
programming support library for a language, you could
284
00:19:10.925 --> 00:19:11.745
say, like,
285
00:19:12.685 --> 00:19:15.345
so stop doing like that and
286
00:19:16.970 --> 00:19:20.670
write code this way and use my library. And this is very weak proposition.
287
00:19:21.290 --> 00:19:22.250
Or you can,
288
00:19:22.570 --> 00:19:23.790
have some utils
289
00:19:25.315 --> 00:19:26.375
which actually solve,
290
00:19:26.755 --> 00:19:27.654
problems that
291
00:19:27.955 --> 00:19:29.815
people have. Like, some
292
00:19:30.115 --> 00:19:33.095
utils with, to work with sequences and
293
00:19:33.539 --> 00:19:38.600
utils, to work with collections in the like red. And then, you say
294
00:19:39.140 --> 00:19:40.840
my with my utils,
295
00:19:41.220 --> 00:19:42.919
we are very customizable
296
00:19:43.284 --> 00:19:43.784
because,
297
00:19:44.404 --> 00:19:44.904
we
298
00:19:45.684 --> 00:19:46.664
accept functions.
299
00:19:46.965 --> 00:19:54.160
So, we have all the functions which accept functions so you can use them for many tasks with your sequences of collections.
300
00:19:54.540 --> 00:19:54.860
And,
301
00:19:55.580 --> 00:19:57.040
this way you're solving
302
00:19:57.740 --> 00:20:01.045
some problems some problems people already have and,
303
00:20:02.005 --> 00:20:08.930
this way people can Google your library and, Google it by Googling their problems not Googling
304
00:20:09.330 --> 00:20:11.030
functional programming in Python,
305
00:20:11.890 --> 00:20:12.370
which,
306
00:20:12.850 --> 00:20:14.070
I think, far
307
00:20:14.610 --> 00:20:15.810
less people do. And,
308
00:20:16.455 --> 00:20:18.235
when people start using these
309
00:20:18.695 --> 00:20:20.635
utilities for sequences and collections,
310
00:20:21.095 --> 00:20:25.990
we start we start thinking of how we're we're formed with utilities,
311
00:20:26.290 --> 00:20:31.510
which are high order functions. And, we start to look around in your library and
312
00:20:31.835 --> 00:20:35.054
maybe introduce to our concepts like function composition.
313
00:20:35.674 --> 00:20:36.575
So it
314
00:20:37.115 --> 00:20:44.669
should not be really intimidating. And so back to why is why is it, besides its name. I think,
315
00:20:45.390 --> 00:20:46.990
any anything that's,
316
00:20:47.630 --> 00:20:47.870
not
317
00:20:48.590 --> 00:20:52.965
people are not used to could be intimidating. Also, we have this,
318
00:20:53.285 --> 00:20:58.030
thing that you can pack lots of things on single line and sometimes,
319
00:20:59.210 --> 00:21:02.510
people abuse it and this makes the code
320
00:21:02.890 --> 00:21:03.390
unreadable
321
00:21:03.690 --> 00:21:08.815
especially for people who are not used to things like WAVs. This could be intimidating. Also,
322
00:21:09.275 --> 00:21:13.180
some features like monads could be intimidating because it's really hard
323
00:21:14.060 --> 00:21:17.760
to explain why do you need those. Because you don't need those in most languages.
324
00:21:18.460 --> 00:21:23.045
And so this is my take. Maybe Matthew can add more. Yeah. No. I I totally agree
325
00:21:23.505 --> 00:21:26.565
with, the right way to brand functional programming and how,
326
00:21:27.105 --> 00:21:30.540
the again, the conversation isn't functional or not functional. It's,
327
00:21:31.100 --> 00:21:37.680
here's here's good ways to write software and and less good ways to write software. And that's really where we should be focusing if we wanna have a
328
00:21:38.435 --> 00:21:43.895
a pragmatic, if we want to have a positive effect. I would say that I think everyone learned some derivative of c historically.
329
00:21:44.195 --> 00:21:50.680
And so if you knew a functional language, it was usually your second language. And as a result, I felt a little bit fringe or elite.
330
00:21:51.220 --> 00:22:01.365
And this this impression of it being elite, I think, was very damaging to the ideas within functional programming. I mean, I think even within the the core Python community, there's still a little bit of hesitancy about any
331
00:22:01.745 --> 00:22:03.765
concept that's that's called functional
332
00:22:04.330 --> 00:22:05.870
because of this this bad history.
333
00:22:06.730 --> 00:22:08.590
But in truth, I think a lot of the concepts
334
00:22:09.370 --> 00:22:11.710
of functional programming are actually much simpler
335
00:22:12.245 --> 00:22:13.225
than people think.
336
00:22:13.845 --> 00:22:17.465
For example, I think that macros, which are maybe a little bit strange to people,
337
00:22:17.765 --> 00:22:18.985
are simpler intrinsically
338
00:22:19.285 --> 00:22:21.990
than, say, class inheritance in object oriented programming.
339
00:22:22.950 --> 00:22:26.650
However, I say recently so you asked about the recent rise in popularity.
340
00:22:27.110 --> 00:22:33.005
And I would say that recently, there's been some hardware changes, like multicore architectures have brought on parallel computing.
341
00:22:33.305 --> 00:22:36.365
And there's been a lot of need around highly concurrent systems.
342
00:22:36.665 --> 00:22:42.010
And for these, the fully stateful model of of imperative programming and mutation and state and object oriented
343
00:22:43.190 --> 00:22:44.490
aren't particularly ideal.
344
00:22:45.270 --> 00:22:51.985
And so sort of the functional language community was sort of off in the side, this very small minority saying, hey. We've actually got all of these nice tools
345
00:22:52.285 --> 00:22:54.065
that are really well suited to handle
346
00:22:54.445 --> 00:22:55.585
parallelism and concurrency
347
00:22:56.399 --> 00:22:58.100
and the building of large systems.
348
00:22:58.880 -->
00:22:59.779
And that really
349
00:23:00.080 --> 00:23:01.620
that utility, that pragmatism,
350
00:23:02.080 --> 00:23:06.645
really helped to reverse public opinion. So I think there's now maybe a sort of revitalization of functional thinking
351
00:23:07.025 --> 00:23:12.725
to the point where most model languages coming out now include a decently large subset of functional concepts within them.
352
00:23:13.690 --> 00:23:17.310
And this this really makes those concepts much more accessible to single language developers.
353
00:23:17.850 --> 00:23:22.110
The fact that in Python, you can use functions as data or you can use higher order functions
354
00:23:22.715 --> 00:23:27.455
that's accessible to someone who's just learning programming is is really powerful, I think.
355
00:23:28.154 --> 00:23:33.020
Yeah. And I think too that part of it might be attributable to the recent rise in popularity
356
00:23:34.120 --> 00:23:34.780
of JavaScript
357
00:23:35.320 --> 00:23:38.700
because while it does have some object oriented capabilities,
358
00:23:40.015 --> 00:23:45.795
it's generally used more in a functional aspect of passing functions around as variables and as data.
359
00:23:46.335 --> 00:23:46.835
And
360
00:23:47.215 --> 00:23:47.715
also,
361
00:23:48.655 --> 00:23:54.860
you know, there's there's a new generation of developers coming up. And as you said, you know, a number of developers started with some
362
00:23:55.400 --> 00:24:01.205
variant of the c programming language, and therefore, so a lot of the functional principles are a little bit more foreign.
363
00:24:01.665 --> 00:24:12.039
But given that there has been a slight rise in function in the popularity of functional languages, like you said, because of the recent changes in CPU architectures,
364
00:24:12.820 --> 00:24:13.320
then
365
00:24:13.620 --> 00:24:19.695
the newer programmers who are getting started don't necessarily have that same shared history and are a little bit more willing to experiment.
366
00:24:20.635 --> 00:24:23.690
And, also going back a little bit to what you were saying
367
00:24:24.250 --> 00:24:25.070
about functional
368
00:24:25.929 --> 00:24:27.950
paradigms being not as,
369
00:24:29.769 --> 00:24:33.005
not as widely advertised or encouraged in Python.
370
00:24:33.465 --> 00:24:46.410
1 thing that I've seen in a number of places, both in some of the official documentation and also in some company style guides, is, for instance, avoiding the use of the map filter and reduce functions in favor of,
371
00:24:47.190 --> 00:24:48.745
list dictionary comprehensions.
372
00:24:49.285 --> 00:24:50.585
And 1 thing that
373
00:24:51.445 --> 00:24:52.265
it seems
374
00:24:52.725 --> 00:24:55.865
it it seems that in some respects, that might be a disservice
375
00:24:56.245 --> 00:24:56.745
because
376
00:24:58.259 --> 00:25:01.720
I haven't tried it specifically, but it seems to me that the,
377
00:25:02.580 --> 00:25:07.455
comprehensions wouldn't
be as composable as some of the, like, map and reduce functionality
378
00:25:07.755 --> 00:25:10.154
that are more intended to be a,
379
00:25:10.794 --> 00:25:14.139
that are that are more intended to be used in a functional approach?
380
00:25:16.039 --> 00:25:17.820
I'm actually totally fine,
381
00:25:18.600 --> 00:25:25.325
making map and filter a little bit, forbidden. The way I think of it, I think I'm gonna steal some ideas that Alexandra has spoken about.
382
00:25:25.625 --> 00:25:28.125
But think of all the ways that you use a for loop.
383
00:25:28.750 --> 00:25:33.010
Right? You might use a for loop to apply some function onto some lists, or you might use a for loop to,
384
00:25:33.950 --> 00:25:39.485
filter some lists by some predicate, or you might use a for loop to reduce some list to some single number, like a sum.
385
00:25:39.865 --> 00:25:51.770
And if you're familiar with map and filter, you realize that, you know, 70% of your use cases of for loops fall into the cases of map, filter, or reduce. And so I think that map and filter are strictly better than a for loop.
386
00:25:52.150 --> 00:25:52.650
But,
387
00:25:53.030 --> 00:25:54.170
the core language designers
388
00:25:54.790 --> 00:25:56.955
recognized this, and they elevated
389
00:25:57.414 --> 00:25:58.715
these common patterns
390
00:25:59.174 --> 00:26:05.960
to to syntax, to the level of syntax. They elevated it to the list comprehension. And when you get up to that level, that's that's even 1 step
391
00:26:06.840 --> 00:26:08.299
better than map and filter.
392
00:26:08.679 --> 00:26:14.785
I think that if list comprehension didn't exist, map and filter would be very, very useful. But because list comprehensions do
393
00:26:15.165 --> 00:26:16.545
exist, they're not so much necessary.
394
00:26:17.245 --> 00:26:19.665
I think what libraries, like what
395
00:26:20.045 --> 00:26:21.505
funky and tools provide,
396
00:26:22.320 --> 00:26:22.900
is they
397
00:26:23.440 --> 00:26:26.179
provide a bunch of other functions like map and filter
398
00:26:26.480 --> 00:26:30.100
that do the same sorts of things. They represent common patterns that people use,
399
00:26:30.465 --> 00:26:32.885
but that haven't yet been elevated to the level of syntax.
400
00:26:33.425 --> 00:26:41.840
So I would say, yes, don't use map and filter. Use list comprehension. They're usually more readable. But map and filter have cousins, and maybe it's good to look at those cousins.
401
00:26:44.060 --> 00:26:47.120
I think, both map filter and, comprehensions
402
00:26:47.420 --> 00:26:48.160
are fine.
403
00:26:49.115 --> 00:26:49.775
The usual
404
00:26:50.955 --> 00:26:53.054
the usual reason to limit
405
00:26:54.554 --> 00:26:56.815
use of, some functions or some language
406
00:26:57.200 --> 00:26:58.419
constructs is to
407
00:26:58.720 --> 00:27:00.580
make your code base more
408
00:27:02.000 --> 00:27:03.220
more simple like,
409
00:27:04.080 --> 00:27:08.085
less, features
in it. Mhmm. So either it could lead
410
00:27:08.705 --> 00:27:10.165
to to less errors
411
00:27:10.705 --> 00:27:13.125
if you have this code style but,
412
00:27:13.585 --> 00:27:15.125
for me both is fine.
413
00:27:15.860 --> 00:27:16.360
Regarding
414
00:27:17.140 --> 00:27:17.539
with and,
415
00:27:19.059 --> 00:27:19.799
when you,
416
00:27:20.419 --> 00:27:22.919
go from my open filter which are
417
00:27:23.735 --> 00:27:24.715
special purpose,
418
00:27:26.215 --> 00:27:27.115
to this
419
00:27:27.575 --> 00:27:28.075
list
420
00:27:28.455 --> 00:27:29.515
of our comprehensions
421
00:27:30.830 --> 00:27:33.390
via more general purpose. So this is,
422
00:27:33.790 --> 00:27:34.690
try to
423
00:27:36.270 --> 00:27:37.010
go from
424
00:27:38.245 --> 00:27:39.065
some specific
425
00:27:39.445 --> 00:27:40.485
solutions to,
426
00:27:41.045 --> 00:27:41.545
DSL.
427
00:27:42.645 --> 00:27:43.145
And
428
00:27:45.580 --> 00:27:46.960
we have, overlaps
429
00:27:47.260 --> 00:27:48.480
in what we provide.
430
00:27:50.060 --> 00:27:51.100
But, I think,
431
00:27:51.580 --> 00:27:55.455
you have you still have a usage for filter and for map.
432
00:27:57.695 --> 00:27:59.635
Yeah. I really like that idea, actually.
433
00:28:00.815 --> 00:28:02.515
Map and filter are good at
434
00:28:03.100 --> 00:28:04.880
giving a name to a common pattern.
435
00:28:06.220 --> 00:28:12.215
And so even if you use a list of operations or use for loops, it's nice to have that name. Oh, I'm doing a map now.
436
00:28:12.615 --> 00:28:14.715
And map and filters, you say, are are more explicit
437
00:28:15.335 --> 00:28:17.034
than our list comprehension.
438
00:28:18.294 --> 00:28:19.995
Sometimes you can also
439
00:28:20.830 --> 00:28:23.170
use a partial of map and or filter,
440
00:28:24.030 --> 00:28:26.130
but this doesn't work with,
441
00:28:26.840 --> 00:28:27.340
comprehensions.
442
00:28:29.125 --> 00:28:32.985
Right. And that's part of what I was thinking of in terms of the composability
443
00:28:33.524 --> 00:28:36.830
where map and filter are a little bit superior in that respect.
444
00:28:37.530 --> 00:28:39.050
But like you said, Matthew, the
445
00:28:40.010 --> 00:28:43.790
in in a lot of cases, the comprehensions can be more immediately
446
00:28:44.595 --> 00:28:51.095
readable and clear as to what they're trying to do. But also, there are cases where you can get a little carried away using comprehensions
447
00:28:51.395 --> 00:28:57.140
where, for instance, you're trying to do 2 or 3 different levels of filtering within 1 comprehension. And then
448
00:28:57.520 --> 00:28:59.780
then it might be make more sense to
449
00:29:00.160 --> 00:29:00.660
either
450
00:29:00.960 --> 00:29:10.155
compose those comprehensions together and nest them a bit or to actually take advantage of the explicit filter call as Alexander was describing.
451
00:29:14.440 --> 00:29:17.980
Actually, I think, comprehension language, could be expanded.
452
00:29:18.615 --> 00:29:18.775
Like,
453
00:29:19.495 --> 00:29:21.915
the most, thing I miss is a lot
454
00:29:22.295 --> 00:29:22.875
in comprehension.
455
00:29:24.215 --> 00:29:28.075
Yeah. When you assign something and then use it later in if or.
456
00:29:29.080 --> 00:29:30.700
That would be very helpful. I agree.
457
00:29:31.400 --> 00:29:31.900
Yeah.
458
00:29:32.680 --> 00:29:33.180
Because,
459
00:29:34.600 --> 00:29:37.260
now sometimes you need to just
460
00:29:40.185 --> 00:29:41.005
duplicate an expression.
461
00:29:42.665 --> 00:29:44.505
For example, in if and in,
462
00:29:46.585 --> 00:29:47.965
in expression itself
463
00:29:49.210 --> 00:29:50.110
before before.
464
00:29:51.210 --> 00:29:51.950
And so
465
00:29:53.610 --> 00:29:54.110
in
466
00:29:54.490 --> 00:29:56.110
the common list, for example,
467
00:29:56.885 --> 00:29:59.145
they have a very, very complicated
468
00:29:59.685 --> 00:30:00.185
loop
469
00:30:00.725 --> 00:30:01.125
macro which
470
00:30:02.965 --> 00:30:03.465
yeah.
471
00:30:03.960 --> 00:30:04.460
I
472
00:30:05.000 --> 00:30:10.860
hear you. You heard about it. And so it does not look like the rest of the language,
473
00:30:11.240 --> 00:30:12.059
but it
474
00:30:13.305 --> 00:30:14.765
and it's it is
475
00:30:15.865 --> 00:30:16.765
causes contradictions.
476
00:30:17.305 --> 00:30:18.684
So some people hate it
477
00:30:18.985 --> 00:30:19.485
but
478
00:30:20.240 --> 00:30:29.115
for many people, it's very handy because you can make lots of and lots of things with it. And you can, do map and filter and,
479
00:30:29.575 --> 00:30:30.875
aggregations and
480
00:30:31.175 --> 00:30:33.115
do several things at once.
481
00:30:35.030 --> 00:30:36.970
And in Python, you sometimes
482
00:30:37.350 --> 00:30:43.450
need to fall back to common 4 because if you want to do several things at once, then
483
00:30:44.065 --> 00:30:45.445
it's usually the
484
00:30:45.905 --> 00:30:47.125
most efficient and
485
00:30:47.745 --> 00:30:49.365
sometimes the only way.
486
00:30:51.690 --> 00:30:53.149
And so we've
487
00:30:53.769 --> 00:31:02.055
done a pretty good coverage of list comprehensions. What other aspects of the Python language lend themselves to being used in a functional manner, and where do they start to fall down?
488
00:31:03.795 --> 00:31:05.335
Maybe the most noticeable
489
00:31:05.795 --> 00:31:07.495
is the absence of
490
00:31:07.960 --> 00:31:08.460
special
491
00:31:08.880 --> 00:31:09.380
immutable
492
00:31:09.800 --> 00:31:10.620
data structures.
493
00:31:12.760 --> 00:31:13.260
But
494
00:31:14.760 --> 00:31:15.260
you
495
00:31:15.640 --> 00:31:16.700
you can usually
496
00:31:17.575 --> 00:31:19.915
be okay with, like, immutable by
497
00:31:20.215 --> 00:31:20.715
convention
498
00:31:22.055 --> 00:31:22.715
which is
499
00:31:23.495 --> 00:31:25.915
not very efficient but, works.
500
00:31:26.429 --> 00:31:28.450
And you can mess it up or also.
501
00:31:29.950 --> 00:31:31.410
But but still.
502
00:31:32.990 --> 00:31:39.085
Yeah. So I would say that Python actually has a pretty good functional core. I would think ideas like streaming iterators,
503
00:31:39.625 --> 00:31:42.125
functions as data, functions as first class objects,
504
00:31:42.745 --> 00:31:44.205
really good core data structures.
505
00:31:44.910 --> 00:31:48.210
There's the legacy things like, Lambda map and filter.
506
00:31:48.590 --> 00:31:55.705
But higher order functions generally, these are all functional ideas that are really common in Python. That Python programmers do a very good job of using
507
00:31:56.245 --> 00:31:57.225
in common style.
508
00:31:57.605 --> 00:32:00.025
So to a large extent, Python is a functional language.
509
00:32:00.620 --> 00:32:03.120
There are some things that doesn't that Python doesn't do,
510
00:32:03.660 --> 00:32:09.645
like, as Alexander was saying, immutable data structures. Well, there's actually been some good work in this. There's I think, fn.py
511
00:32:10.105 --> 00:32:10.925
has some
512
00:32:11.785 --> 00:32:13.805
persistent data structure with structural sharing.
513
00:32:14.185 --> 00:32:16.125
Kurt Smith has a project called Ozymandias,
514
00:32:16.720 --> 00:32:18.740
where he's implementing persistent data structures,
515
00:32:19.520 --> 00:32:24.820
at the sea level, actually, using Cython. And they're very efficient. It's still work in progress, but it's a fun project.
516
00:32:26.255 --> 00:32:27.955
Some things I would love to see
517
00:32:28.495 --> 00:32:28.995
is
518
00:32:31.615 --> 00:32:39.740
better type systems, and there seems to be some push on this in Python 3.5. That's where the type modules coming out. Not doing anything yet, but it's still present.
519
00:32:40.440 --> 00:32:46.285
And macros. I'd actually really love to see macros. As someone who builds user APIs, I would love to see macros.
520
00:32:47.865 --> 00:32:51.325
There's actually a Python library that I came across recently
521
00:32:51.945 --> 00:32:52.445
where
522
00:32:52.850 --> 00:33:03.095
the that that that does provide some macro capabilities. So I'll see if I can find the link, and I'll put it in the show notes. K. There are actually a couple of good ones. There's macro py, and there's also code transformer
523
00:33:03.395 --> 00:33:06.695
by Scott Sanderson and Joe Devnick at Quantropion.
524
00:33:07.235 --> 00:33:11.880
These both do really cool things. They're manipulating the bytecodes. They're doing lots of cool things.
525
00:33:14.020 --> 00:33:14.520
Nothing
526
00:33:15.140 --> 00:33:17.535
will ever work without core language functionality
527
00:33:17.915 --> 00:33:20.095
in the interpreter. Most of my users
528
00:33:20.555 --> 00:33:23.695
are actually using the Python interpreter or like IPython or the Jupyter Notebook.
529
00:33:24.110 --> 00:33:26.130
And it's really hard to provide macros,
530
00:33:26.830 --> 00:33:28.690
without some kind of import hook
531
00:33:29.710 --> 00:33:31.465
or some kind of decorator. Yeah.
532
00:33:32.184 --> 00:33:42.700
There's also I don't know if you're familiar with it, but the highlighting project, which is the yes. We we had them on in a previous episode. And, anyway, the the the project creator,
533
00:33:43.080 --> 00:33:47.740
was describing how he was able to back port some of the Python 3 capabilities
534
00:33:48.365 --> 00:33:51.025
to Python 2 using macros that
535
00:33:51.325 --> 00:33:57.200
he it was the yield from operator that he backported from Python 3 using high using a high macro.
536
00:33:58.380 --> 00:34:03.760
But, yes, like you said, it does require some level of import to be able to inject that into the interpreter.
537
00:34:05.425 --> 00:34:07.525
I also have a point on macros.
538
00:34:08.785 --> 00:34:11.045
It's it's a contradictory 1 because
539
00:34:11.905 --> 00:34:14.085
it's very fascinating to
540
00:34:15.099 --> 00:34:17.680
to have this opportunity to extend the language,
541
00:34:18.460 --> 00:34:18.960
but,
542
00:34:19.579 --> 00:34:21.760
this is a very dangerous 1.
543
00:34:23.260 --> 00:34:24.000
If you
544
00:34:26.045 --> 00:34:27.825
add your markers to your language,
545
00:34:28.125 --> 00:34:28.625
you
546
00:34:29.325 --> 00:34:30.605
get this danger of,
547
00:34:31.725 --> 00:34:32.785
degenerated into
548
00:34:33.270 --> 00:34:34.970
this language into some dialects.
549
00:34:36.310 --> 00:34:44.125
So some people will start using some macros and, our people our ones and when we don't just understand each other.
550
00:34:44.425 --> 00:34:46.445
We can't read each other's code.
551
00:34:47.545 --> 00:34:48.045
It
552
00:34:49.000 --> 00:34:49.500
happens,
553
00:34:50.040 --> 00:34:55.660
more with smaller languages, like scheme when you have a small language and when you
554
00:34:56.120 --> 00:34:57.155
you guys can
555
00:34:57.635 --> 00:34:59.015
build on it with macros.
556
00:35:00.195 --> 00:35:00.695
But
557
00:35:01.875 --> 00:35:05.815
for languages like Python, it's, could also be
558
00:35:06.440 --> 00:35:06.940
dangerous.
559
00:35:09.799 --> 00:35:11.559
The other alternative is that,
560
00:35:12.839 --> 00:35:14.539
you don't use much macros.
561
00:35:15.465 --> 00:35:16.745
Like, you have them, but,
562
00:35:17.385 --> 00:35:17.945
you it's
563
00:35:19.225 --> 00:35:20.445
you should be very
564
00:35:21.625 --> 00:35:23.325
cautious about using them.
565
00:35:25.090 --> 00:35:27.510
And I think in common Lisbon, enclosure,
566
00:35:27.970 --> 00:35:28.470
we're
567
00:35:29.170 --> 00:35:30.550
a bit adopting it.
568
00:35:33.585 --> 00:35:36.805
Yeah. I think that's a totally valid point. And macros are definitely
569
00:35:37.505 --> 00:35:42.840
a language feature that can destroy a language because it provides too much power to library developers
570
00:35:43.140 --> 00:35:45.560
to separate themselves into their own domain specific language.
571
00:35:45.940 --> 00:35:55.645
And along with macros, you must include some sort of community standard that you should never use a macro, and that's absolutely necessary. I think if you look at, again, languages like Clojure or Julia,
572
00:35:56.025 --> 00:35:57.325
this has become the norm.
573
00:35:57.819 --> 00:36:00.720
Very few people use macros. It's considered to be a faux pas.
574
00:36:02.299 --> 00:36:04.400
So yeah. And also, you could also say that, you know, Python
575
00:36:04.945 --> 00:36:12.805
provides lots of opportunities for us to segment ourselves. There's lots of, you know, very, you know, say, 2 powerful ideas in Python, like metaclasses,
576
00:36:13.585 --> 00:36:14.085
that
577
00:36:14.490 --> 00:36:16.590
that enable this sort of bad behavior already.
578
00:36:17.770 --> 00:36:18.830
And monkey patching.
579
00:36:19.290 --> 00:36:24.235
And monkey patching. Yeah. Where Python is very permissive in lots of ways. There's a way that's not permissive.
580
00:36:24.535 --> 00:36:26.875
But as a library writer who serves,
581
00:36:27.415 --> 00:36:28.955
user scientific user communities,
582
00:36:29.255 --> 00:36:40.150
it's something I look at r, look at Julia, and I I just really want the ability to write macros. Because there's there's certain APIs that users love that are just impossible to write without
583
00:36:40.964 --> 00:36:42.345
language support for macros.
584
00:36:43.045 --> 00:36:47.340
But I agree. You know, I only want sort of 10 people in the world to be able to write Python macros. Everyone else not allowed.
585
00:36:53.900 --> 00:37:06.170
And so can you each describe what your respective libraries provide in terms of functional capabilities and what their particular focuses are? And also wondering if they're distinct enough from each other that it would make sense to use them both in a single project.
586
00:37:09.750 --> 00:37:11.050
The most obvious,
587
00:37:11.590 --> 00:37:13.905
thing that, FANSI provides is
588
00:37:15.985 --> 00:37:16.485
sequences
589
00:37:17.185 --> 00:37:18.885
and collections and utilities.
590
00:37:20.145 --> 00:37:20.645
And,
591
00:37:21.905 --> 00:37:22.645
this is
592
00:37:23.380 --> 00:37:25.240
within people use the most.
593
00:37:26.099 --> 00:37:26.599
Because,
594
00:37:27.220 --> 00:37:31.799
you don't need to get functional programming and stuff to use it. I already
595
00:37:32.525 --> 00:37:33.825
talked a bit about it.
596
00:37:34.444 --> 00:37:34.944
When,
597
00:37:35.565 --> 00:37:36.545
Fantic has
598
00:37:37.565 --> 00:37:38.065
some
599
00:37:39.085 --> 00:37:39.585
extension
600
00:37:39.964 --> 00:37:41.665
of fun tools, I guess,
601
00:37:42.500 --> 00:37:43.000
with,
602
00:37:44.180 --> 00:37:44.680
particles,
603
00:37:45.220 --> 00:37:46.680
right particles, and,
604
00:37:47.619 --> 00:37:49.560
composition and some other things.
605
00:37:50.835 --> 00:37:52.375
And when we are just,
606
00:37:53.954 --> 00:37:54.454
some
607
00:37:56.115 -->
00:37:57.335
utilities for
608
00:37:58.520 --> 00:38:00.380
lots of things, like string utilities.
609
00:38:03.320 --> 00:38:03.640
Where
610
00:38:04.360 --> 00:38:09.155
1 example is that I always hated how you need to first
611
00:38:09.615 --> 00:38:15.235
in Python, first match the thing with regular expression and then write if it match then go
612
00:38:16.175 --> 00:38:16.675
with
613
00:38:19.160 --> 00:38:23.900
new it's like you always need at least 3 lines of codes to
614
00:38:24.200 --> 00:38:25.500
use a regular expression.
615
00:38:26.345 --> 00:38:33.785
And, I came before Python, I wrote Perl a lot and it's much easier there and,
616
00:38:35.150 --> 00:38:37.090
usually it's 1 line. So
617
00:38:38.590 --> 00:38:40.450
I added some things to
618
00:38:41.150 --> 00:38:42.530
work with regular expressions.
619
00:38:45.225 --> 00:38:48.685
When some, flow control utilities and some
620
00:38:49.465 --> 00:38:50.445
primitives and
621
00:38:51.170 --> 00:38:54.390
ironically even some utilities for you
622
00:38:54.770 --> 00:38:57.190
to use with objects like cached property.
623
00:38:59.835 --> 00:39:03.695
We we we now also have this cash property in Django,
624
00:39:05.195 --> 00:39:06.575
I believe, from 1.6.
625
00:39:07.800 --> 00:39:08.300
And,
626
00:39:09.080 --> 00:39:11.180
this is really handy when
627
00:39:13.400 --> 00:39:15.900
you have these objects like ORM
628
00:39:17.145 --> 00:39:17.645
objects
629
00:39:19.145 --> 00:39:22.125
and you need some properties which shouldn't be calculated
630
00:39:22.425 --> 00:39:24.525
twice for for an instance.
631
00:39:25.260 --> 00:39:26.300
And, when,
632
00:39:26.860 --> 00:39:29.280
there is a debug debugging utilities
633
00:39:29.900 --> 00:39:31.280
for functional program because,
634
00:39:32.300 --> 00:39:34.480
when you debug a functional program,
635
00:39:35.925 --> 00:39:36.665
you sometimes
636
00:39:37.925 --> 00:39:40.425
can't go the way you usually do.
637
00:39:40.805 --> 00:39:41.465
You can't
638
00:39:42.085 --> 00:39:42.905
add print
639
00:39:43.205 --> 00:39:44.185
because you
640
00:39:45.510 --> 00:39:47.690
everything is happening on a single line
641
00:39:48.150 --> 00:39:49.210
and or you
642
00:39:49.910 --> 00:39:51.769
can't use watches because
643
00:39:53.485 --> 00:39:54.365
and stepping because,
644
00:39:55.005 --> 00:39:55.505
where
645
00:39:55.885 --> 00:39:57.585
file is named variables.
646
00:39:58.525 --> 00:40:01.470
You just pass everything to every 1
647
00:40:01.770 --> 00:40:04.109
1 function to our function and like that.
648
00:40:05.530 --> 00:40:08.350
So it has some debugging utilities like,
649
00:40:09.450 --> 00:40:09.950
decorators,
650
00:40:10.555 --> 00:40:12.655
logging and sprinting and timing
651
00:40:13.195 --> 00:40:14.335
and things like that.
652
00:40:16.715 --> 00:40:20.095
Yeah. So PyTools, that's t o l z,
653
00:40:20.430 --> 00:40:21.890
is a collection of functions
654
00:40:22.990 --> 00:40:26.290
that are good for managing iterators and functions and dictionaries.
655
00:40:27.310 --> 00:40:28.050
These functions
656
00:40:28.845 --> 00:40:31.025
are pure, composable, lazy, and tuned.
657
00:40:32.365 --> 00:40:36.065
And they sort of follow the standard functions you'll find in a variety of languages.
658
00:40:36.420 --> 00:40:38.840
So if you look at languages like Clojure or Scala
659
00:40:39.460 --> 00:40:42.120
or in JavaScript, libraries like underscore or lodash,
660
00:40:43.355 --> 00:40:45.615
or a variety of languages in Ruby and whatnot,
661
00:40:46.395 --> 00:40:47.535
you'll find that
662
00:40:48.395 --> 00:40:53.055
there's a bunch of functions that work on iterators. Things like map and filter, but there's far more, like groupby,
663
00:40:54.460 --> 00:40:56.240
and sliding window, and repeat,
664
00:40:56.860 --> 00:40:58.640
and memoize, and curry.
665
00:40:59.180 --> 00:41:04.125
And you'll find these functions all over the place. And they end up being really useful when you
666
00:41:04.905 --> 00:41:05.805
build an ecosystem
667
00:41:06.105 --> 00:41:14.640
based on core data structures. So earlier in the show, we were talking about the value of just dealing with lists and dictionaries and iterators and normal functions,
668
00:41:15.260 --> 00:41:17.039
and how this is really good for an ecosystem.
669
00:41:17.725 --> 00:41:20.145
And once you start to
670
00:41:20.685 --> 00:41:21.585
use only
671
00:41:22.125 --> 00:41:23.425
these core data structures,
672
00:41:23.805 --> 00:41:30.130
you find there's actually a lot of functions you often want, And there's there's not in the Python standard library, but there are lots of other standard libraries.
673
00:41:30.670 --> 00:41:39.625
And so a lot of other languages sort of converge to this common, like, this common function set, and tools was just the instantiation of that common function set in Python.
674
00:41:40.565 --> 00:41:46.240
You asked about how these libraries might relate to each other if they're different. Whatever you use both in the same same project.
675
00:41:46.780 --> 00:41:51.425
I think that they probably overlap enough that it's not usually necessary to use in the same project.
676
00:41:51.825 --> 00:41:55.045
They're also both simple enough that you could. They're both very lightweight dependencies.
677
00:41:55.745 --> 00:41:57.445
I would say that tools is,
678
00:41:58.065 --> 00:41:58.545
notable,
679
00:41:58.865 --> 00:42:01.045
compared to the all the other functional libraries,
680
00:42:01.990 --> 00:42:04.730
in that it's very low tech. It's usually just functions,
681
00:42:05.990 --> 00:42:08.090
and it's also so it's very accessible.
682
00:42:08.525 --> 00:42:09.825
You shouldn't ever need
683
00:42:10.125 --> 00:42:12.305
to learn some new concept, the same with Funcie.
684
00:42:13.325 --> 00:42:16.625
It's also notable in his performance. Tools has been heavily benchmarked,
685
00:42:17.030 -->
00:42:18.490
largely thanks to Eric Welch,
686
00:42:18.950 --> 00:42:23.290
who does most maintenance today. And Eric also rewrote PyTools in Cython,
687
00:42:24.150 --> 00:42:26.535
which makes the all the iterator stuff
688
00:42:27.095 --> 00:42:31.275
about as fast as you'd find in a compiled language like Java on dynamic data structures.
689
00:42:32.135 --> 00:42:34.075
So tools has tended to be sort of
690
00:42:35.410 --> 00:42:38.550
very much like what you expect if you're familiar with other languages,
691
00:42:39.010 --> 00:42:39.830
and it's
692
00:42:40.609 --> 00:42:41.430
very fast.
693
00:42:45.595 --> 00:42:46.815
The 1 thing that
694
00:42:47.755 --> 00:42:48.255
you
695
00:42:49.035 --> 00:42:49.695
the 1
696
00:42:50.049 --> 00:42:50.789
obvious difference that
697
00:42:51.170 --> 00:42:55.510
with is that you use the the other iterator versions of
698
00:42:55.890 --> 00:42:56.789
all the functions.
699
00:42:58.005 --> 00:42:58.505
But,
700
00:42:59.205 --> 00:43:01.465
yeah, the font is a bit different.
701
00:43:02.005 --> 00:43:03.465
It, first, it has
702
00:43:04.700 --> 00:43:09.280
2 versions for many functions, like, the list version and the other version.
703
00:43:09.740 --> 00:43:10.060
And,
704
00:43:10.780 --> 00:43:13.760
they are named differently for Python 2 and for Python
705
00:43:14.765 --> 00:43:16.945
3. In Python 2, it's like
706
00:43:18.045 --> 00:43:22.225
start if it starts with I, then it's iterative version, like map and imap.
707
00:43:23.290 --> 00:43:23.530
And,
708
00:43:24.090 --> 00:43:27.790
for Python 3, it's like map and l map. So,
709
00:43:28.170 --> 00:43:28.650
in,
710
00:43:29.210 --> 00:43:33.825
Python 3, by default, you don't have a list version of map and field and, like, wait, and
711
00:43:34.204 --> 00:43:34.944
I don't
712
00:43:35.565 --> 00:43:36.865
really like this change
713
00:43:37.405 --> 00:43:38.704
because, this is
714
00:43:39.165 --> 00:43:41.105
so confusing for many people.
715
00:43:43.210 --> 00:43:46.250
You you, like, call this map of filter and,
716
00:43:47.370 --> 00:43:48.910
you it does not actually
717
00:43:50.295 --> 00:43:50.795
do
718
00:43:51.095 --> 00:43:53.515
the the job. You pass this and,
719
00:43:54.615 --> 00:43:56.395
when later, you can
720
00:43:58.049 --> 00:44:00.150
get something from it. And, say,
721
00:44:00.450 --> 00:44:01.190
you can,
722
00:44:02.770 --> 00:44:07.155
forget with its iterator. You can try to add it to a list or you can
723
00:44:08.755 --> 00:44:10.775
try to iterate it, twice,
724
00:44:11.395 --> 00:44:13.395
which gives you an error in a different,
725
00:44:15.150 --> 00:44:16.050
in a different,
726
00:44:16.510 --> 00:44:18.210
point when you actually
727
00:44:19.070 --> 00:44:20.050
get this error,
728
00:44:20.990 --> 00:44:22.930
when you actually made it.
729
00:44:24.615 --> 00:44:26.395
So I should not I don't think,
730
00:44:27.495 --> 00:44:29.435
the iterator should go by default.
731
00:44:30.855 --> 00:44:32.955
This for me it's more like optimization.
732
00:44:33.590 --> 00:44:36.330
So the Python 2 way of doing things,
733
00:44:37.590 --> 00:44:39.050
makes more sense to me.
734
00:44:41.005 --> 00:44:45.744
Right. So this is actually this this difference so you're right. So everything in tools where possible
735
00:44:46.045 --> 00:44:46.865
is an iterator,
736
00:44:47.484 --> 00:44:53.540
even in Python 2. So if you import map from tools in Python 2, you'll get the equivalent of iter toolsimap.
737
00:44:55.840 --> 00:44:58.100
And this is by design for 2 reasons.
738
00:44:59.085 --> 00:44:59.585
1,
739
00:45:01.325 --> 00:45:04.225
1 of the desires here is that tools should be minimally creative.
740
00:45:04.845 --> 00:45:08.200
We didn't want to design an API or make those decisions.
741
00:45:08.580 --> 00:45:15.775
The core Python library core Python language is going lazy. We decided to go lazy too. And second, this also probably comes down to
742
00:45:16.234 --> 00:45:17.055
our backgrounds.
743
00:45:18.234 --> 00:45:24.630
In web programming, I suspect that many times you want map. You actually have a small dataset, and you want that to be a list in memory.
744
00:45:25.170 --> 00:45:33.795
Where for me, I do a lot of data processing. And for me, it's very valuable to have everything be lazy by default. I can stream through tens of gigabytes from disk on my machine,
745
00:45:34.495 --> 00:45:37.875
and be very happy. And tools just works in a small space.
746
00:45:38.690 --> 00:45:39.170
But that
747
00:45:40.050 --> 00:45:46.470
I agree with you that in many cases, when I do work on, you know, just very small lists of 5 elements, I really do
748
00:45:47.615 --> 00:45:50.515
find writing list, you know, list map sequence,
749
00:45:51.135 --> 00:45:52.675
to be very annoying all the time.
750
00:45:54.319 --> 00:45:56.819
Still, I see this, using the teratas
751
00:45:57.200 --> 00:45:58.339
as an optimization.
752
00:45:59.359 --> 00:46:01.220
So it shouldn't go by default.
753
00:46:02.885 --> 00:46:06.185
And, this way I don't agree how Python 3
754
00:46:07.125 --> 00:46:08.905
map and filter does things.
755
00:46:10.829 --> 00:46:17.569
So you should go with lists by default and if it's too slow or too memory consuming when you switch to iterators.
756
00:46:19.505 --> 00:46:20.485
It's like
757
00:46:21.105 --> 00:46:22.325
premature optimization
758
00:46:24.705 --> 00:46:26.885
and Python 3 forces you in it.
759
00:46:27.329 --> 00:46:27.829
And,
760
00:46:28.690 --> 00:46:30.150
I it's rather so we
761
00:46:30.450 --> 00:46:32.950
are inherently more complex than lists.
762
00:46:34.049 --> 00:46:34.549
They're
763
00:46:35.170 --> 00:46:36.390
more hard to
764
00:46:36.805 --> 00:46:37.305
handle.
765
00:46:38.565 --> 00:46:39.705
So you write
766
00:46:40.645 --> 00:46:43.865
a more harder to to handle code by default
767
00:46:45.099 --> 00:46:46.000
and this is wrong.
768
00:46:48.619 --> 00:46:49.359
It's also
769
00:46:49.740 --> 00:46:56.035
very useful sometimes, though. I think if the whole community gets on board with the concept of lazy iterators,
770
00:46:57.135 --> 00:46:58.435
then the whole ecosystem
771
00:46:58.975 --> 00:47:08.570
is lazy by default. And that that has a lot of value, again, when you're when you're processing data, which for me is the common case. Again, I think this is the difference between the web community and the science community,
772
00:47:09.760 --> 00:47:16.525
where it might be an optimization, but it's an optimization we want almost all the time. And it's it is quite tricky to make sure that,
773
00:47:17.065 --> 00:47:18.925
all the libraries you're using everywhere
774
00:47:19.440 --> 00:47:20.900
are being lazy by default.
775
00:47:23.040 --> 00:47:26.020
It's sort of very easy in your own code just to use lists.
776
00:47:26.375 --> 00:47:33.035
But if you're writing infrastructural code, it's gonna be used by many different projects at the same time, you sort of want to be lazy by default.
777
00:47:33.335 --> 00:47:35.595
Because a lot of those users, again, in my community,
778
00:47:35.910 --> 00:47:37.930
want the laziness. It's a high value.
779
00:47:39.190 --> 00:47:42.010
But we can disagree. That's that's that's fun.
780
00:47:44.455 --> 00:47:47.835
And so what inspired each of you to create your libraries?
781
00:47:50.935 --> 00:47:52.875
I started from writing some
782
00:47:54.270 --> 00:47:55.810
data utilities and
783
00:47:56.350 --> 00:47:56.850
functional
784
00:47:57.710 --> 00:47:58.210
utilities.
785
00:47:59.070 --> 00:48:00.850
It was a a part of
786
00:48:02.605 --> 00:48:04.225
a part of a web application.
787
00:48:06.205 --> 00:48:07.585
Only later I extracted,
788
00:48:08.685 --> 00:48:10.305
it just grow this part
789
00:48:10.720 --> 00:48:11.940
over time. And then
790
00:48:12.400 --> 00:48:14.260
I extracted it in the library,
791
00:48:14.880 --> 00:48:15.380
and
792
00:48:15.760 --> 00:48:17.460
it was obviously very
793
00:48:18.295 --> 00:48:19.435
limited in
794
00:48:20.055 --> 00:48:22.155
in in its in its scope because
795
00:48:22.535 --> 00:48:27.035
I only added, the things that, I needed for a project.
796
00:48:27.900 --> 00:48:28.400
So
797
00:48:29.100 --> 00:48:30.720
it's not like a complete library.
798
00:48:31.260 --> 00:48:34.480
So I looked at, Closure Standard Library
799
00:48:34.780 --> 00:48:36.480
and Corio Library and,
800
00:48:37.095 --> 00:48:38.155
on, underscore
801
00:48:38.455 --> 00:48:39.355
JS library.
802
00:48:40.055 --> 00:48:41.035
And, I
803
00:48:42.535 --> 00:48:43.735
looked carefully what,
804
00:48:44.215 --> 00:48:45.355
things could be
805
00:48:45.660 --> 00:48:47.200
useful in Python and,
806
00:48:47.900 --> 00:48:50.640
if, if I need to change them when I'm
807
00:48:51.020 --> 00:48:52.400
moving them to Python.
808
00:48:53.805 --> 00:48:54.205
And,
809
00:48:56.125 --> 00:48:57.025
I
implemented
810
00:48:57.325 --> 00:48:57.825
all
811
00:48:59.325 --> 00:49:00.260
functions from
812
00:49:01.140 --> 00:49:01.640
ClosureCoreClosureCore
813
00:49:02.180 --> 00:49:04.760
library and from underscore JS library,
814
00:49:06.260 --> 00:49:06.760
which
815
00:49:07.540 --> 00:49:09.640
made sense to to move.
816
00:49:10.405 --> 00:49:11.145
And so
817
00:49:11.845 --> 00:49:13.145
so, Fancy was born.
818
00:49:14.325 --> 00:49:17.465
Later, I added some some features that I
819
00:49:18.160 --> 00:49:19.140
needed myself
820
00:49:19.760 --> 00:49:22.740
or some features that people asked
821
00:49:23.920 --> 00:49:24.420
or
822
00:49:25.095 --> 00:49:25.595
contributed.
823
00:49:26.455 --> 00:49:27.435
The idea of,
824
00:49:28.135 --> 00:49:30.075
Alivery also it's that it
825
00:49:31.575 --> 00:49:32.315
aims to
826
00:49:32.855 --> 00:49:35.260
limit its scope at some point. So,
827
00:49:35.720 --> 00:49:37.740
like, you add to it,
828
00:49:39.000 --> 00:49:40.540
but at some point,
829
00:49:41.160 --> 00:49:42.140
it's complete.
830
00:49:43.384 --> 00:49:44.345
Nowadays, we usually
831
00:49:45.305 --> 00:49:46.845
most projects and libraries
832
00:49:47.224 --> 00:49:48.365
were never complete.
833
00:49:49.065 --> 00:49:55.300
We, like, go with new and new versions and you add and add features but I don't think it's,
834
00:49:55.620 --> 00:50:02.165
a good idea for library like this. At some you need to limit scope at some point and,
835
00:50:02.645 --> 00:50:05.385
save it. This is out of scope, like, immutable
836
00:50:06.405 --> 00:50:06.905
structures,
837
00:50:07.605 --> 00:50:09.605
out of fancy scope definitely. And,
838
00:50:10.590 --> 00:50:11.410
some other
839
00:50:11.710 --> 00:50:14.850
things need to stop at some point. So
840
00:50:15.470 --> 00:50:15.970
Matthew?
841
00:50:17.310 --> 00:50:27.630
Yeah. So my experience was very similar. I was writing a lot of Clojure at the time. And I I found that what I really liked about the language wasn't any of the fancier features. I just really liked that the whole ecosystem was based
842
00:50:28.509 --> 00:50:37.555
on everyone using core data structures. And then again, I talked about, you know, HTTP requests were just nested nest addict. Database connections just took in dictionaries of iterators of dictionaries.
843
00:50:38.735 --> 00:50:50.150
And this is I found to be really valuable. And so I took that same thinking into Python, but I found that a lot of the functions that I was used to using weren't present. And so I, again, looked at closure, I looked at underscore, I looked at Ruby's enumerable,
844
00:50:50.850 --> 00:50:53.350
and I started implementing them in in in tools.
845
00:50:54.805 --> 00:50:58.825
And I found that, you know, it's a really nice approach to use core data structures.
846
00:50:59.285 --> 00:51:02.025
And if you have these a good set of functions, you know,
847
00:51:02.460 --> 00:51:03.599
20 or so functions,
848
00:51:03.980 --> 00:51:04.960
that actually encompasses
849
00:51:05.579 --> 00:51:08.880
90% of what you want to do on those data structures for most programs.
850
00:51:09.259 --> 00:51:13.835
And so I was able to lift up my thinking out of 4 loops, out of maps to more,
851
00:51:14.415 --> 00:51:16.075
to more higher level concepts.
852
00:51:17.530 --> 00:51:22.510
It was really interesting looking at all these languages and the set of names that they had converged to, the set of functions.
853
00:51:23.210 --> 00:51:26.095
And so it was actually really an easy project to to build.
854
00:51:26.415 --> 00:51:28.435
All the API was set out.
855
00:51:29.215 --> 00:51:32.755
Actually, so at the time, right after building this, I started working on a few different projects.
856
00:51:33.215 --> 00:51:38.740
It was actually the same algorithm, but in different different languages. And And it was really cool to use python underscore enclosure
857
00:51:39.440 --> 00:51:44.085
and have sort of a lingua franca, a common language, among all of these different languages,
858
00:51:44.565 --> 00:51:49.065
where I sort of depend on this API being present. I found that really valuable.
859
00:51:50.325 --> 00:51:51.545
To go with Alexander's
860
00:51:52.085 --> 00:51:55.450
point, I think that the scope of these libraries should be small.
861
00:51:56.549 --> 00:51:56.710
I
862
00:51:58.230 --> 00:52:00.569
most pull requests today, I I reject
863
00:52:01.135 --> 00:52:02.994
just because they're a little bit weird.
864
00:52:03.535 --> 00:52:07.075
I don't think they just stop. I think they should keep growing. I think we're gonna keep learning. But
865
00:52:07.694 --> 00:52:13.290
I think both tools and Funcie have done a very good job of limiting their scope and of being correct.
866
00:52:13.830 --> 00:52:16.730
You know, there there are almost no bug fixes. It's it's actually really cool.
867
00:52:17.135 --> 00:52:18.435
They they just work, and everyone
868
00:52:18.975 --> 00:52:21.155
understands and trusts everything in the libraries.
869
00:52:22.815 --> 00:52:24.435
The reason is, let's say, implementation
870
00:52:24.735 --> 00:52:25.475
of many,
871
00:52:26.510 --> 00:52:28.530
most of functions is very easy,
872
00:52:29.230 --> 00:52:30.130
very simple.
873
00:52:30.510 --> 00:52:31.010
Yeah.
874
00:52:31.550 --> 00:52:32.430
Sometimes it's,
875
00:52:32.990 --> 00:52:33.490
it's
876
00:52:33.954 --> 00:52:36.454
more complex because you optimize it, but
877
00:52:36.994 --> 00:52:38.855
it's still usually very simple where
878
00:52:39.954 --> 00:52:43.494
the most value is in, like, concepts and ideas.
879
00:52:45.660 --> 00:52:47.359
Yeah. Yeah. So I think that,
880
00:52:47.819 --> 00:52:49.440
it's very simple also because
881
00:52:49.900 --> 00:52:50.640
they're communicating
882
00:52:51.265 --> 00:52:55.045
off a very simple shared abstraction. They communicate off of lists or iterators or dicts.
883
00:52:55.745 --> 00:52:57.685
So usually, bugs occur in software
884
00:52:58.065 --> 00:53:02.329
because we have 2 things that are independently correct. But when they interact,
885
00:53:02.630 --> 00:53:05.369
there's some complex behavior and and a bug arises.
886
00:53:05.990 --> 00:53:10.785
And so, usually, a lot of bugs are tracking down, you know, this piece into this widget.
887
00:53:11.244 --> 00:53:13.665
And all of the functions and tools and all the functions in Func
888
00:53:14.355 --> 00:53:15.585
are completely isolated.
889
00:53:15.960 --> 00:53:20.060
And we just trust the underlying Python mechanisms, iterators, dictionaries, etcetera,
890
00:53:20.520 --> 00:53:21.020
to
891
00:53:21.320 --> 00:53:24.140
be the shared substrate along which they communicate.
892
00:53:24.645 --> 00:53:25.385
Which again,
893
00:53:26.005 --> 00:53:27.065
that lack of interaction
894
00:53:27.685 --> 00:53:28.185
really
895
00:53:28.965 --> 00:53:30.345
makes developing these libraries
896
00:53:30.725 --> 00:53:31.305
a pleasure.
897
00:53:32.740 --> 00:53:47.525
And for anybody who isn't aware of it anyway, there's a functools module in the Python standard library that provides some methods that enable some functional paradigms. I'm wondering where that module falls short and how your respective libraries either augment or replace some of the functionality found therein.
898
00:53:48.660 --> 00:53:53.940
1st, fun tools is, not like a library with the idea. It's like,
899
00:53:54.500 --> 00:53:57.880
storage for very loosely connected things,
900
00:53:58.225 --> 00:54:00.565
which, gets added to into Python,
901
00:54:01.825 --> 00:54:04.465
over time. A more interesting 1 is,
902
00:54:05.185 --> 00:54:05.925
iter tools,
903
00:54:06.230 --> 00:54:08.970
which, actually has some structure and,
904
00:54:09.590 --> 00:54:10.090
has
905
00:54:10.630 --> 00:54:11.530
some utilities,
906
00:54:12.070 --> 00:54:15.050
to work with sequences, which are more,
907
00:54:15.435 --> 00:54:17.215
probably more popular and mono.
908
00:54:17.755 --> 00:54:18.575
If you
909
00:54:18.955 --> 00:54:22.475
you speak about fun tools when, for example, it lets,
910
00:54:22.875 --> 00:54:23.855
even compose
911
00:54:24.609 --> 00:54:25.589
function composition,
912
00:54:26.049 --> 00:54:26.549
probably
913
00:54:26.849 --> 00:54:29.509
very basic thing. So it's very easy.
914
00:54:29.890 --> 00:54:38.345
I'll I'll agree with Alexander here that I'm looking through the fun tools API right now. And the things that I reasonably use are LRU cache in Python 3,
915
00:54:38.645 --> 00:54:39.145
partial,
916
00:54:39.845 --> 00:54:40.665
single dispatch,
917
00:54:41.100 --> 00:54:44.240
and reduce, because it's been sort of evicted from the the core language.
918
00:54:44.700 --> 00:54:46.640
And there's really not a whole lot there otherwise.
919
00:54:47.180 --> 00:54:48.160
Things like composition,
920
00:54:48.460 --> 00:54:49.555
pipe, curry,
921
00:54:50.815 -->
00:54:55.075
aren't present. And these are sort of, like, standard things you'd expect to be in any functional
922
00:54:55.615 --> 00:54:56.115
library.
923
00:54:56.495 --> 00:54:58.470
So it's not really that it's not,
924
00:54:58.930 --> 00:55:00.470
it's really not sort of the love
925
00:55:00.849 --> 00:55:01.910
of the core developers.
926
00:55:02.690 --> 00:55:12.055
I'd also say so I think functools and iter tools have a lot of really great functionality in them. But sometimes so as someone who tries to adhere to standards met by other communities,
927
00:55:12.435 --> 00:55:14.695
sometimes they deviate a little bit from
928
00:55:15.130 --> 00:55:22.895
from common from common interfaces. So for example, the group by function iter tools is not like the group by function in any other language. It's a
929
00:55:23.195 --> 00:55:24.095
false cognate.
930
00:55:24.795 --> 00:55:26.575
I think the LRU cache and functools,
931
00:55:26.955 --> 00:55:28.815
I would much rather have seen
932
00:55:29.195 --> 00:55:29.695
them
933
00:55:30.075 --> 00:55:35.270
accept some mutable mapping as a as an input rather than bake in LRU
934
00:55:35.650 --> 00:55:44.595
to the caching mechanism. There's just some ideas that I think it would have been better if the community had looked out to what other languages have been doing for these exact same concepts and taken a little bit more more inspiration.
935
00:55:44.974 --> 00:55:50.900
That being said, I think it's also fine. I think that a lot of this code shouldn't be in the standard library. It should be in 3rd party libraries.
936
00:55:51.280 --> 00:55:53.300
You know, packaging is good enough now
937
00:55:53.600 --> 00:56:01.105
that, you know, fun c or tools or f on py can just be the standard functional programming library for Python. It doesn't really need to be baked in.
938
00:56:01.645 --> 00:56:05.265
Yeah. That was 1 thing I was going to ask about is whether you thought it would be
939
00:56:05.910 --> 00:56:15.210
useful or something that you would consider pushing for to try and get, you know, some or all of or at least some small portions of some of these
940
00:56:15.555 --> 00:56:23.630
other functional languages baked into the standard library to either augment or replace the existing func tools. But I also know that there's a
941
00:56:24.250 --> 00:56:32.335
common meme that the standard library is where modules go to die because it's a much slower rate of change, and it's not as easy to
942
00:56:34.175 --> 00:56:35.795
introduce updates or modifications
943
00:56:36.175 --> 00:56:38.994
to those modules once they do become part of the standard lib.
944
00:56:39.455 --> 00:56:39.935
I think,
945
00:56:40.335 --> 00:56:41.980
now that we have,
946
00:56:42.520 --> 00:56:45.880
better packaging tools, like, where are still imperfect but,
947
00:56:46.440 --> 00:56:47.500
where better.
948
00:56:48.040 --> 00:56:48.360
It,
949
00:56:49.080 --> 00:56:55.085
makes less sense to keep a big standard library library. So we we have obvious things
950
00:56:55.625 --> 00:56:56.525
like requests
951
00:56:56.985 --> 00:56:59.485
which is far better when building
952
00:56:59.785 --> 00:57:01.085
or or lib and
953
00:57:01.770 --> 00:57:03.390
2 or or even
954
00:57:03.690 --> 00:57:04.750
slightly fixed
955
00:57:05.290 --> 00:57:06.990
fixed in Python 3.
956
00:57:07.369 --> 00:57:15.454
But still we don't have this request in standard library and I think it's okay. The main and the main reason is is better packaging tools. And,
957
00:57:16.154 --> 00:57:17.295
this situation
958
00:57:17.920 --> 00:57:18.420
permits,
959
00:57:19.040 --> 00:57:20.180
each project
960
00:57:20.480 --> 00:57:27.974
to develop separately and usually faster. We can see how even better packaging tools like in Node. Js community.
961
00:57:28.674 --> 00:57:33.734
They lead to, like, lots of lots of packages and, they lead to smaller packages.
962
00:57:34.035 --> 00:57:39.270
So the better your packaging tools, the more and the smaller packages you have. And,
963
00:57:39.910 --> 00:57:40.570
we develop
964
00:57:40.870 --> 00:57:43.130
the faster way. We don't need to to keep
965
00:57:43.725 --> 00:57:45.105
everything in standard library.
966
00:57:45.485 --> 00:57:46.705
It's a at this stage.
967
00:57:47.325 --> 00:57:48.765
Yeah. I I agree. I think,
968
00:57:49.485 --> 00:57:53.265
the only the only value that it would serve, I think, is the value of social proofing
969
00:57:53.790 --> 00:58:13.650
that, okay, here's something that, you know, an external authority has sort of declared as good. But I think we've we've handled social proofing through things like GitHub stars. You know, both Funcie and tools are well downloaded. There are they have lots of stars. We both publish continuous integration tests, And that's sort of enough to convince people that they're worth worth doing. The other value would be with distribution.
970
00:58:14.030 --> 00:58:25.415
But I think, I mean, at least in the scientific world, you know, sort of the, you know, anaconda distribution and, you know, grain of salt, I work for this company. But the anaconda screen has become sort of a de facto standard among numeric developers.
971
00:58:25.890 --> 00:58:30.869
And, you know, requests is there. All of the libraries that you sort of think of as being standard
972
00:58:31.330 --> 00:58:36.955
are distributed along with what people download when they install Python. And you can you can trust that it will be on the user's machine.
973
00:58:37.655 --> 00:58:40.635
You guys have mentioned it a couple of times, the, fn.py
974
00:58:41.095 --> 00:58:41.595
library.
975
00:58:41.975 --> 00:58:48.369
And I'm wondering if you guys can compare and contrast that with your own work or if it makes sense to dig into that further.
976
00:58:48.829 --> 00:58:52.769
If if you don't think it makes sense to dig into that further, then I'm happy moving on to the next question.
977
00:58:53.185 --> 00:58:54.645
I looked at it and I,
978
00:58:55.025 --> 00:59:01.925
read the code of it. And, it's a near interesting library, but I think of it more like
979
00:59:02.400 --> 00:59:07.460
research project, not some practical tool. It's a bit like Haskell.
980
00:59:09.755 --> 00:59:10.255
It
981
00:59:10.555 --> 00:59:11.935
has many interesting
982
00:59:12.315 --> 00:59:13.775
things in it. And
983
00:59:14.555 --> 00:59:17.375
this is why I read it's it's code.
984
00:59:17.835 --> 00:59:18.335
But
985
00:59:19.619 --> 00:59:22.760
I never even tried to use, it's, in
986
00:59:23.619 --> 00:59:25.160
in any real code.
987
00:59:27.375 --> 00:59:29.715
I think that f n.py is a really cool project.
988
00:59:30.175 --> 00:59:33.155
It has actually a lot of really cool stuff in it, like persistent data structures,
989
00:59:33.535 --> 00:59:36.755
a lot of neat syntax tricks to make pretty lambdas and streams.
990
00:59:37.710 --> 00:59:40.589
There's for very clearly some heavy sort of scholar or Haskell inter
991
00:59:40.990 --> 00:59:41.970
influence in it.
992
00:59:42.750 --> 00:59:43.890
I would say that
993
00:59:44.285 --> 00:59:48.625
it differs from the 2 products we've written, and that its its scope is much, much larger.
994
00:59:49.085 --> 00:59:56.180
So it's it's encompassing lots of fun ideas inside functional programming, Where funsci and tools are both restricting themselves to what is clearly pragmatic,
995
00:59:56.720 --> 00:59:57.700
clearly Pythonic.
996
00:59:58.160 --> 01:00:00.820
And you can sort of insert into a workplace
997
01:00:01.345 --> 01:00:03.765
without trumpeting the sort of functional programming
998
01:00:04.385 --> 01:00:04.885
idea.
999
01:00:05.265 --> 01:00:08.720
But I think that for the conversation, we should probably have Alexei here,
1000
01:00:09.280 --> 01:00:10.099
and that's,
1001
01:00:10.560 --> 01:00:11.540
that's not present.
1002
01:00:13.040 --> 01:00:28.580
And so there are a number of concepts involved in functional programming, such as carrying function composition, immutable data, and pure functions. And we've covered a number of those, but, for any of them that we didn't cover, can you describe some of those concepts and explain any that you,
1003
01:00:28.880 --> 01:00:31.620
tried to incorporate into your libraries if we haven't already covered
1004
01:00:32.000 --> 01:00:33.540
them? I tried to incorporate
1005
01:00:34.465 --> 01:00:36.625
some practical and useful tools and,
1006
01:00:37.185 --> 01:00:39.605
still I have things like, function composition.
1007
01:00:40.065 --> 01:00:40.945
I have even,
1008
01:00:41.585 --> 01:00:42.450
helpers for
1009
01:00:43.329 --> 01:00:44.790
immutable data to use
1010
01:00:45.410 --> 01:00:47.430
common collections like dicts,
1011
01:00:48.450 --> 01:00:51.109
as immutable by convention collections
1012
01:00:51.744 --> 01:00:52.404
which I
1013
01:00:52.865 --> 01:00:57.765
I do doubt is useful but but still. So my approach not like
1014
01:00:58.065 --> 01:01:02.010
adding more concepts in the library is I don't
1015
01:01:02.390 --> 01:01:03.290
think I have
1016
01:01:03.910 --> 01:01:04.730
much more
1017
01:01:05.350 --> 01:01:08.170
than those. And current composition, maybe particles,
1018
01:01:08.925 --> 01:01:11.905
a bit immaterial data help us. That's it.
1019
01:01:12.365 --> 01:01:14.305
So I'll say a few things. So,
1020
01:01:14.765 --> 01:01:15.585
like Alexander,
1021
01:01:16.440 --> 01:01:21.260
tools has sort of the basic pure things, pure functions, core data structures, function composition, iterators.
1022
01:01:21.800 --> 01:01:29.315
But some of the concepts we haven't talked about are present, not in our libraries, but in other libraries that are accessible in the Python ecosystem. So I'll talk about those for a little moment.
1023
01:01:30.414 --> 01:01:36.240
There's things like dispatch. There's single dispatch and fun tools. There's a library called multiple dispatch that I also wrote that does
1024
01:01:36.880 --> 01:01:40.420
allows you to dispatch a function based on all the types of all the arguments,
1025
01:01:40.720 --> 01:01:43.460
which can be handy to write some generic functions.
1026
01:01:44.215 --> 01:01:45.435
There's things like macros.
1027
01:01:45.735 --> 01:01:47.435
So a macro is
1028
01:01:47.975 --> 01:01:49.195
something like a function
1029
01:01:49.575 --> 01:01:50.075
which
1030
01:01:50.535 --> 01:01:52.315
gets its full input unevaluated.
1031
01:01:52.930 --> 01:01:56.710
So for example, if I call a function f on the input 1+2,
1032
01:01:57.330 --> 01:02:05.035
the function doesn't get 1+2, it gets the evaluator result 3. So when I go into the function call f, I get the value 3, and f can work on 3.
1033
01:02:05.414 --> 01:02:08.234
Macros actually give the unevaluated result
1034
01:02:08.535 --> 01:02:09.035
1+2
1035
01:02:10.099 --> 01:02:13.000
as an abstract syntax tree to the function.
1036
01:02:13.540 --> 01:02:20.515
This can be useful if you want to make a nice plot function. You want the user to say you know, plot x equals y squared.
1037
01:02:20.895 --> 01:02:22.275
And you actually want to have,
1038
01:02:22.655 --> 01:02:25.635
that full expression being handed to the the macro
1039
01:02:26.529 --> 01:02:28.230
without being evaluated ahead of time.
1040
01:02:28.690 --> 01:02:30.630
So the common use case here is
1041
01:02:31.089 --> 01:02:35.509
the assert statement in Python. So with assert, we say assert x equals y.
1042
01:02:35.845 --> 01:02:46.119
And assert gets true or false. And if it's true, then it does nothing. If it's false, it raises an error. But if it's false and it raises an error, we actually wanna give a good error message to the user.
1043
01:02:46.500 --> 01:02:54.165
And we want to, you know, say, you know, x, you know, x, which happens at the value 3, doesn't equal y, which happens at the value 7.
1044
01:02:54.885 --> 01:03:02.470
And we
sort of gotten around this in very hacky ways. If you look at the project Pytest, Pytest actually compiles your code a little bit to do this for you.
1045
01:03:02.950 --> 01:03:05.130
There's a really cool project called code transformer
1046
01:03:05.510 --> 01:03:07.369
by Joe Jevnick and Scott Sanderson,
1047
01:03:07.910 --> 01:03:09.369
that does a lot of sort of compilation
1048
01:03:09.910 --> 01:03:12.185
of of Python functions in a really cool way.
1049
01:03:12.745 --> 01:03:18.605
It's you know, I'd never use it pragmatically, but it's it's very neat. It's worth looking at. This is, like, immutable data structures with persistence
1050
01:03:18.985 --> 01:03:21.485
and structural sharing. This is another concept.
1051
01:03:21.910 --> 01:03:23.290
So if I have,
1052
01:03:23.590 --> 01:03:28.170
say, some some data structure like a dict, and I want to change some element of it, but
1053
01:03:28.605 --> 01:03:33.184
I've already handed that dict dict off to some other process running some other thread or some other coroutine.
1054
01:03:33.805 --> 01:03:36.065
I wanna make sure that that other
1055
01:03:36.765 --> 01:03:49.885
user of that data doesn't see my change results. So in Python, currently, I'd have to make a copy. This is what Alexander was talking about, immutable by convention. But if it's it's possible with sort of very fancy underlying data structures
1056
01:03:50.425 --> 01:03:50.925
to
1057
01:03:51.225 --> 01:03:54.685
make a slight copy of a data structurally dict or a list
1058
01:03:55.080 --> 01:03:55.580
that
1059
01:03:56.120 --> 01:03:59.820
even though logically they're completely separate, these 2 copies, 1 slightly changed,
1060
01:04:00.120 --> 01:04:05.785
in memory, they actually share a lot of the same structure. So this is persistent data structures with structural sharing.
1061
01:04:06.165 --> 01:04:15.769
There's a project by Kurt Smith called Ozymandias, which is sort of new and and rocky, but it does this on the sea level. So it's it's it's quite fast. There's general thinking about types.
1062
01:04:16.470 --> 01:04:28.005
And there's, again, some nice work from the the core language team to think about, you know, adding function annotations into Python 3. There's a whole types module, allows you to spell interesting types. So some some interesting work happening there as well.
1063
01:04:28.305 --> 01:04:32.060
So just to be clear, there's lots of functional programming happening in Python.
1064
01:04:32.440 --> 01:04:32.940
And
1065
01:04:33.240 --> 01:04:37.580
what Alexander and I have done is just sort of a small part of it. There's lots of other libraries to look at.
1066
01:04:37.974 --> 01:04:40.075
Maybe not a concept, but
1067
01:04:40.375 --> 01:04:43.115
but another touch on, functional programming
1068
01:04:43.575 --> 01:04:46.155
thing that you can do in Python is,
1069
01:04:46.540 --> 01:04:47.040
like,
1070
01:04:47.500 --> 01:04:55.600
utilities to to abstract away flow control. And, in fancy, I have some of those, like, in the some for
1071
01:04:56.045 --> 01:04:56.944
handling errors
1072
01:04:57.244 --> 01:04:58.625
and, some for,
1073
01:04:58.925 --> 01:05:04.224
like, making the retries or things like that. I also have some back ports from Python
1074
01:05:04.550 --> 01:05:06.730
3. Like, in Python 3.4,
1075
01:05:07.510 --> 01:05:09.450
we have, this, suppress
1076
01:05:09.830 --> 01:05:15.215
context manager. Like, you like, we've suppressed some exception and a block which,
1077
01:05:15.855 --> 01:05:19.875
ignores this exception. You can write it in Python too also and and
1078
01:05:20.175 --> 01:05:21.155
so I beg and
1079
01:05:21.900 --> 01:05:23.260
edit a few other things,
1080
01:05:23.740 --> 01:05:25.040
like this to.
1081
01:05:25.340 --> 01:05:27.520
And about immutable data,
1082
01:05:27.900 --> 01:05:34.395
it's, like, if you have them, by default, like, in closure in all where all
1083
01:05:34.775 --> 01:05:36.475
your collections are by default,
1084
01:05:37.095 --> 01:05:39.510
immutable and they share structure.
1085
01:05:39.970 --> 01:05:46.950
This is a very different thing when when you have this immutable collections as a separate library because,
1086
01:05:47.330 --> 01:05:48.845
in this situation
1087
01:05:49.305 --> 01:05:53.705
when we have separate library, it's a very special purpose tool which,
1088
01:05:54.025 --> 01:06:00.090
very small amount of people will use. And, I think, in in Python, it will be like that.
1089
01:06:00.630 --> 01:06:03.290
That's true. But I would I would suggest that
1090
01:06:03.645 --> 01:06:07.744
as long as we adhere to some of the interfaces, like mutable mapping or immutable mapping,
1091
01:06:08.285 --> 01:06:10.704
we can pass these things around and get some more coverage.
1092
01:06:11.240 --> 01:06:16.220
So for example, I think that Python has actually done a really good job about having some shared interfaces, like the file interface,
1093
01:06:16.760 --> 01:06:18.220
the underscore iter interface.
1094
01:06:18.520 --> 01:06:20.985
In the scientific world, we've got the array interface.
1095
01:06:21.845 --> 01:06:22.745
And these allow
1096
01:06:23.125 --> 01:06:27.145
a lot of more custom applications. So many people have written array libraries.
1097
01:06:27.570 --> 01:06:30.869
And they can all interoperate because they all use these these shared protocols.
1098
01:06:31.250 --> 01:06:33.430
And so I think that with things like immutable data structures,
1099
01:06:33.730 --> 01:06:35.404
the sort of hook into the Jamet
1100
01:06:35.865 --> 01:06:40.765
ecosystem is going to be with those shared protocols. They're gonna implement mutable mapping. They're gonna implement sequence.
1101
01:06:41.464 --> 01:06:42.365
Mhmm. Agree.
1102
01:06:42.819 --> 01:06:43.059
This,
1103
01:06:43.700 --> 01:06:46.359
protocols in in Python were very beneficial.
1104
01:06:46.740 --> 01:06:48.119
Very recently with
1105
01:06:48.740 --> 01:06:51.815
a big thing in, closure called transducers.
1106
01:06:52.195 --> 01:06:52.515
And,
1107
01:06:53.235 --> 01:06:55.095
when people started to
1108
01:06:55.795 --> 01:06:59.335
make their own transducers in all sorts of languages
1109
01:06:59.940 --> 01:07:01.960
and, this includes Python.
1110
01:07:02.420 --> 01:07:05.000
So there are probably several implementations
1111
01:07:05.300 --> 01:07:08.055
of, that for Python. And then
1112
01:07:08.355 --> 01:07:09.415
if you're looking
1113
01:07:09.715 --> 01:07:10.615
a bit closer
1114
01:07:10.915 --> 01:07:15.280
to it, it, does not actually makes much sense for Python because,
1115
01:07:16.160 --> 01:07:16.660
transducers
1116
01:07:17.120 --> 01:07:17.620
for
1117
01:07:18.320 --> 01:07:19.360
closure, we solve,
1118
01:07:19.920 --> 01:07:22.260
we we solve a problem that Python
1119
01:07:22.640 --> 01:07:24.545
had solved years ago with,
1120
01:07:25.085 --> 01:07:28.945
iterator protocol and with, things like that. So, protocols
1121
01:07:29.485 --> 01:07:30.930
like iterator protocols and
1122
01:07:31.890 --> 01:07:35.829
abstract base classes like sequence. They help a lot. Agree.
1123
01:07:36.450 --> 01:07:43.785
And, what are some of the resources that you guys have found to be most helpful when trying to learn and apply different functional paradigms to your programs?
1124
01:07:45.045 --> 01:07:45.545
I'm
1125
01:07:45.845 --> 01:07:55.470
in a way self taught programmer, so I just try things. But there there are probably this talk by Rich Hickey called Simple Made Easy,
1126
01:07:56.365 --> 01:07:56.765
which,
1127
01:07:57.165 --> 01:07:57.665
advocates,
1128
01:07:58.925 --> 01:08:02.145
the use in data instead of classes and generally
1129
01:08:02.525 --> 01:08:05.665
simpler things like, things with, those
1130
01:08:06.270 --> 01:08:09.490
less amount of parts. Not sure if you heard it, but
1131
01:08:09.790 --> 01:08:15.650
you probably should. I've heard of it a few times, but I haven't yet watched it. So I'll definitely take some time to do that.
1132
01:08:15.974 --> 01:08:17.114
Yeah. It's only,
1133
01:08:17.574 --> 01:08:20.635
half an hour, and it's it's really good. Has
1134
01:08:21.015 --> 01:08:21.755
other good,
1135
01:08:22.545 --> 01:08:29.170
talks, and many of them are very very opinionated. Like, he advocates things that he
1136
01:08:29.550 --> 01:08:30.370
doesn't closure.
1137
01:08:30.805 --> 01:08:32.025
And for
1138
01:08:32.485 --> 01:08:39.225
many languages these things they don't make that much sense or don't make sense at all. But still
1139
01:08:40.019 --> 01:08:44.440
we're usually interesting. But but if I chose the single thing,
1140
01:08:44.900 --> 01:08:47.080
single talk about functional programming,
1141
01:08:47.435 --> 01:08:54.335
this will be this symbol made easy. It's also very interesting to look at, functional programming languages
1142
01:08:54.989 --> 01:08:57.170
and, at the standard
libraries,
1143
01:08:58.030 --> 01:08:58.530
like,
1144
01:08:59.150 --> 01:09:04.725
Closure and Haskell and ML, different sorts of ML. So these are my sources.
1145
01:09:05.265 --> 01:09:21.895
Yeah. I'll definitely repeat the suggestion for Simple Made Easy. I think it's a great talk. I've I I watch it every once in a while, honestly. It's a little shameful, but it's a great talk. Rich Hickey is a great speaker or he's a very opinionated speaker, and he distills his ideas very well. Whether you agree with him or not, up up to you. But SimpleMedes is certainly,
1146
01:09:23.175 --> 01:09:24.315
certainly worth reading.
1147
01:09:24.855 --> 01:09:28.155
I would I'd also like to repeat, I would say language tutorials.
1148
01:09:28.679 --> 01:09:32.139
I would say that in the last few years, there's been a good surge to
1149
01:09:32.440 --> 01:09:34.380
increase the friendliness of languages,
1150
01:09:35.960 --> 01:09:41.395
especially in the functional space. And so, you know, things like learn you Haskell. There's probably learn you an Erlang or something.
1151
01:09:42.094 --> 01:09:43.315
I would generally encourage
1152
01:09:44.094 --> 01:09:45.554
anyone who's interested in programming
1153
01:09:46.094 --> 01:09:46.594
to
1154
01:09:47.570 --> 01:09:50.230
gain a little bit of exposure to a lot of languages.
1155
01:09:50.929 --> 01:10:12.739
And this really helps you to understand much more about your own language. Just how when you learn a second natural language, like if you speak English and you learn French, it teaches a lot more about the structure of language. So you're not necessarily learning these languages to use them, but it's nice to learn go and see how that affects, you know, how you might use tornado or asyncio inside of python. Actually, in my most recent project, I was using
1156
01:10:13.535 --> 01:10:16.355
concurrent programming a lot. I sort of settled on tornado,
1157
01:10:16.895 --> 01:10:21.555
and before that I read lots of talks on go, or I listened to lots of talks on go,
1158
01:10:22.660 --> 01:10:26.680
I read all the ClosureCore async documentation. I read the asyncio documentation.
1159
01:10:28.020 --> 01:10:28.520
And
1160
01:10:29.135 --> 01:10:36.915
really by exposing yourself to a broad set of ideas from a VOD community, you really become a, I think, a much more well informed programmer. So learn functional programming
1161
01:10:37.215 --> 01:10:38.515
from many different languages,
1162
01:10:38.969 --> 01:10:40.429
but come back home and
1163
01:10:40.730 --> 01:10:43.469
write fairly normal Python. Just slightly
1164
01:10:43.929 --> 01:10:51.725
slightly different. If you are learning languages for concepts, then, I think it's better to find, the least,
1165
01:10:52.205 --> 01:10:52.705
least,
1166
01:10:53.165 --> 01:11:05.300
familiar language and with different paradigm. Like, you it's enough to learn, like, some Lisp and some ML for for functional programming. And then you better
go with some
1167
01:11:05.695 --> 01:11:21.120
logical programming and, when something like stack based, like Forth, this will benefit you more than learning 2 ml type languages or 2 lisps. This is a a bit of aside from functional programming. That's more like an inter learning concepts.
1168
01:11:21.820 --> 01:11:24.240
Yeah. I can definitely agree that learning
1169
01:11:24.745 --> 01:11:27.085
different languages of the same general
1170
01:11:27.465 --> 01:11:34.364
family aren't necessarily going to give you any new they they may give you some new ideas, but they're not going to give you any new concepts
1171
01:11:34.730 --> 01:11:34.969
or,
1172
01:11:35.610 --> 01:11:54.260
drastically improve your worldview when it comes to programming. So for instance, I've had experience in programming both Ruby and Python. They're both pretty familiar once you've started to learn 1 of them. But I've started delving a little bit into the Elixir and Erlang languages, and that's definitely an extreme paradigm shift. So it's
1173
01:11:54.560 --> 01:11:58.579
not as, easy to pick them up because of some of the functional concepts involved.
1174
01:11:59.040 --> 01:12:03.865
Yeah. I'm quite excited about Elixir, actually. That's all the time on my list to to find a project to work in.
1175
01:12:04.245 --> 01:12:08.905
Yeah. Agreed. That and Rust are the 2, the 2 biggest ones on my radar at the moment.
1176
01:12:09.620 --> 01:12:10.120
So,
1177
01:12:11.060 --> 01:12:23.364
is there anything that I didn't ask that you guys think that I should have or anything else that you guys wanted to bring up before we move on? No. I think we've covered it pretty well. Okay. Great. Yeah. Another little point to this discussion about languages.
1178
01:12:24.210 --> 01:12:24.710
Sure.
1179
01:12:25.090 --> 01:12:28.550
Sometimes, even if you look at language with which you
1180
01:12:28.850 --> 01:12:30.469
immediately understand because
1181
01:12:31.170 --> 01:12:41.025
it's similar to whatever you know already. It's, still beneficial to look through its standard library or some ecosystem or its packaging tools
1182
01:12:41.520 --> 01:12:51.265
because, this could be different or or this could be better. So besides Python, I'm also using Node. Js and I look
1183
01:12:51.565 --> 01:12:52.125
at this
1184
01:12:52.685 --> 01:12:54.225
at where package manager
1185
01:12:54.685 --> 01:12:55.565
npm and,
1186
01:12:56.205 --> 01:13:00.989
look at, how we install libraries and how we handle, dependencies.
1187
01:13:01.690 --> 01:13:04.190
And, it's, it has a
1188
01:13:04.730 --> 01:13:05.230
very
1189
01:13:05.610 --> 01:13:10.745
significant difference. Like, you in single process, you you can have, several
1190
01:13:11.045 --> 01:13:14.825
several copies of a semi library of different versions.
1191
01:13:15.210 --> 01:13:17.310
This makes you go with dependencies,
1192
01:13:17.850 --> 01:13:19.070
entirely entirely
1193
01:13:19.449 --> 01:13:25.765
our way. You have some new problems, but you have solved some problems that you have in Python. And,
1194
01:13:26.385 --> 01:13:26.885
if
1195
01:13:27.265 --> 01:13:32.940
you say about, Python and Ruby, which are some somewhat similar but you can,
1196
01:13:33.660 --> 01:13:35.600
read this, like, enumerable
1197
01:13:36.300 --> 01:13:39.360
class and all its methods. And, it is
1198
01:13:39.660 --> 01:13:53.080
interesting to read at least first time. So I will actually say 1 thing. We should probably add a disclaimer that both Alexander and I, come from a similar conceptual background. We both largely agree on we both largely agree on things.
1199
01:13:53.540 --> 01:13:58.840
I think if you had another programmer who is, say, more familiar from the Haskell community, they would have very different opinions.
1200
01:13:59.585 --> 01:14:00.085
So,
1201
01:14:00.545 --> 01:14:10.699
1 shouldn't take our our words here as being, what what functional programmers think. There's lots of functional programmers that would disagree with a lot of what we just said. Functional programming is a very broad,
1202
01:14:11.560 --> 01:14:16.300
very broad term. And a functional a Haskell program would come on to this show and say
1203
01:14:16.675 --> 01:14:18.535
a completely different set of of
1204
01:14:18.915 --> 01:14:19.975
very valid points.
1205
01:14:20.355 --> 01:14:23.895
I agree. We could have added also some object oriented
1206
01:14:24.560 --> 01:14:26.500
advocates, like, coming from Java
1207
01:14:26.800 --> 01:14:28.340
or at least from Ruby.
1208
01:14:29.360 --> 01:14:39.355
Although there are a number of functional aspects to Ruby as well and and a number of people in that community who are embracing some of those functional paradigms, much the same as people in Python are.
1209
01:14:39.920 --> 01:14:41.780
Absolutely. Functional programming is,
1210
01:14:42.320 --> 01:14:43.940
becoming much more hip these days.
1211
01:14:44.640 --> 01:14:50.864
So for anybody who wants to follow you guys and keep up to date with what you're up to, what would be the best way for them to do that?
1212
01:14:51.324 --> 01:14:52.784
Alexander, how about you go first?
1213
01:14:53.725 --> 01:14:55.105
I think it's, Twitter,
1214
01:14:55.804 --> 01:15:00.480
Hackflow. And I also have a blog which I don't write often,
1215
01:15:00.940 --> 01:15:02.880
but still it's a hack hackflow.com.
1216
01:15:04.139 --> 01:15:06.239
Okay. And, Matthew, how about you?
1217
01:15:06.865 --> 01:15:09.045
For me, so my professional website is matthewrocklin.com,
1218
01:15:10.625 --> 01:15:11.125
matthewrocklin.com.
1219
01:15:14.190 --> 01:15:26.255
You go to slash blog, and I write fairly regularly there. I just wrote a thing on testing. And there's various links to all the projects I work on. I wrote tools years ago. I, and I wrote tools years ago, but I work on lots of other projects today.
1220
01:15:26.955 --> 01:15:30.495
For tools, you go to tools, t
o 0 l z,
1221
01:15:31.070 --> 01:15:33.810
dot read the docs dot org. I'm also mrocklin@github
1222
01:15:34.590 --> 01:15:36.849
or Twitter or most things.
1223
01:15:37.469 --> 01:15:43.155
Great. So with that, I'll move on to the picks. And my pick today is going to be Datadog.
1224
01:15:43.614 --> 01:15:53.310
It's a software as a service platform that makes it really easy to get up and going with server metrics and has a number of different integrations for being able to
1225
01:15:53.690 --> 01:15:59.885
get some pretty detailed information out of a number of services you might be using. So for instance, if you're using Mongo or MySQL
1226
01:16:00.985 --> 01:16:08.260
or Nginx, you can add those integrations and start getting some pretty fine grained metrics about how those systems are running.
1227
01:16:08.960 --> 01:16:14.455
That makes it pretty easy to integrate with a pager service. So things like Opsgenie or
1228
01:16:14.935 --> 01:16:15.755
or PagerDuty,
1229
01:16:17.255 --> 01:16:21.470
as well as your chat tool that you might be using, for instance, HipChat or Slack.
1230
01:16:22.430 --> 01:16:23.010
So recently
1231
01:16:23.470 --> 01:16:29.325
added this into the environment of, where I just started working because we were using Zenos, which,
1232
01:16:29.805 --> 01:16:32.225
well, it's a good tool in its own right. It
1233
01:16:32.685 --> 01:16:36.385
doesn't seem to handle as well a cloud workload.
1234
01:16:38.050 --> 01:16:46.469
So Datadog just gave us a really quick and easy way to get some pretty solid metrics up and running and start integrating that with some of the other services that we're using.
1235
01:16:47.555 --> 01:16:51.655
And with that, I'll pass it to you, Alexander. What do you have for us for picks today?
1236
01:16:52.355 --> 01:16:53.910
I'll choose a TV show.
1237
01:16:54.390 --> 01:16:55.370
It's a an Expanse,
1238
01:16:55.910 --> 01:17:01.850
sci fi 1 because I I love everything about space. And, we don't have much
1239
01:17:02.555 --> 01:17:04.575
TV shows about space now nowadays.
1240
01:17:05.515 --> 01:17:10.735
But, and this looks good because it's based on a book. It's a it has a solid
1241
01:17:11.350 --> 01:17:11.850
story.
1242
01:17:12.310 --> 01:17:13.850
It goes great so far.
1243
01:17:14.230 --> 01:17:14.970
If you want
1244
01:17:15.430 --> 01:17:19.690
something more practical, then it could be I can type in this.
1245
01:17:20.375 --> 01:17:23.335
Okay. K. The more practical thing will will be,
1246
01:17:23.815 --> 01:17:25.594
the Revolut app. It's
1247
01:17:25.975 --> 01:17:30.790
an app which sends you bank cards which works in every country without any,
1248
01:17:32.070 --> 01:17:33.449
fees in bank accounts
1249
01:17:33.909 --> 01:17:39.530
and translates money. It's the best, cost you it it best rate you can
1250
01:17:40.535 --> 01:17:41.995
achieve. I've been traveling
1251
01:17:42.775 --> 01:17:44.955
last month and intend to travel
1252
01:17:45.335 -->
01:17:46.155
1 month
1253
01:17:46.615 --> 01:17:48.570
more, so it's very handy for me.
1254
01:17:49.210 --> 01:18:01.625
Yeah. So I've got, 3 picks. First is a library that I ran into just last week on a tools issue. It's Riemann, r I e m a n n. This is a a closure library, so sort of in inside the same theme,
1255
01:18:02.725 --> 01:18:08.230
for they got a really nice stream abstraction, and it's for monitoring distributed systems. If you go to that page, riemondotio,
1256
01:18:08.849 --> 01:18:17.885
there's a there's a video on the front, that I really enjoyed. So if you're watching this podcast, it's it's just some of these ideas. It's nice seeing some of those ideas in practice
1257
01:18:18.425 --> 01:18:21.804
in sort of real concrete problems. It's a it's a nice it's a nice video.
1258
01:18:22.120 --> 01:18:29.045
Yeah. That's a project that I've been keeping a close eye on myself with the intent of putting that into the infrastructures that I've worked
1259
01:18:29.445 --> 01:18:33.304
on. I have not yet had the opportunity to do so. I'll second that recommendation wholeheartedly.
1260
01:18:33.605 --> 01:18:38.105
Yeah. I just learned about it last week. I thought it was really cool. The second 1, more of sort of a humanities
1261
01:18:38.460 --> 01:18:42.240
pick, it was a movie 5 Dances. I think it's like a an independent film.
1262
01:18:42.700 --> 01:18:46.000
And if you're into dance or sort of human movement, it's it's really cool.
1263
01:18:46.380 --> 01:18:49.675
I think in the first 5 minutes, you'll figure out if you love it or hate it.
1264
01:18:50.295 --> 01:18:53.435
But it's just it's a bunch of dancers. There's a little bit of plot, but it's mostly them,
1265
01:18:54.215 --> 01:18:56.395
in a in a practice room going around.
1266
01:18:56.720 --> 01:18:58.900
So I think it's really cool, but, again, it's not for everyone.
1267
01:18:59.600 --> 01:19:04.420
The 3rd 3rd pick is actually a little bit self serving. What I've been working on for the last few months is a project called Distributed,
1268
01:19:05.295 --> 01:19:08.355
which is a dynamic task scheduler, so something like Make,
1269
01:19:08.895 --> 01:19:11.395
but for sort of data science, analytic operations.
1270
01:19:11.855 --> 01:19:13.635
And it's on distributed systems
1271
01:19:14.480 --> 01:19:16.179
using the concurrent futures interface.
1272
01:19:16.719 --> 01:19:19.860
So if you like concurrent futures, but you wanna use a full cluster,
1273
01:19:20.639 --> 01:19:21.540
check out distributed.
1274
01:19:21.855 --> 01:19:28.114
Again, it's sort of new. It's in beta, but I'd love to have some feedback on it, especially from, your your your viewership.
1275
01:19:29.100 --> 01:19:48.340
So that's it. I was also gonna recommend Ty, but we also mentioned that earlier on in the show. So Very cool. Well, thank you both for taking the time out of your day to join us and talk about functional programming and your work on some libraries to make that easier to do in Python. And thank you for having us. This is really fun. Yeah. Thank you too. Alright. Adios. Bye. Bye.