The Past, Present, and Future of Twisted with Moshe Zadka

WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 7/6/2024 5:13:57 PM
Duration: 2082.435
Channels: 1
1
00:00:13.184 --> 00:00:31.914
Hello, and welcome to podcast dot in it, the podcast about Python and the people who make it great. When you're ready to launch your next app, you'll need somewhere to deploy it, so check out Linode. With private networking, shared block storage, node balancers, and a 200 gigabit network, all controlled by a brand new API, you've got everything you need to scale. Go to podcastinit.com/linode to get a $20 credit and launch a new server in under a minute.
2
00:00:35.830 --> 00:00:36.730
Go to podcastinit.com/gocd
3
00:00:38.310 --> 00:00:43.450
to learn more about their professional support services and enterprise add ons, and visit the site at podcastinit.com
4
00:00:44.635 --> 00:01:04.255
subscribe to the show, sign up for the newsletter, and read the show notes. Your host as usual is Tobias Macy, and today, I'm interviewing Moshe Zadka about Twisted, the original multifunction tool for asynchronous operations and network protocols in Python. So Moshe, could you start by introducing yourself? So my name is Moshe Zadka. I've been using Python since 1998.
5
00:01:04.875 --> 00:01:06.255
I've done some core contributing,
6
00:01:06.795 --> 00:01:13.920
and I've been part of the Twixys project since its its inception around 2, 001. And do you remember how you first got introduced to Python?
7
00:01:14.300 --> 00:01:16.560
I was, looking around for a language
8
00:01:16.940 --> 00:01:20.395
which had, a good XML support with,
9
00:01:20.795 --> 00:01:22.655
validation for schemas,
10
00:01:23.275 --> 00:01:30.960
and Python was the only language that had actually good libraries that I was willing to use, so I decided to run Python.
11
00:01:31.260 --> 00:01:39.315
And for anybody who isn't familiar with the Twisted Library, can you share a brief overview of what it is and,
12
00:01:39.715 --> 00:01:51.210
what it does, and what is some of the things that people use it for? It is a library for doing event based programming, and that turns out mostly to be about network events, although not,
13
00:01:51.750 --> 00:01:55.130
uniquely. So most of the things people end up doing with Twisted
14
00:01:55.510 --> 00:01:56.810
are, network
15
00:01:57.415 --> 00:01:57.815
programs.
16
00:01:58.215 --> 00:02:06.200
That includes both network clients and servers of various sorts all the way from various web servers to, things that
17
00:02:07.240 --> 00:02:08.060
send SNMP
18
00:02:08.680 --> 00:02:09.900
commands to routers.
19
00:02:10.440 --> 00:02:12.540
And so you mentioned that you were
20
00:02:12.920 --> 00:02:21.675
part of the, team at the point when Twisted was first being created. So can you share a bit about the origin story and how you got involved with the project?
21
00:02:22.055 --> 00:02:26.040
So Glyph wanted to write a library for, game programming
22
00:02:26.340 --> 00:02:27.160
that involved,
23
00:02:27.700 --> 00:02:28.200
networks,
24
00:02:28.820 --> 00:02:29.620
and he was,
25
00:02:29.940 --> 00:02:32.120
talking about it in, channel Python
26
00:02:32.885 --> 00:02:33.785
on, IRC,
27
00:02:34.325 --> 00:02:35.705
what is now free node.
28
00:02:36.085 --> 00:02:36.565
And,
29
00:02:36.885 --> 00:02:41.705
I got involved mostly because I had some interesting ideas about how to implement, protocols,
30
00:02:42.020 --> 00:02:52.955
and I I want really wanted a place to get a chance to to to try that out. Do you know if Glyph was ever successful in actually creating the game, or did he get too far down the rabbit hole of building Twisted?
31
00:02:54.375 --> 00:02:58.635
That's actually a funny story. So the original name for, Twisted
32
00:02:59.280 --> 00:03:03.220
was because the game framework was supposed to be Twisted Reality,
33
00:03:03.920 --> 00:03:05.620
and so Twisted dot reality
34
00:03:06.080 --> 00:03:09.925
was, the subpackage that was supposed to be the game.
35
00:03:10.305 --> 00:03:20.420
If you look at Cambridge, you'll see that, reality has been deleted because it never actually got to the, a functional point. So not only did we did we not end up, writing a game,
36
00:03:20.739 --> 00:03:33.270
we didn't end up even writing the proper game support library. Just the networking part of it. As is the as is the way with such things. You, set out with the best of intentions and then get waylaid and instead
37
00:03:34.230 --> 00:03:35.770
successful networking library.
38
00:03:36.950 --> 00:03:43.265
I Gliff keeps threatening that 1 day he lights a a game. I think the code name for that is Divinal.
39
00:03:43.645 --> 00:03:50.940
So if you look at his, blog, he actually wrote something about it a while ago. As somebody who's been a maintainer of the twisted library
40
00:03:51.320 --> 00:04:03.905
for such a long time, what is your current role in the team and how many people are currently involved in keeping the, project moving forward? So I don't have a specific role, although
41
00:04:04.364 --> 00:04:13.170
I've kind of done quite a bit of preparing teaching materials on that. But, occasionally, I'll try to review patches, and, occasionally, I'll help people,
42
00:04:13.550 --> 00:04:14.370
fix bugs
43
00:04:15.065 --> 00:04:17.405
that, bother them. I think
44
00:04:18.025 --> 00:04:18.845
there's, like
45
00:04:19.385 --> 00:04:21.945
depending on how you count, anywhere from,
46
00:04:22.505 --> 00:04:23.005
5
47
00:04:23.390 --> 00:04:31.570
to 15 people who kind of semi regularly work on it. It's a very changing crew depending on who has time and who wants to do something specific.
48
00:04:32.205 --> 00:04:32.705
And
49
00:04:33.165 --> 00:04:39.825
so 1 of the things that I heard a lot when I first, learned about Twisted was that it can be difficult
50
00:04:40.580 --> 00:04:48.745
to learn and get started with and that people sometimes have difficulty wrapping their head around the programming model that it uses, particularly
51
00:04:49.205 --> 00:04:53.785
given the fact that it is so heavily asynchronous. So what are some of the
52
00:04:54.165 --> 00:04:54.665
resources
53
00:04:55.000 --> 00:05:00.380
or materials that you found to be most useful for people who want to learn and get started with using Twisted?
54
00:05:00.840 --> 00:05:08.755
So first, I'll I'll I'll say, like, what is the easiest way to get started with Twisted because this this has changed, and I think it's it's changed a lot for the better.
55
00:05:09.215 --> 00:05:12.835
So if you are starting now in 2018,
56
00:05:13.570 --> 00:05:16.790
just start using Python 3 6 with Twisted,
57
00:05:17.410 --> 00:05:22.070
and this means you can use the modern syntax support with async and await functions,
58
00:05:22.705 --> 00:05:29.570
and that will make a lot of things just a lot easier to get started with. So 1 resource that,
59
00:05:30.210 --> 00:05:41.375
Glyph and I prepared, which actually doesn't use the async and await, but using uses a very similar approach to kind of ease people up, is a tutorial we gave in PyCon 2017,
60
00:05:41.995 --> 00:05:43.935
which is obviously on PyVideo.
61
00:05:44.635 --> 00:05:45.775
We can put a
62
00:05:46.080 --> 00:05:47.460
link in the show notes.
63
00:05:47.760 --> 00:05:48.660
And we also,
64
00:05:49.920 --> 00:05:55.265
had, like, the notebook that we use to give this tutorial as a as a resource for people who don't want to watch
65
00:05:55.905 --> 00:06:04.790
a 3 hour long video but just want to go through the exercises can do that. I think this is 1 of the nicest thing. It's kind of a pre it it it tries to,
66
00:06:05.169 --> 00:06:09.190
teach from a very modern approach twisted. And I guess the other thing,
67
00:06:09.810 --> 00:06:16.045
is that somehow a lot of other people are using async programming nowadays because Node. Js
68
00:06:16.585 --> 00:06:18.765
is 1 of the most popular platforms,
69
00:06:19.390 --> 00:06:22.690
and browser JavaScript is even more popular than that.
70
00:06:23.150 --> 00:06:26.835
So I think some of it is just maybe people just
71
00:06:27.555 --> 00:06:32.134
heard that and and and kind of are a bit too scared, but it's actually easier than it looks,
72
00:06:32.435 --> 00:06:37.560
and literally everybody else is doing that. And as 1 of the early movers
73
00:06:37.940 --> 00:06:40.680
in terms of building an asynchronous
74
00:06:41.139 --> 00:06:44.680
library and doing event driven programming, do you
75
00:06:45.205 --> 00:06:46.165
ever sort of,
76
00:06:46.885 --> 00:06:52.265
start shaking your fist at people who are praising the newfangled async frameworks
77
00:06:52.760 --> 00:07:11.840
and, telling them that you've been doing it since they were in diapers? Or Well, so I I I I you know, a lot of people expected that recent people to react badly to that, but we're very happy. It feels like after, like, literally 15 years of shouting at everybody that AC is not that hard, suddenly everybody gets a message and everybody's doing that.
78
00:07:12.400 --> 00:07:19.220
So we're actually very happy to kind of finally have, proven that, this this is this is a good way to go forward.
79
00:07:19.544 --> 00:07:20.764
You have vindicated
80
00:07:21.065 --> 00:07:21.465
in your,
81
00:07:21.945 --> 00:07:24.685
in your beliefs. Enough to say yes. Definitely.
82
00:07:26.505 --> 00:07:28.730
And I know that at least
83
00:07:29.190 --> 00:07:32.250
originally, both Twisted and a number of the other asynchronous
84
00:07:32.950 --> 00:07:34.090
style libraries
85
00:07:34.470 --> 00:07:36.890
were heavily built around the notion of callbacks,
86
00:07:37.375 --> 00:07:53.030
which a lot of people will bemoan occasionally because of the fact that they can become very deeply nested, and it becomes difficult to reason about the order of operations and when things are going to fire. And I know that with a number of frameworks have moved more towards using,
87
00:07:53.409 --> 00:07:54.710
futures or promises
88
00:07:55.135 --> 00:07:56.035
that are
89
00:07:56.335 --> 00:08:04.910
a little bit easier to reason about. So has Twisted followed along on that path as well? In many ways, Twisted has led along that path. I think Twisted had deferred,
90
00:08:06.270 --> 00:08:16.865
since a lot of these did not have them. And you know what they say that, the pioneers have the arrows in their back. So deferreds were very early. And so they're pretty nice, but, definitely,
91
00:08:17.165 --> 00:08:33.024
a lot of the promises in futures, I have to say, are a little bit nicer than the diff twisted deferreds just because we had a chance to learn from the deferreds and not make some of the mistakes we made with them. But still, deferreds are pretty nice, and they really avoid the whole nesting thing with callbacks.
92
00:08:33.565 --> 00:08:34.865
And we've been doing that
93
00:08:35.245 --> 00:08:40.625
for, like, the last 15 years, so at least they're very stable. And with the recent addition
94
00:08:41.005 --> 00:08:46.290
of the asynchronous support in Python 3.5, I think was the first inclusion
95
00:08:46.910 --> 00:08:48.370
and more improvements
96
00:08:48.750 --> 00:08:50.290
with the async await
97
00:08:50.735 --> 00:08:52.275
and yield from syntax
98
00:08:52.655 --> 00:08:57.155
in the more recent versions of Python 3. Has that eased your
99
00:08:58.020 --> 00:09:01.640
workload in terms of adding new feature support to Twisted?
100
00:09:02.180 --> 00:09:04.040
And what does the integration
101
00:09:04.340 --> 00:09:10.335
between the Twisted event loop and then some of the other async engines in the Python ecosystem look like? So,
102
00:09:10.795 --> 00:09:21.459
I guess these these are 2 different questions and but but it's good that you have them together because people often confuse the 2 issues. So while ACK08 was originally invented to help the ACKO library,
103
00:09:21.920 --> 00:09:36.450
Twisted supported it since it since it began. So now I think even when Python 3.6 was in beta, we already had, some async await support. So async await await works pretty well in Twisted just as well as it works in,
104
00:09:36.990 --> 00:09:37.730
in asyncio,
105
00:09:38.510 --> 00:09:45.225
and it really has made wizard programming really nice. So in in many ways, I said that we were happy about other libraries,
106
00:09:45.685 --> 00:09:53.740
coming along. And so the the biggest way ACKO has helped us is because it's developed by Python Python Core. They're adding,
107
00:09:54.199 --> 00:09:55.660
special syntax support
108
00:09:56.040 --> 00:10:01.904
for, async programming, and we get to take advantage of it that in Twisted. So this is actually pretty nice.
109
00:10:02.285 --> 00:10:04.065
As far as asyncio integration,
110
00:10:04.764 --> 00:10:11.220
we've had the ability to run the twisted event loop on top of the asyncio event loop for quite a while,
111
00:10:11.760 --> 00:10:18.180
which means you can actually fully intermix it. We don't have the ability to do the other thing to run the asyncio
112
00:10:18.640 --> 00:10:32.120
event loop on top of the Twisted event loop. And, mostly, that means that you don't get certain features in asyncio because you can get every feature in Twisted by running it on top of the asyncio event loop. But if you, say, wanted to integrate asyncio
113
00:10:32.500 --> 00:10:33.560
with, GTK,
114
00:10:34.260 --> 00:10:47.225
then the easiest way would actually to be to have ACK running on top of Twisted and take advantage of Twisted supports for GTK. I think people are working on that. That's definitely something we plan on supporting, but we don't have that yet.
115
00:10:47.800 --> 00:10:49.899
And 1 of the things that
116
00:10:50.440 --> 00:10:55.019
I am often surprised by, and I know others are as well, is the breadth of
117
00:10:55.365 --> 00:11:03.145
protocols that are built into Twisted and that you can use for being able to tie together very disparate,
118
00:11:03.910 --> 00:11:04.970
types of operations
119
00:11:05.270 --> 00:11:06.250
within the 1,
120
00:11:06.950 --> 00:11:09.930
library of twisted. So what are some of the most
121
00:11:18.230 --> 00:11:20.490
so 1 pretty cool thing that they think,
122
00:11:21.350 --> 00:11:29.765
is now defunct, unfortunately, but, company was using Twisted to basically do router health management. So they were using,
123
00:11:30.145 --> 00:11:33.525
Twisted's ability to support SNMP to send SNMP,
124
00:11:34.305 --> 00:11:46.310
commands to a lot of routers and and and figure out what SMP is a simple network management protocol, and that's basically the native protocol most commercial routers use to report health and to manage themselves.
125
00:11:47.255 --> 00:11:58.720
And they had, like, this device that basically would manage a whole fleet of routers. And, actually, Twisted was so efficient. It would occasionally kill the Cisco routers because it would send them, requests too fast. And
126
00:11:59.579 --> 00:12:07.735
this is actually interesting because some of the support we have interested is, like, you you don't want to obviously slow down your stuff, but you want to make sure you don't kill
127
00:12:08.115 --> 00:12:11.260
anything behind you. So we have actually pretty sophisticated,
128
00:12:12.040 --> 00:12:27.870
support to make sure that you rate limit how much you send to a specific point or to a specific service. And this is always the story I tell when I introduce those tools. I I I I tell the stories of the dying Cisco routers. For the internals of Twisted, is it pure Python,
129
00:12:28.170 --> 00:12:30.750
or do you have any sort of siphonized
130
00:12:31.290 --> 00:12:31.790
or
131
00:12:32.145 --> 00:12:33.445
c bindings for
132
00:12:33.745 --> 00:12:34.805
performance improvements?
133
00:12:35.585 --> 00:12:36.085
So
134
00:12:36.465 --> 00:12:38.085
most of it is pure Python.
135
00:12:38.465 --> 00:12:40.810
There are a few little bits of c that
136
00:12:41.270 --> 00:12:50.045
basically the plan is to spin them out to a dedicated library so that we can do an optional dependency on them. I think 1 is for performance,
137
00:12:50.745 --> 00:12:58.350
but it's it's kind of an edge case. It's for a protocol very few people use, which is 1 of the reasons we want to make it an optional dependency.
138
00:12:58.730 --> 00:13:00.430
And 1 is actually for bindings
139
00:13:00.890 --> 00:13:06.030
to see functions that the usual Python library doesn't bind. I think some of the more,
140
00:13:06.810 --> 00:13:07.310
esoteric,
141
00:13:08.315 --> 00:13:10.495
unique system calls that you need to,
142
00:13:11.115 --> 00:13:15.295
do funny things with unique domain sockets to send file descriptors on.
143
00:13:15.949 --> 00:13:22.850
They're not supported by by Python natively, so we had to write our own binding. But, again, that's kind of, like, a use
144
00:13:23.470 --> 00:13:26.765
case. So the plan is to spin some of those out into,
145
00:13:27.225 --> 00:13:29.725
a dedicated library that just does the binding
146
00:13:30.185 --> 00:13:37.160
and then have Twist and do an optional optional dependency on them. 1 of the things that I've seen in a few places
147
00:13:37.620 --> 00:13:40.040
is that people recommend using Twisted
148
00:13:40.420 --> 00:13:41.335
as the
149
00:13:41.875 --> 00:13:47.815
WSGI layer for web servers in place of some of the other options such as gunicorn
150
00:13:48.195 --> 00:13:52.950
or micro wizgi because of its performance capabilities and its asynchronous nature?
151
00:13:53.650 --> 00:13:55.270
And are there any other
152
00:13:55.650 --> 00:13:56.150
places
153
00:13:56.610 --> 00:14:03.545
that you tend to use Twisted either in your own work or that you recommend others use it for where it,
154
00:14:04.084 --> 00:14:04.824
isn't necessarily
155
00:14:05.204 --> 00:14:07.305
the first thing that people reach for?
156
00:14:08.190 --> 00:14:09.330
Sure. But first,
157
00:14:09.950 --> 00:14:24.565
about the WSGI support, I actually have something to say about that. I don't think it's more performance than you WSGI, but I think I think it still has benefits because very few people will run into the performance issues. But it's actually nice to have a pure Python library,
158
00:14:25.209 --> 00:14:45.320
doing the web serving. That means that a lot of problems, sometimes people wanted to use with how it's built. For example, we have a problem at work with how it's built against SSL, and it's doing something weird. And using using Twisted for that just avoids these problems, and because Twisted was started out as a regular web server with the movie capabilities,
159
00:14:45.779 --> 00:14:48.680
it has built in ability to do SSL termination
160
00:14:49.300 --> 00:15:02.370
and to do file service. So often, you don't need to do this, thing where you put NGINX in front of your gunicorn or in front of your, USB. You can just use Twisted for what you would have used NGINX for as well as what you
161
00:15:03.170 --> 00:15:06.870
as as well as the Willgi server, and that simplifies operational complexity.
162
00:15:07.194 --> 00:15:09.214
The other thing I like to do with Twisted
163
00:15:09.595 --> 00:15:11.694
is I like to use it instead of Supervisory.
164
00:15:12.235 --> 00:15:15.649
Supervisory is certainly a respectable async support that Python 2.7 had that was so bad we had to write,
165
00:15:23.535 --> 00:15:29.235
Twisted, and that's kind of a universal acceptance that was really bad. Asyncio was written because
166
00:15:29.535 --> 00:15:35.779
async chat and async core were not very good, and, supervisor is written on top of basically a deprecated
167
00:15:36.160 --> 00:15:41.779
thing in Python 2.7 that is gone from Python 3. So instead of that, I like to use n Colony,
168
00:15:42.105 --> 00:15:48.605
which is a twisted based, process manager. And it has a bunch of nice things like how it, dynamically
169
00:15:49.199 --> 00:15:51.540
stops and starts services based on detecting
170
00:15:51.920 --> 00:15:54.019
files getting created and deleted.
171
00:15:54.480 --> 00:16:08.850
So it's pretty easy to, do dynamic things. I like to put it inside my Docker containers. So when I dock exec into the Docker container, I can literally just remove a file and have the main service shut down so I can start my own with debugging,
172
00:16:09.710 --> 00:16:12.450
on or something like that. Obviously, it's not great
173
00:16:13.230 --> 00:16:33.275
to have to do it, but in a situation where you have to do it, it's it's pretty cool to be able to do it. And what are some of the cases where Twisted is the wrong choice of library to use or examples of places that you've seen people misuse it or abuse it where they would have been better served by some other piece of technology?
174
00:16:33.895 --> 00:16:39.675
So, first of all, the simplest thing to write always is just a plain synchronous program
175
00:16:40.055 --> 00:16:40.490
that,
176
00:16:40.970 --> 00:16:42.510
has 1 thread of execution,
177
00:16:43.130 --> 00:16:46.830
starts up, does something, then does the other thing. And
178
00:16:47.290 --> 00:16:48.105
if you're using
179
00:16:48.505 --> 00:16:51.324
for no good reason, it'll be harder to debug
180
00:16:51.704 --> 00:16:56.445
than just using something with 1 thread of execution. And the other place
181
00:16:56.900 --> 00:17:03.880
that Twister would not shine in is if you have something very processor heavy, then you're probably better off just
182
00:17:04.295 --> 00:17:04.795
either,
183
00:17:05.655 --> 00:17:06.535
using some,
184
00:17:06.935 --> 00:17:13.115
some c extensions and then using threads and releasing the gill in the c extensions or using multiple processors.
185
00:17:13.549 --> 00:17:26.825
But, definitely, twisted is not for the case where where your main bottleneck is a CPU because it just just like always with Python, you can only use 1 core at a time, so it's not gonna be good use of your machine if you're CPU bound.
186
00:17:27.125 --> 00:17:33.850
And what are some of the ways that Twisted has evolved and changed over the number of years that it's been around?
187
00:17:34.309 --> 00:17:38.170
So, obviously, we've added features, removed bugs, but I think the biggest,
188
00:17:38.790 --> 00:17:40.490
evolution we've had is,
189
00:17:41.030 --> 00:17:42.809
adopting much more rigorous
190
00:17:43.655 --> 00:17:45.434
development process. So
191
00:17:45.895 --> 00:17:55.330
we've sort of gone from 1 extreme to the other. In the beginning, it was kind of cowboy, and everything got checked into right into master without any thought.
192
00:17:55.710 --> 00:17:58.289
And we've developed a very rigorous and
193
00:17:59.184 --> 00:18:01.425
exception free process called,
194
00:18:02.225 --> 00:18:04.645
the ultimate quality development system
195
00:18:05.025 --> 00:18:05.525
that
196
00:18:05.905 --> 00:18:08.804
I think by now is pretty much what everybody does.
197
00:18:09.169 --> 00:18:10.390
But back in
198
00:18:10.850 --> 00:18:16.795
2004, it was kind of an innovation because we had this strict policy of branch based development. So you always
199
00:18:17.275 --> 00:18:20.095
do any feature development on a branch, and
200
00:18:20.635 --> 00:18:23.855
you never start a feature unless it's a corresponding ticket.
201
00:18:24.200 --> 00:18:25.340
And you don't,
202
00:18:26.039 --> 00:18:30.299
you're not finished until the the unit tests that checks that you've done it correctly,
203
00:18:30.840 --> 00:18:33.500
and nothing gets merged without a peer reviewer.
204
00:18:34.184 --> 00:18:37.085
And I think most people, like, now in 2018,
205
00:18:38.024 --> 00:18:41.965
follow some approximation of that, but basic back in 2004,
206
00:18:43.260 --> 00:18:46.960
with the very limited branching capabilities that we had there,
207
00:18:47.820 --> 00:18:53.765
it was really weird to do that. And I'm very happy we did it because it made that Twisted has gotten
208
00:18:54.305 --> 00:18:56.085
much more solid over the years.
209
00:18:56.385 --> 00:18:56.885
And
210
00:18:57.185 --> 00:19:00.930
as part of the life cycle and evolution of Twisted,
211
00:19:08.450 --> 00:19:11.375
Python 3. And from casual conversations
212
00:19:11.835 --> 00:19:19.330
with you and some of the other members of the team, I understand that that was a bit fraud at times. So I don't know if you wanna talk a bit about the,
213
00:19:19.870 --> 00:19:26.770
efforts that you had to go through to get twisted to work with Python 3, particularly given the Unicode
214
00:19:27.150 --> 00:19:30.915
versus bytes issues at the network layer that is often
215
00:19:31.535 --> 00:19:32.035
problematic.
216
00:19:32.415 --> 00:19:38.250
Yeah. So I I think among other things, I think Twisted put a very good pressure on Python itself
217
00:19:38.630 --> 00:19:39.130
to
218
00:19:39.510 --> 00:19:40.890
have better transitional
219
00:19:41.510 --> 00:19:44.890
plans. For example, adding the u prefix to strings
220
00:19:45.325 --> 00:19:48.705
in Python 3 so that you can have code that's 2 and 3 compatible
221
00:19:49.325 --> 00:19:50.145
will definitely
222
00:19:50.685 --> 00:19:55.789
be a big part of it. And in general, the ability to write code that is 23 compatible.
223
00:19:56.490 --> 00:19:59.710
In the beginning, the plan was that you'll write your code
224
00:20:00.010 --> 00:20:02.545
in 2, and then you'll you'll you'll automatically,
225
00:20:03.485 --> 00:20:19.710
do 2 to 3, and you'll publish 2 libraries. And that's both operationally very complex, and I don't think anyone actually managed to follow that plan. And kind of part of a a lot of libraries, but the moment we had a hard time dealing with that. And and, eventually, I think around 3
226
00:20:20.195 --> 00:20:25.815
Python 33, Python 34, core Python started adding enough capability to make writing
227
00:20:26.115 --> 00:20:27.255
code that is both
228
00:20:27.910 --> 00:20:31.050
compatible, more realistic, as well as, the 6,
229
00:20:31.670 --> 00:20:41.904
library coming along and making it easier. And I think at that point, yes, we had a lot to think about the unicode and bytes. I think a lot of that thought process was really good. Unfortunately,
230
00:20:42.445 --> 00:20:44.065
the Internet is really complicated.
231
00:20:45.070 --> 00:20:47.650
A lot of the protocols that we support are, like,
232
00:20:48.110 --> 00:20:49.250
really, really old.
233
00:20:49.710 --> 00:20:52.690
IRC in particular has some very, very interesting,
234
00:20:53.745 --> 00:21:04.290
thoughts about the semantics of bytes. So kind of reconciling them with the Python model of bytes are for lower level things and strings are for human visible things,
235
00:21:04.670 --> 00:21:05.410
was nonprevious.
236
00:21:05.790 --> 00:21:09.970
But in general, I think that this this distinction was very good for twisted
237
00:21:10.350 --> 00:21:10.850
because
238
00:21:11.825 --> 00:21:17.925
a networking library does have to make the distinction between what is a string that is supposed to be shown to human people
239
00:21:18.385 --> 00:21:19.270
and what is
240
00:21:19.750 --> 00:21:23.530
the sequence of bytes that is supposed to be sending over the network and
241
00:21:23.830 --> 00:21:24.730
when the encoding
242
00:21:25.030 --> 00:21:34.975
information and all of that. So, overall, I think Python 3 was good for Tristan, but I think it took time for core Python to
243
00:21:35.540 --> 00:21:44.200
really think about what a realistic plan for library transition looks like. And as somebody who's been working with Twisted for so long,
244
00:21:44.565 --> 00:21:48.105
what are some of the things that you have learned in the process
245
00:21:48.565 --> 00:21:49.945
that were unexpected
246
00:21:50.485 --> 00:21:51.625
or most beneficial?
247
00:21:52.005 --> 00:21:54.470
So I think 1 of the biggest lessons I've learned
248
00:21:55.350 --> 00:21:56.409
is how useful
249
00:21:57.110 --> 00:21:58.730
really, really good unit tests
250
00:21:59.030 --> 00:22:01.929
are. In in in many in many aspects of Twisted,
251
00:22:02.575 --> 00:22:10.115
I've I've kind of learned this lesson over and over again. And, actually, the last point about this, the transition to Python 3 is really interesting.
252
00:22:10.840 --> 00:22:14.460
Part of transition into 3 was actually writing a lot of unit tests
253
00:22:14.840 --> 00:22:24.505
to make sure that we actually do run correctly on Python 3. So all the things that were in the past, especially the old code, it wasn't very well covered by unit tests,
254
00:22:24.885 --> 00:22:28.850
writing unit tests for that and finding a lot of bugs. I I I really think
255
00:22:29.230 --> 00:22:30.450
that Twist is really
256
00:22:31.070 --> 00:22:31.570
hammered
257
00:22:32.030 --> 00:22:36.770
well in me the concept of unit tests are really, really important for code quality.
258
00:22:37.855 --> 00:22:45.955
And 1 of the things that I found interesting in a previous episode that I did with Glyph about the
259
00:22:46.380 --> 00:22:50.059
event machine library that he wrote as a,
260
00:22:50.460 --> 00:22:53.679
corollary to the work you're doing with Twisted is that,
261
00:22:54.345 --> 00:23:01.725
by virtue of adding that event machine capability, you assessed out a few more bugs that didn't get caught by the unit tests. So are there any other,
262
00:23:02.550 --> 00:23:03.850
additional libraries
263
00:23:04.150 --> 00:23:20.725
or tangential projects that have been spawned either as a result of the work on Twisted or to support the work that you're doing? So I think 1 of the things that has been spawned, but I think it it never really it it so far, it hasn't really solidified.
264
00:23:21.570 --> 00:23:25.269
I keep hoping that someone will have time to work on it. It's a pure
265
00:23:25.649 --> 00:23:26.950
Python SSL implementation
266
00:23:27.250 --> 00:23:36.855
that is based on very declarative ideas. Automatt would be a part of it, but a big part of it would be how you declare the structure, the various structures that SSL
267
00:23:42.330 --> 00:23:43.470
lot of the historical
268
00:23:44.095 --> 00:23:47.075
a lot of the historical security flaws in open SSL,
269
00:23:47.775 --> 00:23:56.700
come from the fact that 2 parts, in the code don't don't reveal the interpretation of a certain sequence of bytes. I I think we actually had something that,
270
00:23:57.240 --> 00:23:57.740
started
271
00:23:58.120 --> 00:24:05.085
that, but it turns out to be a really hard project. And that actually spawned, like, a bunch of us kind of going off
272
00:24:05.545 --> 00:24:06.685
and kind of researching
273
00:24:07.145 --> 00:24:11.860
how how to how to do protocols correctly and how to implement protocols correctly
274
00:24:12.240 --> 00:24:17.780
and kind of it sparks a lot of interest in things like TLA plus, you know, the, to this community.
275
00:24:18.705 --> 00:24:21.445
So I I I guess it's it kind of relates to the
276
00:24:22.065 --> 00:24:23.285
to to unit tests,
277
00:24:23.665 --> 00:24:27.809
in the sense that traditionally in computing, there are 2 ways to,
278
00:24:28.210 --> 00:24:37.765
improve the correctness of something. You can test it a lot, and unit test is 1 way of testing it, or you can prove it correct. And automatic in many ways is part of the other
279
00:24:38.465 --> 00:24:50.620
the other tradition of of proving things correct. And I think we've we've kind of gotten so good at unit tests. We've reached the limit of how well they can make your code higher quality, and now we've started to look into
280
00:24:51.240 --> 00:25:01.605
what kind of things we can get from the, from the other traditional stream, from the from the side of proving things correct, from the making sure that things can't be,
281
00:25:01.925 --> 00:25:03.545
can't be incorrect by definition.
282
00:25:04.170 --> 00:25:10.910
And your discussion about doing protocols correctly also brings up some of the recent movement
283
00:25:11.485 --> 00:25:12.545
of decoupling
284
00:25:12.925 --> 00:25:13.905
protocol implementations
285
00:25:14.525 --> 00:25:22.290
from the IO aspects. So I'm wondering how much of that is present and twisted where the protocol implementation
286
00:25:22.750 --> 00:25:23.570
can be taken
287
00:25:24.110 --> 00:25:25.970
on its own without necessarily
288
00:25:26.270 --> 00:25:30.929
having to tie into the asynchronous event loop that is the core of Twisted.
289
00:25:32.485 --> 00:25:36.725
So there's a lot of, side to that. So when we started writing the,
290
00:25:37.045 --> 00:25:48.029
protocol, I think the the big the big plan was to actually make them what would now be called San Diego. And some of the simpler protocols are actually written in San Diego.
291
00:25:48.365 --> 00:26:01.470
But, unfortunately, it's just too easy, the protocol implementation, to kind of reach into the event loop and do something, and sometimes that makes, you know, it it it it's very at the moment, it looks like a good idea. So a lot of the protocols
292
00:26:02.010 --> 00:26:02.510
unwisely,
293
00:26:03.290 --> 00:26:07.595
reach into the event loop. So we do have we we do have some plans around that.
294
00:26:07.995 --> 00:26:09.935
Among them, there's, cubes,
295
00:26:10.555 --> 00:26:11.935
which is a library that's,
296
00:26:12.715 --> 00:26:18.269
written kind of on top of, but you could actually decouple it and use it, however you
297
00:26:18.570 --> 00:26:21.390
want, which is supposed to be this hierarchical
298
00:26:21.850 --> 00:26:22.909
protocol framework,
299
00:26:23.955 --> 00:26:25.815
that is really supposed to just,
300
00:26:26.355 --> 00:26:33.210
be bytes in, bytes out and not think about event loop. And the other thing is that we are really happy to throw away
301
00:26:33.750 --> 00:26:37.770
all the protocol parsing code we have. We have a really good library.
302
00:26:38.230 --> 00:26:42.524
So we're pretty much in the process of throwing away a lot of our HTP implementation
303
00:26:43.225 --> 00:26:44.585
in favor of the,
304
00:26:45.065 --> 00:26:46.345
projects from the,
305
00:26:46.745 --> 00:26:47.885
hyper Python,
306
00:26:48.985 --> 00:26:51.960
Python project. So h 2 and h 1 1.
307
00:26:52.580 --> 00:26:55.320
I think we we never had an implementation
308
00:26:55.620 --> 00:27:01.275
of, h t p 2, and we just decided to not have an implementation of h t p 2. We use,
309
00:27:01.895 --> 00:27:07.995
h 2, and we the way that's visit supports h h t p 2 is just by using the San Diego library.
310
00:27:08.670 --> 00:27:12.770
And we're in the process of basically removing the h t p 1 1 implementation,
311
00:27:13.630 --> 00:27:17.010
again, in favor of just using the, h 1 1 library.
312
00:27:17.390 --> 00:27:20.915
So I think the biggest plan is that these things just don't belong in Twisted,
313
00:27:21.695 --> 00:27:22.095
and,
314
00:27:22.575 --> 00:27:29.030
Twisted will happily use any San Diego library that it can in in favor of instead of writing its own implementation.
315
00:27:29.650 --> 00:27:40.995
I I think in the old days, it was harder, but, Python packaging has gotten a lot better nowadays, so we don't have any problem just depending on whatever libraries we need. Given how long Twisted has been around,
316
00:27:41.455 --> 00:27:43.394
it's easy to sometimes
317
00:27:44.160 --> 00:27:46.100
allow it to fade into the background
318
00:27:46.720 --> 00:27:56.615
and forget that it is there and has played such an important role in the Python community for a number of years. So what are some of the places that people
319
00:27:57.315 --> 00:27:58.535
are using Twisted
320
00:27:59.100 --> 00:28:00.000
that aren't necessarily
321
00:28:00.860 --> 00:28:04.639
the first thing that people think about or that are maybe not quite as visible?
322
00:28:05.580 --> 00:28:07.759
So it's it's always these open source projects.
323
00:28:08.245 --> 00:28:13.525
Tracking usage is is a little bit tricky. So often, the only way we know that people use it is,
324
00:28:13.845 --> 00:28:17.610
either they report the part of the mailing list or they,
325
00:28:17.930 --> 00:28:19.770
hire 1 of us to help them.
326
00:28:20.170 --> 00:28:21.870
So I don't do security.
327
00:28:22.385 --> 00:28:30.800
Pretty much use uses Twisted for all their web facing stuff, so including, like, the, actual security stuff. And,
328
00:28:31.340 --> 00:28:33.660
Apple uses Twisted for their,
329
00:28:33.980 --> 00:28:42.245
calendar server. So if you have an iPhone and your iPhone has a calendar on it and it's correct, then Twister is involved in that story.
330
00:28:42.545 --> 00:28:44.645
And the other the the the newest,
331
00:28:45.105 --> 00:28:57.865
user of Twister is Metrics, which is supposed to be the kind of new replacement to IRC and and, like, really more than I am, decentralized, and all the good stuff. And among other things, the French government,
332
00:28:58.645 --> 00:29:02.665
have decided to to standardize the metrics to be the internal governmental,
333
00:29:03.445 --> 00:29:10.389
communication tool. So now Twist, I guess, is gonna be in charge of, securing all of France's government's con con communications.
334
00:29:11.490 --> 00:29:12.635
It's a big responsibility.
335
00:29:13.255 --> 00:29:13.755
Yeah.
336
00:29:15.495 --> 00:29:19.434
And what are some of the ways that people can support
337
00:29:19.880 --> 00:29:23.660
the twisted project and the people working on it or get involved
338
00:29:24.280 --> 00:29:27.420
and help to continue to bring it forward?
339
00:29:27.764 --> 00:29:31.065
So there's a few things. 1st, we're always,
340
00:29:32.164 --> 00:29:33.384
seem to be bottlenecked
341
00:29:33.764 --> 00:29:35.544
on, having good reviews.
342
00:29:36.050 --> 00:29:38.470
As I said, we don't allow any codes,
343
00:29:38.930 --> 00:29:57.529
into the master branch without without the peer review, and that is a good way to get in because often people are a little bit worried about writing code for the project. But even if you're completely new to Python, just making sure that the code is clear and that that it's obvious that it's doing the right thing is is is great. So just
344
00:29:57.830 --> 00:29:58.809
hoping a review
345
00:29:59.429 --> 00:30:01.850
is is is super useful to the project.
346
00:30:02.295 --> 00:30:09.195
Of course, if you want to fix bugs and twisted, we're more than happy to to help. The the mailing list and IRC channel
347
00:30:09.790 --> 00:30:10.530
was always,
348
00:30:10.990 --> 00:30:17.250
some of us happy to to guide if there's an issue. And the other thing is if you work for a company that use Twisted
349
00:30:17.934 --> 00:30:20.115
and they would like to support us financially,
350
00:30:20.654 --> 00:30:24.434
Twisted has a record of using financial contributions very efficiently.
351
00:30:24.850 --> 00:30:27.750
A few tens of 1, 000 of dollars a year
352
00:30:28.050 --> 00:30:30.550
is usually enough for us to manage to find,
353
00:30:31.010 --> 00:30:33.110
someone to work, on Twisted
354
00:30:33.490 --> 00:30:37.215
for a year. And, basically, when we get that kind of funding,
355
00:30:37.595 --> 00:30:38.495
JP Calderon
356
00:30:38.795 --> 00:30:42.415
and Amber Brown have done it in the past, and they've done, like,
357
00:30:43.100 --> 00:30:45.600
really a lot of stuff on cleaning up,
358
00:30:46.380 --> 00:30:59.775
twisted and making it much better. So if you if you do have a if you are working for a company that use Twisted and and you they want to become a Twisted sponsor, we use money very efficiently, and we are tax deductible.
359
00:31:00.160 --> 00:31:03.860
And what are some of the plans that you have for the future of Twisted?
360
00:31:04.320 --> 00:31:07.780
So some of the plans are really, really good integration with asyncio.
361
00:31:08.534 --> 00:31:10.154
We've already shown some,
362
00:31:10.615 --> 00:31:20.130
some proof of concept about, like, literally running the a AIO HTTP server on top of the recent event loop, which is useful for some use cases.
363
00:31:20.670 --> 00:31:21.809
But we want to do,
364
00:31:22.430 --> 00:31:30.034
much deeper integration, so a lot of people are working on that. And then 1 of the other things where we keep working on is actually taking
365
00:31:30.335 --> 00:31:31.634
bits from
366
00:31:31.934 --> 00:31:35.410
like I said, Twisted, in the old days, packaging was really hard,
367
00:31:35.890 --> 00:31:37.590
so Trisid became very big because,
368
00:31:38.290 --> 00:31:43.430
we didn't want to break it into multiple packages because every extra dependency was a huge pain.
369
00:31:43.895 --> 00:31:45.115
Now with modern PIPI,
370
00:31:45.735 --> 00:31:46.235
dependencies
371
00:31:46.695 --> 00:31:48.955
are are a breeze. So we try to take
372
00:31:49.335 --> 00:31:52.155
bits of Twisted that could stand alone as a library
373
00:31:52.510 --> 00:31:56.289
and break them out into their own library. We move the code from Twisted
374
00:31:56.669 --> 00:31:59.090
and and and have Twisted depend on them.
375
00:31:59.549 --> 00:32:03.515
So a couple of the things we've done it with is the URL support.
376
00:32:04.054 --> 00:32:19.565
Now it's, a hyper project called hyperlink, and now it really doesn't have any URL support. It just use a hyperlink URL support, and we hope to basically, eventually, not have anything that's really cold twisted. Maybe it's gonna be a meta package that just depends on a lot of packages,
377
00:32:20.025 --> 00:32:25.885
but, that's gonna be a slow process. But every single bit we can break out and make into its own library,
378
00:32:26.679 --> 00:32:32.700
end up being a huge game. So that's kind of 1 of the long term plans for, how this is going to evolve.
379
00:32:33.355 --> 00:32:36.655
And are there any other aspects of Twisted
380
00:32:37.035 --> 00:32:38.735
or your work with the project
381
00:32:39.195 --> 00:32:55.215
or anything related to it that we should discuss further before we start to close out the show? No. I think we covered everything I had to say. Alright. Well, for anybody who wants to keep in touch with you and follow the work that you're doing and that is happening with Twisted. I'll have you add your preferred contact information to the show notes.
382
00:32:55.674 --> 00:33:08.490
And with that, I'll move us on into the picks. And this week, I'm going to choose the Leatherman wave plier tool. I've been eyeballing it for a while now, and I finally picked 1 up. And it's a pretty great little multi tool.
383
00:33:08.870 --> 00:33:11.515
It's pretty compact, has a lot of great little
384
00:33:12.054 --> 00:33:15.595
tools built into it, and I've been enjoying using that. Actually,
385
00:33:15.895 --> 00:33:24.290
came in handy a few times last week when I was building a deck, so, it's great to have on hand. So with that, I'll pass it to you, Moshe. Do you have any picks for us this week?
386
00:33:24.750 --> 00:33:26.210
Yes. So I wanted to,
387
00:33:27.005 --> 00:33:28.465
recommend 1 of my favorite,
388
00:33:28.845 --> 00:33:29.825
online books,
389
00:33:30.205 --> 00:33:37.460
called unsung book. It's kind of hard to describe, but the easiest way I found to describe it, it's urban fantasy,
390
00:33:38.320 --> 00:33:40.660
but based on, Jewish mysticism
391
00:33:41.200 --> 00:33:43.059
as the fantasy layer. And
392
00:33:43.360 --> 00:33:49.505
it's a really great book, and I think, not to spoil it too much, but I think that's a really good bit there for programmers.
393
00:33:50.045 --> 00:34:14.135
So, I highly recommend if anyone finds that kind of stuff fun, they should check out the book. Alright. Well, thank you very much for taking the time to join me today and discuss the work that you're doing with Twisted. It's a very interesting library and 1 that has been a crucial piece of the Python ecosystem for a long time. So thank you for that, and I hope you enjoy the rest of your day. Yep. Thank you. Too, Jackie. And, bye.