Add Configuration Best Practices To Your Application In An Afternoon With Dynaconf

WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 7/6/2024 5:04:25 PM
Duration: 3839.378
Channels: 1
1
00:00:00.080 --> 00:00:11.144
Hello, and welcome to podcast dot in it, the podcast about Python and the people who make it great. When you're ready to launch your next app or want to try a project you hear about on the show, you'll need somewhere to deploy it. So check out our friends over at Linode.
2
00:00:11.844 --> 00:00:19.145
With their managed Kubernetes platform, it's easy to get started with the next generation of deployment and scaling powered by the battle tested Linode platform,
3
00:00:19.510 --> 00:00:25.530
including simple pricing, node balancers, 40 gigabit networking, and dedicated CPU and GPU instances.
4
00:00:26.310 --> 00:00:33.885
And now you can launch a managed MySQL, Postgres, or Mongo database cluster in minutes to keep your critical data safe with automated backups and failover.
5
00:00:34.185 --> 00:00:35.405
Go to python podcast.com/linode
6
00:00:36.800 --> 00:00:48.295
today to get a $100 credit to try out their new database service, and don't forget to thank them for their continued support of this show. Your host as usual is Tobias Macy. And today, I'm interviewing Bruno Rocha about Dynaconf,
7
00:00:48.675 --> 00:00:54.730
a powerful and flexible framework for managing your application's configuration settings. So, So, Bruno, can you start by introducing yourself?
8
00:00:55.190 --> 00:01:00.649
Hi. Thanks for inviting me for this podcast. My name is Bruno Rocha. I live in Portugal.
9
00:01:01.190 --> 00:01:05.215
Some things about me, I'm a member of the Python Software Foundation.
10
00:01:05.755 --> 00:01:08.494
I'm a senior engineer on Red Hat Ansible.
11
00:01:08.955 --> 00:01:11.295
I have a YouTube and a Twitch stream channel
12
00:01:11.950 --> 00:01:16.930
targeting mostly Portuguese speaking audience where I talk about Python and programming.
13
00:01:17.630 --> 00:01:23.455
And besides that, I like bicycles. That's all about me. And do you remember how you first got introduced to Python?
14
00:01:24.075 --> 00:01:28.255
Oh, yeah. Let me remind this. It was around 2, 003.
15
00:01:29.970 --> 00:01:31.350
I worked it in a factory,
16
00:01:31.730 --> 00:01:35.510
and my job was maintenance of 100 of Linux desktops.
17
00:01:36.530 --> 00:01:38.855
And we use a Linux distribution
18
00:01:39.395 --> 00:01:39.895
called
19
00:01:40.995 --> 00:01:43.335
Linux, which is a Brazilian Linux distribution.
20
00:01:44.035 --> 00:01:47.920
And 1 of the features of that Linux was called Magic Icons.
21
00:01:48.300 --> 00:01:50.640
So it was like a GUI application
22
00:01:51.260 --> 00:01:54.720
with buttons where user can click and run
23
00:01:55.385 --> 00:01:57.725
bash scripts or things to automate Linux
24
00:01:58.345 --> 00:02:00.045
stuff. And then I
25
00:02:00.585 --> 00:02:04.940
work it written those scripts in bash and pearl for a long
26
00:02:05.320 --> 00:02:10.860
time and adding more day to day off c activities to those icons like merging files,
27
00:02:11.655 --> 00:02:16.474
printing and doing all the offsy day to day job using Pearl basically.
28
00:02:17.015 --> 00:02:23.840
But at some point I started contributing. I think that's was actually my first open source project that I was a contributor.
29
00:02:24.620 --> 00:02:26.319
And when I started contributing
30
00:02:26.620 --> 00:02:29.840
back to the, magic icon system on on Linux,
31
00:02:30.545 --> 00:02:36.245
I found that they were moving from Perl to Python. And then I had to learn Python to rewrite
32
00:02:36.625 --> 00:02:51.025
those script bits, that distribution. Today, that distribution is no more tainted, but that was a good experience. And that's why I I first learned Python. And from that point, I stopped using Perl and then kept using Python for everything. Yeah.
33
00:02:51.325 --> 00:03:03.500
That brings us to now the topic at hand, which is the DynaCon project. I'm curious if you can share a bit about what it is and some of the story behind how it got started and why you decided to build this project.
34
00:03:04.055 --> 00:03:10.715
Well, so yeah. Dynacomf is is a library that I like to define it as a settings client.
35
00:03:11.290 --> 00:03:11.790
So
36
00:03:12.330 --> 00:03:16.350
settings usually are defined in multiple locations like
37
00:03:16.730 --> 00:03:23.345
static files. You can have a py file, toml, yaml, JSON, or any of the other common formats
38
00:03:23.725 --> 00:03:26.225
usually you do for application settings.
39
00:03:26.930 --> 00:03:28.950
The settings is overridable
40
00:03:29.330 --> 00:03:30.870
by environment variables
41
00:03:31.330 --> 00:03:36.655
and even by external servers like in memory database or secrets vault.
42
00:03:37.115 --> 00:03:40.335
And then at some point you need to gather all those information
43
00:03:40.635 --> 00:03:47.599
in a single object. And this object is DynaConf like DynaConf is a I like to think it as a settings client
44
00:03:48.299 --> 00:03:49.519
that I can configure
45
00:03:49.819 --> 00:04:08.770
and point to multiple sources. And then it gives us a single settings object, a single single Python object where you can access and deal with those settings objects. So that is basically DynaConf and and if I want to if I can explain it simply, it's like it's a library to access settings.
46
00:04:09.295 --> 00:04:10.995
However, the settings are located,
47
00:04:11.295 --> 00:04:16.355
then it brings you a a bridge for you to access and and get those settings together.
48
00:04:17.190 --> 00:04:20.090
And the other question was the story behind it. So
49
00:04:20.710 --> 00:04:27.135
I work at in a company that was running Django and Flask application on the Cloud on AWS.
50
00:04:27.995 --> 00:04:29.135
It was 2014
51
00:04:31.195 --> 00:04:35.294
and the main website in API, they use it load balance
52
00:04:35.800 --> 00:04:37.020
and auto scaling.
53
00:04:37.400 --> 00:04:38.220
So eventually,
54
00:04:38.760 --> 00:04:42.675
there were 100 of EC 2 instances running on the cloud.
55
00:04:43.155 --> 00:04:52.055
And they wanted to be able to share the same settings with all those instances that were running. So you can think like applications will start having
56
00:04:52.569 --> 00:04:53.389
on the environment
57
00:04:53.849 --> 00:05:07.435
or hard coded settings like in a constant file. But what they wanted in that project is to have a single place where the settings is stored, like an s 3 or any other shared file system.
58
00:05:07.815 --> 00:05:14.680
And all the instance could read from the same place, the settings. So this was like the first mission we we did.
59
00:05:15.220 --> 00:05:15.960
And then
60
00:05:16.340 --> 00:05:19.720
the second thing that we did in the project was like
61
00:05:20.145 --> 00:05:21.925
turning the settings dynamic.
62
00:05:22.305 --> 00:05:25.605
So for that time, we use it already
63
00:05:26.225 --> 00:05:26.725
instance
64
00:05:27.350 --> 00:05:29.770
and already is stored key values
65
00:05:30.150 --> 00:05:34.010
and then all the instances read key values from the ready's instance
66
00:05:34.389 --> 00:05:35.690
inside the AWS
67
00:05:36.070 --> 00:05:36.570
network.
68
00:05:37.235 --> 00:05:38.534
So that's how
69
00:05:38.835 --> 00:05:45.095
the library started. So I just went to the main project and I created a folder called dynamic
70
00:05:45.634 --> 00:05:46.134
config.
71
00:05:46.720 --> 00:05:52.979
And then I started like coding using Python dynamic features like overriding get attributes,
72
00:05:53.365 --> 00:05:55.305
trying to do like lazy evaluation.
73
00:05:55.765 --> 00:06:00.745
And then I ended up with something really nice which we were able to
74
00:06:01.080 --> 00:06:07.340
use it to read files locally. If it was not found locally, it would fall back to this to the environment
75
00:06:07.880 --> 00:06:09.820
and then to external services.
76
00:06:10.275 --> 00:06:15.254
So we decided to like promote it to, to a package. And then
77
00:06:15.555 --> 00:06:25.409
instead of dynamic config, we shorted a little bit to Dyna Conf and created a package. It started being used by all the projects, all the internal projects on that company.
78
00:06:25.710 --> 00:06:26.930
So it was 2014
79
00:06:27.949 --> 00:06:28.449
Heroku
80
00:06:29.485 --> 00:06:29.985
released
81
00:06:30.285 --> 00:06:32.705
a guide called 12 factor apps.
82
00:06:33.164 --> 00:06:35.585
And 12 factor apps was like
83
00:06:35.965 --> 00:06:39.639
our main drive for deciding how the library would be written.
84
00:06:40.099 --> 00:06:43.240
Because after this first version, we did many refactorings
85
00:06:44.020 --> 00:06:47.800
and we base it a lot on 12 factor guide.
86
00:06:48.225 --> 00:06:51.365
The 3rd topic on the 12 factor is the config
87
00:06:51.745 --> 00:07:00.099
and that says that configuration must belong to environment and then we started like coding DynaConf to be like environment first.
88
00:07:00.400 --> 00:07:01.539
And from that point,
89
00:07:02.159 --> 00:07:07.695
we we added we added many other features. But that's how it started. It started, like,
90
00:07:08.315 --> 00:07:12.335
just as a Python model inside a big monolithic project,
91
00:07:12.875 --> 00:07:16.460
dynamic config, and I remember that we pulled that to
92
00:07:17.340 --> 00:07:20.000
off that project and created the Dynacom.
93
00:07:20.780 --> 00:07:22.160
And yeah. Now
94
00:07:22.539 --> 00:07:30.905
it grow like it's no more like a single small library. It's more like you said on the beginning of the show. It's a framework for managing settings
95
00:07:31.365 --> 00:07:35.520
together with this settings client. So, yeah, that's a little bit of the time.
96
00:07:35.900 --> 00:07:37.120
Are there any particular
97
00:07:37.740 --> 00:07:40.160
goals or kind of design philosophies
98
00:07:40.620 --> 00:07:41.120
or
99
00:07:41.895 --> 00:07:46.555
strong opinions about how to manage settings that you have for DynaCon?
100
00:07:47.335 --> 00:07:50.395
The fact is that the project is evolving organically.
101
00:07:50.840 --> 00:07:54.300
So it's based on the needs and feedback of the users and contributors.
102
00:07:54.840 --> 00:07:55.980
So we are constantly
103
00:07:56.760 --> 00:07:57.980
adding new things
104
00:07:58.360 --> 00:08:00.380
and fixing things based on that.
105
00:08:00.705 --> 00:08:03.365
So we don't have, like, a really strict guideline
106
00:08:03.905 --> 00:08:04.405
on
107
00:08:05.425 --> 00:08:07.525
on the style that we are following.
108
00:08:08.225 --> 00:08:13.740
I can say that the goal as a project is to be reliable and simple to use.
109
00:08:14.280 --> 00:08:22.645
So I all see that I want Dynacomf to be a go to choice when you think about how to manage settings on your Python application.
110
00:08:23.265 --> 00:08:25.605
I already see that on the companies
111
00:08:26.225 --> 00:08:40.785
and communities where I participate because I try to influence people sometimes to use the library and they say, yeah, we need to deal with settings. So it's gonna be like, just put Dynacom here. And to be able to be like a go to choice and simple,
112
00:08:41.165 --> 00:08:42.945
you have to be very
113
00:08:43.405 --> 00:08:44.225
like frictionless.
114
00:08:44.900 --> 00:08:49.960
So the project should be something you can plug into your project
115
00:08:50.340 --> 00:08:59.355
and you can unplug from your project easily. So I think this is a thing that we try to keep on Dynacomf. You can take your project right now, add Dynacomf
116
00:08:59.735 --> 00:09:09.579
and it works. And if you need or if you don't like Dynacomf anymore, you can just remove it. And it's not going to be a hard work because the way Dynacomf is implemented.
117
00:09:10.495 --> 00:09:19.180
As far as the kind of types or categories of projects that you have in mind as you build and iterate on Dynaconf, are there any particular ecosystems
118
00:09:19.880 --> 00:09:25.180
or use cases that you are more focused on or that tend to work better with Dynacon?
119
00:09:25.945 --> 00:09:26.605
Yeah. Definitely.
120
00:09:26.985 --> 00:09:31.165
I think web applications running on cloud is the main target.
121
00:09:31.465 --> 00:09:35.005
So as we base it on 12 factor app, I think
122
00:09:35.360 --> 00:09:36.820
whenever you have environment
123
00:09:37.600 --> 00:09:40.259
and you have this environment varying
124
00:09:40.560 --> 00:09:42.180
across multiple deployments,
125
00:09:42.615 --> 00:09:49.194
I think Dynacomf is a very good fit for it. So it can be running on virtual machine, containers, Kubernetes,
126
00:09:49.495 --> 00:09:54.279
everything about the 12 factor apps, which is like payment for SaaS applications.
127
00:09:54.660 --> 00:10:02.665
I think it's a perfect fit for Dynacom. But we have users using Dynacom for other projects. So have people and contributors
128
00:10:03.125 --> 00:10:07.625
running for machine learning pipelines. So they configure all the models
129
00:10:08.200 --> 00:10:09.100
using Dynacomf
130
00:10:09.640 --> 00:10:11.660
loading of of Dynacom files.
131
00:10:12.040 --> 00:10:18.305
We have CLI applications that people create and then the end user of the CLI application
132
00:10:19.245 --> 00:10:21.345
like puts a settings file together,
133
00:10:21.725 --> 00:10:29.440
in the folder they are running the CLI and DynaCon floats this the local settings. We have desktop applications and testing frameworks.
134
00:10:29.980 --> 00:10:30.480
So
135
00:10:30.975 --> 00:10:45.350
the first focus was cloud applications, but people users starts to be, like, more creative in using it, whatever you you imagine. So I think DynaConf can fit on many cases, but the main target
136
00:10:45.650 --> 00:10:48.470
is when we are looking to, like,
137
00:10:48.845 --> 00:10:52.145
change things, when we are looking to add new
138
00:10:52.525 --> 00:10:56.785
functionalities, we always think about the cloud first. Yeah.
139
00:10:57.339 --> 00:10:58.720
As far as the
140
00:10:59.740 --> 00:11:04.240
category of settings management for applications, it's 1 of those things that is
141
00:11:04.985 --> 00:11:14.850
seemingly very simple, but is deceptively complex as you dig deeper into the problem space where at the outset, it's just, oh, I just need a file that says to, you know, set these different values,
142
00:11:15.630 --> 00:11:23.795
as you were describing, move into different environments, or you need to be able to support different run times or kind of application structures,
143
00:11:24.415 --> 00:11:35.779
you start to realize, oh, now I need to be able to pull things from multiple places. I need to be able to manage things like overrides or, you know, how do I manage secrets? And I'm wondering what are some of the
144
00:11:36.399 --> 00:11:47.665
kind of general philosophies that you've encountered about the so called, quote, unquote, right way to do it and some of the ways that you think about what philosophies are best suited for
145
00:11:47.980 --> 00:11:57.440
applications that you're trying to kind of encode into how Dynacon is able to manage some of those things like overlays and environment specific attributes and secrets,
146
00:11:58.045 --> 00:11:59.345
etcetera. I think that
147
00:11:59.725 --> 00:12:05.665
settings is like an accessory for your application. It's not the main focus of any application.
148
00:12:06.204 --> 00:12:10.450
So your application should focus on the things it should resolve.
149
00:12:11.150 --> 00:12:11.650
And
150
00:12:12.270 --> 00:12:12.770
as
151
00:12:13.470 --> 00:12:18.235
less you see the settings the best because it will be, like, transparent
152
00:12:18.855 --> 00:12:26.570
and it's not, like, being on your way from you and your code. So I think 1 of most complex thing
153
00:12:27.110 --> 00:12:30.010
to do when you are creating your own settings management
154
00:12:30.710 --> 00:12:41.485
is trying to do this in a more like frictionless way, transparent way. I see lots of settings objects where you just to read a single variable,
155
00:12:41.820 --> 00:12:43.839
you need to call methods like
156
00:12:44.540 --> 00:12:46.000
get, read, parse,
157
00:12:46.380 --> 00:12:50.320
and you need to do your own specific type testing on every place.
158
00:12:50.665 --> 00:12:53.964
And then when you look to the code, you have more, like,
159
00:12:54.345 --> 00:12:58.685
interaction with the settings than other things. So I think
160
00:12:59.190 --> 00:13:02.330
what I learned at that is if you do that,
161
00:13:02.710 --> 00:13:04.010
you are like coupling
162
00:13:04.470 --> 00:13:06.890
your code with your settings library.
163
00:13:07.464 --> 00:13:11.725
And the main philosophy philosophy for Dynacomf is to be like decouple.
164
00:13:12.265 --> 00:13:16.204
We want Dynacomf to be decoupled and transparent from the project.
165
00:13:16.930 --> 00:13:18.070
The Dynaconf API
166
00:13:18.529 --> 00:13:20.709
behaves like a normal Python model
167
00:13:21.010 --> 00:13:28.584
or a dict like object because a Python model with attribute access or a dict like object when you look up by keys
168
00:13:29.045 --> 00:13:38.520
is the 2 ways people read settings. If you have a constant file, it's gonna be attribute. If you have a JSON dictionaries like after parsing. So Dynacom supports
169
00:13:38.900 --> 00:13:59.940
both APIs and you can like treat Dynacom just as normal Python object. You don't have any kind of special that you need to to call or need to prepare your application. You just use as normal Python variables. I think this is important because, as I said, at any point in time, you might need to change your settings manager.
170
00:14:00.465 --> 00:14:07.445
Even if you have your own written settings manager, you might want to migrate to DynaConf. And if your
171
00:14:07.880 --> 00:14:30.490
settings are really agnostic, like simple Python objects, it's gonna be easy to plug Dynacomf in your project. In the same way, the other way, like, if you want to remove Dynacom, it's just like removing the object created by Dynacom for placing to another object that behaves like a Python object and everything works. So I think this is important to decouple the settings from the
172
00:14:31.095 --> 00:14:31.595
project.
173
00:14:32.135 --> 00:14:32.635
So
174
00:14:33.415 --> 00:14:38.475
there are challenges that you're gonna be having dealing with settings. Like,
175
00:14:39.269 --> 00:14:39.930
if you
176
00:14:40.470 --> 00:14:43.209
want, like, dynamic features like laser evaluation
177
00:14:43.829 --> 00:14:44.810
and overload
178
00:14:45.269 --> 00:14:48.089
access from the settings, like, you you want to put middlewares
179
00:14:48.505 --> 00:14:53.245
between who is accessing your settings and where the settings is doing transformations.
180
00:14:54.025 --> 00:14:59.130
So this is gonna be easier if you do this in a transparent way because Python offers
181
00:14:59.430 --> 00:15:03.370
lots of features for dynamic programming. You can override.
182
00:15:03.750 --> 00:15:07.210
You can do laser evaluation. You can override attribute access.
183
00:15:08.105 --> 00:15:12.685
So it's not that difficult to use Python to create a layer
184
00:15:13.065 --> 00:15:15.325
of abstraction for assessing.
185
00:15:15.660 --> 00:15:17.280
We see that with ORMs
186
00:15:17.740 --> 00:15:18.480
for databases
187
00:15:19.180 --> 00:15:23.615
and then a constraint to fit that for the settings. So, yeah. And
188
00:15:24.395 --> 00:15:29.055
about the right way, what would be like the right way to treat the settings?
189
00:15:29.835 --> 00:15:33.680
I still stick with the 12 factor apps. I think settings
190
00:15:34.140 --> 00:15:35.360
belongs to environment.
191
00:15:35.900 --> 00:15:36.400
So
192
00:15:36.940 --> 00:15:37.600
the application
193
00:15:38.220 --> 00:15:39.760
must have reasonable
194
00:15:40.300 --> 00:15:40.800
defaults.
195
00:15:41.375 --> 00:15:44.755
It should be able to run without any specific setting.
196
00:15:45.375 --> 00:15:50.995
Application could have, like, required settings, but it should be able to run with minimal settings.
197
00:15:51.460 --> 00:15:53.560
But everything should be, like,
198
00:15:53.940 --> 00:16:00.715
belonging to the environment where application is running. So the code is actually not. I think it's true
199
00:16:01.095 --> 00:16:04.955
for cloud applications because you can run-in multiple cloud providers.
200
00:16:05.415 --> 00:16:12.250
It's true for CLI applications if you distribute it to multiple targets and and multiple ways of running.
201
00:16:12.629 --> 00:16:13.129
So,
202
00:16:13.509 --> 00:16:14.009
yeah,
203
00:16:14.310 --> 00:16:20.694
I think that's what I have to say. 1 of the things that you noted in there was the question of being able to
204
00:16:21.154 --> 00:16:33.620
cast the different values into their appropriate types And with Python being a dynamic language and also the, I guess, not necessarily evenly distributed adoption of type annotations, etcetera,
205
00:16:34.305 --> 00:16:38.725
that is an interesting question for being able to manage settings in Python applications.
206
00:16:39.745 --> 00:16:50.870
And what you're saying about things like parse pool, parseInt, I've definitely seen that all over the place, both in widely used libraries as well as in some of the libraries that I've seen some of my teams kind of build in house.
207
00:16:51.330 --> 00:16:51.830
And
208
00:16:52.475 --> 00:16:56.735
I'm interested in understanding your thoughts on kind of how to
209
00:16:57.915 --> 00:17:20.570
encode some of that type information in the settings that you're loading for your application as far as, like, what type should it be. And then particularly in the case where maybe it's a string, but it should only be 1 of a particular set. So you, you know, maybe you wanna have an enum and just being able to understand, like, what are the possible range of valid settings for this attribute and then being able to bring in things like
210
00:17:20.870 --> 00:17:21.370
annotations
211
00:17:21.670 --> 00:17:22.490
or documentation
212
00:17:22.950 --> 00:17:33.045
around the purpose for a given setting so that as you come into a brand new project and it has the settings defined, you can say, okay. I understand what this value is supposed to do and why.
213
00:17:33.425 --> 00:17:36.965
About typing, in the first version of DynaConf, we decided
214
00:17:38.290 --> 00:17:39.990
to be transparent and frictionless.
215
00:17:40.290 --> 00:17:48.195
I love this word like frictionless, like you just do settings dot key and you fetch the value you want
216
00:17:48.735 --> 00:17:51.794
and with the type it's defined by Dynaculf.
217
00:17:52.414 --> 00:17:58.370
So we wanted to avoid the need for you to call settings dot key dot cast
218
00:17:58.750 --> 00:18:20.679
or settings dot get and then you put the key and then type that you want that to be tested. So we want to avoid this extra work and and give you the final data to use. So the first approach, which is already supported today but it's not highly used, we added type annotations to the variable itself. So as it's based on environment variable
219
00:18:21.220 --> 00:18:22.760
and environment variables
220
00:18:23.140 --> 00:18:25.784
as our first source of truth
221
00:18:26.164 --> 00:18:27.144
on an application
222
00:18:27.524 --> 00:18:28.024
environment.
223
00:18:28.644 --> 00:18:37.300
You can only export strings and Python reads all environment variables as strings. So what we did, we created some tokens, some markers.
224
00:18:37.760 --> 00:18:39.940
It has nothing to do with type annotations.
225
00:18:40.720 --> 00:18:45.225
It's just you put at JSON and then you can put the JSON string
226
00:18:46.005 --> 00:18:47.865
and export it to your environment.
227
00:18:48.405 --> 00:18:54.640
You can put, like, at int, at string, at boolean. All the basic types are we have a marker.
228
00:18:55.020 --> 00:18:58.975
So when you are exporting environment variable, you mark and then
229
00:18:59.355 --> 00:19:10.830
space and the actual value you want to be parsed. So the type is on the source, is on the data itself and DynaConf parses based on on that. So this was the first implementation.
230
00:19:11.530 --> 00:19:20.165
As I said, it works. You can actually add your own transformations. So you can create your own markers, and then you put this in your own annotation.
231
00:19:20.625 --> 00:19:29.100
Like, you have, like, a parser for a phone number or email address. You can mark this as an email, and then Dynacom will try to parse this and include
232
00:19:29.505 --> 00:19:36.085
and give you back when you access, like, a structure, an object, class, anything you want. So it's very customizable.
233
00:19:37.299 --> 00:19:38.519
We call it converters,
234
00:19:38.980 --> 00:19:40.600
inside the the source core
235
00:19:41.139 --> 00:19:43.960
code. But then when we release the DynaCon 3.0,
236
00:19:44.845 --> 00:19:46.065
we figure out that
237
00:19:46.605 --> 00:19:47.985
most of the users
238
00:19:48.525 --> 00:19:54.540
are were migrating to TOML. TOM obvious minimal language, which is used today
239
00:19:55.320 --> 00:20:00.380
on many projects like by project dot toml, Rust uses on cargo dot toml.
240
00:20:00.680 --> 00:20:06.165
We see this being adopted like as a very standard on on application configuration.
241
00:20:06.705 --> 00:20:10.165
So we decided to adopt toml as our default parser.
242
00:20:10.840 --> 00:20:16.460
So that makes things like easier because you can export a variable as a string
243
00:20:17.000 --> 00:20:22.345
and then we pass this when reading to the toml parser and if it's a valid toml
244
00:20:22.965 --> 00:20:26.905
integer then a conf reads as an integer. If it's a valid toml
245
00:20:27.269 --> 00:20:49.290
float, it reads as a float. So you can obviously replace this behavior. You can choose your dynaconf to use yaml as a parser, to use JSON as a parser. But the default 1 is Toml. So you can just export, like if it looks like a string, DynaConf will infer it's a string. So there is a type inference based on environment variables.
246
00:20:49.750 --> 00:20:57.655
When you are working with files that has typing, for example, YAML has its own typing, toml, JSON, then DynaConf
247
00:20:57.955 --> 00:21:03.550
respects the typing coming from the the source. Yeah. But for all the other string based
248
00:21:03.850 --> 00:21:04.350
sources,
249
00:21:04.810 --> 00:21:08.910
typing will be inferred unless you annotate it explicitly.
250
00:21:09.565 --> 00:21:10.865
As I was mentioning
251
00:21:11.245 --> 00:21:11.745
earlier,
252
00:21:12.525 --> 00:21:21.690
basically every engineer who writes an application gets to the point where they say, Oh, I need to be able to change this setting. And so they start to implement their own settings management
253
00:21:22.149 --> 00:21:22.649
system.
254
00:21:22.950 --> 00:21:26.625
You know, oftentimes, it's very basic. But as you
255
00:21:27.005 --> 00:21:39.600
evolve the project and iterate on it, your settings management becomes increasingly complicated. And I'm wondering what are some of the, I guess, antipatterns that you see people develop as they build their own homegrown systems for
256
00:21:40.060 --> 00:21:45.115
addressing settings in their applications and some of the ways that it can lead to unmaintainable
257
00:21:45.815 --> 00:21:47.755
code and overly complex
258
00:21:48.135 --> 00:21:49.035
settings management
259
00:21:49.415 --> 00:21:54.610
approaches or overly rigid or opinionated ways of being able to specify those values.
260
00:21:55.150 --> 00:21:57.410
Dealing with multiple sources of data
261
00:21:57.765 --> 00:21:58.585
is difficult,
262
00:21:59.125 --> 00:22:01.225
especially if you will have layered environments.
263
00:22:01.605 --> 00:22:06.585
For example, you have a special section for testing variables, production, development,
264
00:22:07.070 --> 00:22:11.730
and this varies across all the environments you are going to be deploying. So
265
00:22:12.030 --> 00:22:13.890
when you get all those data together,
266
00:22:14.235 --> 00:22:18.575
there is a very hard decision to make. So let's say you read, like,
267
00:22:19.115 --> 00:22:25.590
2 static files like toml or yml and you read multiple prefixed environment variables.
268
00:22:26.530 --> 00:22:39.185
And now you have lot of data coming from different sources. How do you merge it? How do you if you have the same key on the settings file and the same key is overrided by an environment variable,
269
00:22:39.820 --> 00:22:52.305
And this is not just a simple primitive type like an integer or a string. It's a data structure. So this happens a lot on Django projects. For example, You have like a key called databases, which is a dictionary,
270
00:22:52.684 --> 00:22:58.945
and inside databases you have like nested keys. And then the user wants to override
271
00:22:59.390 --> 00:23:00.210
only the password,
272
00:23:00.669 --> 00:23:11.055
not the whole dictionary for that key. So how do you do that? So I see that in many cases you will be requiring the user to override the whole dictionary,
273
00:23:11.595 --> 00:23:17.730
but we ended up adding a way for you to override only the nested keys. So you can traverse
274
00:23:18.030 --> 00:23:19.890
using the variable names
275
00:23:20.270 --> 00:23:22.130
to the next nested structures
276
00:23:22.590 --> 00:23:28.885
and then merge data. But when we are talking about more complex data structures like lists,
277
00:23:29.265 --> 00:23:31.025
and let's say you have a list of,
278
00:23:31.665 --> 00:23:34.085
middlewares that your application will
279
00:23:34.450 --> 00:23:37.910
use and you want to contribute with another middleware,
280
00:23:38.370 --> 00:23:46.355
how do you decide if it's gonna be on the first item of the list, last item of the list? How do you ensure it's not duplicating?
281
00:23:46.815 --> 00:23:53.075
Like, you if you add the middleware twice, maybe you are, adding overhead in our application. So
282
00:23:53.919 --> 00:23:56.179
those decisions are really hard because
283
00:23:56.559 --> 00:24:02.019
there are multiple opinions on it. So there are people that thinks it should be more strict. Like, if you wanna
284
00:24:02.375 --> 00:24:04.955
override something, you need to override the entire object.
285
00:24:05.255 --> 00:24:11.115
We wanted to add more flexibility. So there are multiple merging strategies that we offer on DynaConf.
286
00:24:11.655 --> 00:24:17.820
So when you are building your variable or your dictionary, you can mark it with some
287
00:24:18.200 --> 00:24:21.820
special keys or markers, and this will tell Dynacomf
288
00:24:22.424 --> 00:24:27.485
what is the merging strategy for that. So what I see when people starts doing
289
00:24:27.945 --> 00:24:31.085
in house settings management is that they, like,
290
00:24:31.710 --> 00:24:37.890
they don't think about it at the first time. And then when the application starts growing and having multiple environments,
291
00:24:38.270 --> 00:24:39.890
they start needing
292
00:24:40.190 --> 00:24:41.010
data merging
293
00:24:41.365 --> 00:24:44.265
in multiple environments, and it becomes a challenge
294
00:24:44.725 --> 00:24:46.825
especially if you have already running
295
00:24:47.125 --> 00:24:50.105
environments that you don't want to change a lot. So
296
00:24:50.440 --> 00:25:01.365
that's 1 of the things. Another thing is that based on Unix philosophy, I think environment variables first. Yeah. So you should always look to environment variables.
297
00:25:01.665 --> 00:25:07.685
We offer ways to define specific subset of environment variables you want to be overridable.
298
00:25:08.490 --> 00:25:14.990
But based on the subset of environment variables you have, this should be the source of truth on your application.
299
00:25:15.665 --> 00:25:27.929
And so digging into DynaConf itself, can you describe a bit about some of the implementation details and some of the ways that you designed the project to be able to make it sustainable and maintainable
300
00:25:28.309 --> 00:25:29.450
and some of the
301
00:25:29.750 --> 00:25:40.654
evolution that it has gone through from when you first started building it as a 1 off solution to a specific application to where it is now, where it is this generalized framework for settings management?
302
00:25:41.330 --> 00:25:46.150
Yeah. So I think the first important thing on this topic is tests. So
303
00:25:46.770 --> 00:25:48.550
from the time we decided to
304
00:25:48.995 --> 00:25:49.735
promote DynaConf
305
00:25:50.115 --> 00:25:51.335
to, package,
306
00:25:51.795 --> 00:25:58.700
we decided that 100% of test coverage will be the target. So we have 100% of unit testing.
307
00:25:59.160 --> 00:26:07.925
I know that unit testing is not enough for all the dynamic cases. So that's why we also have functional tests, which are growing.
308
00:26:08.305 --> 00:26:12.005
For every issue, we add more functional tests. So tests
309
00:26:12.385 --> 00:26:14.565
is, like, helping a lot for
310
00:26:14.880 --> 00:26:17.059
keeping the library reliable.
311
00:26:17.600 --> 00:26:19.539
Yeah. Because it's it's really important.
312
00:26:20.240 --> 00:26:22.100
And yeah. In terms of implementation,
313
00:26:22.825 --> 00:26:26.285
Dynacomf is just a class. Like, it's a Python class
314
00:26:26.665 --> 00:26:28.445
and it's implemented using
315
00:26:28.825 --> 00:26:29.965
lazy evaluation.
316
00:26:30.505 --> 00:26:31.005
So
317
00:26:31.580 --> 00:26:38.560
there's a functional model inside Dynacomf. In this functional, we created some types that has lazy evaluation.
318
00:26:39.020 --> 00:26:40.320
So the user declare
319
00:26:40.935 --> 00:26:41.435
configuration.
320
00:26:42.135 --> 00:26:42.795
For example,
321
00:26:43.335 --> 00:26:44.635
paths, validators,
322
00:26:45.575 --> 00:26:48.315
all the kinds of preferences they want when instantiate
323
00:26:48.919 --> 00:26:50.700
the settings object from Dynacomf.
324
00:26:51.559 --> 00:27:02.684
And at that point, nothing is really performed by Dynacomf. Dynacomf just creates a class with bunch of data in it, the declaration of how we want DynaConf to be behaving,
325
00:27:03.304 --> 00:27:03.804
and
326
00:27:04.424 --> 00:27:08.980
then you can, like, start your application without need to, like, have all the DynaConf
327
00:27:09.280 --> 00:27:12.980
process. So you don't block the import time of your application
328
00:27:13.280 --> 00:27:24.445
because of settings. This is also another thing I see people doing. Like, when you have a settings library that during our time, you need to load all the files and do all the transformations, then you can block
329
00:27:24.825 --> 00:27:35.289
your application startup time just to read some settings files that is gonna be used only later. So that's why we implemented this as a lazy evaluation class.
330
00:27:35.955 --> 00:27:39.254
When the first variable is assessed in dynaconf,
331
00:27:39.634 --> 00:27:40.695
in your application
332
00:27:41.154 --> 00:27:41.654
lifetime,
333
00:27:42.195 --> 00:27:44.134
then dynaconf starts building
334
00:27:44.580 --> 00:27:45.799
its loaders.
335
00:27:46.100 --> 00:27:55.445
So it works with a pool of loaders and you can add more loaders to it. It has a collection of loaders that are built in. For example, it has,
336
00:27:56.164 --> 00:27:56.985
JSON loader,
337
00:27:57.524 --> 00:28:02.105
dotini loader, toml loader, yaml loader, environment variable loader.
338
00:28:02.485 --> 00:28:11.355
So those are included in a pool in a specific order. All of this is configurable and you can easily create your own loader and plug there.
339
00:28:11.835 --> 00:28:12.575
And loaders
340
00:28:12.875 --> 00:28:14.815
uses the strategy pattern
341
00:28:15.355 --> 00:28:19.215
where you can take the base loader and then you pass your
342
00:28:19.600 --> 00:28:28.340
file reader, your file writer, and the loader methods that will be returning data. So after the data is loaded, it's just a Python dictionary
343
00:28:29.125 --> 00:28:31.145
that's coming from all the different
344
00:28:31.605 --> 00:28:46.320
sources. So you see this makes it flexible because on the loader you can read from a database. For example, we have cases where people wrote a loader for MySQL. They go and do a select in MySQL, get some key pairs from MySQL, build a dictionary
345
00:28:46.620 --> 00:28:49.585
and feed dynaconf with that data. So
346
00:28:49.885 --> 00:28:50.385
after
347
00:28:50.765 --> 00:28:54.800
every loader runs, we put each dictionary in a chain map.
348
00:28:55.280 --> 00:28:58.980
Chain map is a Python object that is a stack of dictionaries.
349
00:28:59.360 --> 00:29:06.404
So you put your chain, lots of dictionaries together in a single object And then when you try to access a key,
350
00:29:06.705 --> 00:29:11.580
if the key is found on the first dictionary, the last latest inserted, then
351
00:29:12.120 --> 00:29:18.299
it returned. If not, then the chain map is gonna be trying all the other dictionaries in order until
352
00:29:18.745 --> 00:29:22.125
it do, like, tribute error or key error. So
353
00:29:22.745 --> 00:29:31.220
when DynaConf does it, it's loading. It puts in a in a chain map and then we do all the things we need in the chain map like merging, parsing.
354
00:29:31.600 --> 00:29:34.435
All the transformations are inside the chain map.
355
00:29:35.395 --> 00:29:37.175
So the process of of loading
356
00:29:38.035 --> 00:29:47.840
from source involves parsing. So because, as I said, we can define type annotations. So when you read an environment variable, if it's marked with int, bold, JSON,
357
00:29:48.140 --> 00:29:51.765
etcetera, it should be, like, tested to the appropriate type.
358
00:29:52.245 --> 00:29:53.065
Also, dynaconf
359
00:29:53.445 --> 00:29:55.065
supports some lathe formatting.
360
00:29:55.525 --> 00:29:57.305
So you can have, for example,
361
00:29:57.685 --> 00:30:01.065
environment variable or a settings variable in a text file
362
00:30:01.529 --> 00:30:02.909
with jing jett and plating.
363
00:30:03.289 --> 00:30:08.510
So people usually use this to calculate how many processors you have in the machine
364
00:30:08.925 --> 00:30:16.065
or to call some external thing and and build the settings on the go or to reuse some environment variable.
365
00:30:16.525 --> 00:30:26.820
So this is the process. So this is the heavy process that Dynacomf does. So at the end of the day when your settings is loaded, what you have is a settings, which is
366
00:30:27.164 --> 00:30:30.225
a class using all the dynamic Python features
367
00:30:30.605 --> 00:30:31.265
such as
368
00:30:31.645 --> 00:30:36.225
overloading of get attributes and all those methods that makes
369
00:30:36.630 --> 00:30:38.010
it makes it
370
00:30:38.549 --> 00:31:02.730
easy to get a key and look up that on the chain map that it has. And all of this is very quick because it's all using Python. On the chain map, we have almost constant time to access. So after the loading, everything is very very simple. So I just see the if you look the overview of the architecture is not like too much. It's just a simple class. But then we plug the loaders
371
00:31:03.335 --> 00:31:06.794
and the strategies user on the loaders. Then we have the filters
372
00:31:07.174 --> 00:31:17.659
and, especially, we have the validators so you can ensure that the settings look the way you want. So, yeah, that's more or less how it's implemented. And as I said, it growed,
373
00:31:18.039 --> 00:31:18.539
evolved
374
00:31:19.320 --> 00:31:24.605
mostly organic. So from that small project, it started becoming like a framework. And people
375
00:31:24.985 --> 00:31:25.805
from different
376
00:31:26.105 --> 00:31:33.360
backgrounds are contributing and bringing more features from sometimes these features in other settings libraries they bring to DynaConf.
377
00:31:33.900 --> 00:31:36.000
So we have different ways of doing.
378
00:31:36.515 --> 00:31:38.775
Given the fact that it has these interfaces
379
00:31:39.075 --> 00:31:41.655
for custom loaders and different strategies,
380
00:31:42.275 --> 00:31:44.615
it definitely is very extensible.
381
00:31:45.210 --> 00:31:46.910
And I'm wondering how you
382
00:31:47.370 --> 00:31:49.710
thought about the process of deciding
383
00:31:50.010 --> 00:31:54.190
which strategies, which loaders to incorporate into the DynaCon project
384
00:31:54.545 --> 00:31:56.405
itself as it's packaged and released,
385
00:31:57.025 --> 00:31:58.565
and what capabilities
386
00:31:59.025 --> 00:31:59.845
you are
387
00:32:00.225 --> 00:32:08.230
consciously deciding not to bring into the core because either it's a bespoke solution for 1 person or it's something that is not
388
00:32:09.169 --> 00:32:13.075
a sort of core functionality, and you want to kind of push that more into
389
00:32:13.455 --> 00:32:19.634
allowing people to do what they need to do without overloading the project and overloading your kind of maintainer capabilities
390
00:32:20.390 --> 00:32:25.450
into some of the ways that people are able to hook into DynaConft to be able to add those customizations.
391
00:32:26.630 --> 00:32:31.425
When I started, I got very excited about adding more features.
392
00:32:31.725 --> 00:32:36.465
It was like my vision. Like, I want Dynaconf to be able to load all the possible
393
00:32:37.019 --> 00:32:37.519
configuration
394
00:32:37.980 --> 00:32:39.760
formats and all the possible
395
00:32:40.139 --> 00:32:40.639
services
396
00:32:41.019 --> 00:32:44.639
targeting for configuration like, etcetera, Redis, Vault,
397
00:32:45.260 --> 00:32:45.760
etcetera.
398
00:32:46.145 --> 00:32:55.125
And then at that point, we added too many features into the library. I know that. So all those loaders that we call the core loaders
399
00:32:55.610 --> 00:32:57.470
for multiple file types
400
00:32:58.090 --> 00:33:02.270
are not really necessary on all Dynacomf installations.
401
00:33:02.945 --> 00:33:05.765
But currently, if you do pip install DynaConf,
402
00:33:06.225 --> 00:33:08.805
you are getting all those together.
403
00:33:09.265 --> 00:33:13.340
So at some point, there was no extensive thinking about this.
404
00:33:13.800 --> 00:33:20.745
When we released the 3 dot 0, I started having more contributions and we formed it like a board of co maintainers
405
00:33:21.205 --> 00:33:27.625
and people started thinking about like, we don't want to add more things here. We need things to be added as a plugin.
406
00:33:28.350 --> 00:33:37.865
So now we have like a initial plugin system. So for example, there is a pytest plugin to make it easier to use Dynacom for when you are mutating
407
00:33:38.325 --> 00:33:39.945
settings during your settings.
408
00:33:40.404 --> 00:33:46.024
And we decided to not accept this as a core feature and then the person that sent the PR
409
00:33:46.380 --> 00:33:48.480
created that Dynacomp Pytest repository
410
00:33:48.940 --> 00:33:49.920
with a plugin.
411
00:33:50.700 --> 00:33:55.680
So it was like 1 year ago or 2 years ago, I don't remember exactly, but
412
00:33:56.205 --> 00:33:57.585
we started, like, rejecting
413
00:33:58.125 --> 00:34:02.945
more features to focus on fixing and improving features that we have in the library.
414
00:34:03.325 --> 00:34:05.745
And now we want to improve our plugging system,
415
00:34:06.250 --> 00:34:11.390
and we have plans to have more pluggings in the future and even removing
416
00:34:11.770 --> 00:34:13.550
some features from the core,
417
00:34:14.145 --> 00:34:27.470
turning them into plugins. This is 1 of the things in our road map. Yeah. But right now, we are not accepting, like, new features, like new default core loaders. If you want to load something, for example,
418
00:34:27.850 --> 00:34:31.470
for a new file format or a new external service,
419
00:34:31.770 --> 00:34:32.430
we will
420
00:34:32.995 --> 00:34:35.415
suggest you to create, for example, an plugin.
421
00:34:35.715 --> 00:34:36.535
There are documentation
422
00:34:36.915 --> 00:34:38.135
on how you can use it
423
00:34:38.435 --> 00:34:41.095
and yeah. Currently, that's our approach
424
00:34:41.530 --> 00:34:44.110
to fix and prove things before we
425
00:34:44.410 --> 00:34:53.955
we add more features. And try to get the library. It's small. It's very small, but we need to get it more small and and have only the core on the on the main repository.
426
00:34:54.735 --> 00:34:56.115
And then as far as
427
00:34:57.470 --> 00:35:00.130
promoting kind of reusability and collaboration
428
00:35:00.430 --> 00:35:05.970
among the kind of ecosystem of people building on top of DynaConf, do you have any,
429
00:35:06.305 --> 00:35:07.205
I guess, naming
430
00:35:07.745 --> 00:35:18.640
convention for packages that plug into DynaConf to add custom functionality, such as if I wanted to add something that allowed for loading key value data from console, for instance? Or
431
00:35:19.099 --> 00:35:25.635
is there any kind of GitHub organization that you have for being able to allow people to contribute their plugins to DynaConf
432
00:35:26.015 --> 00:35:31.315
so that people can kind of discover all the options that are out there rather than just reinventing the wheel all the time?
433
00:35:31.760 --> 00:35:35.380
Yeah. We have a GitHub organization. It's called Dynacomf. Github.com/dinacomf.
434
00:35:37.359 --> 00:35:38.740
There is the website, Dynacomf.com,
435
00:35:39.440 --> 00:35:45.145
and there, I think the 2 plugins that exists right now is listed somewhere there.
436
00:35:45.685 --> 00:35:47.145
And we have the discussions
437
00:35:48.100 --> 00:35:49.320
forum on GitHub.
438
00:35:49.700 --> 00:35:50.440
So if
439
00:35:50.740 --> 00:35:57.560
people want like to contribute with a new loader or new functionality, we discuss there or on our matrix channel
440
00:35:57.915 --> 00:36:08.130
and we give the help needed to create this plugin. It's not a hard work because you can create a loader by subclassing the base loader and adding those 3 objects,
441
00:36:08.430 --> 00:36:09.890
reader, writer, and loader.
442
00:36:10.270 --> 00:36:15.890
And also we have now support for hooks. So we got some inspiration from Pytest.
443
00:36:16.225 --> 00:36:22.885
So you can file with hooks that Dynacomf will call on certain points of its loading process.
444
00:36:23.425 --> 00:36:30.930
So it's possible to, like, customize lots of things in in Dynacomf right now. Yeah. But the main channel is GitHub. Github.com/dynacomf.
445
00:36:32.695 --> 00:36:44.619
You mentioned a bit about the process for kind of onboarding onto Dynaconf as part of your project, where it's largely just PIP install and start using it. I'm wondering what are some of the ways that the
446
00:36:45.000 --> 00:36:45.500
implementation
447
00:36:45.880 --> 00:36:51.735
and usage of DynaConf scales with the complexity of a project or the complexity of the
448
00:36:52.135 --> 00:36:52.635
environments
449
00:36:53.575 --> 00:36:56.635
or kind of support matrix of a given project as it evolves?
450
00:36:57.175 --> 00:37:04.650
The goal for the project is to be really simple. Like, you you really create your Python settings object.
451
00:37:05.030 --> 00:37:05.530
Really,
452
00:37:05.910 --> 00:37:08.650
the idea is that it's not going to be something
453
00:37:09.025 --> 00:37:12.885
that you need to care a lot in your project besides the validators,
454
00:37:13.185 --> 00:37:14.805
yeah, which is completely
455
00:37:15.265 --> 00:37:15.925
the user
456
00:37:16.385 --> 00:37:16.885
responsibility
457
00:37:17.185 --> 00:37:19.829
to write. But everything else is really like
458
00:37:20.130 --> 00:37:31.775
transparent and outside your project. And yeah. So the usage scale has been very good. I think there are lots of users right now. I don't remember right now, but I think in GitHub, we have more than
459
00:37:32.315 --> 00:37:32.815
2, 000
460
00:37:34.120 --> 00:37:37.180
repositories that depends on DynaConf right now.
461
00:37:37.640 --> 00:37:38.780
And recently,
462
00:37:39.160 --> 00:37:40.815
PIPI market DynaConf has
463
00:37:41.375 --> 00:37:45.474
critical project, which I think it means that it's on the top 1%
464
00:37:45.775 --> 00:37:47.474
most downloaded libraries.
465
00:37:48.349 --> 00:37:48.849
And
466
00:37:49.150 --> 00:37:50.930
in terms of support, like,
467
00:37:51.470 --> 00:37:52.770
we have few contributors
468
00:37:53.150 --> 00:37:59.165
that are really active in the GitHub issues. So if you need something, if you need if you have questions,
469
00:37:59.705 --> 00:38:01.885
it's really quick to get information.
470
00:38:02.425 --> 00:38:07.590
On the matrix channel, we don't have lots of people because the channel is more like for devs,
471
00:38:07.890 --> 00:38:11.350
but it's also a good source of information if you need implementing
472
00:38:12.130 --> 00:38:13.430
DynaConf in your project.
473
00:38:13.845 --> 00:38:19.305
And I would say that GitHub is our our main place in the discussion tab or issues.
474
00:38:19.684 --> 00:38:26.480
And usually, it's quick. Like, I tend to answer questions really fast and we have other people like helping. So
475
00:38:27.020 --> 00:38:32.075
it's scaling good. As for example, the company I work on is, like,
476
00:38:32.535 --> 00:38:37.510
sponsoring me to work on Dynacom for a certain time on my work.
477
00:38:37.830 --> 00:38:48.845
I have, like, paid time to work on the library. So I'm trying to address lots of issues. So we work with open source, and DynaConf is 1 of the libraries in our open source. Yeah.
478
00:38:49.145 --> 00:38:53.724
1 of the other topics that probably deserves its whole entire episode
479
00:38:54.170 --> 00:38:57.870
that we touched on briefly earlier is the question of secrets management,
480
00:38:58.250 --> 00:38:58.750
where
481
00:38:59.050 --> 00:39:02.510
it's 1 of the things that, again, sometimes seems like it's straightforward
482
00:39:02.970 --> 00:39:04.125
but can be
483
00:39:04.585 --> 00:39:08.765
critical if you get it wrong because it, you know, has the potential to
484
00:39:09.225 --> 00:39:13.640
expose your application, make it vulnerable to exploit or kind of malfeasance.
485
00:39:14.100 --> 00:39:20.280
And I'm wondering, what are some of the ways that you have worked in DynaConf to be able to strike a balance between
486
00:39:20.755 --> 00:39:21.494
that frictionless
487
00:39:21.875 --> 00:39:25.974
use case that you're aiming for while providing appropriate guardrails
488
00:39:26.275 --> 00:39:29.655
to prevent kind of credential leakage or prevent people
489
00:39:30.049 --> 00:39:30.869
kind of circumventing
490
00:39:31.170 --> 00:39:34.869
best practice for how to manage sensitive data in their application?
491
00:39:35.490 --> 00:39:37.430
I think the best way to manage
492
00:39:37.809 --> 00:39:38.789
secrets today
493
00:39:39.105 --> 00:39:43.445
is using encrypted vault services, especially if you are working in the cloud.
494
00:39:43.745 --> 00:39:46.165
So you're gonna have tooling that will
495
00:39:46.670 --> 00:39:53.570
it will depend on the environment you are running. But for example, you have a w s secret manager, you have HashCorp Vault
496
00:39:54.035 --> 00:39:58.535
and you can use encrypted environment variables depending on the on the place you are
497
00:39:58.915 --> 00:39:59.895
deploying your application.
498
00:40:00.515 --> 00:40:02.855
I think that's the best because you are delegating
499
00:40:03.720 --> 00:40:04.220
responsibility
500
00:40:04.520 --> 00:40:05.020
for
501
00:40:05.560 --> 00:40:14.515
this where the secrets are stored and how it's encrypted to another service that is really good on doing this and well tested for this specifically.
502
00:40:15.135 --> 00:40:16.675
And Dynacom supports
503
00:40:17.055 --> 00:40:20.950
out of the box loading data from HashCorp Vault today.
504
00:40:21.330 --> 00:40:23.350
So you can like just pass the configuration,
505
00:40:23.650 --> 00:40:32.585
the host, and your token for Vault, and then it you can pass, I don't remember exactly, but I think the name space and then it goes there and reads your secrets.
506
00:40:33.205 --> 00:40:38.099
But you can add folder for other services. It's not hard to do.
507
00:40:38.400 --> 00:40:49.875
But this is something that probably you're gonna do in a production environment when you have like a team or as a re or DevOps engineer that will provide the service for you as a developer.
508
00:40:50.575 --> 00:40:54.914
During your development environment, I think you don't wanna have this work of, like,
509
00:40:55.400 --> 00:41:16.750
deploying or running a vault service during the development. So we decided to make things easier when you're running in development environment. So Dynacom supports you to add a secret dot toml file. So using the same Dynacom structure when you are working, you have a settings dot toml or dot yaml. The format doesn't matter. You can create a file together
510
00:41:17.530 --> 00:41:24.495
at any settings that is called dot secrets and this extension you are using. This is just a settings file,
511
00:41:25.035 --> 00:41:27.695
but the first way it helps is that
512
00:41:28.075 --> 00:41:31.700
it's added to the git ignore if you use dynamic CLI to initialize
513
00:41:32.160 --> 00:41:32.900
your project.
514
00:41:33.280 --> 00:41:36.420
So you ensure you are not pushing it to your repository.
515
00:41:37.359 --> 00:41:38.755
But it's a separate file
516
00:41:39.135 --> 00:41:41.475
so you can ignore it from your project.
517
00:41:41.935 --> 00:41:46.994
But you also can encrypt this file if you want. So you can use Dynacom CLI
518
00:41:47.350 --> 00:41:50.250
to add new secrets and the secrets will be encrypted
519
00:41:50.550 --> 00:41:52.010
using a private key.
520
00:41:52.390 --> 00:42:07.529
And then your local vault is just a simple file with encrypted data inside it. This is not something that you should use in production system. I know people are using this encrypted secret file. In production, we can't control what users does.
521
00:42:07.829 --> 00:42:16.085
So in production, I recommend using an external service, but this is something we provide for making development and testing easier.
522
00:42:16.464 --> 00:42:18.325
Another thing that we did is
523
00:42:19.025 --> 00:42:20.005
Dynacomf identifies
524
00:42:20.870 --> 00:42:23.610
when your settings comes from a secret.
525
00:42:24.710 --> 00:42:35.465
And then it will not print it out in logs. It will not print it out in the standard outputs. And when you are using banner conf CLI to inspect your environment,
526
00:42:35.765 --> 00:42:37.865
it will also just show the
527
00:42:38.200 --> 00:42:43.900
part of the the secret, not the whole secret. So there's some kind of obfuscation on on the variable
528
00:42:44.279 --> 00:42:45.579
to make it secure.
529
00:42:46.224 --> 00:42:49.525
As users become more advanced or sophisticated
530
00:42:49.825 --> 00:42:54.565
with their use of settings in their application and in their application of Dynaconf,
531
00:42:54.870 --> 00:42:58.490
what are some of the kind of advanced or maybe underutilized
532
00:42:58.790 --> 00:43:00.010
or overlooked capabilities
533
00:43:00.310 --> 00:43:02.170
that are available in the framework?
534
00:43:02.775 --> 00:43:05.435
Yeah. So besides the basic things, I think
535
00:43:05.735 --> 00:43:09.435
yeah. As I said, on env vars, you can override,
536
00:43:09.750 --> 00:43:12.010
you can traverse to nested keys.
537
00:43:12.310 --> 00:43:22.245
That's very powerful, especially if you are working in an existing project. I see lots of Django users that is not using it, but it's very powerful
538
00:43:22.705 --> 00:43:26.245
because we adopted the Django way of doing it, which is
539
00:43:26.590 --> 00:43:27.890
you export a variable.
540
00:43:28.910 --> 00:43:29.970
If it's a dictionary
541
00:43:30.270 --> 00:43:32.770
inside your settings, you can use double underscore
542
00:43:33.310 --> 00:43:34.530
to go nesting
543
00:43:34.895 --> 00:43:35.635
1 level
544
00:43:36.015 --> 00:43:39.474
in our data structure, and then you can keep using double underscore
545
00:43:40.015 --> 00:43:48.930
to traverse to all the other nesting levels. And then you can specifically change that variable you want and that together with the validation
546
00:43:49.230 --> 00:43:51.744
that DynaConf provides, you can have,
547
00:43:52.045 --> 00:43:55.904
easy way to override settings. So this is an underutilized
548
00:43:56.924 --> 00:43:57.424
capability
549
00:43:57.724 --> 00:44:00.750
that I want to promote more and see people using this
550
00:44:01.550 --> 00:44:07.730
better. Other thing is that Dynacom supports lazy values. You can actually create your own lazy formatter.
551
00:44:08.030 --> 00:44:09.490
Right now it has Jinja
552
00:44:09.875 --> 00:44:10.695
and it has
553
00:44:11.315 --> 00:44:11.895
the math
554
00:44:12.355 --> 00:44:15.255
to do like, common math expressions like multiplication,
555
00:44:15.635 --> 00:44:29.494
division, and so on. You can export the variable and mark it as a math or a jinga and put the template or the expression. And then during the load, then a conf will evaluate it. It also works together with validation to be more effective.
556
00:44:30.355 --> 00:44:35.015
And besides the the loaders that we talked, like you can add new loaders to Dynacomf.
557
00:44:35.400 --> 00:44:37.980
Now Dynacomf has a hook system. So you can
558
00:44:38.440 --> 00:44:44.140
write functions that will be hooked into Dynacomf on specific parts of the loading process.
559
00:44:45.035 --> 00:44:47.454
And from the hooks, you return a dictionary,
560
00:44:47.994 --> 00:44:54.590
and that dictionary will have declarations that will contribute to the settings. So you can do like data transformation.
561
00:44:55.050 --> 00:44:56.510
You can do data validation.
562
00:44:56.810 --> 00:45:00.670
You can do all sorts of custom things with this. And I think
563
00:45:01.005 --> 00:45:03.665
the thing that is more underutilized
564
00:45:04.365 --> 00:45:05.185
in Dynacomf
565
00:45:05.965 --> 00:45:06.785
is validators.
566
00:45:07.325 --> 00:45:13.130
And it's really important because by default, Dynaconf is schema less. You just create a settings
567
00:45:13.430 --> 00:45:18.330
that equals Dynaconf class instance, and then you start looking for
568
00:45:18.765 --> 00:45:23.585
keys inside it. But there is no definition of schema by default. It's very flexible.
569
00:45:24.125 --> 00:45:29.570
But we provide a way to define a schema. So you have a validator class where you can provide
570
00:45:29.870 --> 00:45:40.665
validator for each key you want and you can define the type of the key, the range of the key. I think you asked it before. How do you define a range if you have like a number, a port, for example, for a service?
571
00:45:40.965 --> 00:45:47.270
So how do you define exactly which options are available there? So you can use the validators to define it.
572
00:45:47.650 --> 00:45:53.990
And another good feature of validator is that it has a text field, which is a doc for that settings.
573
00:45:54.295 --> 00:45:55.434
So when you define,
574
00:45:55.974 --> 00:45:57.115
collection of validators,
575
00:45:57.655 --> 00:46:01.115
you put the rules that Dynacomf will validate and you put the description
576
00:46:01.575 --> 00:46:05.380
of this. You can use Dynacomf CLI to generate a documentation
577
00:46:05.680 --> 00:46:08.500
for the supported settings for your application.
578
00:46:08.960 --> 00:46:18.855
So this is really under utilized it. I think that's my fault because I need to promote it more. We are focusing more on the other basic features, but it's very powerful.
579
00:46:19.609 --> 00:46:21.310
And I guess 1
580
00:46:22.010 --> 00:46:25.950
kind of side topic that I'm interested in personally is
581
00:46:26.410 --> 00:46:26.910
another
582
00:46:27.210 --> 00:46:30.575
project that I've seen that has some similar capabilities,
583
00:46:31.035 --> 00:46:33.055
at least in terms of being able to
584
00:46:33.675 --> 00:46:34.175
assign
585
00:46:34.475 --> 00:46:35.935
schema and type information
586
00:46:36.235 --> 00:46:38.735
to your settings and be able to do some validations
587
00:46:39.309 --> 00:46:41.890
at load time is the Pydantic project.
588
00:46:42.270 --> 00:46:43.170
And I'm wondering
589
00:46:43.630 --> 00:46:46.130
if you see that as a complementary
590
00:46:46.510 --> 00:46:47.650
or an alternative
591
00:46:48.295 --> 00:46:50.474
solution for being able to approach settings?
592
00:46:51.095 --> 00:47:00.300
Yeah. So the answer for these questions is on Dynacom pull request tab. If you go there, you see that there is a pull request that is going to be the Dynacom 40.
593
00:47:00.840 --> 00:47:04.780
And on Dynacom 40, we are bringing by then take to Dynacomf.
594
00:47:05.080 --> 00:47:06.605
So these validators
595
00:47:06.905 --> 00:47:12.765
I told you now is we created the validation class. It works very well.
596
00:47:13.110 --> 00:47:14.730
We want to keep it working,
597
00:47:15.190 --> 00:47:20.010
but what we're gonna do is we're gonna provide a way for the users to create a schema,
598
00:47:20.325 --> 00:47:21.545
use subclass dynaconf,
599
00:47:21.845 --> 00:47:24.425
and then you create a schema and then use type annotations
600
00:47:24.725 --> 00:47:25.945
and use bydentic
601
00:47:26.245 --> 00:47:26.745
fields.
602
00:47:27.125 --> 00:47:29.225
And then Dynacomf will follow
603
00:47:29.550 --> 00:47:30.050
bydantic,
604
00:47:30.670 --> 00:47:34.130
style of schema. So we see that Python community
605
00:47:34.430 --> 00:47:37.330
is moving to this thing called modern Python.
606
00:47:37.855 --> 00:47:41.075
All the things are going to this way of programming.
607
00:47:41.375 --> 00:47:45.875
So using more data class, more bydentic with type annotations.
608
00:47:46.610 --> 00:47:58.025
So we are moving this way also. It's taking a time because we are not like lots of contributors, but this is in our roadmap for the next version. Yeah. In your experience
609
00:47:58.325 --> 00:48:08.220
of building and using Dynacon for yourself and working with the community of people who are adopting it for their own projects, what are some of the most interesting or innovative or unexpected ways that you've seen it applied?
610
00:48:08.680 --> 00:48:09.660
Oh, that's interesting.
611
00:48:10.360 --> 00:48:14.435
I have been hired some months ago by a company to give a consultants
612
00:48:15.055 --> 00:48:17.075
on a project they were using, DynaConf.
613
00:48:17.855 --> 00:48:21.240
And it was a very old software. They had like this data collectors,
614
00:48:29.265 --> 00:48:32.885
like, scan things and this generates a dot CSV file.
615
00:48:33.265 --> 00:48:33.765
And
616
00:48:34.145 --> 00:48:39.970
at the end of the day, the person collecting the data will select an upload field on the application,
617
00:48:40.510 --> 00:48:46.850
select the CSV file or multiple files and upload to a back end. So they were able to
618
00:48:47.244 --> 00:48:49.505
create a new back end using Flask
619
00:48:50.125 --> 00:48:52.065
but they are not able to upgrade
620
00:48:52.365 --> 00:48:53.425
this outdated
621
00:48:53.805 --> 00:48:54.705
data collectors.
622
00:48:55.130 --> 00:49:02.190
So on the application they have on the client side, they have only a single upload field to send files to the server
623
00:49:02.734 --> 00:49:04.755
and they wanted to gather more information
624
00:49:05.055 --> 00:49:17.300
from the client parts. So they wanted to have a form with the name, the location, and all the information, but it was not possible because I think this is outdated. They can't, like, update that. So
625
00:49:17.680 --> 00:49:18.580
what they did,
626
00:49:19.120 --> 00:49:21.215
they added a toml
627
00:49:21.675 --> 00:49:25.775
file in the mobile device with the information they want, like the name of the user
628
00:49:26.075 --> 00:49:38.150
and all the serial numbers and things. And then the user, when sending the upload, they select the CSV file and select this toml file with the information from the the local place
629
00:49:38.535 --> 00:49:41.275
and upload it to the Flask server. And Flask,
630
00:49:41.815 --> 00:49:43.275
instead of parsing this
631
00:49:43.575 --> 00:49:53.460
manually, they decided to use Dynacom for it, which is not like the go for Dynacomf, but work it because during upload, they drop this file in the configuration file
632
00:49:54.155 --> 00:49:57.855
and then when running their pipeline, I think it was an airflow application,
633
00:49:58.395 --> 00:49:58.975
a pipeline,
634
00:49:59.435 --> 00:50:01.775
they read settings and Dyna conf would
635
00:50:02.190 --> 00:50:08.849
load that toml file and know how to process that CSV file. So just to resume this,
636
00:50:09.155 --> 00:50:13.495
what they did is instead of creating a new form in the application,
637
00:50:14.115 --> 00:50:15.495
they replaced the form
638
00:50:15.795 --> 00:50:23.780
with a toml file. They use a dynaconf to parse it. And dynaconf is not like a library for doing this. They use a dynaconf as,
639
00:50:24.625 --> 00:50:28.405
ORM or something to process data coming from request
640
00:50:28.865 --> 00:50:30.885
and not as a settings library,
641
00:50:31.265 --> 00:50:34.790
but that was working. They didn't want it to change so
642
00:50:35.090 --> 00:50:37.590
they got a problem where the end users
643
00:50:38.130 --> 00:50:38.630
discovered
644
00:50:39.010 --> 00:50:39.990
that they can
645
00:50:40.345 --> 00:50:45.724
edit the toml file and override settings that they should not override.
646
00:50:46.265 --> 00:50:48.444
And then we ended like creating
647
00:50:48.880 --> 00:50:50.900
a custom format, not toml,
648
00:50:51.359 --> 00:50:54.900
and adding a new loader to Dynacomf with more validation
649
00:50:55.280 --> 00:50:57.780
to prevent users to hack the system.
650
00:50:58.234 --> 00:51:03.295
And it's running with Dynacomf today. So they have lots of collectors. They run multiple
651
00:51:03.755 --> 00:51:04.895
pipelines on airflow,
652
00:51:05.515 --> 00:51:06.655
and they use Dynacomf
653
00:51:06.955 --> 00:51:08.270
like in unexpected
654
00:51:09.050 --> 00:51:11.230
way. So DynaConf is their request
655
00:51:11.610 --> 00:51:12.590
parsing library
656
00:51:13.050 --> 00:51:15.550
and yeah. I don't think there is any other
657
00:51:17.515 --> 00:51:20.335
personal or project using like this. And yeah.
658
00:51:21.275 --> 00:51:22.335
Yeah. It's definitely,
659
00:51:23.355 --> 00:51:30.230
interesting, the different ways that the kind of old trope of when you have a hammer, everything looks like a nail, some of the ways that that manifests.
660
00:51:31.090 --> 00:51:39.255
Yeah. You can control what the user does. We try. Yeah. I learn with this and other projects I work on. We always try to, like, build fences
661
00:51:39.635 --> 00:51:42.360
to prevent users for of doing this.
662
00:51:42.680 --> 00:51:46.300
But, yeah, when you work in a project like this that grows organically,
663
00:51:46.760 --> 00:51:47.740
when you find
664
00:51:48.200 --> 00:51:50.140
the hole that you need to close,
665
00:51:50.805 --> 00:52:04.650
all the users are already using that. So you can't just, like, remove this feature because everyone relies on the feature. So instead of a bug, you just document and it becomes a feature. Then life goes on, yeah, with lots of testing. So,
666
00:52:05.030 --> 00:52:06.090
yeah, that's interesting.
667
00:52:06.630 --> 00:52:07.770
In your own experience
668
00:52:08.070 --> 00:52:09.515
of building DynaConf
669
00:52:09.895 --> 00:52:15.914
and maintaining it and evolving it, what are some of the most interesting or unexpected or challenging lessons that you've learned in the process?
670
00:52:16.609 --> 00:52:17.910
Okay. So well,
671
00:52:18.450 --> 00:52:18.950
interesting.
672
00:52:19.330 --> 00:52:19.910
I think
673
00:52:20.690 --> 00:52:21.589
when I
674
00:52:22.450 --> 00:52:23.750
started doing the project,
675
00:52:24.185 --> 00:52:26.045
I thought that more documentation
676
00:52:26.505 --> 00:52:28.365
would be better. If I documented
677
00:52:28.745 --> 00:52:31.290
everything I did would be the best thing to do,
678
00:52:31.850 --> 00:52:34.830
and that's not true. I found out that
679
00:52:35.530 --> 00:52:36.030
sometimes
680
00:52:36.410 --> 00:52:37.550
more documentation
681
00:52:38.010 --> 00:52:38.750
is bad.
682
00:52:39.145 --> 00:52:41.725
So the previous documentation of Dynaconf
683
00:52:42.105 --> 00:52:44.445
have too many docs. We use it to
684
00:52:44.745 --> 00:52:50.790
dock, for example, APIs and function calls and using that old style Python way of documenting
685
00:52:51.330 --> 00:52:51.830
everything
686
00:52:52.130 --> 00:52:53.430
on your library.
687
00:52:53.890 --> 00:52:56.150
But as Dynacomf is more like a client,
688
00:52:56.495 --> 00:52:57.955
not really a library.
689
00:52:58.335 --> 00:53:02.035
You stunted as a client. You are the end users. You are not really
690
00:53:02.415 --> 00:53:04.595
extending it a lot in terms of API.
691
00:53:05.180 --> 00:53:05.840
I learned
692
00:53:06.140 --> 00:53:09.040
that today, and I think it's something that is changing
693
00:53:09.340 --> 00:53:11.600
on software industry. If you look to the newest
694
00:53:12.065 --> 00:53:14.244
software that is created on open source,
695
00:53:14.545 --> 00:53:15.045
documentation
696
00:53:15.665 --> 00:53:18.405
is becoming more like a set of tutorials
697
00:53:19.105 --> 00:53:23.900
written in a book form. If you go to excellent documentations like fast API,
698
00:53:24.200 --> 00:53:25.020
I can mention.
699
00:53:25.400 --> 00:53:30.714
You see that the style of reading docs are more like a tutorial. You are guiding the person
700
00:53:31.175 --> 00:53:32.474
to know workflows
701
00:53:32.934 --> 00:53:34.315
on how to use your application
702
00:53:34.694 --> 00:53:35.835
instead of documenting
703
00:53:36.454 --> 00:53:46.359
every method or every single thing your application can do. You put everything together. So this is was an interesting thing I learned, a different way of writing docs,
704
00:53:46.660 --> 00:53:58.589
and I'm trying to do this to apply this to Dynacomp. There is a lot of things we need to change there in this way. And this is more difficult because you can't just look to the code and document based on code.
705
00:53:58.970 --> 00:54:01.309
You need to think about use cases
706
00:54:01.930 --> 00:54:02.670
and document
707
00:54:02.970 --> 00:54:03.950
each use case
708
00:54:04.315 --> 00:54:07.775
separately, but it's more powerful in terms of documentation.
709
00:54:09.275 --> 00:54:10.815
An unexpected thing
710
00:54:11.289 --> 00:54:14.190
that I figured out is that Django settings
711
00:54:14.490 --> 00:54:16.269
is really difficult to customize.
712
00:54:17.130 --> 00:54:23.785
While in Flask, you can easily subclass and override the config class because it's like designed for that.
713
00:54:24.244 --> 00:54:28.825
On Django, the only way to customize the settings interface is by patching.
714
00:54:29.160 --> 00:54:35.180
Yeah. So I discussed it with some Django core developers ways of making the Django settings
715
00:54:35.560 --> 00:54:39.965
because Django has its settings manager when you do from Django dot conf import settings,
716
00:54:40.425 --> 00:54:44.765
it does similar things as Dynacomf, like lazy loading and things like this.
717
00:54:45.065 --> 00:54:48.970
You can add plugins to it. You can customize it in a good
718
00:54:49.270 --> 00:54:53.130
way. So what we did, we patch it. All the other libraries
719
00:54:53.590 --> 00:54:56.010
for Django are patching the settings,
720
00:54:56.355 --> 00:55:00.375
and we are constantly trying to get Django developers more involved
721
00:55:00.755 --> 00:55:02.945
in a way to make it easier.
722
00:55:03.450 --> 00:55:12.109
And the other thing was challenging, I think, is merging data from multiple sources. It's the biggest challenge right now. The source of our,
723
00:55:12.685 --> 00:55:15.745
most of our bugs and requests and and confusion
724
00:55:16.125 --> 00:55:20.465
is how do you take multiple data and merge conflicting data
725
00:55:20.830 --> 00:55:23.650
and how to make the decision of the precedence,
726
00:55:24.110 --> 00:55:28.275
which 1 wins and how the final data will look. So, yeah.
727
00:55:28.595 --> 00:55:35.174
We are still learning and trying to improve better ways of giving flexibility for people that wants to merge data from
728
00:55:35.474 --> 00:55:36.295
different sources.
729
00:55:36.900 --> 00:55:38.200
And so for
730
00:55:38.820 --> 00:55:42.120
an individual or a team who's looking for
731
00:55:42.740 --> 00:55:54.450
a way to manage settings for their application or their project? What are the cases where DynaConf is the wrong choice and maybe they're better suited with a different framework or building their own custom
732
00:55:55.410 --> 00:56:01.990
solution? Well, I like to think that DynaConf is a good fit for any project that needs the couple settings from code.
733
00:56:02.395 --> 00:56:24.855
Yeah? So if your project doesn't need to decouple the setting from your code, and you are good with just a constant dot py file and everything static there, I think then maybe DynaConf is not needed because you were adding the loading overhead and all the things, and you actually doesn't have the need. But I like to think that this is simple, small, and you can use it in any project.
734
00:56:25.315 --> 00:56:30.820
From a simple project having only a single settings dot toml or dot yml file from
735
00:56:31.200 --> 00:56:35.460
a project using multiple environments, thousands of files and external
736
00:56:35.920 --> 00:56:36.420
services.
737
00:56:37.119 --> 00:56:37.619
However,
738
00:56:38.005 --> 00:56:39.865
if your project is a library
739
00:56:40.165 --> 00:56:41.065
or a plugin
740
00:56:41.445 --> 00:56:43.865
to be added to another project,
741
00:56:44.405 --> 00:56:51.119
I don't recommend you using Dynacomf. And I saw people trying to do this and coming to the issues saying that
742
00:56:51.500 --> 00:56:53.680
there were some kind of bug or problem.
743
00:56:54.125 --> 00:56:56.065
And actually, I think libraries
744
00:56:56.525 --> 00:56:59.185
or plugins doesn't have to define settings.
745
00:56:59.485 --> 00:57:00.545
Libraries and plugins
746
00:57:00.925 --> 00:57:01.425
will
747
00:57:01.760 --> 00:57:03.780
consume the settings from the application
748
00:57:04.080 --> 00:57:06.900
that are consuming it. So I think
749
00:57:07.360 --> 00:57:07.860
that
750
00:57:08.320 --> 00:57:12.135
that will be the good choice. If you are doing a library or a plugin,
751
00:57:12.595 --> 00:57:13.095
you
752
00:57:13.555 --> 00:57:31.445
add interfaces that you can receive the settings from the application that you are plugging to and data conf will definitely not be a good solution. You're gonna be, like, frustrated because data conf depends on environment and you don't control environment when you are releasing a library or a plug
753
00:57:31.825 --> 00:57:48.525
in. And as you continue to build and maintain and evolve the Dynacont project, you mentioned a little bit of the work that is kind of in process, but I'm wondering what do you have planned for the near to medium term or any areas of feedback or contribution that you're looking for help with?
754
00:57:48.905 --> 00:57:54.445
Yeah. So first, I would like to thank the contributors and co maintainers of the project. We gather, like,
755
00:57:54.990 --> 00:58:00.530
amount of people that helps me, like, taking decisions and giving the project to the right direction.
756
00:58:00.990 --> 00:58:05.965
And we recently defined the road map for the version 4 0, and there are
757
00:58:06.345 --> 00:58:08.525
3 big changes. We have small fixes,
758
00:58:09.305 --> 00:58:11.885
improvements, but the 3 biggest changes is
759
00:58:12.250 --> 00:58:13.150
we are implementing
760
00:58:13.530 --> 00:58:18.590
that new way to validate settings. This is based on Pydantic and Python type annotations.
761
00:58:19.130 --> 00:58:25.325
So we want to keep all the current way of doing things working without breaking any compatibility,
762
00:58:25.704 --> 00:58:31.790
but we are adding this new way for you to define schema in the Pydantic style that will work good.
763
00:58:32.250 --> 00:58:33.150
The second thing
764
00:58:33.850 --> 00:58:39.470
is we are splitting the code base and removing the core features and transforming in plugins.
765
00:58:40.295 --> 00:58:43.515
So the Dynacomf package, when you install Dynacomf,
766
00:58:43.975 --> 00:58:45.675
will keep being a bundle
767
00:58:46.295 --> 00:58:47.435
with all the
768
00:58:48.200 --> 00:58:49.740
parts installed together.
769
00:58:50.119 --> 00:58:52.619
So it's not going to break any installation.
770
00:58:53.320 --> 00:58:54.700
But then we are creating
771
00:58:55.000 --> 00:58:57.005
external projects on the organization
772
00:58:57.385 --> 00:58:59.005
that will be Dynacom Core,
773
00:58:59.464 --> 00:59:07.080
Dynacom Vault, Dynacom Redis, Dynacom VM. So every single loader, every single functionality will be split into,
774
00:59:07.700 --> 00:59:08.760
separate repository.
775
00:59:09.300 --> 00:59:12.520
Not everything, but all the things that make sense to be separated.
776
00:59:13.065 --> 00:59:26.910
So those can evolve like plugins, and when deciding which feature you want, you can pull only the feature you need to build your settings library. So this is like, technical depth we had on the first implementation,
777
00:59:27.210 --> 00:59:29.790
and we are solving in the version 40.
778
00:59:30.605 --> 00:59:31.345
And I
779
00:59:31.645 --> 00:59:35.745
think the other 1, which is the hardest 1, and I'm working a lot on it,
780
00:59:36.045 --> 00:59:43.089
is the data merging algorithm. I said to you that data merging is the most difficult thing to do in a settings library
781
00:59:44.190 --> 00:59:51.045
and data conf today has a part written in Rust. If you go to crates. Io, you have a project there called
782
00:59:51.425 --> 00:59:51.925
HydroConf.
783
00:59:52.705 --> 00:59:53.925
HydroConf is
784
00:59:54.385 --> 00:59:57.365
the same philosophy, the same strategies of DynaConf
785
00:59:57.900 --> 01:00:00.160
written in Rust for Rust web applications.
786
01:00:01.180 --> 01:00:05.920
And when doing this in Rust, we found that the merging algorithm
787
01:00:06.765 --> 01:00:07.825
is more efficient
788
01:00:08.125 --> 01:00:08.785
in Rust.
789
01:00:09.244 --> 01:00:09.744
So
790
01:00:10.125 --> 01:00:12.385
what we're gonna do is we're gonna write this
791
01:00:12.765 --> 01:00:14.865
as a separate crate on the Rust
792
01:00:15.280 --> 01:00:17.940
and export that crate to a Python model.
793
01:00:18.400 --> 01:00:30.515
And then we use this in Python because it's something that today is really easy to do to create a Python plugin, Python model in Rust. And then we're gonna fix the problem of data merging to be really more efficient
794
01:00:31.190 --> 01:00:36.730
using Rust. So those are the the 3 things that are on the road map for the version 4 0.
795
01:00:37.109 --> 01:00:40.045
And, yeah, hopefully, it comes this year,
796
01:00:40.425 --> 01:00:41.085
I hope.
797
01:00:41.465 --> 01:01:06.174
Well, for anybody who wants to get in touch with you and follow along, I'll have you add your preferred contact information to the show notes. And so with that, I'll move us into the picks. This week, my pick is on the theme of secrets management for applications. So I've been using the SOPS project for a while for some of my work. It's a great utility for being able to encrypt data in flat files and do it in a manner that is
798
01:01:06.690 --> 01:01:34.204
relatively secure because it's able to hook into things like either HashiCorp Vault or a different cloud key management solution so that you can use their secret key material to be able to manage the encryption. So it's a bit safer to be able to actually store it in your source repositories and version your secrets along with your code. So definitely recommend taking a look at that for anybody who's interested. It's a project out of the Mozilla group. And so with that, I'll pass it to you, Bruno. Do you have any picks this week?
799
01:01:34.505 --> 01:01:46.130
Yeah. So first, a note on on your pick. I think it's interesting. I heard about it, never use it. I hope it becomes like a Dynacomps plugging in the future to use the SOPS, the encryption
800
01:01:46.670 --> 01:01:54.055
stuff. We are using currently a thing called Jose, Jose. I don't know how to mention it, but it also does some kind of encryption.
801
01:01:54.355 --> 01:02:01.890
And, yeah, with our plug in system, probably, we're gonna have subs integrated with Dynacomp and yeah. So now going to my picks.
802
01:02:02.349 --> 01:02:06.690
Well, I don't have a pick in terms of software and as you said it can be anything.
803
01:02:07.145 --> 01:02:08.525
So I recently
804
01:02:08.985 --> 01:02:09.485
watched
805
01:02:09.785 --> 01:02:11.965
the series, very interested, called Severance,
806
01:02:12.425 --> 01:02:20.510
and I think it's really great. I just finished it and it's on my mind. I can't start thinking about what's happening there. So I recommend
807
01:02:20.810 --> 01:02:26.455
for people who like a little bit of sci fi, but just a little bit and also computers. This is really interesting.
808
01:02:27.315 --> 01:02:39.640
Yeah. So my other pick is gonna be learn Rust. I think the future of Python is full of Rust. Yeah. We're gonna see more and more Rust in the future for doing better Python projects. So
809
01:02:39.955 --> 01:02:42.055
it's gonna be a skill for every
810
01:02:42.355 --> 01:02:45.495
Python person to learn a little bit of Rust. Yeah.
811
01:02:45.875 --> 01:03:14.895
Alright. Well, thank you very much for taking the time today to join me and share the work that you've been doing on the DynaConference project. It's definitely a very interesting and full featured framework, and settings management is a important capability for any application. So I appreciate all of the time and energy that you and the other maintainers and contributors of that project have put into it. So thank you again for your time, and I hope you enjoy the rest of your day. Yeah. Thank you. Thanks for the invitation. It was very nice to participate and, yeah, have a nice rest of the day.
812
01:03:17.275 --> 01:03:30.335
Thank you for listening. Don't forget to check out our other shows, the Data Engineering Podcast, which covers the latest on modern data management, and the Machine Learning Podcast, which helps you go from idea to production with machine learning. Visit the site at pythonpodcast.com
813
01:03:31.355 --> 01:03:39.760
to subscribe to the show, sign up for the mailing list, and read the show notes. And if you learned something or tried out a project from the show, then tell us about it. Email host at pythonpodcast.com
814
01:03:40.619 --> 01:03:41.440
with your story.
815
01:03:41.740 --> 01:03:46.720
And to help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers.