1
0:0:0,06 --> 0:0:2,72
Nik: Hello, hello, this is Postgres.FM,
as usual.
2
0:0:2,8 --> 0:0:6,3399997
I'm Nik, PostgresAI, and as usual
with me Michael, pgMustard.
3
0:0:6,68 --> 0:0:7,46
Hi, Michael.
4
0:0:8,26 --> 0:0:9,059999
Michael: Hi, Nik.
5
0:0:9,559999 --> 0:0:11,42
Nik: We have a great guest today.
6
0:0:11,719999 --> 0:0:13,16
Did we have you already?
7
0:0:13,48 --> 0:0:15,04
Álvaro from Ongres.
8
0:0:15,72 --> 0:0:16,220001
Álvaro: Hello.
9
0:0:16,46 --> 0:0:18,28
No, it's my 1st time here.
10
0:0:18,54 --> 0:0:20,64
At least as a speaker, not as a
listener.
11
0:0:21,7 --> 0:0:22,54
Nik: Thank you for coming.
12
0:0:22,54 --> 0:0:24,439999
I know you're a very active listener.
13
0:0:25,08 --> 0:0:30,02
We definitely have you on Postgres
TV channel, on YouTube channel,
14
0:0:30,02 --> 0:0:31,2
talking about licenses.
15
0:0:31,22 --> 0:0:32,46
That's why I'm confused.
16
0:0:32,48 --> 0:0:33,3
Do you remember?
17
0:0:34,06 --> 0:0:34,56
Álvaro: Yeah.
18
0:0:35,08 --> 0:0:39,0
Nik: And I know we will touch a little
bit licenses today, just knowing
19
0:0:39,0 --> 0:0:39,64
the topic.
20
0:0:39,760002 --> 0:0:41,02
Yeah, thank you for coming.
21
0:0:41,4 --> 0:0:45,4
I'm especially excited that today's
topic will connect so many
22
0:0:45,4 --> 0:0:46,9
dots, right?
23
0:0:47,16 --> 0:0:52,12
Performance, sharding, Kubernetes
operator, what your company
24
0:0:52,12 --> 0:0:52,84
is doing.
25
0:0:53,08 --> 0:0:55,68
It's so open source and true open
source, right?
26
0:0:55,68 --> 0:1:0,1
I'm very excited, looking forward
to dive into this chat because
27
0:1:0,1 --> 0:1:4,62
many topics I'm very interested
in myself and my company also,
28
0:1:5,32 --> 0:1:6,72
they are connected today.
29
0:1:7,72 --> 0:1:9,92
Álvaro: No, thank you very much
for having me.
30
0:1:9,92 --> 0:1:14,22
It's been a lot of episodes listening
with you over the year
31
0:1:14,34 --> 0:1:16,96
and now being present here, it's
very exciting for me.
32
0:1:16,96 --> 0:1:17,82
So thank you.
33
0:1:18,54 --> 0:1:22,58
Nik: And today we will talk about
benchmark you haven't published
34
0:1:22,58 --> 0:1:23,08
yet.
35
0:1:23,479996 --> 0:1:27,68
So thank you for coming with results
being still prepared.
36
0:1:27,84 --> 0:1:31,36
That's great because I know we
scheduled this episode before
37
0:1:31,36 --> 0:1:33,4
I think you decided to work on
it.
38
0:1:33,4 --> 0:1:34,2
So this is like...
39
0:1:34,2 --> 0:1:35,46
Álvaro: Right, that's correct.
40
0:1:36,98 --> 0:1:38,56
Nik: Action happening in real time.
41
0:1:38,56 --> 0:1:39,3
That's great.
42
0:1:39,52 --> 0:1:44,14
And yeah, I received your pictures,
3 charts about benchmark
43
0:1:44,14 --> 0:1:47,72
data, benchmark results this morning,
so I'm catching up.
44
0:1:47,98 --> 0:1:49,46
Where should we start actually?
45
0:1:49,6 --> 0:1:52,44
Because this is a great topic,
right?
46
0:1:52,44 --> 0:1:55,18
So how Postgres can scale, what
numbers we can achieve.
47
0:1:55,58 --> 0:1:58,94
Maybe we should start mentioning
that Michael and I had an episode
48
0:1:58,94 --> 0:2:3,48
about our company benchmarks on
1 node, no sharding at all, just
49
0:2:3,48 --> 0:2:7,72
1 node, we achieved 4 million TPS
a couple of years ago.
50
0:2:7,72 --> 0:2:9,44
It was a couple of years ago, right?
51
0:2:9,9 --> 0:2:14,52
With SELECT only primary key
lookup just to see how it's not
52
0:2:14,54 --> 0:2:19,24
super huge compared to some other
database systems, which might
53
0:2:19,24 --> 0:2:22,48
be faster in specific workloads,
but it's great that general
54
0:2:22,48 --> 0:2:23,98
Postgres can achieve that.
55
0:2:24,48 --> 0:2:29,8
And you can give us numbers much
bigger because you involve sharding,
56
0:2:29,8 --> 0:2:30,3
right?
57
0:2:31,42 --> 0:2:32,2
Álvaro: That's correct.
58
0:2:32,38 --> 0:2:36,26
If you want, I can start a bit
with my thesis about sharding
59
0:2:36,26 --> 0:2:37,62
in general for Postgres.
60
0:2:38,94 --> 0:2:43,84
I think it's a field where we're
seeing a lot of innovation happening,
61
0:2:43,86 --> 0:2:44,86
and I love this.
62
0:2:45,22 --> 0:2:48,98
Innovation and competition are
great, are what drives us forward.
63
0:2:49,4 --> 0:2:53,74
Nik: Let me interrupt you with
plugging in episodes with Sugu,
64
0:2:53,74 --> 0:2:57,94
Multigres I think we had 1 or
2, and definitely we had 2 episodes
65
0:2:57,98 --> 0:2:59,68
with Lev PgDog.
66
0:2:59,76 --> 0:3:4,02
So There is an innovation starting
last year, which is great.
67
0:3:4,08 --> 0:3:5,78
And I know you have your own opinion.
68
0:3:6,58 --> 0:3:7,84
Apologies for interrupting.
69
0:3:8,08 --> 0:3:8,86
Let's continue.
70
0:3:9,72 --> 0:3:9,96
Álvaro: Yeah.
71
0:3:9,96 --> 0:3:10,2
Yeah.
72
0:3:10,2 --> 0:3:11,52
I do have my opinion.
73
0:3:11,98 --> 0:3:16,3
Basically databases, they take
a long time to mature.
74
0:3:16,64 --> 0:3:20,28
Databases need to be the places
where you can trust your data,
75
0:3:20,28 --> 0:3:22,7
that it's gonna be there, that
you're durable, and they obviously
76
0:3:22,72 --> 0:3:25,62
obey the rest of the properties
that you expect from a database.
77
0:3:26,14 --> 0:3:30,3
And typically new databases are
risky or tend to lag adoption
78
0:3:30,3 --> 0:3:31,58
because they need to mature.
79
0:3:32,08 --> 0:3:35,24
When you're sharding, my thoughts
keep being the same.
80
0:3:35,34 --> 0:3:39,24
You really need the mature and
trust solution, a boring, I would
81
0:3:39,24 --> 0:3:42,1
say, technology to run your shards
because now you don't have
82
0:3:42,1 --> 0:3:43,68
1 cluster, you have many clusters.
83
0:3:44,28 --> 0:3:47,72
Each shard, each portion of the
database, it's a cluster on its
84
0:3:47,72 --> 0:3:48,22
own.
85
0:3:48,42 --> 0:3:51,96
So the software that manages all
that needs to also be well-established,
86
0:3:52,28 --> 0:3:53,04
let's say.
87
0:3:53,68 --> 0:3:56,94
And again, in competition and innovation
are absolutely great
88
0:3:56,94 --> 0:3:58,24
and very, very welcome.
89
0:3:58,74 --> 0:4:1,8
But I believe that there's a great
technology out there, open
90
0:4:1,8 --> 0:4:5,6
source for sharding Postgres that's
been in the market for, I
91
0:4:5,6 --> 0:4:8,16
guess, like more than 10 years
called Citus.
92
0:4:9,24 --> 0:4:10,58
And Citus is just great.
93
0:4:10,76 --> 0:4:13,94
Might not be perfect, but it's
really good enough or more than
94
0:4:13,94 --> 0:4:14,68
that even.
95
0:4:15,14 --> 0:4:18,9
And a lot of people may have forgotten
about it or put it a little
96
0:4:18,9 --> 0:4:19,46
bit aside.
97
0:4:19,46 --> 0:4:23,86
Myself, the same several years
ago, until I started with thinking,
98
0:4:23,9 --> 0:4:27,46
OK, people need to shard Postgres
and people need to scale Postgres.
99
0:4:27,94 --> 0:4:32,56
And not only because of the performance
directly, because as
100
0:4:32,56 --> 0:4:37,22
you demonstrated, you can run very
high TPS on a single node.
101
0:4:37,38 --> 0:4:40,46
But sometimes it's also about the
blast radius.
102
0:4:40,92 --> 0:4:45,06
Like if you're running a SaaS shop
or an e-commerce and your
103
0:4:45,06 --> 0:4:47,82
database cluster goes down for
whatever reason.
104
0:4:47,9 --> 0:4:51,44
Sometimes it's a configuration
issue with the network, whatever
105
0:4:51,44 --> 0:4:52,16
it is, right?
106
0:4:52,16 --> 0:4:55,02
Your cluster is down, then your
whole thing is down.
107
0:4:55,58 --> 0:5:0,78
If you shard and you split your
database into, let's say 16 logical
108
0:5:0,84 --> 0:5:1,5
shards, right?
109
0:5:1,5 --> 0:5:2,22
Or workers.
110
0:5:2,68 --> 0:5:6,06
And then 1 goes down because of
a configuration error, then it's
111
0:5:6,06 --> 0:5:8,94
only a small percentage of your
users that are effectively down,
112
0:5:8,94 --> 0:5:11,56
which it's not good, but it's definitely
not 100%.
113
0:5:11,64 --> 0:5:14,18
Your PR is completely different
night and day, right?
114
0:5:14,18 --> 0:5:17,44
So a lot of people now need to
shard either because of performance
115
0:5:17,44 --> 0:5:20,42
reasons, because they cannot scale
the writes to a single primary,
116
0:5:20,82 --> 0:5:25,46
and or because of high availability
and blast radius reasons.
117
0:5:26,28 --> 0:5:28,88
And we always had Citus there.
118
0:5:29,38 --> 0:5:31,16
And Citus is fully featured.
119
0:5:32,02 --> 0:5:35,16
It has almost everything that you
need for sharding, for running
120
0:5:35,16 --> 0:5:38,34
complex queries that go down, they're
pushed down to the workers,
121
0:5:39,0 --> 0:5:42,48
and has the tools for rebalancing,
for dealing with hot shards,
122
0:5:42,74 --> 0:5:45,48
for almost all operational activities
that you need to perform
123
0:5:45,48 --> 0:5:47,86
on a sharded database and it just
works.
124
0:5:48,32 --> 0:5:52,44
I believe because their initial
focus was more OLAP style or
125
0:5:52,44 --> 0:5:57,24
mixed OLTP OLAP that people started
considering Citus only for
126
0:5:57,24 --> 0:6:0,8
that and it's definitely good fit
for that but it's also really
127
0:6:0,8 --> 0:6:1,7
good for OLTP.
128
0:6:3,06 --> 0:6:4,24
Nik: 2 points here.
129
0:6:4,24 --> 0:6:7,32
1st of all, you said we always
had.
130
0:6:7,54 --> 0:6:11,14
Let me disagree with you because
when I actually was looking
131
0:6:11,14 --> 0:6:13,62
into Citus, it was several years
ago.
132
0:6:13,96 --> 0:6:17,66
The Key features were not open
source.
133
0:6:17,8 --> 0:6:19,4
For example, online rebalancing.
134
0:6:19,94 --> 0:6:22,58
So we didn't have it in the open
source area.
135
0:6:23,26 --> 0:6:26,42
It was like you need to have a
contract with them to have online
136
0:6:26,42 --> 0:6:30,24
rebalancing, so it quite quickly
becomes out of scope if you
137
0:6:30,24 --> 0:6:33,02
want to rely on open source and
avoid vendor lock-in.
138
0:6:33,08 --> 0:6:34,1
Now it's different.
139
0:6:34,2 --> 0:6:36,24
It has changed a few years ago.
140
0:6:36,34 --> 0:6:39,88
All features after acquisition
by Microsoft, at some point all
141
0:6:39,88 --> 0:6:42,6
features went to open source, which
is great.
142
0:6:43,26 --> 0:6:45,26
But it's not like we always had,
right?
143
0:6:45,26 --> 0:6:46,78
Álvaro: Yeah, you're absolutely
right.
144
0:6:46,8 --> 0:6:47,8
And thanks for that.
145
0:6:47,8 --> 0:6:52,5
I think, if I'm not mistaken, Citus
version 11 was the 1st 1
146
0:6:52,5 --> 0:6:54,82
that included everything as open
source.
147
0:6:55,24 --> 0:6:57,32
And that was, I think, 2022.
148
0:6:57,64 --> 0:7:2,22
So 4 years ago, 4 years and 0.5
ago, that was released as fully
149
0:7:2,22 --> 0:7:3,58
open source and has been since
then.
150
0:7:3,58 --> 0:7:4,7
So yes, you're right.
151
0:7:4,74 --> 0:7:5,68
For this podcast,
152
0:7:6,22 --> 0:7:9,5
Nik: you're right also, because
this podcast exists for years.
153
0:7:9,72 --> 0:7:10,18
Exactly.
154
0:7:10,18 --> 0:7:12,62
So, good.
155
0:7:13,38 --> 0:7:16,52
And the 2nd point is OLAP versus
OLTP.
156
0:7:16,56 --> 0:7:21,34
My concern, like, last 4 years,
I think I didn't dive into Citus.
157
0:7:21,34 --> 0:7:25,96
I remember, diving means like I
run it, inspect myself, the benchmark
158
0:7:25,96 --> 0:7:26,44
and so on.
159
0:7:26,44 --> 0:7:29,26
I did it before we started podcast
actually.
160
0:7:29,26 --> 0:7:32,9
I remember I was reporting some
bugs And since then I didn't
161
0:7:32,9 --> 0:7:37,58
dive into it, but my concern is
we didn't see great examples
162
0:7:38,4 --> 0:7:43,82
of some well-known companies, SaaS
companies, for example, running
163
0:7:43,82 --> 0:7:44,32
it.
164
0:7:44,54 --> 0:7:48,76
Like, you know, if somebody like
GitHub scale would be running
165
0:7:48,76 --> 0:7:51,3
Citus, I wouldn't have any concerns
at all.
166
0:7:51,72 --> 0:7:52,94
What are the great examples?
167
0:7:53,16 --> 0:7:54,6
OLTP style.
168
0:7:56,04 --> 0:7:56,92
That's my concern.
169
0:7:56,92 --> 0:7:59,86
And I think it's still like, I
agree, like what numbers you bring,
170
0:7:59,86 --> 0:8:0,8
and this is interesting.
171
0:8:0,8 --> 0:8:2,46
We dive into them today.
172
0:8:2,8 --> 0:8:6,42
And I was in talks with Citus team
some time ago, maybe a couple
173
0:8:6,42 --> 0:8:7,32
of years ago.
174
0:8:7,48 --> 0:8:12,54
And they presented me some examples
which were not OLTP.
175
0:8:12,62 --> 0:8:17,6
For example, some search for, I
think, lawyers of video, where
176
0:8:17,6 --> 0:8:22,22
I thought like, actually lawyers
can wait 10 seconds to find
177
0:8:22,22 --> 0:8:24,3
problematic materials and so on.
178
0:8:24,38 --> 0:8:28,58
It's not OLTP when we know 200 milliseconds,
our very 1st episode
179
0:8:28,58 --> 0:8:33,04
in this podcast means that humans
already notice delay.
180
0:8:33,46 --> 0:8:36,18
Álvaro: So I think there's 2 angles
to what you're saying.
181
0:8:36,18 --> 0:8:40,36
So 1 is the technical 1, and the
other 1 is the marketing or
182
0:8:40,36 --> 0:8:41,98
trust 1, let's say.
183
0:8:42,28 --> 0:8:46,02
On the trust 1, I would say you're
right, meaning that there's
184
0:8:46,02 --> 0:8:47,22
no such great examples.
185
0:8:47,22 --> 0:8:49,54
And this actually piggybacks on
what I was saying.
186
0:8:49,54 --> 0:8:52,24
It looks like it's a technology
that didn't get the momentum
187
0:8:52,24 --> 0:8:56,1
it could have had, or it could
still has, because of the technological
188
0:8:56,32 --> 0:8:58,04
means that it conveys, right?
189
0:8:58,18 --> 0:9:1,72
I can tell you that we have right
now significant examples of
190
0:9:1,72 --> 0:9:2,64
this kind of usage.
191
0:9:2,64 --> 0:9:6,92
We have now a customer running
Citus on StackGres that is actually
192
0:9:7,02 --> 0:9:10,62
growing database towards 2.2 petabytes
by the end of the year.
193
0:9:10,92 --> 0:9:11,12
And...
194
0:9:11,12 --> 0:9:13,54
Nik: Wait, hold on 1 2nd, 2.2 petabytes?
195
0:9:14,44 --> 0:9:14,94
Yes.
196
0:9:15,48 --> 0:9:17,46
It's just 1 big Citus cluster?
197
0:9:18,08 --> 0:9:18,28
Yes.
198
0:9:18,28 --> 0:9:19,24
Álvaro: Running on StackGres?
199
0:9:20,02 --> 0:9:20,52
Yes.
200
0:9:21,04 --> 0:9:21,1
Nik: Okay.
201
0:9:21,1 --> 0:9:22,06
How many nodes?
202
0:9:23,0 --> 0:9:23,5
Álvaro: 24.
203
0:9:24,78 --> 0:9:30,26
Each node is, I'm not sure if it's
64 or 96 cores, and 0.5 a
204
0:9:30,26 --> 0:9:31,94
terabyte or more of RAM.
205
0:9:33,34 --> 0:9:34,5
Nik: Wow, okay.
206
0:9:34,86 --> 0:9:36,6
Álvaro: Those numbers are actually
quite impressive.
207
0:9:37,36 --> 0:9:42,44
It's an IoT system and they ingest
data massively from millions
208
0:9:42,44 --> 0:9:45,52
and millions of devices that send
heartbeats every 5 minutes.
209
0:9:46,1 --> 0:9:48,72
And they have a 10 day retention
period.
210
0:9:49,34 --> 0:9:52,84
So basically they're deleting 1
10th of the database and ingesting
211
0:9:52,84 --> 0:9:54,94
1 10th of the database every day.
212
0:9:55,44 --> 0:9:58,74
Right now they're in ramping up
period, and as of today, they're
213
0:9:58,74 --> 0:10:3,9
ingesting some 30 plus terabytes
per day, and this is gonna grow
214
0:10:3,9 --> 0:10:5,74
to 200 terabytes per day.
215
0:10:5,74 --> 0:10:8,08
So they're growing by 6X by the
end of the year.
216
0:10:8,92 --> 0:10:11,26
And yeah, this is running on a
single Citus cluster.
217
0:10:11,64 --> 0:10:14,98
We can discuss whether this is
a little bit OLTP or OLAP.
218
0:10:15,06 --> 0:10:16,22
It's a bit of both.
219
0:10:16,36 --> 0:10:17,98
Ingestion is OLTP.
220
0:10:18,84 --> 0:10:21,28
Queries, there are a little bit
of both.
221
0:10:21,28 --> 0:10:25,38
Also OLTP queries when they're
accessing health information about
222
0:10:25,38 --> 0:10:26,4
a given device.
223
0:10:26,88 --> 0:10:31,16
Or more aggregated reports, like
how many devices are down or
224
0:10:31,16 --> 0:10:32,22
having some issues.
225
0:10:32,44 --> 0:10:35,74
So it's a bit of a mixed workload,
but I think it works.
226
0:10:35,74 --> 0:10:39,52
Again, I agree, there are not so
many public great examples about
227
0:10:39,52 --> 0:10:40,46
using Citus.
228
0:10:40,52 --> 0:10:44,06
Citus also introduced some releases
ago, the support for sharding
229
0:10:44,06 --> 0:10:48,84
by tenant, which is actually directly
towards SaaS enterprises,
230
0:10:48,9 --> 0:10:49,14
right?
231
0:10:49,14 --> 0:10:51,98
The SaaS applications where you
don't shard by row, you shard
232
0:10:51,98 --> 0:10:55,02
by tenant, which is a great fit
for those cases too.
233
0:10:55,24 --> 0:10:57,94
Typically, these cases where you
create like a lot of tables
234
0:10:58,0 --> 0:10:59,08
for every tenant.
235
0:10:59,28 --> 0:11:1,74
So it's another way of isolating
those workloads.
236
0:11:1,98 --> 0:11:4,18
But from a technical perspective
and speaking.
237
0:11:5,22 --> 0:11:8,54
Nik: Yeah, I just, I just, I'm,
I'm, I'm smiling at my glasses
238
0:11:8,56 --> 0:11:11,8
because they said, they started
arguing with me saying that there
239
0:11:11,8 --> 0:11:16,62
are public well-known names like
Heap and the Cloudflare named
240
0:11:16,62 --> 0:11:20,14
on, on Citus pages for, for
years.
241
0:11:20,14 --> 0:11:24,18
But, but both of these cases are
analytical cases, 1st of all,
242
0:11:24,18 --> 0:11:27,38
and both these cases went away
from Citus, as we know.
243
0:11:27,62 --> 0:11:33,06
Like, I was at Meetup when Cloudflare
explained how they moved
244
0:11:33,06 --> 0:11:35,64
away from Citus to ClickHouse.
245
0:11:36,06 --> 0:11:39,44
And I know Heap Analytics moved
somewhere else as well.
246
0:11:39,56 --> 0:11:42,04
Heap Analytics was a great case,
but again, analytics.
247
0:11:42,72 --> 0:11:45,56
And Cloudflare, it was analytics,
DNS analytics.
248
0:11:45,78 --> 0:11:46,86
So it's not OLTP.
249
0:11:47,2 --> 0:11:48,7
It can be HTAP.
250
0:11:48,7 --> 0:11:49,4
Yeah, I agree.
251
0:11:49,4 --> 0:11:50,7
And your case is interesting.
252
0:11:50,74 --> 0:11:54,02
It's not what I want to see, like
I said, GitHub or some social
253
0:11:54,02 --> 0:11:56,46
media or something, when there
are many humans working.
254
0:11:56,52 --> 0:12:1,82
Your case is a great case for sharding
Specifically because it
255
0:12:1,92 --> 0:12:6,78
shows exactly where the problem
with the biggest pain of single
256
0:12:6,78 --> 0:12:8,5
cluster is a lot of writes.
257
0:12:9,38 --> 0:12:14,18
This case is highlighting this
particular use case, which is
258
0:12:14,18 --> 0:12:14,66
great.
259
0:12:14,66 --> 0:12:18,52
But I'm still like, my heart is
with social media, SaaS, like
260
0:12:18,52 --> 0:12:21,86
these things, like e-commerce,
online stuff, where a lot of humans
261
0:12:21,86 --> 0:12:26,18
try to deal with some system and
then they want greater responsiveness
262
0:12:26,92 --> 0:12:28,28
of all API calls.
263
0:12:28,28 --> 0:12:31,18
Álvaro: So let's talk about that
because part of my response
264
0:12:31,2 --> 0:12:34,08
also was coming from a technical
angle and from a technical angle
265
0:12:34,08 --> 0:12:38,56
we've been running some benchmarks
and 1 of the questions whether
266
0:12:38,56 --> 0:12:42,56
to say the system is ready for
OLTP or not is the number of concurrent
267
0:12:42,56 --> 0:12:45,32
queries that it can run and the
latency at which those queries
268
0:12:45,32 --> 0:12:45,82
run.
269
0:12:46,32 --> 0:12:51,82
And Citus can run on millions of
transactions per 2nd, even under
270
0:12:51,82 --> 0:12:53,5
1 millisecond p50.
271
0:12:54,62 --> 0:12:55,94
I would call that OLTP.
272
0:12:56,8 --> 0:12:57,7
Nik: I agree here.
273
0:12:57,7 --> 0:13:1,68
If you check very fast SELECTs,
we have benchmarks.
274
0:13:1,78 --> 0:13:6,94
We did it multiple times for actually
all sharding systems, open
275
0:13:6,94 --> 0:13:10,46
source sharding systems we can
reach, including SPQR, Citus,
276
0:13:10,84 --> 0:13:14,74
Multigres actually we also tested,
PgDog, something else.
277
0:13:15,3 --> 0:13:20,26
And Citus, you checked for latency
overhead, off latency, latency
278
0:13:20,26 --> 0:13:21,66
overhead benchmark, simple.
279
0:13:22,54 --> 0:13:26,74
Which like if primary key lookup is
sub millisecond, you don't want
280
0:13:26,74 --> 0:13:29,66
another few milliseconds to be
introduced.
281
0:13:29,7 --> 0:13:33,96
And Citus was actually looking
very well.
282
0:13:34,64 --> 0:13:38,04
It was introducing like 200 microseconds
or so, as I remember.
283
0:13:38,8 --> 0:13:42,14
Álvaro: Yeah, I actually have drilled
down on the CPU consumption
284
0:13:42,2 --> 0:13:44,0
per layer for this kind of queries.
285
0:13:44,44 --> 0:13:48,84
And the Total aggregated CPU consumption
per query, including
286
0:13:49,5 --> 0:13:53,4
the coordinator, including the
worker, including the connection
287
0:13:53,4 --> 0:13:58,22
pooling, excluding the network,
is somewhere in between 300 to
288
0:13:58,44 --> 0:13:59,02
400 microseconds.
289
0:13:59,34 --> 0:14:0,52
Nik: Microseconds, exactly.
290
0:14:0,54 --> 0:14:1,02
Yeah, yeah.
291
0:14:1,02 --> 0:14:1,5
Okay.
292
0:14:1,5 --> 0:14:4,46
Slightly different from what I
remember from our benchmarks but
293
0:14:4,46 --> 0:14:8,48
well the point here is that it
can be very quickly eaten by something
294
0:14:8,48 --> 0:14:12,48
like network complexity right you
won't notice this small overhead
295
0:14:12,48 --> 0:14:16,0
behind something else like
just hops between yeah
296
0:14:16,0 --> 0:14:18,72
Álvaro: the networks that can be
worse than that yeah yeah
297
0:14:18,74 --> 0:14:20,06
Nik: that's I agree yeah
298
0:14:20,34 --> 0:14:23,24
Michael: Álvaro when you query
when you're doing that check are
299
0:14:23,24 --> 0:14:25,9
you going through a coordinator
in Citus?
300
0:14:26,54 --> 0:14:29,7
Because I know now you can query
directly 1 of the workers.
301
0:14:30,06 --> 0:14:31,9
Would that reduce the overhead?
302
0:14:33,52 --> 0:14:37,96
Álvaro: So, yes, the usual way
you wire up a Citus cluster is
303
0:14:37,96 --> 0:14:42,18
you have workers, which are where
you split the data into and
304
0:14:42,18 --> 0:14:47,5
then you have the coordinator,
which acts as a metadata entry
305
0:14:47,5 --> 0:14:49,46
point, but also as a query router.
306
0:14:49,96 --> 0:14:52,7
And so it is the entry point by
default.
307
0:14:53,14 --> 0:14:56,26
It's true that Citus at version
11 introduced a functionality
308
0:14:56,46 --> 0:14:59,44
where you can query also the worker
nodes directly.
309
0:14:59,44 --> 0:15:1,1
There's 2 reasons to do this.
310
0:15:1,28 --> 0:15:4,78
1, if you know from the application
perspective, the sharding
311
0:15:4,78 --> 0:15:7,66
algorithm and you know which shard
to talk directly, which is
312
0:15:7,66 --> 0:15:10,86
probably not a good idea, but technically
sometimes you can do
313
0:15:10,86 --> 0:15:11,16
this.
314
0:15:11,16 --> 0:15:14,72
It is a good idea actually, but
the algorithm is a little bit
315
0:15:14,72 --> 0:15:16,06
internal to Citus, right?
316
0:15:16,06 --> 0:15:19,4
If it would be like clear, exposed
algorithm, then it would be
317
0:15:19,4 --> 0:15:21,9
very good for applications that
want to implement that algorithm,
318
0:15:21,9 --> 0:15:23,16
then talk directly to Shards.
319
0:15:23,16 --> 0:15:25,68
But at least Citus gives you the
opportunity to do that if you
320
0:15:25,68 --> 0:15:26,24
want to do so.
321
0:15:26,24 --> 0:15:27,72
You can target directly the workers.
322
0:15:27,72 --> 0:15:30,04
And if you target directly the
worker, there's no query routing.
323
0:15:30,04 --> 0:15:32,98
It goes directly and resolves the
query locally, if it's local
324
0:15:33,04 --> 0:15:35,66
or fans out to other workers if
required.
325
0:15:36,22 --> 0:15:39,58
But the other reason is to try
to scale the single coordinator.
326
0:15:39,96 --> 0:15:43,82
And I believe this actually also
piggybacks a little bit with
327
0:15:43,82 --> 0:15:47,5
what you were saying, Nikolay, that
Citus maybe was abandoned
328
0:15:47,5 --> 0:15:49,5
by some project before, like Cloudflare.
329
0:15:50,22 --> 0:15:53,42
And Citus can scale workers horizontally
as much as you need.
330
0:15:53,42 --> 0:15:56,52
I was referring to this 24 node
cluster before, and there's been
331
0:15:56,52 --> 0:16:0,12
reports of much larger clusters,
like 60 nodes or more.
332
0:16:0,3 --> 0:16:2,12
And they scale very well.
333
0:16:2,26 --> 0:16:4,84
However, the coordinator remains
only 1.
334
0:16:5,9 --> 0:16:11,18
And if you outrun the coordinator,
then you're hitting a ceiling.
335
0:16:11,52 --> 0:16:14,58
That ceiling is normally quite
high and actually we've measured
336
0:16:14,58 --> 0:16:17,14
that ceiling, but it exists, right?
337
0:16:17,22 --> 0:16:19,92
So that's why Citus introduced
functionality, you can query the
338
0:16:19,92 --> 0:16:22,62
routers directly, and then you
have like essentially multiple
339
0:16:22,64 --> 0:16:23,14
coordinators.
340
0:16:23,6 --> 0:16:26,84
But I don't think that's the way
to go either, because then you're
341
0:16:26,84 --> 0:16:28,12
now contending workloads.
342
0:16:28,26 --> 0:16:32,5
So you have nodes that act both
as query routers, which consumes
343
0:16:32,5 --> 0:16:36,38
CPU, consume resources, notable,
they require connections, and
344
0:16:36,38 --> 0:16:39,72
they spread all the connections
to other nodes, making also connection
345
0:16:39,72 --> 0:16:40,54
math harder.
346
0:16:41,2 --> 0:16:43,62
And they're also serving data and
queries.
347
0:16:43,84 --> 0:16:49,54
So it's a bit of a mixed workload
that I believe is hard to dimension
348
0:16:49,78 --> 0:16:53,04
and do capacity planning around
that and makes everything much
349
0:16:53,04 --> 0:16:53,68
more complicated.
350
0:16:53,68 --> 0:16:55,74
So it's not a perfect solution
either.
351
0:16:56,48 --> 0:16:59,74
Nik: I think we forgot an important
piece.
352
0:17:0,72 --> 0:17:5,14
Why Álvaro at all talking about
Citus, right?
353
0:17:5,14 --> 0:17:6,3
This is a good question.
354
0:17:6,3 --> 0:17:7,64
You're doing something else.
355
0:17:7,64 --> 0:17:9,66
Why are you talking about Citus
at all?
356
0:17:9,72 --> 0:17:11,46
Citus is in the hands of Microsoft.
357
0:17:12,56 --> 0:17:13,66
What happened here?
358
0:17:14,54 --> 0:17:19,02
You developed StackGres, and then
somehow these paths joined,
359
0:17:19,02 --> 0:17:19,52
right?
360
0:17:19,6 --> 0:17:22,66
They met these things and you have
examples.
361
0:17:23,68 --> 0:17:26,7
That's why you mentioned this 2
petabyte example, right?
362
0:17:26,88 --> 0:17:29,6
Can you explain a little bit what's
happening, why you focus
363
0:17:29,6 --> 0:17:30,98
on Citus so much lately?
364
0:17:32,5 --> 0:17:35,78
Álvaro: Well, it's not me, it's
what customers and users request.
365
0:17:36,42 --> 0:17:40,32
I see a growing concern and need
to scale Postgres beyond the
366
0:17:40,32 --> 0:17:43,4
single primary for the reasons
I mentioned before, write limitations
367
0:17:43,94 --> 0:17:45,36
and also blast radius.
368
0:17:45,78 --> 0:17:48,62
And that's why we are also seeing
an explosion of new sharding
369
0:17:48,62 --> 0:17:49,84
technologies for Postgres.
370
0:17:49,84 --> 0:17:53,5
So this is a real need and is driven
by market demand.
371
0:17:53,5 --> 0:17:57,26
We are seeing a lot of people coming
to us asking how we can
372
0:17:57,26 --> 0:17:58,4
shard our database.
373
0:17:59,44 --> 0:18:2,44
And StackGres, it's a platform for
running Postgres, right?
374
0:18:2,44 --> 0:18:5,28
That's it, it's fully featured,
it's really advanced, but it's
375
0:18:5,28 --> 0:18:6,6
a platform for running Postgres.
376
0:18:6,66 --> 0:18:10,14
And so we saw this need and we
saw early that we had to provide
377
0:18:10,14 --> 0:18:11,86
users a way to shard.
378
0:18:11,98 --> 0:18:15,04
And actually we were in a very
privileged position, I would say,
379
0:18:15,04 --> 0:18:18,66
because what we've done in StackGres
with single clusters is
380
0:18:18,66 --> 0:18:22,2
orchestrate and allow you to run
the whole stack of components
381
0:18:22,2 --> 0:18:25,46
that you need to run Postgres,
connection pooling, backups, monitoring,
382
0:18:25,6 --> 0:18:28,38
all that, all automated and all
packaged.
383
0:18:28,44 --> 0:18:33,28
So deploying it is extremely easy
while also following best practices
384
0:18:33,28 --> 0:18:35,78
that we have from our company's
operational experience.
385
0:18:36,48 --> 0:18:40,18
But then we saw that creating a
Citus cluster involved a lot
386
0:18:40,18 --> 0:18:43,86
of manual work because it's no
longer 1 cluster.
387
0:18:44,14 --> 0:18:47,06
Now, if you think about it, and
the model we follow in StackGres
388
0:18:47,12 --> 0:18:51,78
is that every component of the
Citus cluster, the workers, the
389
0:18:51,78 --> 0:18:56,14
coordinator, all of those are a
Postgres cluster in itself, typically
390
0:18:56,14 --> 0:18:59,02
with more than 1 instance also
for high availability purpose,
391
0:18:59,02 --> 0:18:59,52
right?
392
0:18:59,6 --> 0:19:2,54
And they need to be wired up precisely
with the same configuration
393
0:19:2,64 --> 0:19:5,74
parameters across many places or
otherwise it will not behave
394
0:19:5,74 --> 0:19:6,84
as a single cluster.
395
0:19:7,16 --> 0:19:10,82
So it's not only a lot of work
to wire up as let's say a 24,
396
0:19:10,86 --> 0:19:14,18
48 node the Citus cluster but also
to do it right.
397
0:19:14,66 --> 0:19:17,8
And With our automation and orchestration
that we built for a
398
0:19:17,8 --> 0:19:19,54
StackGres, we could do this work.
399
0:19:19,78 --> 0:19:23,2
And now the reality is that if
you use a StackGres and want to
400
0:19:23,2 --> 0:19:26,12
create a sharded cluster with Citus,
by the way, we support all
401
0:19:26,12 --> 0:19:29,94
the other sharding technologies
in StackGres, not only Citus,
402
0:19:30,18 --> 0:19:32,78
even though it's the most used
and the most feature complete,
403
0:19:32,78 --> 0:19:34,98
I would say solution for sharding
that we have.
404
0:19:35,34 --> 0:19:39,86
But with the 13 lines of YAML,
really high level, you can create
405
0:19:39,86 --> 0:19:42,84
a Citus cluster of 60 nodes
in a few minutes.
406
0:19:43,28 --> 0:19:46,06
You can even do from a web console,
Click, click, click, boom,
407
0:19:46,06 --> 0:19:47,62
and you get the sharded cluster.
408
0:19:47,78 --> 0:19:50,72
There's nothing else compared to
this other than the managed
409
0:19:50,72 --> 0:19:53,04
service and Microsoft offers on
Azure.
410
0:19:53,92 --> 0:19:56,82
And that's the gap we wanted to
cover because we saw demand and
411
0:19:56,82 --> 0:19:57,88
it's paying off.
412
0:19:58,02 --> 0:20:1,36
I would say a significant percentage
of our new customers and
413
0:20:1,36 --> 0:20:3,28
users come for sharding.
414
0:20:4,54 --> 0:20:8,0
It is very widespread and it's
a need that few people are focused
415
0:20:8,0 --> 0:20:11,6
on solving, especially using a
technology that is proven and
416
0:20:11,6 --> 0:20:13,52
well maintained as of today like
Citus.
417
0:20:13,94 --> 0:20:14,18
It's a
418
0:20:14,18 --> 0:20:17,32
Michael: good point about Citus
being on Microsoft's platform
419
0:20:17,32 --> 0:20:21,58
and it could because it's AGPL
it means other platforms are
420
0:20:21,58 --> 0:20:25,78
going to really struggle probably
just can't offer it right so
421
0:20:25,92 --> 0:20:30,06
actually it makes sense that the
if the only kind of 10 year
422
0:20:30,06 --> 0:20:32,74
old sharding actually it's maybe
it's not the only it because
423
0:20:32,74 --> 0:20:36,64
it wasn't SPQR quite old as well
as you say they feature complete
424
0:20:37,14 --> 0:20:40,44
sharding solution on Postgres if
that's 10 years old but you
425
0:20:40,44 --> 0:20:44,28
can't get it on any managed services
except for 1, actually,
426
0:20:44,54 --> 0:20:48,42
it makes sense that there's demand
for, okay, how do we self-manage
427
0:20:48,64 --> 0:20:49,12
this?
428
0:20:49,12 --> 0:20:52,1
Nik: We need to revisit the topic
of AGPL misconception.
429
0:20:52,58 --> 0:20:53,5
How to say?
430
0:20:54,02 --> 0:20:58,62
So AGPL allows you to run it on
your managed service, but you
431
0:20:58,62 --> 0:21:2,08
cannot modify it without not publishing
source code.
432
0:21:2,62 --> 0:21:4,94
And right, this is 1st thing.
433
0:21:4,94 --> 0:21:7,8
Let me explain my words and then
Álvaro you can correct me if
434
0:21:7,8 --> 0:21:8,2
I'm wrong.
435
0:21:8,2 --> 0:21:10,94
I'm just curious if I understand
it right.
436
0:21:11,14 --> 0:21:14,12
And then Álvaro is not doing a
managed service at all.
437
0:21:14,54 --> 0:21:19,3
StackGres is just like operator
plus of course there is enterprise
438
0:21:19,3 --> 0:21:21,1
edition with support and so on.
439
0:21:21,1 --> 0:21:24,64
This is like money coming from
support, but it can be installed
440
0:21:24,64 --> 0:21:25,74
anywhere, right?
441
0:21:25,9 --> 0:21:29,68
Like having AGPL components doesn't
hurt at all.
442
0:21:29,84 --> 0:21:34,7
And, And finally, I know StackGres
was on AGPL until lately, right?
443
0:21:34,84 --> 0:21:36,18
Can you elaborate this?
444
0:21:37,2 --> 0:21:38,16
On 1 bit.
445
0:21:38,44 --> 0:21:39,34
Yeah, am I right?
446
0:21:39,34 --> 0:21:41,08
It's not an AGPL anymore?
447
0:21:41,58 --> 0:21:44,24
Álvaro: It's very good insider
information, yes.
448
0:21:44,24 --> 0:21:45,68
So, 1st of all, AGPL.
449
0:21:46,06 --> 0:21:47,58
I'm not a lawyer, okay?
450
0:21:47,68 --> 0:21:51,42
But I love AGPL and GPL or in
general copyleft licenses.
451
0:21:52,04 --> 0:21:54,88
I believe people say they are viral,
that they restrict your
452
0:21:54,88 --> 0:21:55,28
freedom.
453
0:21:55,28 --> 0:21:56,84
I believe it's exactly the opposite.
454
0:21:56,84 --> 0:22:0,92
They provide forward freedom warranties,
meaning that whatever
455
0:22:0,92 --> 0:22:4,3
software is published under a copyleft
license, it is providing
456
0:22:4,3 --> 0:22:6,6
a warranty that will never be closed
source.
457
0:22:6,6 --> 0:22:9,26
It will always remain under such
a license.
458
0:22:9,4 --> 0:22:12,14
So it's a forward warranty for
users of Freedom.
459
0:22:12,4 --> 0:22:17,02
Whereas a license like the Apache
2 or BSD, they can be turned
460
0:22:17,02 --> 0:22:20,28
into proprietary software, and
that software may cease to exist
461
0:22:20,28 --> 0:22:21,62
as an open source software.
462
0:22:22,2 --> 0:22:23,44
That's why I like them.
463
0:22:23,44 --> 0:22:27,44
Now, the reality is that a lot
of people believe that they cannot
464
0:22:27,44 --> 0:22:30,4
be used, for example, as a managed
service, but you put it very
465
0:22:30,4 --> 0:22:30,9
well.
466
0:22:30,92 --> 0:22:34,84
You can perfectly well use an AGPL
software while offering a
467
0:22:34,84 --> 0:22:35,66
managed service.
468
0:22:35,74 --> 0:22:38,6
The only thing that the license
force you to do is that whomever
469
0:22:38,6 --> 0:22:42,1
interacts over the network with
your software need to provide
470
0:22:42,1 --> 0:22:45,72
the same warranties that you got
for that software, which means
471
0:22:45,72 --> 0:22:47,06
publishing the source code.
472
0:22:47,08 --> 0:22:50,86
So if you use Citus, and even with
Citus, it's arguable because
473
0:22:50,86 --> 0:22:52,62
you don't talk directly to Citus.
474
0:22:52,94 --> 0:22:56,32
But even if we do a little more
loose interpretation, or because
475
0:22:56,32 --> 0:23:0,04
I have Citus, then I'm bound by
these terms directly, it would
476
0:23:0,04 --> 0:23:3,48
suffice to publish the source code
of Citus, which is a link
477
0:23:3,66 --> 0:23:5,9
and a disclosure that you use Citus,
right?
478
0:23:6,14 --> 0:23:10,52
Then if you patch Citus and then
you improve Citus, even if you
479
0:23:10,52 --> 0:23:13,68
don't submit upstream, your users
have the right to get those
480
0:23:13,68 --> 0:23:16,38
benefits too, which is why I love
this kind of licenses.
481
0:23:17,4 --> 0:23:19,96
But some lawyers disagree with
me, probably they're right because
482
0:23:19,96 --> 0:23:21,3
they're lawyers, I'm not.
483
0:23:21,88 --> 0:23:26,5
And not many people want to use
AGPL on their managed services.
484
0:23:26,58 --> 0:23:30,36
But in my understanding, limited
understanding, I think that's
485
0:23:30,36 --> 0:23:31,9
not necessarily correct.
486
0:23:32,54 --> 0:23:35,58
Nik: Imagine if Postgres was on
AGPL version 3.
487
0:23:35,82 --> 0:23:39,38
RDS would need to publish all the
changes they made to implement
488
0:23:39,48 --> 0:23:41,5
rds_superuser and so on.
489
0:23:42,9 --> 0:23:44,36
It's simplified as well.
490
0:23:45,06 --> 0:23:48,18
Michael: But this is a really great
point because I think Postgres
491
0:23:48,18 --> 0:23:51,18
has benefited from being on a permissive
license in terms of
492
0:23:51,18 --> 0:23:52,54
how much usage it's got.
493
0:23:52,54 --> 0:23:55,32
And maybe we're seeing the exact
opposite with Citus.
494
0:23:55,32 --> 0:23:58,44
Maybe we're seeing that AGPL is
actually costing in terms
495
0:23:58,44 --> 0:23:59,16
Nik: of usage.
496
0:23:59,16 --> 0:23:59,94
Repelled usage.
497
0:24:0,12 --> 0:24:0,52
Repelled, yeah.
498
0:24:0,52 --> 0:24:3,68
Like Google, for example, have
internal, at least in the past,
499
0:24:3,68 --> 0:24:7,58
they had internal directive that
let's avoid the GPL version
500
0:24:7,58 --> 0:24:8,98
3 software completely.
501
0:24:9,52 --> 0:24:11,34
Álvaro: Yeah, it's a big if.
502
0:24:11,52 --> 0:24:12,74
We don't know the answer.
503
0:24:13,52 --> 0:24:17,96
Most likely, because The market
has this, what I believe it's
504
0:24:17,96 --> 0:24:22,86
a misconception, but this general
wisdom that AGPL or GPL licenses
505
0:24:23,04 --> 0:24:27,38
are this viral and so on, then
yes, it looks like copyleft software
506
0:24:27,8 --> 0:24:32,42
is perceived as unideal and permissive
licenses are winning.
507
0:24:33,06 --> 0:24:35,94
And certainly this is the reason
I don't want to avoid the question,
508
0:24:36,22 --> 0:24:41,82
yes, StackGres, and I can announce
this publicly, it's gonna be
509
0:24:41,82 --> 0:24:46,08
licensed under the Apache 2 license,
instead of AGPL, as much
510
0:24:46,08 --> 0:24:50,08
as I wish to do otherwise, but
the market and the users demand
511
0:24:50,08 --> 0:24:51,36
this, so we're happy to
512
0:24:51,36 --> 0:24:52,28
Nik: do that and we will
513
0:24:52,28 --> 0:24:55,02
Álvaro: do that in the next days
actually.
514
0:24:55,52 --> 0:24:59,34
Nik: While still Citus remains
on AGPL and it's just plugged
515
0:24:59,34 --> 0:25:0,12
in, right?
516
0:25:0,78 --> 0:25:4,28
Álvaro: Yep, we support, in StackGres
we support more than 200
517
0:25:4,28 --> 0:25:4,78
extensions.
518
0:25:4,78 --> 0:25:8,1
They come with a lot of licenses
and it's a user's responsibility
519
0:25:8,48 --> 0:25:12,16
to, to know the license they're,
they're dealing with and their
520
0:25:12,18 --> 0:25:12,6
wording.
521
0:25:12,6 --> 0:25:15,04
We just provide the software and
the platform.
522
0:25:15,16 --> 0:25:18,74
Nik: It's like a big Lego and some
pieces are different colors
523
0:25:18,74 --> 0:25:19,78
and so on, right?
524
0:25:20,32 --> 0:25:20,82
Álvaro: Yeah.
525
0:25:21,46 --> 0:25:25,58
And to complete the picture, I
think I want to go back a little
526
0:25:25,58 --> 0:25:27,42
bit to what you were mentioning,
Michael.
527
0:25:28,38 --> 0:25:32,04
That also relates maybe to 1 of
the reasons, or potentially,
528
0:25:32,04 --> 0:25:35,08
I don't know exactly the reasons,
but why I believe that maybe
529
0:25:35,08 --> 0:25:39,86
some companies abandoned Citus
at some point, which is that Citus,
530
0:25:40,02 --> 0:25:43,14
the workers, where you split the
data, you can, again, you can
531
0:25:43,14 --> 0:25:47,06
add as many as you need, but coordinator
is only 1.
532
0:25:47,68 --> 0:25:51,66
And if you saturate the coordinator,
then you're out of luck,
533
0:25:51,66 --> 0:25:51,98
right?
534
0:25:51,98 --> 0:25:55,92
You can use, as we said, you can
query the nodes directly, the
535
0:25:55,92 --> 0:26:0,6
workers directly, but that it's
complicated to operate and you
536
0:26:0,6 --> 0:26:2,78
still also will get as far as you
can.
537
0:26:3,42 --> 0:26:7,44
This is something we saw some time
ago and decided to do something
538
0:26:7,44 --> 0:26:8,18
about it.
539
0:26:8,32 --> 0:26:11,0
And in the most recent version
of StackGres, StackGres
540
0:26:11,0 --> 0:26:14,12
1.19, we have introduced something
called query routers.
541
0:26:15,06 --> 0:26:18,34
Query routers does nothing that
we have actually patched on Citus
542
0:26:18,34 --> 0:26:19,28
or anything like that.
543
0:26:19,28 --> 0:26:20,58
It's unmodified Citus.
544
0:26:20,58 --> 0:26:24,02
It's just an architectural pattern
that we identified that we
545
0:26:24,02 --> 0:26:27,94
could apply to Citus to allow to
have essentially multiple coordinators,
546
0:26:28,26 --> 0:26:29,16
put that simple.
547
0:26:30,06 --> 0:26:34,3
So a query router is like a coordinator,
so it can route queries.
548
0:26:34,54 --> 0:26:37,18
It doesn't have data, it just route
queries.
549
0:26:37,36 --> 0:26:39,82
The only thing it cannot do is
the DDL operations.
550
0:26:39,96 --> 0:26:42,28
The DDL operations go through the
coordinator.
551
0:26:42,9 --> 0:26:45,82
But now you can add as many query
routers as you want.
552
0:26:46,08 --> 0:26:50,32
And so the limitation that Citus
may have had until now with
553
0:26:50,32 --> 0:26:52,5
a single coordinator, it's gone.
554
0:26:52,8 --> 0:26:55,84
And now you can scale by adding
as many query routers as you
555
0:26:55,84 --> 0:27:0,52
want and scale, not only now the
worker side, the layer where
556
0:27:0,52 --> 0:27:4,58
it keeps the data, but also the
router layer to almost any size.
557
0:27:4,66 --> 0:27:7,6
I'm publishing some benchmarks
that we have been running in the
558
0:27:7,6 --> 0:27:8,9
last days about this.
559
0:27:9,64 --> 0:27:14,88
And ballpark numbers, 64 core workers
running simple queries
560
0:27:15,06 --> 0:27:19,44
indexed by primary key, but not
fully in memory, but with a high
561
0:27:19,44 --> 0:27:20,24
hit ratio.
562
0:27:20,68 --> 0:27:25,76
A single coordinator with 3 workers,
the coordinator being also
563
0:27:25,76 --> 0:27:30,64
64 cores, was maxing out at around
550, 000 transactions per
564
0:27:30,64 --> 0:27:33,9
2nd with a pretty good latency,
like under 1 millisecond.
565
0:27:35,08 --> 0:27:36,08
But that's it.
566
0:27:36,34 --> 0:27:39,78
More than 500000, a little bit
over 500000, and the coordinator
567
0:27:39,96 --> 0:27:40,66
is saturated.
568
0:27:41,18 --> 0:27:44,92
The reason why this number is because
these queries are quite
569
0:27:44,92 --> 0:27:48,42
fast to process, and the proportion
of overhead per query processing
570
0:27:48,44 --> 0:27:49,68
is maxed out, right?
571
0:27:49,74 --> 0:27:53,1
It's a lot of overhead per very
small query, per tiny query.
572
0:27:53,1 --> 0:27:55,76
Of course, if you hit all the queries
and they involve I/O.
573
0:27:55,76 --> 0:27:59,38
And reading work, you can scale
further before hitting the limit
574
0:27:59,38 --> 0:28:0,14
of the coordinator.
575
0:28:0,14 --> 0:28:3,54
So we designed this benchmark to
actually find the limit as soon
576
0:28:3,54 --> 0:28:6,1
as possible of the coordinator
at still very high rates, right?
577
0:28:6,1 --> 0:28:8,16
More than 500000 transactions per
2nd.
578
0:28:8,52 --> 0:28:10,28
So coordinator is maxed out.
579
0:28:10,28 --> 0:28:14,5
Then we add 2 coordinators with
the same 3 shards and then 3
580
0:28:14,5 --> 0:28:18,58
coordinators with the same 3 workers,
sorry, the data nodes.
581
0:28:18,7 --> 0:28:21,18
And the performance basically scaled
almost linear.
582
0:28:21,18 --> 0:28:26,02
So from 550, it went to 1, 050,
000 and then to 1, 500, 000.
583
0:28:26,66 --> 0:28:28,22
So pretty much linear growth.
584
0:28:28,66 --> 0:28:32,16
Michael: When you say 3 coordinators,
Do you mean 3 query routers
585
0:28:32,16 --> 0:28:32,86
in the new?
586
0:28:32,86 --> 0:28:33,34
Yeah.
587
0:28:33,34 --> 0:28:33,76
OK.
588
0:28:33,76 --> 0:28:34,26
OK.
589
0:28:34,28 --> 0:28:34,98
Álvaro: Yeah, sorry.
590
0:28:35,0 --> 0:28:37,86
I use the terms a little bit interchangeably,
because for me,
591
0:28:37,86 --> 0:28:40,46
they're conceptually the same,
even though they're not.
592
0:28:40,58 --> 0:28:41,34
You're right.
593
0:28:41,34 --> 0:28:42,74
It's query routers, yes.
594
0:28:42,74 --> 0:28:44,0
So 3 query routers.
595
0:28:44,2 --> 0:28:45,72
The coordinator remains there.
596
0:28:45,72 --> 0:28:50,56
And actually, I am now recommending
that if you use this functionality
597
0:28:50,78 --> 0:28:53,3
with the StackGres, with the query
routers, that you keep the
598
0:28:53,3 --> 0:28:56,64
coordinator small configuration,
small number of cores, just
599
0:28:56,64 --> 0:29:0,24
to run the DDL through there, and
do not use a coordinator for
600
0:29:0,24 --> 0:29:1,06
query routing.
601
0:29:1,06 --> 0:29:3,04
And then use a layer of query routers.
602
0:29:3,04 --> 0:29:6,68
You can start with 1 if you want
and then scale them up as your
603
0:29:6,68 --> 0:29:8,48
needs grow, but they're already
prepared.
604
0:29:8,48 --> 0:29:11,82
You already, they use a different
service in Kubernetes, so it's
605
0:29:11,82 --> 0:29:14,18
just a way to route the queries
and entry point.
606
0:29:14,34 --> 0:29:15,92
And then you already have that
established.
607
0:29:15,92 --> 0:29:19,6
Your application points to the
service that will balance across
608
0:29:19,6 --> 0:29:21,14
all the query routers you may have.
609
0:29:21,14 --> 0:29:24,08
And then it's just a matter of
changing 1 character in your YAML
610
0:29:24,08 --> 0:29:27,9
file, say, query routers 1, query
routers 2, and you would scale
611
0:29:28,02 --> 0:29:28,52
automatically.
612
0:29:29,48 --> 0:29:33,08
Michael: And You mentioned a while
back that StackGres supports
613
0:29:33,08 --> 0:29:34,46
multiple ways of sharding.
614
0:29:34,7 --> 0:29:38,08
Could I use query routers with
any of them or is this a Citus
615
0:29:38,08 --> 0:29:39,72
specific thing?
616
0:29:39,72 --> 0:29:40,96
Álvaro: It's a very good question.
617
0:29:40,96 --> 0:29:42,42
This is Citus specific.
618
0:29:42,84 --> 0:29:47,16
So if we look at the internals,
Again, this is not, we didn't
619
0:29:47,16 --> 0:29:48,94
touch Citus source code.
620
0:29:48,94 --> 0:29:51,3
We could have if we needed, but
it was not needed.
621
0:29:51,3 --> 0:29:53,1
The building blocks were surprisingly
there.
622
0:29:53,1 --> 0:29:55,82
Actually, what I'm surprised that
nobody saw this before.
623
0:29:56,2 --> 0:30:0,52
For us, it was crystal clear because
it all started with the
624
0:30:0,52 --> 0:30:4,76
concept that every worker can function
as a query router, as
625
0:30:4,76 --> 0:30:5,8
a coordinator, right?
626
0:30:5,8 --> 0:30:7,32
We just call them query routers.
627
0:30:7,54 --> 0:30:9,72
So that was already working there.
628
0:30:9,72 --> 0:30:13,14
The way this works is because they
implement the metadata that
629
0:30:13,14 --> 0:30:15,3
Citus replicates across all nodes.
630
0:30:15,36 --> 0:30:18,0
So you know whom to talk to, essentially.
631
0:30:19,14 --> 0:30:22,44
And there's another property, which
is what we combined into
632
0:30:22,44 --> 0:30:24,58
this pattern that we call the query
router.
633
0:30:24,72 --> 0:30:28,2
There is a property that is called
shouldhaveshards, and you
634
0:30:28,2 --> 0:30:29,56
can set it to false.
635
0:30:30,06 --> 0:30:33,64
Shards are the way that Citus internally
distributes the data,
636
0:30:33,64 --> 0:30:33,96
right?
637
0:30:33,96 --> 0:30:37,06
So also to clear, to make the context
clear for everybody.
638
0:30:37,36 --> 0:30:40,46
On Citus, there is workers, which
let's say they are the physical
639
0:30:40,58 --> 0:30:44,16
nodes that contain the data, physical,
virtual, you name it,
640
0:30:44,16 --> 0:30:46,12
the nodes that contain the data.
641
0:30:46,38 --> 0:30:48,84
But the data in reality is split
into shards.
642
0:30:49,06 --> 0:30:53,52
And then you map shards to workers,
at least 1, but could be
643
0:30:53,52 --> 0:30:54,44
more than 1.
644
0:30:54,64 --> 0:30:58,22
Sometimes some other people call
this virtual shards versus real
645
0:30:58,22 --> 0:30:59,1
shards, right?
646
0:30:59,34 --> 0:31:2,56
Citus terminology is workers,
like the physical thing, and shards
647
0:31:2,56 --> 0:31:3,96
like the virtual thing.
648
0:31:3,96 --> 0:31:7,62
And typically we recommend a higher
number of shards to worker
649
0:31:7,76 --> 0:31:10,96
because it helps you also scale
when you add more workers, you
650
0:31:10,96 --> 0:31:13,82
just move some of the shards to
other workers.
651
0:31:13,82 --> 0:31:15,86
You don't need to re-shard the
whole thing.
652
0:31:15,86 --> 0:31:16,24
Right.
653
0:31:16,24 --> 0:31:17,86
It's much, much better pattern.
654
0:31:17,86 --> 0:31:20,5
Actually, by the way, I'm diving
maybe a little bit too deep
655
0:31:20,5 --> 0:31:22,86
into that topic, but I use the
magic number 120.
656
0:31:23,36 --> 0:31:24,86
Most people use powers of 2.
657
0:31:24,86 --> 0:31:27,96
I think that's a mistake, because
then you can only grow by powers
658
0:31:27,96 --> 0:31:31,92
of 2, which by the growth of powers
of 2, can mean to grow a
659
0:31:31,92 --> 0:31:34,24
lot when maybe you don't need to
grow that much.
660
0:31:34,34 --> 0:31:37,88
Versus having 120 is a number that
has a lot of divisors, and
661
0:31:37,88 --> 0:31:41,2
so you can grow from 30 to 40,
for example.
662
0:31:41,2 --> 0:31:43,94
You don't need to go from 30 to
60, you need to double because
663
0:31:43,94 --> 0:31:46,7
they're still evenly divided across
all your workers.
664
0:31:47,18 --> 0:31:51,72
If you take a worker that can act
as a coordinator or a router,
665
0:31:51,78 --> 0:31:55,6
and you set this property
shouldhaveshards to false, then
666
0:31:55,6 --> 0:31:59,16
Citus will not place any shards
on it, it will be data-less.
667
0:31:59,68 --> 0:32:5,4
So essentially a data-less worker,
and that's a good router.
668
0:32:5,46 --> 0:32:6,88
So the pattern was just there.
669
0:32:6,88 --> 0:32:11,02
We just enabled it through architecture
and automation in StackGres.
670
0:32:11,82 --> 0:32:13,22
Michael: Simple, I like it.
671
0:32:13,66 --> 0:32:16,32
Álvaro: So of course, there's more
meat to it when you look into
672
0:32:16,32 --> 0:32:18,78
the internal details, but conceptually
this is it.
673
0:32:19,28 --> 0:32:19,78
Just
674
0:32:20,24 --> 0:32:22,4
Nik: don't write proxy, just use
Postgres.
675
0:32:25,64 --> 0:32:26,92
In yet another area.
676
0:32:27,8 --> 0:32:28,94
Yeah, it's interesting.
677
0:32:29,04 --> 0:32:34,16
Speaking of benchmarks, these numbers,
what concerns me that
678
0:32:34,16 --> 0:32:39,0
we talk about only read-only queries,
I would like to see the
679
0:32:39,96 --> 0:32:44,12
series 2 of benchmarks talking
about writes, because as we already
680
0:32:44,12 --> 0:32:50,72
discussed, the case you have in
hand is 2.2 petabytes.
681
0:32:52,42 --> 0:32:57,18
It's about writes, not reads, because
scaling reads is less interesting,
682
0:32:57,44 --> 0:32:58,72
less challenging, right?
683
0:32:59,12 --> 0:33:3,42
Álvaro: It is actually challenging
And again, I will not claim
684
0:33:3,42 --> 0:33:4,7
that Citus is perfect.
685
0:33:4,74 --> 0:33:7,36
And it is really good, but it's
not perfect.
686
0:33:7,36 --> 0:33:8,5
I guess nothing is perfect.
687
0:33:8,8 --> 0:33:10,14
And it requires tuning.
688
0:33:10,44 --> 0:33:12,62
I'm not gonna say this worked out
of the box.
689
0:33:12,62 --> 0:33:14,08
No, it requires tuning.
690
0:33:14,1 --> 0:33:18,06
And 1 of The hardest part of Citus
actually is connection tuning,
691
0:33:18,28 --> 0:33:22,12
because when you look at Citus,
at any distributed database for
692
0:33:22,12 --> 0:33:25,6
that matter, any sharded database,
it's no longer a single thing,
693
0:33:25,6 --> 0:33:26,1
right?
694
0:33:26,14 --> 0:33:29,94
Now you're dealing with connections
from the client to the coordinator
695
0:33:29,96 --> 0:33:33,58
or the query routers, and from
them to the workers.
696
0:33:34,06 --> 0:33:36,76
And sometimes these queries actually
have a big fan-out.
697
0:33:36,76 --> 0:33:41,3
For example, on this particular
case of the 2.2 petabyte database,
698
0:33:41,64 --> 0:33:46,42
I just cannot mention the customer
name, but it's a big telco.
699
0:33:46,96 --> 0:33:50,02
They are using COPY to ingest data.
700
0:33:50,02 --> 0:33:53,1
And that's amazing because it's
the most efficient way to ingest
701
0:33:53,1 --> 0:33:54,12
data into Postgres.
702
0:33:54,92 --> 0:33:59,64
And they're using 120 shards on
24 nodes, which divides evenly.
703
0:34:0,24 --> 0:34:7,06
Now, if you do a COPY of 50, 000
records, a single connection,
704
0:34:7,06 --> 0:34:10,04
so a single connection tries to
ingest 50, 000 records via a
705
0:34:10,04 --> 0:34:13,22
COPY, that COPY is going to create
24 sub copies.
706
0:34:14,16 --> 0:34:18,96
Because by law of probability,
all 24 shards are going to receive
707
0:34:18,96 --> 0:34:19,46
data.
708
0:34:19,64 --> 0:34:21,98
1/24th of the 50, 000 records.
709
0:34:22,38 --> 0:34:25,58
Because the way that data is sharded
is by taking a hash of the
710
0:34:25,58 --> 0:34:29,86
primary key, and that hash is a
random, let's say, it's randomly
711
0:34:29,86 --> 0:34:31,78
distributed, so it's going to touch
all the shards.
712
0:34:31,96 --> 0:34:36,14
So 1 connection from 1 client now
will create 24 connections.
713
0:34:36,38 --> 0:34:39,46
Citus implements this very well,
just do 24 sub copies.
714
0:34:39,52 --> 0:34:42,72
But you can imagine the potential
multiplication of connections
715
0:34:42,72 --> 0:34:44,7
that may happen here under high
concurrency.
716
0:34:45,48 --> 0:34:50,04
But by applying proper tuning and
doing proper connection management,
717
0:34:50,14 --> 0:34:51,42
this works very well.
718
0:34:51,5 --> 0:34:52,78
Nik: What is proper here?
719
0:34:53,68 --> 0:34:56,02
Álvaro: Oh, well, it's not an easy
exercise.
720
0:34:56,26 --> 0:35:0,06
The connections from the coordinator
or query routers to the
721
0:35:0,06 --> 0:35:2,98
worker nodes, they're bound by
several parameters.
722
0:35:3,42 --> 0:35:6,98
Essentially, Citus has some kind
of, let's say, connection pooling
723
0:35:7,04 --> 0:35:11,2
system to communicate between the
coordinator or coordinator
724
0:35:11,2 --> 0:35:12,02
to the workers.
725
0:35:12,4 --> 0:35:16,94
But it's not like PgBouncer, because
they open backends that
726
0:35:16,94 --> 0:35:19,78
are Citus is Postgres, for the
good and the bad.
727
0:35:19,84 --> 0:35:22,32
There's many good things and a
few bad things.
728
0:35:22,66 --> 0:35:26,84
And so if there's a client talking
to the coordinator, that creates
729
0:35:26,84 --> 0:35:28,52
a backend on the coordinator, right?
730
0:35:28,52 --> 0:35:30,14
And that backend needs to be open.
731
0:35:30,14 --> 0:35:34,24
Now, this backend may now own multiple
sockets to multiple connections
732
0:35:34,28 --> 0:35:35,04
to the workers.
733
0:35:35,06 --> 0:35:38,3
And there's several parameters
to tune whether you want a single
734
0:35:38,3 --> 0:35:41,32
connection per worker, whether
you want a number of connections
735
0:35:41,32 --> 0:35:41,88
per worker.
736
0:35:41,88 --> 0:35:45,84
You can cache connections already
open per worker, but then there's
737
0:35:45,84 --> 0:35:49,64
also like a global pool so that
you don't want to do more than
738
0:35:49,64 --> 0:35:52,76
X number of connections for a given
coordinator across all the
739
0:35:52,76 --> 0:35:53,26
workers.
740
0:35:53,56 --> 0:35:57,2
And so you need to really understand
that we all know that Postgres
741
0:35:57,66 --> 0:36:2,22
doesn't work well with a huge number
of connections because it
742
0:36:2,22 --> 0:36:3,5
creates a lot of backend.
743
0:36:3,9 --> 0:36:7,0
And so you need to understand the
fan-in rules, the fan-out rules,
744
0:36:7,08 --> 0:36:9,48
whether to put connection pool,
for example, in front of the
745
0:36:9,48 --> 0:36:11,5
coordinator or not, and so on.
746
0:36:11,5 --> 0:36:15,58
So the math ends up being non-trivial
sometimes, especially if
747
0:36:15,58 --> 0:36:17,32
you're talking about huge workloads.
748
0:36:17,44 --> 0:36:20,14
Like for small workloads, even
if you're starting with again,
749
0:36:20,14 --> 0:36:22,84
maybe hundreds of thousand transactions
per 2nd, maybe more or
750
0:36:22,84 --> 0:36:26,14
less trivial, like any server,
modern server will deal comfortably
751
0:36:26,14 --> 0:36:26,52
with that.
752
0:36:26,52 --> 0:36:29,84
But when you grow to these sizes,
you really need to size everything
753
0:36:29,92 --> 0:36:30,64
around this.
754
0:36:30,64 --> 0:36:34,64
But what's actually very fun of
this particular case is that
755
0:36:34,64 --> 0:36:37,64
as of today, on this particular
case, we'll need to blog about
756
0:36:37,64 --> 0:36:38,14
this.
757
0:36:38,24 --> 0:36:43,0
Right now, the database is perceived
as slow, but they're running
758
0:36:43,14 --> 0:36:45,96
some software that I think you're
familiar with called pg_ash,
759
0:36:47,08 --> 0:36:52,58
and it's reporting all the time,
waits are mostly ClientRead.
760
0:36:53,86 --> 0:36:57,2
They don't have an ingestion pipeline
fast enough to saturate
761
0:36:57,2 --> 0:36:57,88
the database.
762
0:36:58,26 --> 0:37:0,94
The database is faster than their
ingestion pipeline.
763
0:37:1,62 --> 0:37:5,14
Because again, COPY is really optimized
in Postgres and can
764
0:37:5,14 --> 0:37:6,98
ingest huge amounts of data.
765
0:37:7,18 --> 0:37:11,64
And we have here 24 shards, really
beefy servers that can ingest
766
0:37:11,64 --> 0:37:13,76
data at really, really high speeds.
767
0:37:13,94 --> 0:37:17,72
We've seen, don't remember the
numbers, but multiple gigabytes
768
0:37:17,72 --> 0:37:19,1
per 2nd being ingested.
769
0:37:19,34 --> 0:37:21,32
And the database is just fine.
770
0:37:21,94 --> 0:37:24,36
But the ingestion pipeline is not.
771
0:37:24,96 --> 0:37:25,82
Nik: That's interesting.
772
0:37:25,9 --> 0:37:29,8
So I see, I hear 3 different directions
here.
773
0:37:29,8 --> 0:37:33,72
1st of all, this benchmark in your
blog post, which is coming,
774
0:37:34,54 --> 0:37:37,44
I guess you will publish all the
details as usual, right?
775
0:37:38,2 --> 0:37:39,26
Álvaro: Yes, of course.
776
0:37:40,84 --> 0:37:44,2
Nik: This is our company's common
value, we share it a lot.
777
0:37:45,14 --> 0:37:47,84
Álvaro: Fully reproducible, StackGres
being open source, we'll
778
0:37:47,84 --> 0:37:49,14
open source all the code.
779
0:37:49,4 --> 0:37:52,76
Nik: It's Postgres's open source
and benchmarks should be open
780
0:37:52,76 --> 0:37:54,64
so anyone could repeat them.
781
0:37:55,16 --> 0:37:55,66
Yes.
782
0:37:55,9 --> 0:37:57,88
Yeah, so that's great.
783
0:37:58,58 --> 0:38:2,32
SELECT only because this is to
compare with different benchmark
784
0:38:2,32 --> 0:38:6,44
which is not so open, right, and
not about open source.
785
0:38:7,24 --> 0:38:8,04
That's great.
786
0:38:8,24 --> 0:38:11,76
And there, I guess, this is interesting
to continue exploring
787
0:38:12,04 --> 0:38:14,28
the limits.
788
0:38:14,44 --> 0:38:16,96
And I think maybe it will also...
789
0:38:17,1 --> 0:38:20,74
I expect more benchmarks to be
done because we have multiple
790
0:38:20,74 --> 0:38:23,85
systems competing in purely open
source areas.
791
0:38:23,85 --> 0:38:26,98
Some of them are AGPL, some of
them are Apache licenses, we mentioned
792
0:38:26,98 --> 0:38:27,48
already.
793
0:38:29,1 --> 0:38:31,74
It's interesting to start comparing
them.
794
0:38:31,88 --> 0:38:34,78
It's great that it's happening
and this is maybe the beginning
795
0:38:34,78 --> 0:38:36,24
of some comparison.
796
0:38:36,6 --> 0:38:37,94
It's time to start comparing.
797
0:38:38,74 --> 0:38:42,68
Right now your bet is fully on
Citus, like almost fully.
798
0:38:43,08 --> 0:38:46,66
I hear you that StackGres supports
other sharding as well.
799
0:38:46,86 --> 0:38:49,6
And I guess at some point you can
start supporting those new
800
0:38:49,6 --> 0:38:50,5
alternatives as well.
801
0:38:50,5 --> 0:38:51,04
Why not?
802
0:38:51,04 --> 0:38:53,8
It's just a Lego style thing, right?
803
0:38:53,8 --> 0:38:54,9
This is 1 direction.
804
0:38:56,82 --> 0:39:1,16
Read-only benchmarks, which I guess
it's not so exciting.
805
0:39:1,62 --> 0:39:4,94
2nd would be probably writing benchmarks
like COPY.
806
0:39:5,38 --> 0:39:9,3
When you say COPY, you mention
specific production case, but
807
0:39:9,3 --> 0:39:13,04
it would be great also to have
some benchmark showing, to start
808
0:39:13,04 --> 0:39:14,74
playing with and showing capabilities.
809
0:39:15,3 --> 0:39:17,38
Do you plan to have this benchmark?
810
0:39:18,26 --> 0:39:21,54
Álvaro: We could do that, but I
believe we will start by publishing
811
0:39:21,68 --> 0:39:25,58
some, the data that we can publish
about 1 particular case that
812
0:39:25,58 --> 0:39:28,66
we have already, like this 1, because
I think that already shows
813
0:39:28,66 --> 0:39:29,28
a lot.
814
0:39:30,06 --> 0:39:34,1
And so to be honest, we have
focused quite a bit on benchmarks.
815
0:39:34,6 --> 0:39:37,44
So let me be clear, we've done
2 different benchmarks.
816
0:39:37,44 --> 0:39:40,62
Both are going to be published
and will be fully reproducible
817
0:39:40,76 --> 0:39:41,96
back by code, right?
818
0:39:42,26 --> 0:39:44,18
And so full transparency on everything.
819
0:39:44,44 --> 0:39:48,04
The 1st 1 is a benchmark to demonstrate
the ceiling of the single
820
0:39:48,04 --> 0:39:51,34
coordinator and to see how we can
scale with query routers, right?
821
0:39:51,34 --> 0:39:54,52
Like I just mentioned the numbers,
it can go from 0.5, essentially
822
0:39:54,52 --> 0:39:58,24
500000 to 1.5 million, so almost
linear scalability by going
823
0:39:58,24 --> 0:40:1,02
from 1 coordinator to 3 query routers,
right?
824
0:40:1,46 --> 0:40:6,44
But then we saw actually that for
this particular workload, like
825
0:40:6,44 --> 0:40:10,96
very, very small primary key queries,
mostly in memory with a
826
0:40:10,96 --> 0:40:16,72
non-uniform distribution, that
you end up needing a similar number
827
0:40:16,72 --> 0:40:20,54
of cores on the routers layer than
on the shards or the, sorry,
828
0:40:20,54 --> 0:40:21,88
the workers layer, right?
829
0:40:22,12 --> 0:40:26,48
And so we did some tuning and around
this number that the ideal
830
0:40:26,48 --> 0:40:33,68
is to have 6 routers of 32 cores
each and 3 workers of 64 cores
831
0:40:33,68 --> 0:40:34,04
each.
832
0:40:34,04 --> 0:40:37,4
So that's the same number of cores
on both layers, but 6 routers
833
0:40:37,4 --> 0:40:38,36
and 3 workers.
834
0:40:38,52 --> 0:40:40,62
And we call this a unit of a scaling.
835
0:40:41,0 --> 0:40:44,84
That package, let's say it's well
tuned so that both layers are
836
0:40:44,96 --> 0:40:47,26
close to saturation but before
saturation.
837
0:40:47,26 --> 0:40:50,68
So they operate at the operational
regime and all math about
838
0:40:50,68 --> 0:40:53,54
connection, tuning connection,
pooling, and all that is well
839
0:40:53,54 --> 0:40:54,04
tuned.
840
0:40:54,34 --> 0:40:58,5
And that's basically delivering
1.5 million transactions per
841
0:40:58,5 --> 0:41:0,36
2nd, that unit of a scale.
842
0:41:0,72 --> 0:41:4,44
And then we decided to multiply
this unit of the scale by 2,
843
0:41:4,44 --> 0:41:5,9
by 4, and by 8.
844
0:41:6,34 --> 0:41:10,56
And this is a 2nd benchmark that
we'll be publishing that also
845
0:41:10,56 --> 0:41:13,78
compares to some data points of
another sharding solution, which
846
0:41:13,78 --> 0:41:16,02
has achieved those numbers and
more actually.
847
0:41:16,56 --> 0:41:20,24
And it also shows really good behavior,
like basically to share
848
0:41:20,24 --> 0:41:24,92
the numbers by multiplying this
by 8x, it passes over 10 million
849
0:41:24,92 --> 0:41:26,14
transactions per 2nd.
850
0:41:26,84 --> 0:41:33,74
With 1.something millisecond p50
latency and 6 milliseconds p99,
851
0:41:35,02 --> 0:41:37,32
at 10 million transactions per
2nd.
852
0:41:37,46 --> 0:41:41,62
To put this into perspective, who
needed to do 10 million transactions
853
0:41:41,68 --> 0:41:42,38
per 2nd?
854
0:41:42,5 --> 0:41:45,36
Probably just a few sites on earth
do more than that.
855
0:41:45,48 --> 0:41:46,88
Nik: So it's...
856
0:41:47,7 --> 0:41:48,48
Maybe more.
857
0:41:48,52 --> 0:41:50,66
We have gigantic workloads these
days.
858
0:41:50,74 --> 0:41:52,54
I think here I disagree.
859
0:41:52,96 --> 0:41:54,02
A lot of more.
860
0:41:54,38 --> 0:41:57,94
We see crazy startups which skyrocket
these days.
861
0:41:58,52 --> 0:42:0,04
And their users are agents.
862
0:42:0,62 --> 0:42:1,92
And they, like, just...
863
0:42:2,44 --> 0:42:4,94
Álvaro: Yeah, but most of the time
what I see is that agents
864
0:42:4,94 --> 0:42:5,88
are small databases.
865
0:42:5,9 --> 0:42:7,38
There are many of them, but a small...
866
0:42:7,38 --> 0:42:10,12
We're talking about single cluster,
right?
867
0:42:10,64 --> 0:42:15,92
But anyway, my point is that anyway,
reaching 10 million transactions
868
0:42:15,96 --> 0:42:19,82
per 2nd with very decent latency
numbers, right?
869
0:42:20,64 --> 0:42:22,46
It's an excellent result.
870
0:42:23,18 --> 0:42:26,56
It's home to most of the clusters
we've ever seen.
871
0:42:27,04 --> 0:42:29,68
And so it's a valid solution for
almost anyone.
872
0:42:29,68 --> 0:42:31,96
And That's actually my takeaway,
right?
873
0:42:31,96 --> 0:42:35,9
It can scale almost as much or
as much as you want with really
874
0:42:35,9 --> 0:42:37,62
decent numbers on this pure OLTP.
875
0:42:38,4 --> 0:42:40,6802
Nik: So I have 4 directions already.
876
0:42:40,6802 --> 0:42:42,1
So you will publish.
877
0:42:42,18 --> 0:42:43,6
I know you will publish this.
878
0:42:43,6 --> 0:42:45,54
You will probably will iterate
this.
879
0:42:45,82 --> 0:42:49,52
I guess this benchmark also not
cheap to run, right?
880
0:42:49,54 --> 0:42:51,8
You need to provision those resources.
881
0:42:52,1 --> 0:42:52,9
Álvaro: Yeah, exactly.
882
0:42:52,9 --> 0:42:54,18
But this is a good point.
883
0:42:54,44 --> 0:42:58,58
We're still a bootstrapped startup,
and we strive to be, we have
884
0:42:58,58 --> 0:43:0,4
to be, very cost efficient.
885
0:43:1,26 --> 0:43:5,02
So running this benchmark, I don't
have the final data, but it's
886
0:43:5,02 --> 0:43:5,78
very approximate.
887
0:43:6,34 --> 0:43:12,08
We've spent around $5, 000 to $6,
000 on running this benchmark,
888
0:43:12,16 --> 0:43:14,58
scaling to 10 million transactions
per 2nd.
889
0:43:14,58 --> 0:43:16,42
Nik: It's AWS or where?
890
0:43:16,84 --> 0:43:17,34
Álvaro: Yep.
891
0:43:17,78 --> 0:43:21,2
Nik: And you just use Spot because
it's not reliable in this
892
0:43:21,2 --> 0:43:21,7
case.
893
0:43:22,36 --> 0:43:26,12
Álvaro: Yeah, it's hard because,
and we had some problems when
894
0:43:26,12 --> 0:43:28,3
you deal with such a large number
of instances.
895
0:43:28,4 --> 0:43:33,22
So on the 10 million transactions
per 2nd, this involved 72 instances
896
0:43:33,28 --> 0:43:36,98
plus the drivers for the workload,
so like around 90 instances.
897
0:43:37,54 --> 0:43:39,74
Just 1 instance behaving wrongly?
898
0:43:40,92 --> 0:43:43,3
Nik: The probability of failure
is too high.
899
0:43:44,34 --> 0:43:44,84
Álvaro: Yeah.
900
0:43:45,12 --> 0:43:48,74
So, yeah, Spot would have most
likely had mean that the test
901
0:43:48,74 --> 0:43:50,72
needs to be repeated over and over
again.
902
0:43:51,76 --> 0:43:52,4
Nik: I see.
903
0:43:52,54 --> 0:43:52,96
That's great.
904
0:43:52,96 --> 0:43:55,82
And that's great cost-wise as well
for 10 million.
905
0:43:55,96 --> 0:43:59,36
So once again, 1 direction is SELECT
only primary key lookup,
906
0:43:59,38 --> 0:44:0,92
trivial workload, I would say.
907
0:44:1,56 --> 0:44:6,94
And I think this is very adjacent
to our benchmarks we did multiple
908
0:44:6,94 --> 0:44:11,04
times for latency overhead is very
in this area, like very fast
909
0:44:11,04 --> 0:44:11,54
queries.
910
0:44:11,6 --> 0:44:15,36
2nd, I still think at some point
there should be some benchmarks
911
0:44:15,58 --> 0:44:18,74
for INSERTs or write workloads,
right?
912
0:44:19,64 --> 0:44:23,64
3rd thing is that you mentioned
many connections is a trouble.
913
0:44:24,52 --> 0:44:30,36
I think, I know Andres Freund did
a lot of great work in Postgres
914
0:44:30,36 --> 0:44:35,08
13 and especially I think 14 to
improve the situation when you
915
0:44:35,08 --> 0:44:36,6
have thousands of connections.
916
0:44:36,98 --> 0:44:42,16
Because before that, thousands
of idle connections affected performance
917
0:44:42,18 --> 0:44:42,9
so badly.
918
0:44:42,94 --> 0:44:44,08
Now it's not so bad.
919
0:44:44,38 --> 0:44:45,64
What does it mean still?
920
0:44:46,5 --> 0:44:48,02
We don't have threads, right?
921
0:44:48,58 --> 0:44:52,62
What's happening if you have 10,
000 connections on single Postgres?
922
0:44:52,72 --> 0:44:54,94
I think this research is very interesting.
923
0:44:54,96 --> 0:44:58,2
I'm talking for myself because
we have a lot of interesting directions.
924
0:44:58,22 --> 0:44:59,44
We have benchmarks.
925
0:44:59,96 --> 0:45:2,82
Our focus mostly on single node
clusters.
926
0:45:3,54 --> 0:45:7,68
Our company is dealing with some
self-implementation, like application
927
0:45:7,74 --> 0:45:8,54
side sharding.
928
0:45:9,0 --> 0:45:11,98
But it's so interesting, I see
some interesting directions here
929
0:45:11,98 --> 0:45:13,22
to benchmark further.
930
0:45:13,82 --> 0:45:19,08
And finally, I'm thinking if you
stop at 10 million, I'm going
931
0:45:19,08 --> 0:45:25,4
to explore the possibility to have
10 million TPS primary key
932
0:45:25,4 --> 0:45:27,42
lookups for a single node.
933
0:45:28,08 --> 0:45:32,06
This probably will encourage you
to go to 50 million or something,
934
0:45:32,28 --> 0:45:34,04
spend a few thousand dollars.
935
0:45:34,14 --> 0:45:38,86
Because if, yeah, I will put it
to midterm to-do list.
936
0:45:39,0 --> 0:45:41,44
If we achieve that, you will be
forced
937
0:45:41,44 --> 0:45:42,02
Álvaro: to cover.
938
0:45:42,02 --> 0:45:44,7
But you know, if you want to do
that, you know, you only have
939
0:45:44,7 --> 0:45:45,56
2 weeks, right?
940
0:45:46,4 --> 0:45:48,48
Nik: No, no, no, no, no.
941
0:45:48,48 --> 0:45:50,02
Go ahead, publish your results.
942
0:45:50,02 --> 0:45:53,2
This is an interesting, friendly
competition and I think some
943
0:45:53,3 --> 0:45:56,3
systems will need it to join.
944
0:45:56,32 --> 0:45:58,58
What I like about this is that
data is shared.
945
0:45:58,58 --> 0:46:0,22
It's about open source components.
946
0:46:0,46 --> 0:46:1,72
Everything is open source.
947
0:46:1,74 --> 0:46:2,78
Data is open source.
948
0:46:2,78 --> 0:46:3,74
Benchmarks are open source.
949
0:46:3,74 --> 0:46:4,72
They can be revisited.
950
0:46:4,74 --> 0:46:5,42
This is great.
951
0:46:5,42 --> 0:46:7,86
This is very good style of competition.
952
0:46:8,3 --> 0:46:10,06
Everyone benefits from, right?
953
0:46:11,42 --> 0:46:14,34
So I'm not going to chase you right
now.
954
0:46:14,34 --> 0:46:16,24
I'm just saying maybe later.
955
0:46:16,78 --> 0:46:22,06
Álvaro: No, I'm very open if anyone
wants to sponsor workloads
956
0:46:22,16 --> 0:46:26,12
to be tested on Citus and see how
Citus can scale to from 10
957
0:46:26,12 --> 0:46:30,42
to 50 million transactions per
2nd or to read-write workloads We
958
0:46:30,42 --> 0:46:32,72
would be more than happy to help
in that regards.
959
0:46:33,48 --> 0:46:34,12
Michael: So yeah.
960
0:46:34,7 --> 0:46:38,24
I think it's, I think what, from
1 of the charts that you sent
961
0:46:38,24 --> 0:46:43,54
over, it's quite an interesting
chart of how the throughput changes
962
0:46:43,58 --> 0:46:44,44
as you scale up.
963
0:46:44,44 --> 0:46:46,88
So I think if you, if you, you
mentioned these units of work,
964
0:46:46,88 --> 0:46:52,06
if, if you add a 2nd unit of work
and the TPS exactly doubles
965
0:46:52,06 --> 0:46:56,5
if it's linear like as you add
more units it's very easy to extrapolate
966
0:46:56,98 --> 0:47:0,84
that of course it can get to 50
I just need 5 times more but
967
0:47:0,84 --> 0:47:5,22
if it isn't if it isn't quite efficient
if you lose 10 each time
968
0:47:5,22 --> 0:47:10,66
if it if it feels like at doubling
the instance size only 1.5x
969
0:47:11,32 --> 0:47:15,26
is the TPS, then it's less believable
that it can scale infinitely.
970
0:47:15,72 --> 0:47:17,78
What kind of numbers were you seeing
there?
971
0:47:19,3 --> 0:47:20,86
Álvaro: So that's a really good
question.
972
0:47:20,86 --> 0:47:25,78
Scaling is slightly sublinear,
with most systems are.
973
0:47:26,2 --> 0:47:32,72
And on our scaling from 1 to 8,
it was 1.94 every time we doubled.
974
0:47:34,06 --> 0:47:34,56
Nik: Okay.
975
0:47:34,64 --> 0:47:35,14
Yeah.
976
0:47:35,5 --> 0:47:39,32
There is a big challenge of scaling
client in this case.
977
0:47:39,38 --> 0:47:39,88
Yes.
978
0:47:40,52 --> 0:47:41,34
That's it.
979
0:47:41,55 --> 0:47:42,44
I mean, I
980
0:47:42,44 --> 0:47:43,86
Álvaro: can go very deep into this
topic.
981
0:47:43,86 --> 0:47:45,14
I guess it's not the moment.
982
0:47:45,48 --> 0:47:49,4
But it's a really, really interesting
capacity planning we had
983
0:47:49,4 --> 0:47:50,58
to do to achieve these numbers.
984
0:47:50,58 --> 0:47:54,52
Again, it sounds like, oh, just
put more nodes and yeah, you
985
0:47:54,52 --> 0:47:56,9
scale 10 million, you just need
to hit enter, right?
986
0:47:56,98 --> 0:47:58,28
No, it wasn't that case.
987
0:47:58,32 --> 0:47:59,84
We had to do a lot of tuning.
988
0:48:0,06 --> 0:48:2,86
The number of connections is actually
mind-blowing because even
989
0:48:2,86 --> 0:48:6,24
though these queries are really
fast and take hundreds of microseconds
990
0:48:6,42 --> 0:48:10,38
to run in reality across all the
fleet, across the CPU aggregated
991
0:48:10,38 --> 0:48:11,64
all the fleet, right?
992
0:48:11,72 --> 0:48:14,02
The number of connections that
you need, the sessions that you
993
0:48:14,02 --> 0:48:16,02
need to drive this is very, very
large.
994
0:48:16,02 --> 0:48:18,46
It's many, many thousands of sessions.
995
0:48:18,82 --> 0:48:23,48
And then they multiply because
if each query router is, I think
996
0:48:23,48 --> 0:48:27,1
where our optimal regime was around
400 connections per query
997
0:48:27,1 --> 0:48:30,7
router on 32 cores, which by the
way, if you look at the numbers,
998
0:48:30,7 --> 0:48:34,64
it's always like the usual rule
for tuning Postgres, right?
999
0:48:34,64 --> 0:48:38,22
The maximum number of connections
is a multiple of the number
1000
0:48:38,22 --> 0:48:38,6
of cores.
1001
0:48:38,6 --> 0:48:41,4
In this case, the multiple is around
7 because they're very,
1002
0:48:41,4 --> 0:48:44,58
very CPU intensive queries because
they almost not touch disk,
1003
0:48:44,58 --> 0:48:45,02
right?
1004
0:48:45,02 --> 0:48:46,88
So the multiple is a bit high.
1005
0:48:46,88 --> 0:48:50,32
Anyway, do you think of 400 connections
coming to each query
1006
0:48:50,32 --> 0:48:54,52
router of which there are 48 on
the 10 million transactions per
1007
0:48:54,52 --> 0:48:56,1
2nd run, right?
1008
0:48:56,5 --> 0:49:2,06
And now each 1 is gonna have outgoing
the same number of connections
1009
0:49:2,58 --> 0:49:5,22
in the same number of sessions,
because they're all, but they
1010
0:49:5,22 --> 0:49:6,52
all go to all shards.
1011
0:49:7,2 --> 0:49:12,18
So the incoming connections at
the shards are the whole fleet.
1012
0:49:12,18 --> 0:49:15,36
So you're seeing the 400 multiplied
by 48.
1013
0:49:15,86 --> 0:49:18,3
Those are incoming connections
to every worker.
1014
0:49:19,7 --> 0:49:23,44
So to actually deal with this,
we had to introduce connection
1015
0:49:23,54 --> 0:49:25,5
pooling before the workers.
1016
0:49:26,32 --> 0:49:30,06
And it's also been an exercise
because StackGres by default uses
1017
0:49:30,06 --> 0:49:33,28
PgBouncer, and PgBouncer is single-threaded.
1018
0:49:33,92 --> 0:49:36,24
So were we saturating 1 thread?
1019
0:49:36,54 --> 0:49:37,04
Definitely.
1020
0:49:37,72 --> 0:49:40,9
So actually this benchmark required,
if I'm not mistaken, 15
1021
0:49:40,9 --> 0:49:45,84
PgBouncers per worker, running
with a SO_REUSEPORT functionality
1022
0:49:46,42 --> 0:49:49,94
and all handled by StackGres as
a custom container.
1023
0:49:50,42 --> 0:49:53,94
So it worked beautifully well,
but you need to do all the dimensioning
1024
0:49:54,14 --> 0:49:56,78
correctly, otherwise it will not
achieve these numbers either.
1025
0:49:57,44 --> 0:50:0,56
Michael: Did you say 15 PgBouncers
per Worker?
1026
0:50:1,34 --> 0:50:1,84
Álvaro: Yep.
1027
0:50:1,86 --> 0:50:2,72
Michael: Oh my goodness.
1028
0:50:4,3 --> 0:50:4,8
Okay.
1029
0:50:5,14 --> 0:50:5,64
Okay.
1030
0:50:5,74 --> 0:50:6,24
Álvaro: Good.
1031
0:50:6,34 --> 0:50:9,68
Because we're dealing with, I think,
19, 000 incoming connections.
1032
0:50:11,76 --> 0:50:15,2
And they're all sending queries
at a really high pace.
1033
0:50:15,86 --> 0:50:18,98
Nik: What would it take to reach
100 million TPS?
1034
0:50:20,32 --> 0:50:22,22
Álvaro: Well, there's 2 things.
1035
0:50:22,48 --> 0:50:24,24
1 is just to keep multiplying.
1036
0:50:24,28 --> 0:50:28,58
I think this unit of scale is very
well tuned as of now, and
1037
0:50:28,58 --> 0:50:29,94
it just keep multiplying.
1038
0:50:30,14 --> 0:50:33,4
As of now, it should be an easy
exercise, other than money-wise.
1039
0:50:34,42 --> 0:50:38,48
But I believe the scalability is
gonna start going, is gonna
1040
0:50:38,48 --> 0:50:40,82
start going less and less over
time.
1041
0:50:41,12 --> 0:50:44,0
And this is, again, because of
how Citus works.
1042
0:50:44,48 --> 0:50:45,94
As I said, it's not perfect.
1043
0:50:46,28 --> 0:50:48,48
You may need to question whether
you need more than 10 million
1044
0:50:48,48 --> 0:50:49,66
transactions per 2nd.
1045
0:50:50,28 --> 0:50:53,62
But it's, so Citus is gonna create
a backend per session, per
1046
0:50:53,62 --> 0:50:54,12
router.
1047
0:50:54,86 --> 0:50:57,78
And this, again, we were just discussing
this, is gonna create,
1048
0:50:57,8 --> 0:51:1,36
that multiplied by the number of
routers will become the number
1049
0:51:1,36 --> 0:51:3,42
of incoming connections to each
worker node.
1050
0:51:3,42 --> 0:51:8,26
So instead of having 19, 200 connections,
if we do this 10 times,
1051
0:51:8,42 --> 0:51:12,56
that will require almost 200, 000
connections incoming per node.
1052
0:51:12,56 --> 0:51:13,94
Not all of them are active.
1053
0:51:14,24 --> 0:51:17,08
Are active actually are that number
divided by the number of
1054
0:51:17,08 --> 0:51:21,6
workers, but still a very high
number, a very, very, very fun
1055
0:51:21,6 --> 0:51:24,34
difference between the fan-in,
the number of connections incoming
1056
0:51:24,34 --> 0:51:27,52
to the workers through the poolers
versus the number of outgoing
1057
0:51:27,52 --> 0:51:30,96
connections, the pool connections
to Postgres, which needs to
1058
0:51:30,96 --> 0:51:35,82
remain constant at around 600 in
the case of the workers or 800,
1059
0:51:35,94 --> 0:51:38,34
something like this, because they
have double the CPUs, right?
1060
0:51:38,94 --> 0:51:43,7
And this difference between this
fan-in and fan-out creates inefficiency
1061
0:51:44,28 --> 0:51:45,74
in the workers themselves.
1062
0:51:45,86 --> 0:51:47,74
So it will start saturating earlier.
1063
0:51:47,9 --> 0:51:51,02
They also create inefficiencies
in the routers because they need
1064
0:51:51,02 --> 0:51:52,22
to talk to more nodes.
1065
0:51:53,08 --> 0:51:55,46
And so, will it reach there?
1066
0:51:55,68 --> 0:51:56,6
I don't know.
1067
0:51:56,72 --> 0:52:0,04
If someone wants to pour the money
into it, we can test it out.
1068
0:52:0,04 --> 0:52:4,64
I believe it may reach to a very
high number, maybe with a slightly
1069
0:52:4,64 --> 0:52:9,3
less efficiency, but it will reach
maybe 50 million, maybe more
1070
0:52:9,76 --> 0:52:12,04
with more effort, but it will do
so.
1071
0:52:12,18 --> 0:52:16,64
It's still compared to other sharding
solutions that can reach
1072
0:52:16,64 --> 0:52:17,38
those numbers.
1073
0:52:17,56 --> 0:52:21,24
I don't think all can, but out
of those that we have tested,
1074
0:52:21,76 --> 0:52:25,64
it doing so, this 10 million transactions
per 2nd, is using 0.5
1075
0:52:25,64 --> 0:52:29,68
of the silicon compared to other
sharding solutions.
1076
0:52:30,06 --> 0:52:32,96
So even though the scaling may
be slightly less linear because
1077
0:52:32,96 --> 0:52:36,82
of this Citus connection model,
which is Postgres connection
1078
0:52:36,82 --> 0:52:41,26
model, it needed 0.5 of the cores
to achieve those results.
1079
0:52:41,82 --> 0:52:44,52
So it's still very impressive,
right?
1080
0:52:44,76 --> 0:52:47,64
Now, Do I see a future different
than this?
1081
0:52:47,64 --> 0:52:48,3
I do.
1082
0:52:48,54 --> 0:52:53,46
I believe there may be an option
where the transport that Citus
1083
0:52:53,46 --> 0:52:57,98
uses could benefit from a multiplexing
transport, something like
1084
0:52:57,98 --> 0:53:2,06
gRPC, which other of the modern
solutions are adopting.
1085
0:53:3,34 --> 0:53:7,86
And that significantly reduces
the fan-in and fan-out connection
1086
0:53:7,9 --> 0:53:9,78
problem from the routers to the
workers.
1087
0:53:10,84 --> 0:53:15,38
So if I were to complete my initial
thesis that Citus is a great
1088
0:53:15,38 --> 0:53:20,14
solution as of today that is very
mature, well proven, featureful,
1089
0:53:20,42 --> 0:53:24,86
and so on, and that new solutions
may take time to mature.
1090
0:53:25,88 --> 0:53:30,68
Maybe, maybe, and it's a big if,
by just improving a bit Citus
1091
0:53:30,68 --> 0:53:34,6
architecture, it could take sharding
even to the next level if
1092
0:53:34,6 --> 0:53:38,3
needed, given that already it reaches
levels that are more than
1093
0:53:38,3 --> 0:53:41,76
enough for mostly everybody, but
even for those who may need
1094
0:53:41,82 --> 0:53:46,12
an even bigger scale, then maybe
It's just a matter of improving
1095
0:53:46,12 --> 0:53:46,62
Citus.
1096
0:53:47,54 --> 0:53:50,64
Nik: Yeah, I have only 1 question
to ask in the end.
1097
0:53:50,64 --> 0:53:55,78
I know you're comparing to other
benchmarks, and I'm just curious
1098
0:53:55,92 --> 0:53:59,48
in terms of, I saw some pictures
you shared with us before this
1099
0:53:59,48 --> 0:54:2,76
recording that was involving money,
like some dozens of thousands
1100
0:54:2,76 --> 0:54:4,7
of dollars to have this cluster.
1101
0:54:5,02 --> 0:54:9,4
I'm very curious, do you compare
with humans involved with support
1102
0:54:9,44 --> 0:54:14,36
or just pure open source and plus
cloud costs and that's it?
1103
0:54:15,78 --> 0:54:20,0
Álvaro: Yeah, so I did 2 comparisons
and they will be clearly
1104
0:54:20,14 --> 0:54:23,04
identified in the blog post that
will announce these benchmarks.
1105
0:54:23,8 --> 0:54:28,3
And so there's the pure infrastructure
cost, which is silicon,
1106
0:54:28,58 --> 0:54:32,06
like number of CPUs, where essentially
it's like around 55%.
1107
0:54:33,48 --> 0:54:36,18
So almost 0.5 the cost in terms
of CPU.
1108
0:54:36,68 --> 0:54:41,42
But then we can compute the total
cost to be very clear.
1109
0:54:41,42 --> 0:54:43,92
And there's nothing here to hide.
1110
0:54:44,06 --> 0:54:47,88
I'm comparing right now with a
benchmark that was run and published
1111
0:54:47,88 --> 0:54:48,54
by Neki.
1112
0:54:49,34 --> 0:54:52,86
And Neki is a managed service,
and they publish their list price.
1113
0:54:52,86 --> 0:54:55,36
And their list price is not the
instance cost.
1114
0:54:55,52 --> 0:54:59,16
It is the instance cost plus their
managing fee or their margin.
1115
0:55:0,04 --> 0:55:4,02
And when comparing with that, the
cost is not 0.5.
1116
0:55:4,3 --> 0:55:8,7
It's depending on what you consider
is between 0.333 or 0.2.
1117
0:55:8,94 --> 0:55:12,5
Actually, a well-dimensioned system
should be just 5 times cheaper.
1118
0:55:12,5 --> 0:55:16,08
Basically, 0.333 is if you compare
on-demand cost, which nobody
1119
0:55:16,08 --> 0:55:20,3
should run like a production cluster
sharded on on-demand cost.
1120
0:55:20,42 --> 0:55:24,44
And if you use a basic saving plans,
then you'll be 5 times cheaper.
1121
0:55:24,56 --> 0:55:25,06
Yeah.
1122
0:55:25,08 --> 0:55:28,16
And this is obviously a managed
service to a software comparison.
1123
0:55:28,58 --> 0:55:32,6
You may discuss that there may
be humans involved in the total
1124
0:55:32,66 --> 0:55:34,3
TCO calculation.
1125
0:55:35,02 --> 0:55:39,56
But my take is that a managed service
and software that does
1126
0:55:39,56 --> 0:55:42,42
a managed service for you is not
so much different.
1127
0:55:42,44 --> 0:55:46,22
Actually, I call StackGres a managed
software, a managed database
1128
0:55:46,22 --> 0:55:49,92
software, because when you say
a managed service, there's no
1129
0:55:49,92 --> 0:55:51,4
human typing, right?
1130
0:55:51,96 --> 0:55:55,44
There may be, if there's a support
ticket or something happens,
1131
0:55:55,44 --> 0:55:57,44
but the operation is just software.
1132
0:55:57,44 --> 0:56:0,36
So it's essentially the same for
me.
1133
0:56:0,8 --> 0:56:8,94
Nik: I know some humans can send
kill -9 in very big companies offering
1134
0:56:8,94 --> 0:56:10,66
managed services sometimes.
1135
0:56:11,0 --> 0:56:13,12
That kind of support is not welcome.
1136
0:56:13,94 --> 0:56:15,04
Not welcome, right?
1137
0:56:15,04 --> 0:56:16,9
kill -9 from support.
1138
0:56:18,28 --> 0:56:21,82
Michael: I also think these machines
are so big, like we're talking
1139
0:56:21,82 --> 0:56:25,36
about sharding and we're talking
about multiple huge machines
1140
0:56:25,46 --> 0:56:26,82
really, like they're expensive.
1141
0:56:27,34 --> 0:56:28,78
Often you don't pay this.
1142
0:56:28,78 --> 0:56:30,26
If you're actually a company,
1143
0:56:30,4 --> 0:56:30,9
Nik: saving
1144
0:56:31,1 --> 0:56:34,24
Michael: those machines, you would,
you would have like savings
1145
0:56:34,24 --> 0:56:37,64
plans either through PlanetScale
and or bring your own cloud
1146
0:56:37,64 --> 0:56:38,2
type thing.
1147
0:56:38,2 --> 0:56:41,32
Nik: So you're comparing not apples
to apples, but the numbers
1148
0:56:41,32 --> 0:56:46,2
you provided on your side, it could
be less with savings plan
1149
0:56:46,26 --> 0:56:50,24
if you pay in advance some number
of years to AWS.
1150
0:56:50,66 --> 0:56:53,76
On the other hand, you need probably
to roughly multiply it by
1151
0:56:53,76 --> 0:56:59,04
2, thinking about humans and the
support needed to surround this
1152
0:56:59,04 --> 0:57:0,04
solution, right?
1153
0:57:0,54 --> 0:57:1,22
Very roughly.
1154
0:57:2,78 --> 0:57:4,24
Álvaro: Yeah, so that's what I
said.
1155
0:57:4,24 --> 0:57:6,82
And so 1st of all, that's why I
want to give both numbers.
1156
0:57:6,82 --> 0:57:10,12
So you can 1st of all, compare
from the pure Silicon perspective
1157
0:57:10,12 --> 0:57:13,6
to pure architecture perspective,
and then from the sale price.
1158
0:57:13,94 --> 0:57:18,42
But as I said, For me, a managed
software, it's just software.
1159
0:57:18,58 --> 0:57:20,52
A managed service is just software,
right?
1160
0:57:20,86 --> 0:57:23,72
And if the software automates and
runs everything for you, which
1161
0:57:23,72 --> 0:57:25,58
is the case in both cases, right?
1162
0:57:25,58 --> 0:57:30,18
Because all managed services, there's
no humans doing operations,
1163
0:57:30,36 --> 0:57:31,4
It's only software.
1164
0:57:31,4 --> 0:57:35,1
So humans only intervene in the
case of issues.
1165
0:57:35,9 --> 0:57:37,36
And in those cases, yes.
1166
0:57:37,36 --> 0:57:41,48
And you can get like a, we sell
enterprise, the StackGres Enterprise,
1167
0:57:41,48 --> 0:57:42,6
which is exactly that.
1168
0:57:42,6 --> 0:57:44,32
Support for those cases, right?
1169
0:57:45,1 --> 0:57:47,56
And that's something optional,
but that's something you would
1170
0:57:47,56 --> 0:57:49,02
add on top of that.
1171
0:57:49,02 --> 0:57:51,6
And then the calculation will not
be 5 times cheaper, it will
1172
0:57:51,6 --> 0:57:52,08
be
1173
0:57:52,08 --> 0:57:52,66
Nik: maybe, I
1174
0:57:52,66 --> 0:57:54,86
Álvaro: don't have the numbers
exactly, 3 times cheaper, 4 times
1175
0:57:54,86 --> 0:57:57,08
cheaper, but still obviously much
cheaper.
1176
0:57:57,44 --> 0:57:59,94
Nik: Yeah, thank you for this clarification,
that's interesting.
1177
0:58:0,48 --> 0:58:3,48
Álvaro: And sorry, also to add,
and that's because talking about
1178
0:58:3,48 --> 0:58:6,66
cloud, but if you talk about on-prem,
which is definitely in
1179
0:58:6,66 --> 0:58:10,06
the space where we also, the StackGres
works anywhere on-prem,
1180
0:58:10,32 --> 0:58:14,08
then the math is even much more
into your favor.
1181
0:58:14,84 --> 0:58:15,53
Nik: Right, if you own hardware.
1182
0:58:15,53 --> 0:58:15,72
If you own
1183
0:58:15,72 --> 0:58:18,92
Álvaro: hardware and data centers,
1184
0:58:19,28 --> 0:58:23,3
Nik: then it's open source, you
can troubleshoot with your engineering
1185
0:58:23,36 --> 0:58:24,34
team if needed.
1186
0:58:24,34 --> 0:58:24,68
That's
1187
0:58:24,68 --> 0:58:25,18
Álvaro: great.
1188
0:58:25,765 --> 0:58:26,265
Thank
1189
0:58:26,85 --> 0:58:29,44
Nik: you so much.
1190
0:58:29,44 --> 0:58:30,12
Very interesting.
1191
0:58:30,18 --> 0:58:34,68
And also I see several paths for
future work.
1192
0:58:35,06 --> 0:58:38,46
Maybe we should have another episode
at some point in the future
1193
0:58:38,76 --> 0:58:41,52
to revisit the topics in some directions.
1194
0:58:41,6 --> 0:58:42,26
Álvaro: Any time.
1195
0:58:42,44 --> 0:58:44,98
I would be more than happy to come
back.
1196
0:58:45,66 --> 0:58:46,4
Nik: Thank you so much.
1197
0:58:46,4 --> 0:58:47,26
It was interesting.
1198
0:58:47,72 --> 0:58:49,02
Very interesting to me.
1199
0:58:49,6 --> 0:58:51,04
Álvaro: Thank you for inviting
me.
0:0:0,06 --> 0:0:2,72
Nik: Hello, hello, this is Postgres.FM,
as usual.
2
0:0:2,8 --> 0:0:6,3399997
I'm Nik, PostgresAI, and as usual
with me Michael, pgMustard.
3
0:0:6,68 --> 0:0:7,46
Hi, Michael.
4
0:0:8,26 --> 0:0:9,059999
Michael: Hi, Nik.
5
0:0:9,559999 --> 0:0:11,42
Nik: We have a great guest today.
6
0:0:11,719999 --> 0:0:13,16
Did we have you already?
7
0:0:13,48 --> 0:0:15,04
Álvaro from Ongres.
8
0:0:15,72 --> 0:0:16,220001
Álvaro: Hello.
9
0:0:16,46 --> 0:0:18,28
No, it's my 1st time here.
10
0:0:18,54 --> 0:0:20,64
At least as a speaker, not as a
listener.
11
0:0:21,7 --> 0:0:22,54
Nik: Thank you for coming.
12
0:0:22,54 --> 0:0:24,439999
I know you're a very active listener.
13
0:0:25,08 --> 0:0:30,02
We definitely have you on Postgres
TV channel, on YouTube channel,
14
0:0:30,02 --> 0:0:31,2
talking about licenses.
15
0:0:31,22 --> 0:0:32,46
That's why I'm confused.
16
0:0:32,48 --> 0:0:33,3
Do you remember?
17
0:0:34,06 --> 0:0:34,56
Álvaro: Yeah.
18
0:0:35,08 --> 0:0:39,0
Nik: And I know we will touch a little
bit licenses today, just knowing
19
0:0:39,0 --> 0:0:39,64
the topic.
20
0:0:39,760002 --> 0:0:41,02
Yeah, thank you for coming.
21
0:0:41,4 --> 0:0:45,4
I'm especially excited that today's
topic will connect so many
22
0:0:45,4 --> 0:0:46,9
dots, right?
23
0:0:47,16 --> 0:0:52,12
Performance, sharding, Kubernetes
operator, what your company
24
0:0:52,12 --> 0:0:52,84
is doing.
25
0:0:53,08 --> 0:0:55,68
It's so open source and true open
source, right?
26
0:0:55,68 --> 0:1:0,1
I'm very excited, looking forward
to dive into this chat because
27
0:1:0,1 --> 0:1:4,62
many topics I'm very interested
in myself and my company also,
28
0:1:5,32 --> 0:1:6,72
they are connected today.
29
0:1:7,72 --> 0:1:9,92
Álvaro: No, thank you very much
for having me.
30
0:1:9,92 --> 0:1:14,22
It's been a lot of episodes listening
with you over the year
31
0:1:14,34 --> 0:1:16,96
and now being present here, it's
very exciting for me.
32
0:1:16,96 --> 0:1:17,82
So thank you.
33
0:1:18,54 --> 0:1:22,58
Nik: And today we will talk about
benchmark you haven't published
34
0:1:22,58 --> 0:1:23,08
yet.
35
0:1:23,479996 --> 0:1:27,68
So thank you for coming with results
being still prepared.
36
0:1:27,84 --> 0:1:31,36
That's great because I know we
scheduled this episode before
37
0:1:31,36 --> 0:1:33,4
I think you decided to work on
it.
38
0:1:33,4 --> 0:1:34,2
So this is like...
39
0:1:34,2 --> 0:1:35,46
Álvaro: Right, that's correct.
40
0:1:36,98 --> 0:1:38,56
Nik: Action happening in real time.
41
0:1:38,56 --> 0:1:39,3
That's great.
42
0:1:39,52 --> 0:1:44,14
And yeah, I received your pictures,
3 charts about benchmark
43
0:1:44,14 --> 0:1:47,72
data, benchmark results this morning,
so I'm catching up.
44
0:1:47,98 --> 0:1:49,46
Where should we start actually?
45
0:1:49,6 --> 0:1:52,44
Because this is a great topic,
right?
46
0:1:52,44 --> 0:1:55,18
So how Postgres can scale, what
numbers we can achieve.
47
0:1:55,58 --> 0:1:58,94
Maybe we should start mentioning
that Michael and I had an episode
48
0:1:58,94 --> 0:2:3,48
about our company benchmarks on
1 node, no sharding at all, just
49
0:2:3,48 --> 0:2:7,72
1 node, we achieved 4 million TPS
a couple of years ago.
50
0:2:7,72 --> 0:2:9,44
It was a couple of years ago, right?
51
0:2:9,9 --> 0:2:14,52
With SELECT only primary key
lookup just to see how it's not
52
0:2:14,54 --> 0:2:19,24
super huge compared to some other
database systems, which might
53
0:2:19,24 --> 0:2:22,48
be faster in specific workloads,
but it's great that general
54
0:2:22,48 --> 0:2:23,98
Postgres can achieve that.
55
0:2:24,48 --> 0:2:29,8
And you can give us numbers much
bigger because you involve sharding,
56
0:2:29,8 --> 0:2:30,3
right?
57
0:2:31,42 --> 0:2:32,2
Álvaro: That's correct.
58
0:2:32,38 --> 0:2:36,26
If you want, I can start a bit
with my thesis about sharding
59
0:2:36,26 --> 0:2:37,62
in general for Postgres.
60
0:2:38,94 --> 0:2:43,84
I think it's a field where we're
seeing a lot of innovation happening,
61
0:2:43,86 --> 0:2:44,86
and I love this.
62
0:2:45,22 --> 0:2:48,98
Innovation and competition are
great, are what drives us forward.
63
0:2:49,4 --> 0:2:53,74
Nik: Let me interrupt you with
plugging in episodes with Sugu,
64
0:2:53,74 --> 0:2:57,94
Multigres I think we had 1 or
2, and definitely we had 2 episodes
65
0:2:57,98 --> 0:2:59,68
with Lev PgDog.
66
0:2:59,76 --> 0:3:4,02
So There is an innovation starting
last year, which is great.
67
0:3:4,08 --> 0:3:5,78
And I know you have your own opinion.
68
0:3:6,58 --> 0:3:7,84
Apologies for interrupting.
69
0:3:8,08 --> 0:3:8,86
Let's continue.
70
0:3:9,72 --> 0:3:9,96
Álvaro: Yeah.
71
0:3:9,96 --> 0:3:10,2
Yeah.
72
0:3:10,2 --> 0:3:11,52
I do have my opinion.
73
0:3:11,98 --> 0:3:16,3
Basically databases, they take
a long time to mature.
74
0:3:16,64 --> 0:3:20,28
Databases need to be the places
where you can trust your data,
75
0:3:20,28 --> 0:3:22,7
that it's gonna be there, that
you're durable, and they obviously
76
0:3:22,72 --> 0:3:25,62
obey the rest of the properties
that you expect from a database.
77
0:3:26,14 --> 0:3:30,3
And typically new databases are
risky or tend to lag adoption
78
0:3:30,3 --> 0:3:31,58
because they need to mature.
79
0:3:32,08 --> 0:3:35,24
When you're sharding, my thoughts
keep being the same.
80
0:3:35,34 --> 0:3:39,24
You really need the mature and
trust solution, a boring, I would
81
0:3:39,24 --> 0:3:42,1
say, technology to run your shards
because now you don't have
82
0:3:42,1 --> 0:3:43,68
1 cluster, you have many clusters.
83
0:3:44,28 --> 0:3:47,72
Each shard, each portion of the
database, it's a cluster on its
84
0:3:47,72 --> 0:3:48,22
own.
85
0:3:48,42 --> 0:3:51,96
So the software that manages all
that needs to also be well-established,
86
0:3:52,28 --> 0:3:53,04
let's say.
87
0:3:53,68 --> 0:3:56,94
And again, in competition and innovation
are absolutely great
88
0:3:56,94 --> 0:3:58,24
and very, very welcome.
89
0:3:58,74 --> 0:4:1,8
But I believe that there's a great
technology out there, open
90
0:4:1,8 --> 0:4:5,6
source for sharding Postgres that's
been in the market for, I
91
0:4:5,6 --> 0:4:8,16
guess, like more than 10 years
called Citus.
92
0:4:9,24 --> 0:4:10,58
And Citus is just great.
93
0:4:10,76 --> 0:4:13,94
Might not be perfect, but it's
really good enough or more than
94
0:4:13,94 --> 0:4:14,68
that even.
95
0:4:15,14 --> 0:4:18,9
And a lot of people may have forgotten
about it or put it a little
96
0:4:18,9 --> 0:4:19,46
bit aside.
97
0:4:19,46 --> 0:4:23,86
Myself, the same several years
ago, until I started with thinking,
98
0:4:23,9 --> 0:4:27,46
OK, people need to shard Postgres
and people need to scale Postgres.
99
0:4:27,94 --> 0:4:32,56
And not only because of the performance
directly, because as
100
0:4:32,56 --> 0:4:37,22
you demonstrated, you can run very
high TPS on a single node.
101
0:4:37,38 --> 0:4:40,46
But sometimes it's also about the
blast radius.
102
0:4:40,92 --> 0:4:45,06
Like if you're running a SaaS shop
or an e-commerce and your
103
0:4:45,06 --> 0:4:47,82
database cluster goes down for
whatever reason.
104
0:4:47,9 --> 0:4:51,44
Sometimes it's a configuration
issue with the network, whatever
105
0:4:51,44 --> 0:4:52,16
it is, right?
106
0:4:52,16 --> 0:4:55,02
Your cluster is down, then your
whole thing is down.
107
0:4:55,58 --> 0:5:0,78
If you shard and you split your
database into, let's say 16 logical
108
0:5:0,84 --> 0:5:1,5
shards, right?
109
0:5:1,5 --> 0:5:2,22
Or workers.
110
0:5:2,68 --> 0:5:6,06
And then 1 goes down because of
a configuration error, then it's
111
0:5:6,06 --> 0:5:8,94
only a small percentage of your
users that are effectively down,
112
0:5:8,94 --> 0:5:11,56
which it's not good, but it's definitely
not 100%.
113
0:5:11,64 --> 0:5:14,18
Your PR is completely different
night and day, right?
114
0:5:14,18 --> 0:5:17,44
So a lot of people now need to
shard either because of performance
115
0:5:17,44 --> 0:5:20,42
reasons, because they cannot scale
the writes to a single primary,
116
0:5:20,82 --> 0:5:25,46
and or because of high availability
and blast radius reasons.
117
0:5:26,28 --> 0:5:28,88
And we always had Citus there.
118
0:5:29,38 --> 0:5:31,16
And Citus is fully featured.
119
0:5:32,02 --> 0:5:35,16
It has almost everything that you
need for sharding, for running
120
0:5:35,16 --> 0:5:38,34
complex queries that go down, they're
pushed down to the workers,
121
0:5:39,0 --> 0:5:42,48
and has the tools for rebalancing,
for dealing with hot shards,
122
0:5:42,74 --> 0:5:45,48
for almost all operational activities
that you need to perform
123
0:5:45,48 --> 0:5:47,86
on a sharded database and it just
works.
124
0:5:48,32 --> 0:5:52,44
I believe because their initial
focus was more OLAP style or
125
0:5:52,44 --> 0:5:57,24
mixed OLTP OLAP that people started
considering Citus only for
126
0:5:57,24 --> 0:6:0,8
that and it's definitely good fit
for that but it's also really
127
0:6:0,8 --> 0:6:1,7
good for OLTP.
128
0:6:3,06 --> 0:6:4,24
Nik: 2 points here.
129
0:6:4,24 --> 0:6:7,32
1st of all, you said we always
had.
130
0:6:7,54 --> 0:6:11,14
Let me disagree with you because
when I actually was looking
131
0:6:11,14 --> 0:6:13,62
into Citus, it was several years
ago.
132
0:6:13,96 --> 0:6:17,66
The Key features were not open
source.
133
0:6:17,8 --> 0:6:19,4
For example, online rebalancing.
134
0:6:19,94 --> 0:6:22,58
So we didn't have it in the open
source area.
135
0:6:23,26 --> 0:6:26,42
It was like you need to have a
contract with them to have online
136
0:6:26,42 --> 0:6:30,24
rebalancing, so it quite quickly
becomes out of scope if you
137
0:6:30,24 --> 0:6:33,02
want to rely on open source and
avoid vendor lock-in.
138
0:6:33,08 --> 0:6:34,1
Now it's different.
139
0:6:34,2 --> 0:6:36,24
It has changed a few years ago.
140
0:6:36,34 --> 0:6:39,88
All features after acquisition
by Microsoft, at some point all
141
0:6:39,88 --> 0:6:42,6
features went to open source, which
is great.
142
0:6:43,26 --> 0:6:45,26
But it's not like we always had,
right?
143
0:6:45,26 --> 0:6:46,78
Álvaro: Yeah, you're absolutely
right.
144
0:6:46,8 --> 0:6:47,8
And thanks for that.
145
0:6:47,8 --> 0:6:52,5
I think, if I'm not mistaken, Citus
version 11 was the 1st 1
146
0:6:52,5 --> 0:6:54,82
that included everything as open
source.
147
0:6:55,24 --> 0:6:57,32
And that was, I think, 2022.
148
0:6:57,64 --> 0:7:2,22
So 4 years ago, 4 years and 0.5
ago, that was released as fully
149
0:7:2,22 --> 0:7:3,58
open source and has been since
then.
150
0:7:3,58 --> 0:7:4,7
So yes, you're right.
151
0:7:4,74 --> 0:7:5,68
For this podcast,
152
0:7:6,22 --> 0:7:9,5
Nik: you're right also, because
this podcast exists for years.
153
0:7:9,72 --> 0:7:10,18
Exactly.
154
0:7:10,18 --> 0:7:12,62
So, good.
155
0:7:13,38 --> 0:7:16,52
And the 2nd point is OLAP versus
OLTP.
156
0:7:16,56 --> 0:7:21,34
My concern, like, last 4 years,
I think I didn't dive into Citus.
157
0:7:21,34 --> 0:7:25,96
I remember, diving means like I
run it, inspect myself, the benchmark
158
0:7:25,96 --> 0:7:26,44
and so on.
159
0:7:26,44 --> 0:7:29,26
I did it before we started podcast
actually.
160
0:7:29,26 --> 0:7:32,9
I remember I was reporting some
bugs And since then I didn't
161
0:7:32,9 --> 0:7:37,58
dive into it, but my concern is
we didn't see great examples
162
0:7:38,4 --> 0:7:43,82
of some well-known companies, SaaS
companies, for example, running
163
0:7:43,82 --> 0:7:44,32
it.
164
0:7:44,54 --> 0:7:48,76
Like, you know, if somebody like
GitHub scale would be running
165
0:7:48,76 --> 0:7:51,3
Citus, I wouldn't have any concerns
at all.
166
0:7:51,72 --> 0:7:52,94
What are the great examples?
167
0:7:53,16 --> 0:7:54,6
OLTP style.
168
0:7:56,04 --> 0:7:56,92
That's my concern.
169
0:7:56,92 --> 0:7:59,86
And I think it's still like, I
agree, like what numbers you bring,
170
0:7:59,86 --> 0:8:0,8
and this is interesting.
171
0:8:0,8 --> 0:8:2,46
We dive into them today.
172
0:8:2,8 --> 0:8:6,42
And I was in talks with Citus team
some time ago, maybe a couple
173
0:8:6,42 --> 0:8:7,32
of years ago.
174
0:8:7,48 --> 0:8:12,54
And they presented me some examples
which were not OLTP.
175
0:8:12,62 --> 0:8:17,6
For example, some search for, I
think, lawyers of video, where
176
0:8:17,6 --> 0:8:22,22
I thought like, actually lawyers
can wait 10 seconds to find
177
0:8:22,22 --> 0:8:24,3
problematic materials and so on.
178
0:8:24,38 --> 0:8:28,58
It's not OLTP when we know 200 milliseconds,
our very 1st episode
179
0:8:28,58 --> 0:8:33,04
in this podcast means that humans
already notice delay.
180
0:8:33,46 --> 0:8:36,18
Álvaro: So I think there's 2 angles
to what you're saying.
181
0:8:36,18 --> 0:8:40,36
So 1 is the technical 1, and the
other 1 is the marketing or
182
0:8:40,36 --> 0:8:41,98
trust 1, let's say.
183
0:8:42,28 --> 0:8:46,02
On the trust 1, I would say you're
right, meaning that there's
184
0:8:46,02 --> 0:8:47,22
no such great examples.
185
0:8:47,22 --> 0:8:49,54
And this actually piggybacks on
what I was saying.
186
0:8:49,54 --> 0:8:52,24
It looks like it's a technology
that didn't get the momentum
187
0:8:52,24 --> 0:8:56,1
it could have had, or it could
still has, because of the technological
188
0:8:56,32 --> 0:8:58,04
means that it conveys, right?
189
0:8:58,18 --> 0:9:1,72
I can tell you that we have right
now significant examples of
190
0:9:1,72 --> 0:9:2,64
this kind of usage.
191
0:9:2,64 --> 0:9:6,92
We have now a customer running
Citus on StackGres that is actually
192
0:9:7,02 --> 0:9:10,62
growing database towards 2.2 petabytes
by the end of the year.
193
0:9:10,92 --> 0:9:11,12
And...
194
0:9:11,12 --> 0:9:13,54
Nik: Wait, hold on 1 2nd, 2.2 petabytes?
195
0:9:14,44 --> 0:9:14,94
Yes.
196
0:9:15,48 --> 0:9:17,46
It's just 1 big Citus cluster?
197
0:9:18,08 --> 0:9:18,28
Yes.
198
0:9:18,28 --> 0:9:19,24
Álvaro: Running on StackGres?
199
0:9:20,02 --> 0:9:20,52
Yes.
200
0:9:21,04 --> 0:9:21,1
Nik: Okay.
201
0:9:21,1 --> 0:9:22,06
How many nodes?
202
0:9:23,0 --> 0:9:23,5
Álvaro: 24.
203
0:9:24,78 --> 0:9:30,26
Each node is, I'm not sure if it's
64 or 96 cores, and 0.5 a
204
0:9:30,26 --> 0:9:31,94
terabyte or more of RAM.
205
0:9:33,34 --> 0:9:34,5
Nik: Wow, okay.
206
0:9:34,86 --> 0:9:36,6
Álvaro: Those numbers are actually
quite impressive.
207
0:9:37,36 --> 0:9:42,44
It's an IoT system and they ingest
data massively from millions
208
0:9:42,44 --> 0:9:45,52
and millions of devices that send
heartbeats every 5 minutes.
209
0:9:46,1 --> 0:9:48,72
And they have a 10 day retention
period.
210
0:9:49,34 --> 0:9:52,84
So basically they're deleting 1
10th of the database and ingesting
211
0:9:52,84 --> 0:9:54,94
1 10th of the database every day.
212
0:9:55,44 --> 0:9:58,74
Right now they're in ramping up
period, and as of today, they're
213
0:9:58,74 --> 0:10:3,9
ingesting some 30 plus terabytes
per day, and this is gonna grow
214
0:10:3,9 --> 0:10:5,74
to 200 terabytes per day.
215
0:10:5,74 --> 0:10:8,08
So they're growing by 6X by the
end of the year.
216
0:10:8,92 --> 0:10:11,26
And yeah, this is running on a
single Citus cluster.
217
0:10:11,64 --> 0:10:14,98
We can discuss whether this is
a little bit OLTP or OLAP.
218
0:10:15,06 --> 0:10:16,22
It's a bit of both.
219
0:10:16,36 --> 0:10:17,98
Ingestion is OLTP.
220
0:10:18,84 --> 0:10:21,28
Queries, there are a little bit
of both.
221
0:10:21,28 --> 0:10:25,38
Also OLTP queries when they're
accessing health information about
222
0:10:25,38 --> 0:10:26,4
a given device.
223
0:10:26,88 --> 0:10:31,16
Or more aggregated reports, like
how many devices are down or
224
0:10:31,16 --> 0:10:32,22
having some issues.
225
0:10:32,44 --> 0:10:35,74
So it's a bit of a mixed workload,
but I think it works.
226
0:10:35,74 --> 0:10:39,52
Again, I agree, there are not so
many public great examples about
227
0:10:39,52 --> 0:10:40,46
using Citus.
228
0:10:40,52 --> 0:10:44,06
Citus also introduced some releases
ago, the support for sharding
229
0:10:44,06 --> 0:10:48,84
by tenant, which is actually directly
towards SaaS enterprises,
230
0:10:48,9 --> 0:10:49,14
right?
231
0:10:49,14 --> 0:10:51,98
The SaaS applications where you
don't shard by row, you shard
232
0:10:51,98 --> 0:10:55,02
by tenant, which is a great fit
for those cases too.
233
0:10:55,24 --> 0:10:57,94
Typically, these cases where you
create like a lot of tables
234
0:10:58,0 --> 0:10:59,08
for every tenant.
235
0:10:59,28 --> 0:11:1,74
So it's another way of isolating
those workloads.
236
0:11:1,98 --> 0:11:4,18
But from a technical perspective
and speaking.
237
0:11:5,22 --> 0:11:8,54
Nik: Yeah, I just, I just, I'm,
I'm, I'm smiling at my glasses
238
0:11:8,56 --> 0:11:11,8
because they said, they started
arguing with me saying that there
239
0:11:11,8 --> 0:11:16,62
are public well-known names like
Heap and the Cloudflare named
240
0:11:16,62 --> 0:11:20,14
on, on Citus pages for, for
years.
241
0:11:20,14 --> 0:11:24,18
But, but both of these cases are
analytical cases, 1st of all,
242
0:11:24,18 --> 0:11:27,38
and both these cases went away
from Citus, as we know.
243
0:11:27,62 --> 0:11:33,06
Like, I was at Meetup when Cloudflare
explained how they moved
244
0:11:33,06 --> 0:11:35,64
away from Citus to ClickHouse.
245
0:11:36,06 --> 0:11:39,44
And I know Heap Analytics moved
somewhere else as well.
246
0:11:39,56 --> 0:11:42,04
Heap Analytics was a great case,
but again, analytics.
247
0:11:42,72 --> 0:11:45,56
And Cloudflare, it was analytics,
DNS analytics.
248
0:11:45,78 --> 0:11:46,86
So it's not OLTP.
249
0:11:47,2 --> 0:11:48,7
It can be HTAP.
250
0:11:48,7 --> 0:11:49,4
Yeah, I agree.
251
0:11:49,4 --> 0:11:50,7
And your case is interesting.
252
0:11:50,74 --> 0:11:54,02
It's not what I want to see, like
I said, GitHub or some social
253
0:11:54,02 --> 0:11:56,46
media or something, when there
are many humans working.
254
0:11:56,52 --> 0:12:1,82
Your case is a great case for sharding
Specifically because it
255
0:12:1,92 --> 0:12:6,78
shows exactly where the problem
with the biggest pain of single
256
0:12:6,78 --> 0:12:8,5
cluster is a lot of writes.
257
0:12:9,38 --> 0:12:14,18
This case is highlighting this
particular use case, which is
258
0:12:14,18 --> 0:12:14,66
great.
259
0:12:14,66 --> 0:12:18,52
But I'm still like, my heart is
with social media, SaaS, like
260
0:12:18,52 --> 0:12:21,86
these things, like e-commerce,
online stuff, where a lot of humans
261
0:12:21,86 --> 0:12:26,18
try to deal with some system and
then they want greater responsiveness
262
0:12:26,92 --> 0:12:28,28
of all API calls.
263
0:12:28,28 --> 0:12:31,18
Álvaro: So let's talk about that
because part of my response
264
0:12:31,2 --> 0:12:34,08
also was coming from a technical
angle and from a technical angle
265
0:12:34,08 --> 0:12:38,56
we've been running some benchmarks
and 1 of the questions whether
266
0:12:38,56 --> 0:12:42,56
to say the system is ready for
OLTP or not is the number of concurrent
267
0:12:42,56 --> 0:12:45,32
queries that it can run and the
latency at which those queries
268
0:12:45,32 --> 0:12:45,82
run.
269
0:12:46,32 --> 0:12:51,82
And Citus can run on millions of
transactions per 2nd, even under
270
0:12:51,82 --> 0:12:53,5
1 millisecond p50.
271
0:12:54,62 --> 0:12:55,94
I would call that OLTP.
272
0:12:56,8 --> 0:12:57,7
Nik: I agree here.
273
0:12:57,7 --> 0:13:1,68
If you check very fast SELECTs,
we have benchmarks.
274
0:13:1,78 --> 0:13:6,94
We did it multiple times for actually
all sharding systems, open
275
0:13:6,94 --> 0:13:10,46
source sharding systems we can
reach, including SPQR, Citus,
276
0:13:10,84 --> 0:13:14,74
Multigres actually we also tested,
PgDog, something else.
277
0:13:15,3 --> 0:13:20,26
And Citus, you checked for latency
overhead, off latency, latency
278
0:13:20,26 --> 0:13:21,66
overhead benchmark, simple.
279
0:13:22,54 --> 0:13:26,74
Which like if primary key lookup is
sub millisecond, you don't want
280
0:13:26,74 --> 0:13:29,66
another few milliseconds to be
introduced.
281
0:13:29,7 --> 0:13:33,96
And Citus was actually looking
very well.
282
0:13:34,64 --> 0:13:38,04
It was introducing like 200 microseconds
or so, as I remember.
283
0:13:38,8 --> 0:13:42,14
Álvaro: Yeah, I actually have drilled
down on the CPU consumption
284
0:13:42,2 --> 0:13:44,0
per layer for this kind of queries.
285
0:13:44,44 --> 0:13:48,84
And the Total aggregated CPU consumption
per query, including
286
0:13:49,5 --> 0:13:53,4
the coordinator, including the
worker, including the connection
287
0:13:53,4 --> 0:13:58,22
pooling, excluding the network,
is somewhere in between 300 to
288
0:13:58,44 --> 0:13:59,02
400 microseconds.
289
0:13:59,34 --> 0:14:0,52
Nik: Microseconds, exactly.
290
0:14:0,54 --> 0:14:1,02
Yeah, yeah.
291
0:14:1,02 --> 0:14:1,5
Okay.
292
0:14:1,5 --> 0:14:4,46
Slightly different from what I
remember from our benchmarks but
293
0:14:4,46 --> 0:14:8,48
well the point here is that it
can be very quickly eaten by something
294
0:14:8,48 --> 0:14:12,48
like network complexity right you
won't notice this small overhead
295
0:14:12,48 --> 0:14:16,0
behind something else like
just hops between yeah
296
0:14:16,0 --> 0:14:18,72
Álvaro: the networks that can be
worse than that yeah yeah
297
0:14:18,74 --> 0:14:20,06
Nik: that's I agree yeah
298
0:14:20,34 --> 0:14:23,24
Michael: Álvaro when you query
when you're doing that check are
299
0:14:23,24 --> 0:14:25,9
you going through a coordinator
in Citus?
300
0:14:26,54 --> 0:14:29,7
Because I know now you can query
directly 1 of the workers.
301
0:14:30,06 --> 0:14:31,9
Would that reduce the overhead?
302
0:14:33,52 --> 0:14:37,96
Álvaro: So, yes, the usual way
you wire up a Citus cluster is
303
0:14:37,96 --> 0:14:42,18
you have workers, which are where
you split the data into and
304
0:14:42,18 --> 0:14:47,5
then you have the coordinator,
which acts as a metadata entry
305
0:14:47,5 --> 0:14:49,46
point, but also as a query router.
306
0:14:49,96 --> 0:14:52,7
And so it is the entry point by
default.
307
0:14:53,14 --> 0:14:56,26
It's true that Citus at version
11 introduced a functionality
308
0:14:56,46 --> 0:14:59,44
where you can query also the worker
nodes directly.
309
0:14:59,44 --> 0:15:1,1
There's 2 reasons to do this.
310
0:15:1,28 --> 0:15:4,78
1, if you know from the application
perspective, the sharding
311
0:15:4,78 --> 0:15:7,66
algorithm and you know which shard
to talk directly, which is
312
0:15:7,66 --> 0:15:10,86
probably not a good idea, but technically
sometimes you can do
313
0:15:10,86 --> 0:15:11,16
this.
314
0:15:11,16 --> 0:15:14,72
It is a good idea actually, but
the algorithm is a little bit
315
0:15:14,72 --> 0:15:16,06
internal to Citus, right?
316
0:15:16,06 --> 0:15:19,4
If it would be like clear, exposed
algorithm, then it would be
317
0:15:19,4 --> 0:15:21,9
very good for applications that
want to implement that algorithm,
318
0:15:21,9 --> 0:15:23,16
then talk directly to Shards.
319
0:15:23,16 --> 0:15:25,68
But at least Citus gives you the
opportunity to do that if you
320
0:15:25,68 --> 0:15:26,24
want to do so.
321
0:15:26,24 --> 0:15:27,72
You can target directly the workers.
322
0:15:27,72 --> 0:15:30,04
And if you target directly the
worker, there's no query routing.
323
0:15:30,04 --> 0:15:32,98
It goes directly and resolves the
query locally, if it's local
324
0:15:33,04 --> 0:15:35,66
or fans out to other workers if
required.
325
0:15:36,22 --> 0:15:39,58
But the other reason is to try
to scale the single coordinator.
326
0:15:39,96 --> 0:15:43,82
And I believe this actually also
piggybacks a little bit with
327
0:15:43,82 --> 0:15:47,5
what you were saying, Nikolay, that
Citus maybe was abandoned
328
0:15:47,5 --> 0:15:49,5
by some project before, like Cloudflare.
329
0:15:50,22 --> 0:15:53,42
And Citus can scale workers horizontally
as much as you need.
330
0:15:53,42 --> 0:15:56,52
I was referring to this 24 node
cluster before, and there's been
331
0:15:56,52 --> 0:16:0,12
reports of much larger clusters,
like 60 nodes or more.
332
0:16:0,3 --> 0:16:2,12
And they scale very well.
333
0:16:2,26 --> 0:16:4,84
However, the coordinator remains
only 1.
334
0:16:5,9 --> 0:16:11,18
And if you outrun the coordinator,
then you're hitting a ceiling.
335
0:16:11,52 --> 0:16:14,58
That ceiling is normally quite
high and actually we've measured
336
0:16:14,58 --> 0:16:17,14
that ceiling, but it exists, right?
337
0:16:17,22 --> 0:16:19,92
So that's why Citus introduced
functionality, you can query the
338
0:16:19,92 --> 0:16:22,62
routers directly, and then you
have like essentially multiple
339
0:16:22,64 --> 0:16:23,14
coordinators.
340
0:16:23,6 --> 0:16:26,84
But I don't think that's the way
to go either, because then you're
341
0:16:26,84 --> 0:16:28,12
now contending workloads.
342
0:16:28,26 --> 0:16:32,5
So you have nodes that act both
as query routers, which consumes
343
0:16:32,5 --> 0:16:36,38
CPU, consume resources, notable,
they require connections, and
344
0:16:36,38 --> 0:16:39,72
they spread all the connections
to other nodes, making also connection
345
0:16:39,72 --> 0:16:40,54
math harder.
346
0:16:41,2 --> 0:16:43,62
And they're also serving data and
queries.
347
0:16:43,84 --> 0:16:49,54
So it's a bit of a mixed workload
that I believe is hard to dimension
348
0:16:49,78 --> 0:16:53,04
and do capacity planning around
that and makes everything much
349
0:16:53,04 --> 0:16:53,68
more complicated.
350
0:16:53,68 --> 0:16:55,74
So it's not a perfect solution
either.
351
0:16:56,48 --> 0:16:59,74
Nik: I think we forgot an important
piece.
352
0:17:0,72 --> 0:17:5,14
Why Álvaro at all talking about
Citus, right?
353
0:17:5,14 --> 0:17:6,3
This is a good question.
354
0:17:6,3 --> 0:17:7,64
You're doing something else.
355
0:17:7,64 --> 0:17:9,66
Why are you talking about Citus
at all?
356
0:17:9,72 --> 0:17:11,46
Citus is in the hands of Microsoft.
357
0:17:12,56 --> 0:17:13,66
What happened here?
358
0:17:14,54 --> 0:17:19,02
You developed StackGres, and then
somehow these paths joined,
359
0:17:19,02 --> 0:17:19,52
right?
360
0:17:19,6 --> 0:17:22,66
They met these things and you have
examples.
361
0:17:23,68 --> 0:17:26,7
That's why you mentioned this 2
petabyte example, right?
362
0:17:26,88 --> 0:17:29,6
Can you explain a little bit what's
happening, why you focus
363
0:17:29,6 --> 0:17:30,98
on Citus so much lately?
364
0:17:32,5 --> 0:17:35,78
Álvaro: Well, it's not me, it's
what customers and users request.
365
0:17:36,42 --> 0:17:40,32
I see a growing concern and need
to scale Postgres beyond the
366
0:17:40,32 --> 0:17:43,4
single primary for the reasons
I mentioned before, write limitations
367
0:17:43,94 --> 0:17:45,36
and also blast radius.
368
0:17:45,78 --> 0:17:48,62
And that's why we are also seeing
an explosion of new sharding
369
0:17:48,62 --> 0:17:49,84
technologies for Postgres.
370
0:17:49,84 --> 0:17:53,5
So this is a real need and is driven
by market demand.
371
0:17:53,5 --> 0:17:57,26
We are seeing a lot of people coming
to us asking how we can
372
0:17:57,26 --> 0:17:58,4
shard our database.
373
0:17:59,44 --> 0:18:2,44
And StackGres, it's a platform for
running Postgres, right?
374
0:18:2,44 --> 0:18:5,28
That's it, it's fully featured,
it's really advanced, but it's
375
0:18:5,28 --> 0:18:6,6
a platform for running Postgres.
376
0:18:6,66 --> 0:18:10,14
And so we saw this need and we
saw early that we had to provide
377
0:18:10,14 --> 0:18:11,86
users a way to shard.
378
0:18:11,98 --> 0:18:15,04
And actually we were in a very
privileged position, I would say,
379
0:18:15,04 --> 0:18:18,66
because what we've done in StackGres
with single clusters is
380
0:18:18,66 --> 0:18:22,2
orchestrate and allow you to run
the whole stack of components
381
0:18:22,2 --> 0:18:25,46
that you need to run Postgres,
connection pooling, backups, monitoring,
382
0:18:25,6 --> 0:18:28,38
all that, all automated and all
packaged.
383
0:18:28,44 --> 0:18:33,28
So deploying it is extremely easy
while also following best practices
384
0:18:33,28 --> 0:18:35,78
that we have from our company's
operational experience.
385
0:18:36,48 --> 0:18:40,18
But then we saw that creating a
Citus cluster involved a lot
386
0:18:40,18 --> 0:18:43,86
of manual work because it's no
longer 1 cluster.
387
0:18:44,14 --> 0:18:47,06
Now, if you think about it, and
the model we follow in StackGres
388
0:18:47,12 --> 0:18:51,78
is that every component of the
Citus cluster, the workers, the
389
0:18:51,78 --> 0:18:56,14
coordinator, all of those are a
Postgres cluster in itself, typically
390
0:18:56,14 --> 0:18:59,02
with more than 1 instance also
for high availability purpose,
391
0:18:59,02 --> 0:18:59,52
right?
392
0:18:59,6 --> 0:19:2,54
And they need to be wired up precisely
with the same configuration
393
0:19:2,64 --> 0:19:5,74
parameters across many places or
otherwise it will not behave
394
0:19:5,74 --> 0:19:6,84
as a single cluster.
395
0:19:7,16 --> 0:19:10,82
So it's not only a lot of work
to wire up as let's say a 24,
396
0:19:10,86 --> 0:19:14,18
48 node the Citus cluster but also
to do it right.
397
0:19:14,66 --> 0:19:17,8
And With our automation and orchestration
that we built for a
398
0:19:17,8 --> 0:19:19,54
StackGres, we could do this work.
399
0:19:19,78 --> 0:19:23,2
And now the reality is that if
you use a StackGres and want to
400
0:19:23,2 --> 0:19:26,12
create a sharded cluster with Citus,
by the way, we support all
401
0:19:26,12 --> 0:19:29,94
the other sharding technologies
in StackGres, not only Citus,
402
0:19:30,18 --> 0:19:32,78
even though it's the most used
and the most feature complete,
403
0:19:32,78 --> 0:19:34,98
I would say solution for sharding
that we have.
404
0:19:35,34 --> 0:19:39,86
But with the 13 lines of YAML,
really high level, you can create
405
0:19:39,86 --> 0:19:42,84
a Citus cluster of 60 nodes
in a few minutes.
406
0:19:43,28 --> 0:19:46,06
You can even do from a web console,
Click, click, click, boom,
407
0:19:46,06 --> 0:19:47,62
and you get the sharded cluster.
408
0:19:47,78 --> 0:19:50,72
There's nothing else compared to
this other than the managed
409
0:19:50,72 --> 0:19:53,04
service and Microsoft offers on
Azure.
410
0:19:53,92 --> 0:19:56,82
And that's the gap we wanted to
cover because we saw demand and
411
0:19:56,82 --> 0:19:57,88
it's paying off.
412
0:19:58,02 --> 0:20:1,36
I would say a significant percentage
of our new customers and
413
0:20:1,36 --> 0:20:3,28
users come for sharding.
414
0:20:4,54 --> 0:20:8,0
It is very widespread and it's
a need that few people are focused
415
0:20:8,0 --> 0:20:11,6
on solving, especially using a
technology that is proven and
416
0:20:11,6 --> 0:20:13,52
well maintained as of today like
Citus.
417
0:20:13,94 --> 0:20:14,18
It's a
418
0:20:14,18 --> 0:20:17,32
Michael: good point about Citus
being on Microsoft's platform
419
0:20:17,32 --> 0:20:21,58
and it could because it's AGPL
it means other platforms are
420
0:20:21,58 --> 0:20:25,78
going to really struggle probably
just can't offer it right so
421
0:20:25,92 --> 0:20:30,06
actually it makes sense that the
if the only kind of 10 year
422
0:20:30,06 --> 0:20:32,74
old sharding actually it's maybe
it's not the only it because
423
0:20:32,74 --> 0:20:36,64
it wasn't SPQR quite old as well
as you say they feature complete
424
0:20:37,14 --> 0:20:40,44
sharding solution on Postgres if
that's 10 years old but you
425
0:20:40,44 --> 0:20:44,28
can't get it on any managed services
except for 1, actually,
426
0:20:44,54 --> 0:20:48,42
it makes sense that there's demand
for, okay, how do we self-manage
427
0:20:48,64 --> 0:20:49,12
this?
428
0:20:49,12 --> 0:20:52,1
Nik: We need to revisit the topic
of AGPL misconception.
429
0:20:52,58 --> 0:20:53,5
How to say?
430
0:20:54,02 --> 0:20:58,62
So AGPL allows you to run it on
your managed service, but you
431
0:20:58,62 --> 0:21:2,08
cannot modify it without not publishing
source code.
432
0:21:2,62 --> 0:21:4,94
And right, this is 1st thing.
433
0:21:4,94 --> 0:21:7,8
Let me explain my words and then
Álvaro you can correct me if
434
0:21:7,8 --> 0:21:8,2
I'm wrong.
435
0:21:8,2 --> 0:21:10,94
I'm just curious if I understand
it right.
436
0:21:11,14 --> 0:21:14,12
And then Álvaro is not doing a
managed service at all.
437
0:21:14,54 --> 0:21:19,3
StackGres is just like operator
plus of course there is enterprise
438
0:21:19,3 --> 0:21:21,1
edition with support and so on.
439
0:21:21,1 --> 0:21:24,64
This is like money coming from
support, but it can be installed
440
0:21:24,64 --> 0:21:25,74
anywhere, right?
441
0:21:25,9 --> 0:21:29,68
Like having AGPL components doesn't
hurt at all.
442
0:21:29,84 --> 0:21:34,7
And, And finally, I know StackGres
was on AGPL until lately, right?
443
0:21:34,84 --> 0:21:36,18
Can you elaborate this?
444
0:21:37,2 --> 0:21:38,16
On 1 bit.
445
0:21:38,44 --> 0:21:39,34
Yeah, am I right?
446
0:21:39,34 --> 0:21:41,08
It's not an AGPL anymore?
447
0:21:41,58 --> 0:21:44,24
Álvaro: It's very good insider
information, yes.
448
0:21:44,24 --> 0:21:45,68
So, 1st of all, AGPL.
449
0:21:46,06 --> 0:21:47,58
I'm not a lawyer, okay?
450
0:21:47,68 --> 0:21:51,42
But I love AGPL and GPL or in
general copyleft licenses.
451
0:21:52,04 --> 0:21:54,88
I believe people say they are viral,
that they restrict your
452
0:21:54,88 --> 0:21:55,28
freedom.
453
0:21:55,28 --> 0:21:56,84
I believe it's exactly the opposite.
454
0:21:56,84 --> 0:22:0,92
They provide forward freedom warranties,
meaning that whatever
455
0:22:0,92 --> 0:22:4,3
software is published under a copyleft
license, it is providing
456
0:22:4,3 --> 0:22:6,6
a warranty that will never be closed
source.
457
0:22:6,6 --> 0:22:9,26
It will always remain under such
a license.
458
0:22:9,4 --> 0:22:12,14
So it's a forward warranty for
users of Freedom.
459
0:22:12,4 --> 0:22:17,02
Whereas a license like the Apache
2 or BSD, they can be turned
460
0:22:17,02 --> 0:22:20,28
into proprietary software, and
that software may cease to exist
461
0:22:20,28 --> 0:22:21,62
as an open source software.
462
0:22:22,2 --> 0:22:23,44
That's why I like them.
463
0:22:23,44 --> 0:22:27,44
Now, the reality is that a lot
of people believe that they cannot
464
0:22:27,44 --> 0:22:30,4
be used, for example, as a managed
service, but you put it very
465
0:22:30,4 --> 0:22:30,9
well.
466
0:22:30,92 --> 0:22:34,84
You can perfectly well use an AGPL
software while offering a
467
0:22:34,84 --> 0:22:35,66
managed service.
468
0:22:35,74 --> 0:22:38,6
The only thing that the license
force you to do is that whomever
469
0:22:38,6 --> 0:22:42,1
interacts over the network with
your software need to provide
470
0:22:42,1 --> 0:22:45,72
the same warranties that you got
for that software, which means
471
0:22:45,72 --> 0:22:47,06
publishing the source code.
472
0:22:47,08 --> 0:22:50,86
So if you use Citus, and even with
Citus, it's arguable because
473
0:22:50,86 --> 0:22:52,62
you don't talk directly to Citus.
474
0:22:52,94 --> 0:22:56,32
But even if we do a little more
loose interpretation, or because
475
0:22:56,32 --> 0:23:0,04
I have Citus, then I'm bound by
these terms directly, it would
476
0:23:0,04 --> 0:23:3,48
suffice to publish the source code
of Citus, which is a link
477
0:23:3,66 --> 0:23:5,9
and a disclosure that you use Citus,
right?
478
0:23:6,14 --> 0:23:10,52
Then if you patch Citus and then
you improve Citus, even if you
479
0:23:10,52 --> 0:23:13,68
don't submit upstream, your users
have the right to get those
480
0:23:13,68 --> 0:23:16,38
benefits too, which is why I love
this kind of licenses.
481
0:23:17,4 --> 0:23:19,96
But some lawyers disagree with
me, probably they're right because
482
0:23:19,96 --> 0:23:21,3
they're lawyers, I'm not.
483
0:23:21,88 --> 0:23:26,5
And not many people want to use
AGPL on their managed services.
484
0:23:26,58 --> 0:23:30,36
But in my understanding, limited
understanding, I think that's
485
0:23:30,36 --> 0:23:31,9
not necessarily correct.
486
0:23:32,54 --> 0:23:35,58
Nik: Imagine if Postgres was on
AGPL version 3.
487
0:23:35,82 --> 0:23:39,38
RDS would need to publish all the
changes they made to implement
488
0:23:39,48 --> 0:23:41,5
rds_superuser and so on.
489
0:23:42,9 --> 0:23:44,36
It's simplified as well.
490
0:23:45,06 --> 0:23:48,18
Michael: But this is a really great
point because I think Postgres
491
0:23:48,18 --> 0:23:51,18
has benefited from being on a permissive
license in terms of
492
0:23:51,18 --> 0:23:52,54
how much usage it's got.
493
0:23:52,54 --> 0:23:55,32
And maybe we're seeing the exact
opposite with Citus.
494
0:23:55,32 --> 0:23:58,44
Maybe we're seeing that AGPL is
actually costing in terms
495
0:23:58,44 --> 0:23:59,16
Nik: of usage.
496
0:23:59,16 --> 0:23:59,94
Repelled usage.
497
0:24:0,12 --> 0:24:0,52
Repelled, yeah.
498
0:24:0,52 --> 0:24:3,68
Like Google, for example, have
internal, at least in the past,
499
0:24:3,68 --> 0:24:7,58
they had internal directive that
let's avoid the GPL version
500
0:24:7,58 --> 0:24:8,98
3 software completely.
501
0:24:9,52 --> 0:24:11,34
Álvaro: Yeah, it's a big if.
502
0:24:11,52 --> 0:24:12,74
We don't know the answer.
503
0:24:13,52 --> 0:24:17,96
Most likely, because The market
has this, what I believe it's
504
0:24:17,96 --> 0:24:22,86
a misconception, but this general
wisdom that AGPL or GPL licenses
505
0:24:23,04 --> 0:24:27,38
are this viral and so on, then
yes, it looks like copyleft software
506
0:24:27,8 --> 0:24:32,42
is perceived as unideal and permissive
licenses are winning.
507
0:24:33,06 --> 0:24:35,94
And certainly this is the reason
I don't want to avoid the question,
508
0:24:36,22 --> 0:24:41,82
yes, StackGres, and I can announce
this publicly, it's gonna be
509
0:24:41,82 --> 0:24:46,08
licensed under the Apache 2 license,
instead of AGPL, as much
510
0:24:46,08 --> 0:24:50,08
as I wish to do otherwise, but
the market and the users demand
511
0:24:50,08 --> 0:24:51,36
this, so we're happy to
512
0:24:51,36 --> 0:24:52,28
Nik: do that and we will
513
0:24:52,28 --> 0:24:55,02
Álvaro: do that in the next days
actually.
514
0:24:55,52 --> 0:24:59,34
Nik: While still Citus remains
on AGPL and it's just plugged
515
0:24:59,34 --> 0:25:0,12
in, right?
516
0:25:0,78 --> 0:25:4,28
Álvaro: Yep, we support, in StackGres
we support more than 200
517
0:25:4,28 --> 0:25:4,78
extensions.
518
0:25:4,78 --> 0:25:8,1
They come with a lot of licenses
and it's a user's responsibility
519
0:25:8,48 --> 0:25:12,16
to, to know the license they're,
they're dealing with and their
520
0:25:12,18 --> 0:25:12,6
wording.
521
0:25:12,6 --> 0:25:15,04
We just provide the software and
the platform.
522
0:25:15,16 --> 0:25:18,74
Nik: It's like a big Lego and some
pieces are different colors
523
0:25:18,74 --> 0:25:19,78
and so on, right?
524
0:25:20,32 --> 0:25:20,82
Álvaro: Yeah.
525
0:25:21,46 --> 0:25:25,58
And to complete the picture, I
think I want to go back a little
526
0:25:25,58 --> 0:25:27,42
bit to what you were mentioning,
Michael.
527
0:25:28,38 --> 0:25:32,04
That also relates maybe to 1 of
the reasons, or potentially,
528
0:25:32,04 --> 0:25:35,08
I don't know exactly the reasons,
but why I believe that maybe
529
0:25:35,08 --> 0:25:39,86
some companies abandoned Citus
at some point, which is that Citus,
530
0:25:40,02 --> 0:25:43,14
the workers, where you split the
data, you can, again, you can
531
0:25:43,14 --> 0:25:47,06
add as many as you need, but coordinator
is only 1.
532
0:25:47,68 --> 0:25:51,66
And if you saturate the coordinator,
then you're out of luck,
533
0:25:51,66 --> 0:25:51,98
right?
534
0:25:51,98 --> 0:25:55,92
You can use, as we said, you can
query the nodes directly, the
535
0:25:55,92 --> 0:26:0,6
workers directly, but that it's
complicated to operate and you
536
0:26:0,6 --> 0:26:2,78
still also will get as far as you
can.
537
0:26:3,42 --> 0:26:7,44
This is something we saw some time
ago and decided to do something
538
0:26:7,44 --> 0:26:8,18
about it.
539
0:26:8,32 --> 0:26:11,0
And in the most recent version
of StackGres, StackGres
540
0:26:11,0 --> 0:26:14,12
1.19, we have introduced something
called query routers.
541
0:26:15,06 --> 0:26:18,34
Query routers does nothing that
we have actually patched on Citus
542
0:26:18,34 --> 0:26:19,28
or anything like that.
543
0:26:19,28 --> 0:26:20,58
It's unmodified Citus.
544
0:26:20,58 --> 0:26:24,02
It's just an architectural pattern
that we identified that we
545
0:26:24,02 --> 0:26:27,94
could apply to Citus to allow to
have essentially multiple coordinators,
546
0:26:28,26 --> 0:26:29,16
put that simple.
547
0:26:30,06 --> 0:26:34,3
So a query router is like a coordinator,
so it can route queries.
548
0:26:34,54 --> 0:26:37,18
It doesn't have data, it just route
queries.
549
0:26:37,36 --> 0:26:39,82
The only thing it cannot do is
the DDL operations.
550
0:26:39,96 --> 0:26:42,28
The DDL operations go through the
coordinator.
551
0:26:42,9 --> 0:26:45,82
But now you can add as many query
routers as you want.
552
0:26:46,08 --> 0:26:50,32
And so the limitation that Citus
may have had until now with
553
0:26:50,32 --> 0:26:52,5
a single coordinator, it's gone.
554
0:26:52,8 --> 0:26:55,84
And now you can scale by adding
as many query routers as you
555
0:26:55,84 --> 0:27:0,52
want and scale, not only now the
worker side, the layer where
556
0:27:0,52 --> 0:27:4,58
it keeps the data, but also the
router layer to almost any size.
557
0:27:4,66 --> 0:27:7,6
I'm publishing some benchmarks
that we have been running in the
558
0:27:7,6 --> 0:27:8,9
last days about this.
559
0:27:9,64 --> 0:27:14,88
And ballpark numbers, 64 core workers
running simple queries
560
0:27:15,06 --> 0:27:19,44
indexed by primary key, but not
fully in memory, but with a high
561
0:27:19,44 --> 0:27:20,24
hit ratio.
562
0:27:20,68 --> 0:27:25,76
A single coordinator with 3 workers,
the coordinator being also
563
0:27:25,76 --> 0:27:30,64
64 cores, was maxing out at around
550, 000 transactions per
564
0:27:30,64 --> 0:27:33,9
2nd with a pretty good latency,
like under 1 millisecond.
565
0:27:35,08 --> 0:27:36,08
But that's it.
566
0:27:36,34 --> 0:27:39,78
More than 500000, a little bit
over 500000, and the coordinator
567
0:27:39,96 --> 0:27:40,66
is saturated.
568
0:27:41,18 --> 0:27:44,92
The reason why this number is because
these queries are quite
569
0:27:44,92 --> 0:27:48,42
fast to process, and the proportion
of overhead per query processing
570
0:27:48,44 --> 0:27:49,68
is maxed out, right?
571
0:27:49,74 --> 0:27:53,1
It's a lot of overhead per very
small query, per tiny query.
572
0:27:53,1 --> 0:27:55,76
Of course, if you hit all the queries
and they involve I/O.
573
0:27:55,76 --> 0:27:59,38
And reading work, you can scale
further before hitting the limit
574
0:27:59,38 --> 0:28:0,14
of the coordinator.
575
0:28:0,14 --> 0:28:3,54
So we designed this benchmark to
actually find the limit as soon
576
0:28:3,54 --> 0:28:6,1
as possible of the coordinator
at still very high rates, right?
577
0:28:6,1 --> 0:28:8,16
More than 500000 transactions per
2nd.
578
0:28:8,52 --> 0:28:10,28
So coordinator is maxed out.
579
0:28:10,28 --> 0:28:14,5
Then we add 2 coordinators with
the same 3 shards and then 3
580
0:28:14,5 --> 0:28:18,58
coordinators with the same 3 workers,
sorry, the data nodes.
581
0:28:18,7 --> 0:28:21,18
And the performance basically scaled
almost linear.
582
0:28:21,18 --> 0:28:26,02
So from 550, it went to 1, 050,
000 and then to 1, 500, 000.
583
0:28:26,66 --> 0:28:28,22
So pretty much linear growth.
584
0:28:28,66 --> 0:28:32,16
Michael: When you say 3 coordinators,
Do you mean 3 query routers
585
0:28:32,16 --> 0:28:32,86
in the new?
586
0:28:32,86 --> 0:28:33,34
Yeah.
587
0:28:33,34 --> 0:28:33,76
OK.
588
0:28:33,76 --> 0:28:34,26
OK.
589
0:28:34,28 --> 0:28:34,98
Álvaro: Yeah, sorry.
590
0:28:35,0 --> 0:28:37,86
I use the terms a little bit interchangeably,
because for me,
591
0:28:37,86 --> 0:28:40,46
they're conceptually the same,
even though they're not.
592
0:28:40,58 --> 0:28:41,34
You're right.
593
0:28:41,34 --> 0:28:42,74
It's query routers, yes.
594
0:28:42,74 --> 0:28:44,0
So 3 query routers.
595
0:28:44,2 --> 0:28:45,72
The coordinator remains there.
596
0:28:45,72 --> 0:28:50,56
And actually, I am now recommending
that if you use this functionality
597
0:28:50,78 --> 0:28:53,3
with the StackGres, with the query
routers, that you keep the
598
0:28:53,3 --> 0:28:56,64
coordinator small configuration,
small number of cores, just
599
0:28:56,64 --> 0:29:0,24
to run the DDL through there, and
do not use a coordinator for
600
0:29:0,24 --> 0:29:1,06
query routing.
601
0:29:1,06 --> 0:29:3,04
And then use a layer of query routers.
602
0:29:3,04 --> 0:29:6,68
You can start with 1 if you want
and then scale them up as your
603
0:29:6,68 --> 0:29:8,48
needs grow, but they're already
prepared.
604
0:29:8,48 --> 0:29:11,82
You already, they use a different
service in Kubernetes, so it's
605
0:29:11,82 --> 0:29:14,18
just a way to route the queries
and entry point.
606
0:29:14,34 --> 0:29:15,92
And then you already have that
established.
607
0:29:15,92 --> 0:29:19,6
Your application points to the
service that will balance across
608
0:29:19,6 --> 0:29:21,14
all the query routers you may have.
609
0:29:21,14 --> 0:29:24,08
And then it's just a matter of
changing 1 character in your YAML
610
0:29:24,08 --> 0:29:27,9
file, say, query routers 1, query
routers 2, and you would scale
611
0:29:28,02 --> 0:29:28,52
automatically.
612
0:29:29,48 --> 0:29:33,08
Michael: And You mentioned a while
back that StackGres supports
613
0:29:33,08 --> 0:29:34,46
multiple ways of sharding.
614
0:29:34,7 --> 0:29:38,08
Could I use query routers with
any of them or is this a Citus
615
0:29:38,08 --> 0:29:39,72
specific thing?
616
0:29:39,72 --> 0:29:40,96
Álvaro: It's a very good question.
617
0:29:40,96 --> 0:29:42,42
This is Citus specific.
618
0:29:42,84 --> 0:29:47,16
So if we look at the internals,
Again, this is not, we didn't
619
0:29:47,16 --> 0:29:48,94
touch Citus source code.
620
0:29:48,94 --> 0:29:51,3
We could have if we needed, but
it was not needed.
621
0:29:51,3 --> 0:29:53,1
The building blocks were surprisingly
there.
622
0:29:53,1 --> 0:29:55,82
Actually, what I'm surprised that
nobody saw this before.
623
0:29:56,2 --> 0:30:0,52
For us, it was crystal clear because
it all started with the
624
0:30:0,52 --> 0:30:4,76
concept that every worker can function
as a query router, as
625
0:30:4,76 --> 0:30:5,8
a coordinator, right?
626
0:30:5,8 --> 0:30:7,32
We just call them query routers.
627
0:30:7,54 --> 0:30:9,72
So that was already working there.
628
0:30:9,72 --> 0:30:13,14
The way this works is because they
implement the metadata that
629
0:30:13,14 --> 0:30:15,3
Citus replicates across all nodes.
630
0:30:15,36 --> 0:30:18,0
So you know whom to talk to, essentially.
631
0:30:19,14 --> 0:30:22,44
And there's another property, which
is what we combined into
632
0:30:22,44 --> 0:30:24,58
this pattern that we call the query
router.
633
0:30:24,72 --> 0:30:28,2
There is a property that is called
shouldhaveshards, and you
634
0:30:28,2 --> 0:30:29,56
can set it to false.
635
0:30:30,06 --> 0:30:33,64
Shards are the way that Citus internally
distributes the data,
636
0:30:33,64 --> 0:30:33,96
right?
637
0:30:33,96 --> 0:30:37,06
So also to clear, to make the context
clear for everybody.
638
0:30:37,36 --> 0:30:40,46
On Citus, there is workers, which
let's say they are the physical
639
0:30:40,58 --> 0:30:44,16
nodes that contain the data, physical,
virtual, you name it,
640
0:30:44,16 --> 0:30:46,12
the nodes that contain the data.
641
0:30:46,38 --> 0:30:48,84
But the data in reality is split
into shards.
642
0:30:49,06 --> 0:30:53,52
And then you map shards to workers,
at least 1, but could be
643
0:30:53,52 --> 0:30:54,44
more than 1.
644
0:30:54,64 --> 0:30:58,22
Sometimes some other people call
this virtual shards versus real
645
0:30:58,22 --> 0:30:59,1
shards, right?
646
0:30:59,34 --> 0:31:2,56
Citus terminology is workers,
like the physical thing, and shards
647
0:31:2,56 --> 0:31:3,96
like the virtual thing.
648
0:31:3,96 --> 0:31:7,62
And typically we recommend a higher
number of shards to worker
649
0:31:7,76 --> 0:31:10,96
because it helps you also scale
when you add more workers, you
650
0:31:10,96 --> 0:31:13,82
just move some of the shards to
other workers.
651
0:31:13,82 --> 0:31:15,86
You don't need to re-shard the
whole thing.
652
0:31:15,86 --> 0:31:16,24
Right.
653
0:31:16,24 --> 0:31:17,86
It's much, much better pattern.
654
0:31:17,86 --> 0:31:20,5
Actually, by the way, I'm diving
maybe a little bit too deep
655
0:31:20,5 --> 0:31:22,86
into that topic, but I use the
magic number 120.
656
0:31:23,36 --> 0:31:24,86
Most people use powers of 2.
657
0:31:24,86 --> 0:31:27,96
I think that's a mistake, because
then you can only grow by powers
658
0:31:27,96 --> 0:31:31,92
of 2, which by the growth of powers
of 2, can mean to grow a
659
0:31:31,92 --> 0:31:34,24
lot when maybe you don't need to
grow that much.
660
0:31:34,34 --> 0:31:37,88
Versus having 120 is a number that
has a lot of divisors, and
661
0:31:37,88 --> 0:31:41,2
so you can grow from 30 to 40,
for example.
662
0:31:41,2 --> 0:31:43,94
You don't need to go from 30 to
60, you need to double because
663
0:31:43,94 --> 0:31:46,7
they're still evenly divided across
all your workers.
664
0:31:47,18 --> 0:31:51,72
If you take a worker that can act
as a coordinator or a router,
665
0:31:51,78 --> 0:31:55,6
and you set this property
shouldhaveshards to false, then
666
0:31:55,6 --> 0:31:59,16
Citus will not place any shards
on it, it will be data-less.
667
0:31:59,68 --> 0:32:5,4
So essentially a data-less worker,
and that's a good router.
668
0:32:5,46 --> 0:32:6,88
So the pattern was just there.
669
0:32:6,88 --> 0:32:11,02
We just enabled it through architecture
and automation in StackGres.
670
0:32:11,82 --> 0:32:13,22
Michael: Simple, I like it.
671
0:32:13,66 --> 0:32:16,32
Álvaro: So of course, there's more
meat to it when you look into
672
0:32:16,32 --> 0:32:18,78
the internal details, but conceptually
this is it.
673
0:32:19,28 --> 0:32:19,78
Just
674
0:32:20,24 --> 0:32:22,4
Nik: don't write proxy, just use
Postgres.
675
0:32:25,64 --> 0:32:26,92
In yet another area.
676
0:32:27,8 --> 0:32:28,94
Yeah, it's interesting.
677
0:32:29,04 --> 0:32:34,16
Speaking of benchmarks, these numbers,
what concerns me that
678
0:32:34,16 --> 0:32:39,0
we talk about only read-only queries,
I would like to see the
679
0:32:39,96 --> 0:32:44,12
series 2 of benchmarks talking
about writes, because as we already
680
0:32:44,12 --> 0:32:50,72
discussed, the case you have in
hand is 2.2 petabytes.
681
0:32:52,42 --> 0:32:57,18
It's about writes, not reads, because
scaling reads is less interesting,
682
0:32:57,44 --> 0:32:58,72
less challenging, right?
683
0:32:59,12 --> 0:33:3,42
Álvaro: It is actually challenging
And again, I will not claim
684
0:33:3,42 --> 0:33:4,7
that Citus is perfect.
685
0:33:4,74 --> 0:33:7,36
And it is really good, but it's
not perfect.
686
0:33:7,36 --> 0:33:8,5
I guess nothing is perfect.
687
0:33:8,8 --> 0:33:10,14
And it requires tuning.
688
0:33:10,44 --> 0:33:12,62
I'm not gonna say this worked out
of the box.
689
0:33:12,62 --> 0:33:14,08
No, it requires tuning.
690
0:33:14,1 --> 0:33:18,06
And 1 of The hardest part of Citus
actually is connection tuning,
691
0:33:18,28 --> 0:33:22,12
because when you look at Citus,
at any distributed database for
692
0:33:22,12 --> 0:33:25,6
that matter, any sharded database,
it's no longer a single thing,
693
0:33:25,6 --> 0:33:26,1
right?
694
0:33:26,14 --> 0:33:29,94
Now you're dealing with connections
from the client to the coordinator
695
0:33:29,96 --> 0:33:33,58
or the query routers, and from
them to the workers.
696
0:33:34,06 --> 0:33:36,76
And sometimes these queries actually
have a big fan-out.
697
0:33:36,76 --> 0:33:41,3
For example, on this particular
case of the 2.2 petabyte database,
698
0:33:41,64 --> 0:33:46,42
I just cannot mention the customer
name, but it's a big telco.
699
0:33:46,96 --> 0:33:50,02
They are using COPY to ingest data.
700
0:33:50,02 --> 0:33:53,1
And that's amazing because it's
the most efficient way to ingest
701
0:33:53,1 --> 0:33:54,12
data into Postgres.
702
0:33:54,92 --> 0:33:59,64
And they're using 120 shards on
24 nodes, which divides evenly.
703
0:34:0,24 --> 0:34:7,06
Now, if you do a COPY of 50, 000
records, a single connection,
704
0:34:7,06 --> 0:34:10,04
so a single connection tries to
ingest 50, 000 records via a
705
0:34:10,04 --> 0:34:13,22
COPY, that COPY is going to create
24 sub copies.
706
0:34:14,16 --> 0:34:18,96
Because by law of probability,
all 24 shards are going to receive
707
0:34:18,96 --> 0:34:19,46
data.
708
0:34:19,64 --> 0:34:21,98
1/24th of the 50, 000 records.
709
0:34:22,38 --> 0:34:25,58
Because the way that data is sharded
is by taking a hash of the
710
0:34:25,58 --> 0:34:29,86
primary key, and that hash is a
random, let's say, it's randomly
711
0:34:29,86 --> 0:34:31,78
distributed, so it's going to touch
all the shards.
712
0:34:31,96 --> 0:34:36,14
So 1 connection from 1 client now
will create 24 connections.
713
0:34:36,38 --> 0:34:39,46
Citus implements this very well,
just do 24 sub copies.
714
0:34:39,52 --> 0:34:42,72
But you can imagine the potential
multiplication of connections
715
0:34:42,72 --> 0:34:44,7
that may happen here under high
concurrency.
716
0:34:45,48 --> 0:34:50,04
But by applying proper tuning and
doing proper connection management,
717
0:34:50,14 --> 0:34:51,42
this works very well.
718
0:34:51,5 --> 0:34:52,78
Nik: What is proper here?
719
0:34:53,68 --> 0:34:56,02
Álvaro: Oh, well, it's not an easy
exercise.
720
0:34:56,26 --> 0:35:0,06
The connections from the coordinator
or query routers to the
721
0:35:0,06 --> 0:35:2,98
worker nodes, they're bound by
several parameters.
722
0:35:3,42 --> 0:35:6,98
Essentially, Citus has some kind
of, let's say, connection pooling
723
0:35:7,04 --> 0:35:11,2
system to communicate between the
coordinator or coordinator
724
0:35:11,2 --> 0:35:12,02
to the workers.
725
0:35:12,4 --> 0:35:16,94
But it's not like PgBouncer, because
they open backends that
726
0:35:16,94 --> 0:35:19,78
are Citus is Postgres, for the
good and the bad.
727
0:35:19,84 --> 0:35:22,32
There's many good things and a
few bad things.
728
0:35:22,66 --> 0:35:26,84
And so if there's a client talking
to the coordinator, that creates
729
0:35:26,84 --> 0:35:28,52
a backend on the coordinator, right?
730
0:35:28,52 --> 0:35:30,14
And that backend needs to be open.
731
0:35:30,14 --> 0:35:34,24
Now, this backend may now own multiple
sockets to multiple connections
732
0:35:34,28 --> 0:35:35,04
to the workers.
733
0:35:35,06 --> 0:35:38,3
And there's several parameters
to tune whether you want a single
734
0:35:38,3 --> 0:35:41,32
connection per worker, whether
you want a number of connections
735
0:35:41,32 --> 0:35:41,88
per worker.
736
0:35:41,88 --> 0:35:45,84
You can cache connections already
open per worker, but then there's
737
0:35:45,84 --> 0:35:49,64
also like a global pool so that
you don't want to do more than
738
0:35:49,64 --> 0:35:52,76
X number of connections for a given
coordinator across all the
739
0:35:52,76 --> 0:35:53,26
workers.
740
0:35:53,56 --> 0:35:57,2
And so you need to really understand
that we all know that Postgres
741
0:35:57,66 --> 0:36:2,22
doesn't work well with a huge number
of connections because it
742
0:36:2,22 --> 0:36:3,5
creates a lot of backend.
743
0:36:3,9 --> 0:36:7,0
And so you need to understand the
fan-in rules, the fan-out rules,
744
0:36:7,08 --> 0:36:9,48
whether to put connection pool,
for example, in front of the
745
0:36:9,48 --> 0:36:11,5
coordinator or not, and so on.
746
0:36:11,5 --> 0:36:15,58
So the math ends up being non-trivial
sometimes, especially if
747
0:36:15,58 --> 0:36:17,32
you're talking about huge workloads.
748
0:36:17,44 --> 0:36:20,14
Like for small workloads, even
if you're starting with again,
749
0:36:20,14 --> 0:36:22,84
maybe hundreds of thousand transactions
per 2nd, maybe more or
750
0:36:22,84 --> 0:36:26,14
less trivial, like any server,
modern server will deal comfortably
751
0:36:26,14 --> 0:36:26,52
with that.
752
0:36:26,52 --> 0:36:29,84
But when you grow to these sizes,
you really need to size everything
753
0:36:29,92 --> 0:36:30,64
around this.
754
0:36:30,64 --> 0:36:34,64
But what's actually very fun of
this particular case is that
755
0:36:34,64 --> 0:36:37,64
as of today, on this particular
case, we'll need to blog about
756
0:36:37,64 --> 0:36:38,14
this.
757
0:36:38,24 --> 0:36:43,0
Right now, the database is perceived
as slow, but they're running
758
0:36:43,14 --> 0:36:45,96
some software that I think you're
familiar with called pg_ash,
759
0:36:47,08 --> 0:36:52,58
and it's reporting all the time,
waits are mostly ClientRead.
760
0:36:53,86 --> 0:36:57,2
They don't have an ingestion pipeline
fast enough to saturate
761
0:36:57,2 --> 0:36:57,88
the database.
762
0:36:58,26 --> 0:37:0,94
The database is faster than their
ingestion pipeline.
763
0:37:1,62 --> 0:37:5,14
Because again, COPY is really optimized
in Postgres and can
764
0:37:5,14 --> 0:37:6,98
ingest huge amounts of data.
765
0:37:7,18 --> 0:37:11,64
And we have here 24 shards, really
beefy servers that can ingest
766
0:37:11,64 --> 0:37:13,76
data at really, really high speeds.
767
0:37:13,94 --> 0:37:17,72
We've seen, don't remember the
numbers, but multiple gigabytes
768
0:37:17,72 --> 0:37:19,1
per 2nd being ingested.
769
0:37:19,34 --> 0:37:21,32
And the database is just fine.
770
0:37:21,94 --> 0:37:24,36
But the ingestion pipeline is not.
771
0:37:24,96 --> 0:37:25,82
Nik: That's interesting.
772
0:37:25,9 --> 0:37:29,8
So I see, I hear 3 different directions
here.
773
0:37:29,8 --> 0:37:33,72
1st of all, this benchmark in your
blog post, which is coming,
774
0:37:34,54 --> 0:37:37,44
I guess you will publish all the
details as usual, right?
775
0:37:38,2 --> 0:37:39,26
Álvaro: Yes, of course.
776
0:37:40,84 --> 0:37:44,2
Nik: This is our company's common
value, we share it a lot.
777
0:37:45,14 --> 0:37:47,84
Álvaro: Fully reproducible, StackGres
being open source, we'll
778
0:37:47,84 --> 0:37:49,14
open source all the code.
779
0:37:49,4 --> 0:37:52,76
Nik: It's Postgres's open source
and benchmarks should be open
780
0:37:52,76 --> 0:37:54,64
so anyone could repeat them.
781
0:37:55,16 --> 0:37:55,66
Yes.
782
0:37:55,9 --> 0:37:57,88
Yeah, so that's great.
783
0:37:58,58 --> 0:38:2,32
SELECT only because this is to
compare with different benchmark
784
0:38:2,32 --> 0:38:6,44
which is not so open, right, and
not about open source.
785
0:38:7,24 --> 0:38:8,04
That's great.
786
0:38:8,24 --> 0:38:11,76
And there, I guess, this is interesting
to continue exploring
787
0:38:12,04 --> 0:38:14,28
the limits.
788
0:38:14,44 --> 0:38:16,96
And I think maybe it will also...
789
0:38:17,1 --> 0:38:20,74
I expect more benchmarks to be
done because we have multiple
790
0:38:20,74 --> 0:38:23,85
systems competing in purely open
source areas.
791
0:38:23,85 --> 0:38:26,98
Some of them are AGPL, some of
them are Apache licenses, we mentioned
792
0:38:26,98 --> 0:38:27,48
already.
793
0:38:29,1 --> 0:38:31,74
It's interesting to start comparing
them.
794
0:38:31,88 --> 0:38:34,78
It's great that it's happening
and this is maybe the beginning
795
0:38:34,78 --> 0:38:36,24
of some comparison.
796
0:38:36,6 --> 0:38:37,94
It's time to start comparing.
797
0:38:38,74 --> 0:38:42,68
Right now your bet is fully on
Citus, like almost fully.
798
0:38:43,08 --> 0:38:46,66
I hear you that StackGres supports
other sharding as well.
799
0:38:46,86 --> 0:38:49,6
And I guess at some point you can
start supporting those new
800
0:38:49,6 --> 0:38:50,5
alternatives as well.
801
0:38:50,5 --> 0:38:51,04
Why not?
802
0:38:51,04 --> 0:38:53,8
It's just a Lego style thing, right?
803
0:38:53,8 --> 0:38:54,9
This is 1 direction.
804
0:38:56,82 --> 0:39:1,16
Read-only benchmarks, which I guess
it's not so exciting.
805
0:39:1,62 --> 0:39:4,94
2nd would be probably writing benchmarks
like COPY.
806
0:39:5,38 --> 0:39:9,3
When you say COPY, you mention
specific production case, but
807
0:39:9,3 --> 0:39:13,04
it would be great also to have
some benchmark showing, to start
808
0:39:13,04 --> 0:39:14,74
playing with and showing capabilities.
809
0:39:15,3 --> 0:39:17,38
Do you plan to have this benchmark?
810
0:39:18,26 --> 0:39:21,54
Álvaro: We could do that, but I
believe we will start by publishing
811
0:39:21,68 --> 0:39:25,58
some, the data that we can publish
about 1 particular case that
812
0:39:25,58 --> 0:39:28,66
we have already, like this 1, because
I think that already shows
813
0:39:28,66 --> 0:39:29,28
a lot.
814
0:39:30,06 --> 0:39:34,1
And so to be honest, we have
focused quite a bit on benchmarks.
815
0:39:34,6 --> 0:39:37,44
So let me be clear, we've done
2 different benchmarks.
816
0:39:37,44 --> 0:39:40,62
Both are going to be published
and will be fully reproducible
817
0:39:40,76 --> 0:39:41,96
back by code, right?
818
0:39:42,26 --> 0:39:44,18
And so full transparency on everything.
819
0:39:44,44 --> 0:39:48,04
The 1st 1 is a benchmark to demonstrate
the ceiling of the single
820
0:39:48,04 --> 0:39:51,34
coordinator and to see how we can
scale with query routers, right?
821
0:39:51,34 --> 0:39:54,52
Like I just mentioned the numbers,
it can go from 0.5, essentially
822
0:39:54,52 --> 0:39:58,24
500000 to 1.5 million, so almost
linear scalability by going
823
0:39:58,24 --> 0:40:1,02
from 1 coordinator to 3 query routers,
right?
824
0:40:1,46 --> 0:40:6,44
But then we saw actually that for
this particular workload, like
825
0:40:6,44 --> 0:40:10,96
very, very small primary key queries,
mostly in memory with a
826
0:40:10,96 --> 0:40:16,72
non-uniform distribution, that
you end up needing a similar number
827
0:40:16,72 --> 0:40:20,54
of cores on the routers layer than
on the shards or the, sorry,
828
0:40:20,54 --> 0:40:21,88
the workers layer, right?
829
0:40:22,12 --> 0:40:26,48
And so we did some tuning and around
this number that the ideal
830
0:40:26,48 --> 0:40:33,68
is to have 6 routers of 32 cores
each and 3 workers of 64 cores
831
0:40:33,68 --> 0:40:34,04
each.
832
0:40:34,04 --> 0:40:37,4
So that's the same number of cores
on both layers, but 6 routers
833
0:40:37,4 --> 0:40:38,36
and 3 workers.
834
0:40:38,52 --> 0:40:40,62
And we call this a unit of a scaling.
835
0:40:41,0 --> 0:40:44,84
That package, let's say it's well
tuned so that both layers are
836
0:40:44,96 --> 0:40:47,26
close to saturation but before
saturation.
837
0:40:47,26 --> 0:40:50,68
So they operate at the operational
regime and all math about
838
0:40:50,68 --> 0:40:53,54
connection, tuning connection,
pooling, and all that is well
839
0:40:53,54 --> 0:40:54,04
tuned.
840
0:40:54,34 --> 0:40:58,5
And that's basically delivering
1.5 million transactions per
841
0:40:58,5 --> 0:41:0,36
2nd, that unit of a scale.
842
0:41:0,72 --> 0:41:4,44
And then we decided to multiply
this unit of the scale by 2,
843
0:41:4,44 --> 0:41:5,9
by 4, and by 8.
844
0:41:6,34 --> 0:41:10,56
And this is a 2nd benchmark that
we'll be publishing that also
845
0:41:10,56 --> 0:41:13,78
compares to some data points of
another sharding solution, which
846
0:41:13,78 --> 0:41:16,02
has achieved those numbers and
more actually.
847
0:41:16,56 --> 0:41:20,24
And it also shows really good behavior,
like basically to share
848
0:41:20,24 --> 0:41:24,92
the numbers by multiplying this
by 8x, it passes over 10 million
849
0:41:24,92 --> 0:41:26,14
transactions per 2nd.
850
0:41:26,84 --> 0:41:33,74
With 1.something millisecond p50
latency and 6 milliseconds p99,
851
0:41:35,02 --> 0:41:37,32
at 10 million transactions per
2nd.
852
0:41:37,46 --> 0:41:41,62
To put this into perspective, who
needed to do 10 million transactions
853
0:41:41,68 --> 0:41:42,38
per 2nd?
854
0:41:42,5 --> 0:41:45,36
Probably just a few sites on earth
do more than that.
855
0:41:45,48 --> 0:41:46,88
Nik: So it's...
856
0:41:47,7 --> 0:41:48,48
Maybe more.
857
0:41:48,52 --> 0:41:50,66
We have gigantic workloads these
days.
858
0:41:50,74 --> 0:41:52,54
I think here I disagree.
859
0:41:52,96 --> 0:41:54,02
A lot of more.
860
0:41:54,38 --> 0:41:57,94
We see crazy startups which skyrocket
these days.
861
0:41:58,52 --> 0:42:0,04
And their users are agents.
862
0:42:0,62 --> 0:42:1,92
And they, like, just...
863
0:42:2,44 --> 0:42:4,94
Álvaro: Yeah, but most of the time
what I see is that agents
864
0:42:4,94 --> 0:42:5,88
are small databases.
865
0:42:5,9 --> 0:42:7,38
There are many of them, but a small...
866
0:42:7,38 --> 0:42:10,12
We're talking about single cluster,
right?
867
0:42:10,64 --> 0:42:15,92
But anyway, my point is that anyway,
reaching 10 million transactions
868
0:42:15,96 --> 0:42:19,82
per 2nd with very decent latency
numbers, right?
869
0:42:20,64 --> 0:42:22,46
It's an excellent result.
870
0:42:23,18 --> 0:42:26,56
It's home to most of the clusters
we've ever seen.
871
0:42:27,04 --> 0:42:29,68
And so it's a valid solution for
almost anyone.
872
0:42:29,68 --> 0:42:31,96
And That's actually my takeaway,
right?
873
0:42:31,96 --> 0:42:35,9
It can scale almost as much or
as much as you want with really
874
0:42:35,9 --> 0:42:37,62
decent numbers on this pure OLTP.
875
0:42:38,4 --> 0:42:40,6802
Nik: So I have 4 directions already.
876
0:42:40,6802 --> 0:42:42,1
So you will publish.
877
0:42:42,18 --> 0:42:43,6
I know you will publish this.
878
0:42:43,6 --> 0:42:45,54
You will probably will iterate
this.
879
0:42:45,82 --> 0:42:49,52
I guess this benchmark also not
cheap to run, right?
880
0:42:49,54 --> 0:42:51,8
You need to provision those resources.
881
0:42:52,1 --> 0:42:52,9
Álvaro: Yeah, exactly.
882
0:42:52,9 --> 0:42:54,18
But this is a good point.
883
0:42:54,44 --> 0:42:58,58
We're still a bootstrapped startup,
and we strive to be, we have
884
0:42:58,58 --> 0:43:0,4
to be, very cost efficient.
885
0:43:1,26 --> 0:43:5,02
So running this benchmark, I don't
have the final data, but it's
886
0:43:5,02 --> 0:43:5,78
very approximate.
887
0:43:6,34 --> 0:43:12,08
We've spent around $5, 000 to $6,
000 on running this benchmark,
888
0:43:12,16 --> 0:43:14,58
scaling to 10 million transactions
per 2nd.
889
0:43:14,58 --> 0:43:16,42
Nik: It's AWS or where?
890
0:43:16,84 --> 0:43:17,34
Álvaro: Yep.
891
0:43:17,78 --> 0:43:21,2
Nik: And you just use Spot because
it's not reliable in this
892
0:43:21,2 --> 0:43:21,7
case.
893
0:43:22,36 --> 0:43:26,12
Álvaro: Yeah, it's hard because,
and we had some problems when
894
0:43:26,12 --> 0:43:28,3
you deal with such a large number
of instances.
895
0:43:28,4 --> 0:43:33,22
So on the 10 million transactions
per 2nd, this involved 72 instances
896
0:43:33,28 --> 0:43:36,98
plus the drivers for the workload,
so like around 90 instances.
897
0:43:37,54 --> 0:43:39,74
Just 1 instance behaving wrongly?
898
0:43:40,92 --> 0:43:43,3
Nik: The probability of failure
is too high.
899
0:43:44,34 --> 0:43:44,84
Álvaro: Yeah.
900
0:43:45,12 --> 0:43:48,74
So, yeah, Spot would have most
likely had mean that the test
901
0:43:48,74 --> 0:43:50,72
needs to be repeated over and over
again.
902
0:43:51,76 --> 0:43:52,4
Nik: I see.
903
0:43:52,54 --> 0:43:52,96
That's great.
904
0:43:52,96 --> 0:43:55,82
And that's great cost-wise as well
for 10 million.
905
0:43:55,96 --> 0:43:59,36
So once again, 1 direction is SELECT
only primary key lookup,
906
0:43:59,38 --> 0:44:0,92
trivial workload, I would say.
907
0:44:1,56 --> 0:44:6,94
And I think this is very adjacent
to our benchmarks we did multiple
908
0:44:6,94 --> 0:44:11,04
times for latency overhead is very
in this area, like very fast
909
0:44:11,04 --> 0:44:11,54
queries.
910
0:44:11,6 --> 0:44:15,36
2nd, I still think at some point
there should be some benchmarks
911
0:44:15,58 --> 0:44:18,74
for INSERTs or write workloads,
right?
912
0:44:19,64 --> 0:44:23,64
3rd thing is that you mentioned
many connections is a trouble.
913
0:44:24,52 --> 0:44:30,36
I think, I know Andres Freund did
a lot of great work in Postgres
914
0:44:30,36 --> 0:44:35,08
13 and especially I think 14 to
improve the situation when you
915
0:44:35,08 --> 0:44:36,6
have thousands of connections.
916
0:44:36,98 --> 0:44:42,16
Because before that, thousands
of idle connections affected performance
917
0:44:42,18 --> 0:44:42,9
so badly.
918
0:44:42,94 --> 0:44:44,08
Now it's not so bad.
919
0:44:44,38 --> 0:44:45,64
What does it mean still?
920
0:44:46,5 --> 0:44:48,02
We don't have threads, right?
921
0:44:48,58 --> 0:44:52,62
What's happening if you have 10,
000 connections on single Postgres?
922
0:44:52,72 --> 0:44:54,94
I think this research is very interesting.
923
0:44:54,96 --> 0:44:58,2
I'm talking for myself because
we have a lot of interesting directions.
924
0:44:58,22 --> 0:44:59,44
We have benchmarks.
925
0:44:59,96 --> 0:45:2,82
Our focus mostly on single node
clusters.
926
0:45:3,54 --> 0:45:7,68
Our company is dealing with some
self-implementation, like application
927
0:45:7,74 --> 0:45:8,54
side sharding.
928
0:45:9,0 --> 0:45:11,98
But it's so interesting, I see
some interesting directions here
929
0:45:11,98 --> 0:45:13,22
to benchmark further.
930
0:45:13,82 --> 0:45:19,08
And finally, I'm thinking if you
stop at 10 million, I'm going
931
0:45:19,08 --> 0:45:25,4
to explore the possibility to have
10 million TPS primary key
932
0:45:25,4 --> 0:45:27,42
lookups for a single node.
933
0:45:28,08 --> 0:45:32,06
This probably will encourage you
to go to 50 million or something,
934
0:45:32,28 --> 0:45:34,04
spend a few thousand dollars.
935
0:45:34,14 --> 0:45:38,86
Because if, yeah, I will put it
to midterm to-do list.
936
0:45:39,0 --> 0:45:41,44
If we achieve that, you will be
forced
937
0:45:41,44 --> 0:45:42,02
Álvaro: to cover.
938
0:45:42,02 --> 0:45:44,7
But you know, if you want to do
that, you know, you only have
939
0:45:44,7 --> 0:45:45,56
2 weeks, right?
940
0:45:46,4 --> 0:45:48,48
Nik: No, no, no, no, no.
941
0:45:48,48 --> 0:45:50,02
Go ahead, publish your results.
942
0:45:50,02 --> 0:45:53,2
This is an interesting, friendly
competition and I think some
943
0:45:53,3 --> 0:45:56,3
systems will need it to join.
944
0:45:56,32 --> 0:45:58,58
What I like about this is that
data is shared.
945
0:45:58,58 --> 0:46:0,22
It's about open source components.
946
0:46:0,46 --> 0:46:1,72
Everything is open source.
947
0:46:1,74 --> 0:46:2,78
Data is open source.
948
0:46:2,78 --> 0:46:3,74
Benchmarks are open source.
949
0:46:3,74 --> 0:46:4,72
They can be revisited.
950
0:46:4,74 --> 0:46:5,42
This is great.
951
0:46:5,42 --> 0:46:7,86
This is very good style of competition.
952
0:46:8,3 --> 0:46:10,06
Everyone benefits from, right?
953
0:46:11,42 --> 0:46:14,34
So I'm not going to chase you right
now.
954
0:46:14,34 --> 0:46:16,24
I'm just saying maybe later.
955
0:46:16,78 --> 0:46:22,06
Álvaro: No, I'm very open if anyone
wants to sponsor workloads
956
0:46:22,16 --> 0:46:26,12
to be tested on Citus and see how
Citus can scale to from 10
957
0:46:26,12 --> 0:46:30,42
to 50 million transactions per
2nd or to read-write workloads We
958
0:46:30,42 --> 0:46:32,72
would be more than happy to help
in that regards.
959
0:46:33,48 --> 0:46:34,12
Michael: So yeah.
960
0:46:34,7 --> 0:46:38,24
I think it's, I think what, from
1 of the charts that you sent
961
0:46:38,24 --> 0:46:43,54
over, it's quite an interesting
chart of how the throughput changes
962
0:46:43,58 --> 0:46:44,44
as you scale up.
963
0:46:44,44 --> 0:46:46,88
So I think if you, if you, you
mentioned these units of work,
964
0:46:46,88 --> 0:46:52,06
if, if you add a 2nd unit of work
and the TPS exactly doubles
965
0:46:52,06 --> 0:46:56,5
if it's linear like as you add
more units it's very easy to extrapolate
966
0:46:56,98 --> 0:47:0,84
that of course it can get to 50
I just need 5 times more but
967
0:47:0,84 --> 0:47:5,22
if it isn't if it isn't quite efficient
if you lose 10 each time
968
0:47:5,22 --> 0:47:10,66
if it if it feels like at doubling
the instance size only 1.5x
969
0:47:11,32 --> 0:47:15,26
is the TPS, then it's less believable
that it can scale infinitely.
970
0:47:15,72 --> 0:47:17,78
What kind of numbers were you seeing
there?
971
0:47:19,3 --> 0:47:20,86
Álvaro: So that's a really good
question.
972
0:47:20,86 --> 0:47:25,78
Scaling is slightly sublinear,
with most systems are.
973
0:47:26,2 --> 0:47:32,72
And on our scaling from 1 to 8,
it was 1.94 every time we doubled.
974
0:47:34,06 --> 0:47:34,56
Nik: Okay.
975
0:47:34,64 --> 0:47:35,14
Yeah.
976
0:47:35,5 --> 0:47:39,32
There is a big challenge of scaling
client in this case.
977
0:47:39,38 --> 0:47:39,88
Yes.
978
0:47:40,52 --> 0:47:41,34
That's it.
979
0:47:41,55 --> 0:47:42,44
I mean, I
980
0:47:42,44 --> 0:47:43,86
Álvaro: can go very deep into this
topic.
981
0:47:43,86 --> 0:47:45,14
I guess it's not the moment.
982
0:47:45,48 --> 0:47:49,4
But it's a really, really interesting
capacity planning we had
983
0:47:49,4 --> 0:47:50,58
to do to achieve these numbers.
984
0:47:50,58 --> 0:47:54,52
Again, it sounds like, oh, just
put more nodes and yeah, you
985
0:47:54,52 --> 0:47:56,9
scale 10 million, you just need
to hit enter, right?
986
0:47:56,98 --> 0:47:58,28
No, it wasn't that case.
987
0:47:58,32 --> 0:47:59,84
We had to do a lot of tuning.
988
0:48:0,06 --> 0:48:2,86
The number of connections is actually
mind-blowing because even
989
0:48:2,86 --> 0:48:6,24
though these queries are really
fast and take hundreds of microseconds
990
0:48:6,42 --> 0:48:10,38
to run in reality across all the
fleet, across the CPU aggregated
991
0:48:10,38 --> 0:48:11,64
all the fleet, right?
992
0:48:11,72 --> 0:48:14,02
The number of connections that
you need, the sessions that you
993
0:48:14,02 --> 0:48:16,02
need to drive this is very, very
large.
994
0:48:16,02 --> 0:48:18,46
It's many, many thousands of sessions.
995
0:48:18,82 --> 0:48:23,48
And then they multiply because
if each query router is, I think
996
0:48:23,48 --> 0:48:27,1
where our optimal regime was around
400 connections per query
997
0:48:27,1 --> 0:48:30,7
router on 32 cores, which by the
way, if you look at the numbers,
998
0:48:30,7 --> 0:48:34,64
it's always like the usual rule
for tuning Postgres, right?
999
0:48:34,64 --> 0:48:38,22
The maximum number of connections
is a multiple of the number
1000
0:48:38,22 --> 0:48:38,6
of cores.
1001
0:48:38,6 --> 0:48:41,4
In this case, the multiple is around
7 because they're very,
1002
0:48:41,4 --> 0:48:44,58
very CPU intensive queries because
they almost not touch disk,
1003
0:48:44,58 --> 0:48:45,02
right?
1004
0:48:45,02 --> 0:48:46,88
So the multiple is a bit high.
1005
0:48:46,88 --> 0:48:50,32
Anyway, do you think of 400 connections
coming to each query
1006
0:48:50,32 --> 0:48:54,52
router of which there are 48 on
the 10 million transactions per
1007
0:48:54,52 --> 0:48:56,1
2nd run, right?
1008
0:48:56,5 --> 0:49:2,06
And now each 1 is gonna have outgoing
the same number of connections
1009
0:49:2,58 --> 0:49:5,22
in the same number of sessions,
because they're all, but they
1010
0:49:5,22 --> 0:49:6,52
all go to all shards.
1011
0:49:7,2 --> 0:49:12,18
So the incoming connections at
the shards are the whole fleet.
1012
0:49:12,18 --> 0:49:15,36
So you're seeing the 400 multiplied
by 48.
1013
0:49:15,86 --> 0:49:18,3
Those are incoming connections
to every worker.
1014
0:49:19,7 --> 0:49:23,44
So to actually deal with this,
we had to introduce connection
1015
0:49:23,54 --> 0:49:25,5
pooling before the workers.
1016
0:49:26,32 --> 0:49:30,06
And it's also been an exercise
because StackGres by default uses
1017
0:49:30,06 --> 0:49:33,28
PgBouncer, and PgBouncer is single-threaded.
1018
0:49:33,92 --> 0:49:36,24
So were we saturating 1 thread?
1019
0:49:36,54 --> 0:49:37,04
Definitely.
1020
0:49:37,72 --> 0:49:40,9
So actually this benchmark required,
if I'm not mistaken, 15
1021
0:49:40,9 --> 0:49:45,84
PgBouncers per worker, running
with a SO_REUSEPORT functionality
1022
0:49:46,42 --> 0:49:49,94
and all handled by StackGres as
a custom container.
1023
0:49:50,42 --> 0:49:53,94
So it worked beautifully well,
but you need to do all the dimensioning
1024
0:49:54,14 --> 0:49:56,78
correctly, otherwise it will not
achieve these numbers either.
1025
0:49:57,44 --> 0:50:0,56
Michael: Did you say 15 PgBouncers
per Worker?
1026
0:50:1,34 --> 0:50:1,84
Álvaro: Yep.
1027
0:50:1,86 --> 0:50:2,72
Michael: Oh my goodness.
1028
0:50:4,3 --> 0:50:4,8
Okay.
1029
0:50:5,14 --> 0:50:5,64
Okay.
1030
0:50:5,74 --> 0:50:6,24
Álvaro: Good.
1031
0:50:6,34 --> 0:50:9,68
Because we're dealing with, I think,
19, 000 incoming connections.
1032
0:50:11,76 --> 0:50:15,2
And they're all sending queries
at a really high pace.
1033
0:50:15,86 --> 0:50:18,98
Nik: What would it take to reach
100 million TPS?
1034
0:50:20,32 --> 0:50:22,22
Álvaro: Well, there's 2 things.
1035
0:50:22,48 --> 0:50:24,24
1 is just to keep multiplying.
1036
0:50:24,28 --> 0:50:28,58
I think this unit of scale is very
well tuned as of now, and
1037
0:50:28,58 --> 0:50:29,94
it just keep multiplying.
1038
0:50:30,14 --> 0:50:33,4
As of now, it should be an easy
exercise, other than money-wise.
1039
0:50:34,42 --> 0:50:38,48
But I believe the scalability is
gonna start going, is gonna
1040
0:50:38,48 --> 0:50:40,82
start going less and less over
time.
1041
0:50:41,12 --> 0:50:44,0
And this is, again, because of
how Citus works.
1042
0:50:44,48 --> 0:50:45,94
As I said, it's not perfect.
1043
0:50:46,28 --> 0:50:48,48
You may need to question whether
you need more than 10 million
1044
0:50:48,48 --> 0:50:49,66
transactions per 2nd.
1045
0:50:50,28 --> 0:50:53,62
But it's, so Citus is gonna create
a backend per session, per
1046
0:50:53,62 --> 0:50:54,12
router.
1047
0:50:54,86 --> 0:50:57,78
And this, again, we were just discussing
this, is gonna create,
1048
0:50:57,8 --> 0:51:1,36
that multiplied by the number of
routers will become the number
1049
0:51:1,36 --> 0:51:3,42
of incoming connections to each
worker node.
1050
0:51:3,42 --> 0:51:8,26
So instead of having 19, 200 connections,
if we do this 10 times,
1051
0:51:8,42 --> 0:51:12,56
that will require almost 200, 000
connections incoming per node.
1052
0:51:12,56 --> 0:51:13,94
Not all of them are active.
1053
0:51:14,24 --> 0:51:17,08
Are active actually are that number
divided by the number of
1054
0:51:17,08 --> 0:51:21,6
workers, but still a very high
number, a very, very, very fun
1055
0:51:21,6 --> 0:51:24,34
difference between the fan-in,
the number of connections incoming
1056
0:51:24,34 --> 0:51:27,52
to the workers through the poolers
versus the number of outgoing
1057
0:51:27,52 --> 0:51:30,96
connections, the pool connections
to Postgres, which needs to
1058
0:51:30,96 --> 0:51:35,82
remain constant at around 600 in
the case of the workers or 800,
1059
0:51:35,94 --> 0:51:38,34
something like this, because they
have double the CPUs, right?
1060
0:51:38,94 --> 0:51:43,7
And this difference between this
fan-in and fan-out creates inefficiency
1061
0:51:44,28 --> 0:51:45,74
in the workers themselves.
1062
0:51:45,86 --> 0:51:47,74
So it will start saturating earlier.
1063
0:51:47,9 --> 0:51:51,02
They also create inefficiencies
in the routers because they need
1064
0:51:51,02 --> 0:51:52,22
to talk to more nodes.
1065
0:51:53,08 --> 0:51:55,46
And so, will it reach there?
1066
0:51:55,68 --> 0:51:56,6
I don't know.
1067
0:51:56,72 --> 0:52:0,04
If someone wants to pour the money
into it, we can test it out.
1068
0:52:0,04 --> 0:52:4,64
I believe it may reach to a very
high number, maybe with a slightly
1069
0:52:4,64 --> 0:52:9,3
less efficiency, but it will reach
maybe 50 million, maybe more
1070
0:52:9,76 --> 0:52:12,04
with more effort, but it will do
so.
1071
0:52:12,18 --> 0:52:16,64
It's still compared to other sharding
solutions that can reach
1072
0:52:16,64 --> 0:52:17,38
those numbers.
1073
0:52:17,56 --> 0:52:21,24
I don't think all can, but out
of those that we have tested,
1074
0:52:21,76 --> 0:52:25,64
it doing so, this 10 million transactions
per 2nd, is using 0.5
1075
0:52:25,64 --> 0:52:29,68
of the silicon compared to other
sharding solutions.
1076
0:52:30,06 --> 0:52:32,96
So even though the scaling may
be slightly less linear because
1077
0:52:32,96 --> 0:52:36,82
of this Citus connection model,
which is Postgres connection
1078
0:52:36,82 --> 0:52:41,26
model, it needed 0.5 of the cores
to achieve those results.
1079
0:52:41,82 --> 0:52:44,52
So it's still very impressive,
right?
1080
0:52:44,76 --> 0:52:47,64
Now, Do I see a future different
than this?
1081
0:52:47,64 --> 0:52:48,3
I do.
1082
0:52:48,54 --> 0:52:53,46
I believe there may be an option
where the transport that Citus
1083
0:52:53,46 --> 0:52:57,98
uses could benefit from a multiplexing
transport, something like
1084
0:52:57,98 --> 0:53:2,06
gRPC, which other of the modern
solutions are adopting.
1085
0:53:3,34 --> 0:53:7,86
And that significantly reduces
the fan-in and fan-out connection
1086
0:53:7,9 --> 0:53:9,78
problem from the routers to the
workers.
1087
0:53:10,84 --> 0:53:15,38
So if I were to complete my initial
thesis that Citus is a great
1088
0:53:15,38 --> 0:53:20,14
solution as of today that is very
mature, well proven, featureful,
1089
0:53:20,42 --> 0:53:24,86
and so on, and that new solutions
may take time to mature.
1090
0:53:25,88 --> 0:53:30,68
Maybe, maybe, and it's a big if,
by just improving a bit Citus
1091
0:53:30,68 --> 0:53:34,6
architecture, it could take sharding
even to the next level if
1092
0:53:34,6 --> 0:53:38,3
needed, given that already it reaches
levels that are more than
1093
0:53:38,3 --> 0:53:41,76
enough for mostly everybody, but
even for those who may need
1094
0:53:41,82 --> 0:53:46,12
an even bigger scale, then maybe
It's just a matter of improving
1095
0:53:46,12 --> 0:53:46,62
Citus.
1096
0:53:47,54 --> 0:53:50,64
Nik: Yeah, I have only 1 question
to ask in the end.
1097
0:53:50,64 --> 0:53:55,78
I know you're comparing to other
benchmarks, and I'm just curious
1098
0:53:55,92 --> 0:53:59,48
in terms of, I saw some pictures
you shared with us before this
1099
0:53:59,48 --> 0:54:2,76
recording that was involving money,
like some dozens of thousands
1100
0:54:2,76 --> 0:54:4,7
of dollars to have this cluster.
1101
0:54:5,02 --> 0:54:9,4
I'm very curious, do you compare
with humans involved with support
1102
0:54:9,44 --> 0:54:14,36
or just pure open source and plus
cloud costs and that's it?
1103
0:54:15,78 --> 0:54:20,0
Álvaro: Yeah, so I did 2 comparisons
and they will be clearly
1104
0:54:20,14 --> 0:54:23,04
identified in the blog post that
will announce these benchmarks.
1105
0:54:23,8 --> 0:54:28,3
And so there's the pure infrastructure
cost, which is silicon,
1106
0:54:28,58 --> 0:54:32,06
like number of CPUs, where essentially
it's like around 55%.
1107
0:54:33,48 --> 0:54:36,18
So almost 0.5 the cost in terms
of CPU.
1108
0:54:36,68 --> 0:54:41,42
But then we can compute the total
cost to be very clear.
1109
0:54:41,42 --> 0:54:43,92
And there's nothing here to hide.
1110
0:54:44,06 --> 0:54:47,88
I'm comparing right now with a
benchmark that was run and published
1111
0:54:47,88 --> 0:54:48,54
by Neki.
1112
0:54:49,34 --> 0:54:52,86
And Neki is a managed service,
and they publish their list price.
1113
0:54:52,86 --> 0:54:55,36
And their list price is not the
instance cost.
1114
0:54:55,52 --> 0:54:59,16
It is the instance cost plus their
managing fee or their margin.
1115
0:55:0,04 --> 0:55:4,02
And when comparing with that, the
cost is not 0.5.
1116
0:55:4,3 --> 0:55:8,7
It's depending on what you consider
is between 0.333 or 0.2.
1117
0:55:8,94 --> 0:55:12,5
Actually, a well-dimensioned system
should be just 5 times cheaper.
1118
0:55:12,5 --> 0:55:16,08
Basically, 0.333 is if you compare
on-demand cost, which nobody
1119
0:55:16,08 --> 0:55:20,3
should run like a production cluster
sharded on on-demand cost.
1120
0:55:20,42 --> 0:55:24,44
And if you use a basic saving plans,
then you'll be 5 times cheaper.
1121
0:55:24,56 --> 0:55:25,06
Yeah.
1122
0:55:25,08 --> 0:55:28,16
And this is obviously a managed
service to a software comparison.
1123
0:55:28,58 --> 0:55:32,6
You may discuss that there may
be humans involved in the total
1124
0:55:32,66 --> 0:55:34,3
TCO calculation.
1125
0:55:35,02 --> 0:55:39,56
But my take is that a managed service
and software that does
1126
0:55:39,56 --> 0:55:42,42
a managed service for you is not
so much different.
1127
0:55:42,44 --> 0:55:46,22
Actually, I call StackGres a managed
software, a managed database
1128
0:55:46,22 --> 0:55:49,92
software, because when you say
a managed service, there's no
1129
0:55:49,92 --> 0:55:51,4
human typing, right?
1130
0:55:51,96 --> 0:55:55,44
There may be, if there's a support
ticket or something happens,
1131
0:55:55,44 --> 0:55:57,44
but the operation is just software.
1132
0:55:57,44 --> 0:56:0,36
So it's essentially the same for
me.
1133
0:56:0,8 --> 0:56:8,94
Nik: I know some humans can send
kill -9 in very big companies offering
1134
0:56:8,94 --> 0:56:10,66
managed services sometimes.
1135
0:56:11,0 --> 0:56:13,12
That kind of support is not welcome.
1136
0:56:13,94 --> 0:56:15,04
Not welcome, right?
1137
0:56:15,04 --> 0:56:16,9
kill -9 from support.
1138
0:56:18,28 --> 0:56:21,82
Michael: I also think these machines
are so big, like we're talking
1139
0:56:21,82 --> 0:56:25,36
about sharding and we're talking
about multiple huge machines
1140
0:56:25,46 --> 0:56:26,82
really, like they're expensive.
1141
0:56:27,34 --> 0:56:28,78
Often you don't pay this.
1142
0:56:28,78 --> 0:56:30,26
If you're actually a company,
1143
0:56:30,4 --> 0:56:30,9
Nik: saving
1144
0:56:31,1 --> 0:56:34,24
Michael: those machines, you would,
you would have like savings
1145
0:56:34,24 --> 0:56:37,64
plans either through PlanetScale
and or bring your own cloud
1146
0:56:37,64 --> 0:56:38,2
type thing.
1147
0:56:38,2 --> 0:56:41,32
Nik: So you're comparing not apples
to apples, but the numbers
1148
0:56:41,32 --> 0:56:46,2
you provided on your side, it could
be less with savings plan
1149
0:56:46,26 --> 0:56:50,24
if you pay in advance some number
of years to AWS.
1150
0:56:50,66 --> 0:56:53,76
On the other hand, you need probably
to roughly multiply it by
1151
0:56:53,76 --> 0:56:59,04
2, thinking about humans and the
support needed to surround this
1152
0:56:59,04 --> 0:57:0,04
solution, right?
1153
0:57:0,54 --> 0:57:1,22
Very roughly.
1154
0:57:2,78 --> 0:57:4,24
Álvaro: Yeah, so that's what I
said.
1155
0:57:4,24 --> 0:57:6,82
And so 1st of all, that's why I
want to give both numbers.
1156
0:57:6,82 --> 0:57:10,12
So you can 1st of all, compare
from the pure Silicon perspective
1157
0:57:10,12 --> 0:57:13,6
to pure architecture perspective,
and then from the sale price.
1158
0:57:13,94 --> 0:57:18,42
But as I said, For me, a managed
software, it's just software.
1159
0:57:18,58 --> 0:57:20,52
A managed service is just software,
right?
1160
0:57:20,86 --> 0:57:23,72
And if the software automates and
runs everything for you, which
1161
0:57:23,72 --> 0:57:25,58
is the case in both cases, right?
1162
0:57:25,58 --> 0:57:30,18
Because all managed services, there's
no humans doing operations,
1163
0:57:30,36 --> 0:57:31,4
It's only software.
1164
0:57:31,4 --> 0:57:35,1
So humans only intervene in the
case of issues.
1165
0:57:35,9 --> 0:57:37,36
And in those cases, yes.
1166
0:57:37,36 --> 0:57:41,48
And you can get like a, we sell
enterprise, the StackGres Enterprise,
1167
0:57:41,48 --> 0:57:42,6
which is exactly that.
1168
0:57:42,6 --> 0:57:44,32
Support for those cases, right?
1169
0:57:45,1 --> 0:57:47,56
And that's something optional,
but that's something you would
1170
0:57:47,56 --> 0:57:49,02
add on top of that.
1171
0:57:49,02 --> 0:57:51,6
And then the calculation will not
be 5 times cheaper, it will
1172
0:57:51,6 --> 0:57:52,08
be
1173
0:57:52,08 --> 0:57:52,66
Nik: maybe, I
1174
0:57:52,66 --> 0:57:54,86
Álvaro: don't have the numbers
exactly, 3 times cheaper, 4 times
1175
0:57:54,86 --> 0:57:57,08
cheaper, but still obviously much
cheaper.
1176
0:57:57,44 --> 0:57:59,94
Nik: Yeah, thank you for this clarification,
that's interesting.
1177
0:58:0,48 --> 0:58:3,48
Álvaro: And sorry, also to add,
and that's because talking about
1178
0:58:3,48 --> 0:58:6,66
cloud, but if you talk about on-prem,
which is definitely in
1179
0:58:6,66 --> 0:58:10,06
the space where we also, the StackGres
works anywhere on-prem,
1180
0:58:10,32 --> 0:58:14,08
then the math is even much more
into your favor.
1181
0:58:14,84 --> 0:58:15,53
Nik: Right, if you own hardware.
1182
0:58:15,53 --> 0:58:15,72
If you own
1183
0:58:15,72 --> 0:58:18,92
Álvaro: hardware and data centers,
1184
0:58:19,28 --> 0:58:23,3
Nik: then it's open source, you
can troubleshoot with your engineering
1185
0:58:23,36 --> 0:58:24,34
team if needed.
1186
0:58:24,34 --> 0:58:24,68
That's
1187
0:58:24,68 --> 0:58:25,18
Álvaro: great.
1188
0:58:25,765 --> 0:58:26,265
Thank
1189
0:58:26,85 --> 0:58:29,44
Nik: you so much.
1190
0:58:29,44 --> 0:58:30,12
Very interesting.
1191
0:58:30,18 --> 0:58:34,68
And also I see several paths for
future work.
1192
0:58:35,06 --> 0:58:38,46
Maybe we should have another episode
at some point in the future
1193
0:58:38,76 --> 0:58:41,52
to revisit the topics in some directions.
1194
0:58:41,6 --> 0:58:42,26
Álvaro: Any time.
1195
0:58:42,44 --> 0:58:44,98
I would be more than happy to come
back.
1196
0:58:45,66 --> 0:58:46,4
Nik: Thank you so much.
1197
0:58:46,4 --> 0:58:47,26
It was interesting.
1198
0:58:47,72 --> 0:58:49,02
Very interesting to me.
1199
0:58:49,6 --> 0:58:51,04
Álvaro: Thank you for inviting
me.
