1
0:0:0,06 --> 0:0:2,06
Nik: Hello, hello, this is Postgre.FM
2
0:0:2,08 --> 0:0:5,8199997
My name is Nik, PostgresAI, and
as usual with me, Michael,
3
0:0:6,66 --> 0:0:7,16
pgMustard.
4
0:0:7,44 --> 0:0:8,18
Hi, Michael.
5
0:0:8,74 --> 0:0:9,5199995
Michael C.: Hi, Nik.
6
0:0:10,08 --> 0:0:13,5
Nik: And we have a very, very interesting
guest today, Michael Malis,
7
0:0:13,5 --> 0:0:18,7
who created pgrust, which has already,
I think, 5,000 stars
8
0:0:18,7 --> 0:0:22,38
on GitHub and a lot of noise around,
like a lot of buzz.
9
0:0:22,7 --> 0:0:24,439999
Hi, Michael, thank you for coming.
10
0:0:24,96 --> 0:0:25,76
Michael M.: Yeah, of course.
11
0:0:25,76 --> 0:0:27,04
Thank you for having me.
12
0:0:28,08 --> 0:0:32,56
Nik: So, of course, I think the
1st question should be how it
13
0:0:32,56 --> 0:0:33,78
all started, why?
14
0:0:33,84 --> 0:0:35,22
Tell us the story, please.
15
0:0:36,98 --> 0:0:40,52
Michael M.: So Jason and I were
looking for projects to work
16
0:0:40,52 --> 0:0:45,56
on, and we wanted to do something
that we knew super well.
17
0:0:45,86 --> 0:0:50,38
And initially, what we were focused
on was reliability and how
18
0:0:50,38 --> 0:0:53,14
can we help people make their websites
more reliable.
19
0:0:54,14 --> 0:0:56,7
And we were working with a bunch
of people.
20
0:0:57,1 --> 0:1:1,3
And the common pattern that emerged
was that A lot of the problems
21
0:1:1,36 --> 0:1:5,16
that caused reliability issues,
a lot of them stemmed from their
22
0:1:5,16 --> 0:1:8,22
database and how they were using
their database, whether it was
23
0:1:8,22 --> 0:1:10,54
like Postgres or Redis or some
other system.
24
0:1:11,12 --> 0:1:14,54
And so we started looking at what
are ways that we can solve
25
0:1:14,54 --> 0:1:15,26
this problem.
26
0:1:15,94 --> 0:1:19,64
And we were throwing around a couple
of things, and we ended
27
0:1:19,64 --> 0:1:23,66
up figuring out that AI is actually
pretty good at rewriting
28
0:1:23,66 --> 0:1:24,78
software at this point.
29
0:1:24,86 --> 0:1:27,1
This is back in April.
30
0:1:27,88 --> 0:1:31,84
And so we're like, hey, why don't
we actually try fixing these
31
0:1:31,84 --> 0:1:35,28
problems at the source and actually
trying to modify Postgres
32
0:1:35,98 --> 0:1:39,56
and fix a lot of the things that
cause people to have different
33
0:1:39,56 --> 0:1:43,2
reliability issues, whether it's
connections or it's like certain
34
0:1:43,2 --> 0:1:46,5
queries just take a really long
time or like 1 long-running query
35
0:1:46,5 --> 0:1:49,62
can take down your database or
vacuums as we're all familiar
36
0:1:49,64 --> 0:1:50,14
with.
37
0:1:50,28 --> 0:1:55,76
And so the idea behind pgrust,
the name's a bit misleading because
38
0:1:55,76 --> 0:1:58,14
it's not actually really about
Rust.
39
0:1:58,14 --> 0:2:0,52
I think that's actually the least
interesting part about it.
40
0:2:0,52 --> 0:2:5,7
It's actually more about how do
we leverage AI to re-architect
41
0:2:6,16 --> 0:2:10,14
Postgres and just build a much
better database.
42
0:2:11,38 --> 0:2:11,88
Nik: Okay.
43
0:2:12,12 --> 0:2:14,08
And when you say better, what do
you mean?
44
0:2:14,9 --> 0:2:15,4
Michael M.: Yeah.
45
0:2:15,66 --> 0:2:21,44
So there's a combination of like
different challenges that like
46
0:2:21,44 --> 0:2:23,94
I've just seen people repeatedly
have with Postgres and I actually
47
0:2:23,94 --> 0:2:27,78
wrote a blog post called it like
the 4 horsemen about a lot of
48
0:2:27,78 --> 0:2:29,24
the challenges that people have.
49
0:2:29,32 --> 0:2:32,64
And some of the most common ones
are like, you misconfigure your
50
0:2:32,64 --> 0:2:35,54
connection limit and like all of
a sudden like nothing can connect
51
0:2:35,54 --> 0:2:36,32
to your database.
52
0:2:36,74 --> 0:2:38,64
You have like JSON support.
53
0:2:38,64 --> 0:2:42,08
Tons and tons of people use JSON
but Postgres doesn't have statistics
54
0:2:42,54 --> 0:2:44,02
for how to actually query JSON.
55
0:2:44,02 --> 0:2:46,32
So when you try to do it, you're
going to get really bad query
56
0:2:46,32 --> 0:2:48,46
plans and everything's just going
to be really slow.
57
0:2:48,68 --> 0:2:52,66
There's like the wraparound vacuum
and you have more than several
58
0:2:52,66 --> 0:2:55,84
billion transactions and like all
of a sudden if your vacuum
59
0:2:55,84 --> 0:2:58,74
can't keep up your database falls
over and there's just like
60
0:2:58,74 --> 0:3:3,92
all these problems that have been
around for such a long time.
61
0:3:3,92 --> 0:3:7,94
And like Postgres is a great product
and like it is a really
62
0:3:7,94 --> 0:3:12,98
great system, but the way the Postgres
core team approaches things
63
0:3:12,98 --> 0:3:16,44
is they approach it in terms of
stability And how do we keep
64
0:3:16,44 --> 0:3:17,56
what we have today?
65
0:3:17,56 --> 0:3:19,14
Like, how do we keep it working?
66
0:3:19,2 --> 0:3:21,26
And how do we make sure we don't
break it?
67
0:3:21,42 --> 0:3:24,44
Because like, there's like billions
of Postgres instances out
68
0:3:24,44 --> 0:3:24,84
there.
69
0:3:24,84 --> 0:3:28,64
And so number 1 priority is just
how do we not break the existing
70
0:3:28,64 --> 0:3:29,14
stuff?
71
0:3:29,54 --> 0:3:32,98
Versus how do we actually fix the
things that are not working.
72
0:3:33,34 --> 0:3:36,84
And so pgrust, because it's a
new project, we can take a little
73
0:3:36,84 --> 0:3:40,64
bit of a different approach of
let's try to actually fix the
74
0:3:40,64 --> 0:3:42,08
things that aren't working.
75
0:3:42,44 --> 0:3:45,54
And maybe some of the existing
stuff, it's not going to be quite
76
0:3:45,54 --> 0:3:48,16
as reliable, at least upfront,
as Postgres is.
77
0:3:48,16 --> 0:3:52,86
But at least we'll be able to fix
some of the long-standing architectural
78
0:3:52,96 --> 0:3:54,82
issues that Postgres has.
79
0:3:56,48 --> 0:3:57,6
Nik: Yeah, it makes sense.
80
0:3:57,84 --> 0:3:58,94
How did you do it?
81
0:3:58,94 --> 0:3:59,76
It's a lot of...
82
0:3:59,76 --> 0:4:1,5
Obviously, it's with Claude, right?
83
0:4:1,72 --> 0:4:4,74
Michael M.: Yeah, it took a couple
of attempts to actually figure
84
0:4:4,74 --> 0:4:6,52
out what is the best way to do
this.
85
0:4:6,82 --> 0:4:12,72
And what ended up working really
well was with Opus, we were
86
0:4:12,72 --> 0:4:19,06
able to take each file of Postgres
and basically transpile it
87
0:4:19,06 --> 0:4:19,74
to Rust.
88
0:4:19,82 --> 0:4:22,12
And so if you look at our code
and look at the Postgres code
89
0:4:22,12 --> 0:4:24,06
side by side, it actually looks
very similar.
90
0:4:24,16 --> 0:4:27,24
Where like the functions are the
same, some of the details may
91
0:4:27,24 --> 0:4:30,24
be a little bit different, but
overall it's actually really close
92
0:4:30,24 --> 0:4:32,5
to just a straight rewrite of Postgres.
93
0:4:32,8 --> 0:4:35,42
And so we did this across all the
files.
94
0:4:35,9 --> 0:4:40,28
There were some things that had
to be changed to actually work
95
0:4:40,28 --> 0:4:40,92
in Rust.
96
0:4:40,92 --> 0:4:44,28
1 of the big ones was how memory
management is done, because
97
0:4:44,28 --> 0:4:47,38
Rust is very particular about how
you allocate memory.
98
0:4:47,86 --> 0:4:51,04
But this approach of going through
all the files, rewriting them,
99
0:4:51,04 --> 0:4:54,44
we had a bunch of Claude agents
like going at this in parallel.
100
0:4:54,72 --> 0:4:57,24
There were like some conflicts
between different files, but we
101
0:4:57,24 --> 0:5:0,62
were able to resolve those and
then get pgrust to actually work.
102
0:5:0,62 --> 0:5:4,08
And then from there, we focused
on the Postgres test suite, getting
103
0:5:4,08 --> 0:5:5,14
all that to pass.
104
0:5:6,04 --> 0:5:10,78
And we have something that actually
looks a lot like Postgres,
105
0:5:11,04 --> 0:5:12,08
but is in Rust.
106
0:5:12,8 --> 0:5:15,6
Michael C.: I have potentially
a minor question on the source
107
0:5:15,6 --> 0:5:16,1
code.
108
0:5:16,3 --> 0:5:18,7
Really interesting that you went
file by file and that worked
109
0:5:18,7 --> 0:5:19,2
well.
110
0:5:19,36 --> 0:5:22,74
1 of my favorite things in the
source code is all the is the
111
0:5:22,74 --> 0:5:27,34
comments and they describe in quite
a lot of detail how things
112
0:5:27,34 --> 0:5:32,62
work and I wonder if you've noticed
how it rewrote those.
113
0:5:32,78 --> 0:5:37,92
I can imagine the code being somewhat
reliable in terms of it
114
0:5:37,92 --> 0:5:41,8
being translated, but I can imagine
the comments actually being
115
0:5:41,8 --> 0:5:43,76
harder in some way to be trustworthy.
116
0:5:44,18 --> 0:5:45,74
What have you found on that front?
117
0:5:47,08 --> 0:5:49,9
Michael M.: Yeah, for the comments,
like Claude has been like,
118
0:5:49,9 --> 0:5:52,36
Claude is very verbose when it
comes to comments.
119
0:5:53,26 --> 0:5:56,48
And like, every file have a whole
like big comment about like,
120
0:5:56,48 --> 0:5:59,84
what's in the file that I've actually
had to give it feedback,
121
0:5:59,84 --> 0:6:2,7
like I had to give it feedback
to comment less.
122
0:6:2,72 --> 0:6:5,84
And I think there's a linter that says
files should not have more
123
0:6:5,84 --> 0:6:7,86
than 5% of the line shouldn't be
comments.
124
0:6:9,96 --> 0:6:13,7
I haven't paid quite close attention
to the actual content of
125
0:6:13,7 --> 0:6:18,28
the comments, but I do know Claude
is making a lot of comments
126
0:6:18,28 --> 0:6:19,2
in the new code.
127
0:6:19,64 --> 0:6:21,3
Nik: It does, yeah.
128
0:6:21,58 --> 0:6:24,78
So I remember, now it's version
0.2, right?
129
0:6:24,96 --> 0:6:25,46
Michael M.: Yep.
130
0:6:25,58 --> 0:6:29,72
Nik: Yeah, and I remember version
0.1, I think there was a claim
131
0:6:29,72 --> 0:6:36,0
that all tests pass, but when I
wrote some silly stuff, like
132
0:6:36,14 --> 0:6:39,86
obviously not SQL syntax, it accepted
it without any errors.
133
0:6:40,02 --> 0:6:40,94
Now it's not so.
134
0:6:40,94 --> 0:6:46,94
In 0.2 it works fine, so it looks
like Postgres syntax is implemented
135
0:6:47,12 --> 0:6:47,88
very well.
136
0:6:48,06 --> 0:6:52,86
Was it about some lack of tests
in Postgres tests for checking
137
0:6:52,86 --> 0:6:53,46
negative stuff?
138
0:6:53,46 --> 0:6:54,3
I don't know.
139
0:6:55,08 --> 0:6:58,42
Or it just wasn't complete implementation
before?
140
0:6:58,44 --> 0:7:1,56
What changed between 0.1 and 0.2?
141
0:7:2,58 --> 0:7:7,86
Michael M.: Yeah, so pgrust 0.1 was
largely rewritten by Opus.
142
0:7:8,44 --> 0:7:13,34
And Opus I found to be a fine model,
but it's a bit difficult
143
0:7:13,34 --> 0:7:13,94
to work with.
144
0:7:13,94 --> 0:7:16,86
They'll mislead you a bit about
how much work is actually completed,
145
0:7:16,86 --> 0:7:19,28
we would ask it to rewrite this
file and it would rewrite 0.5
146
0:7:19,28 --> 0:7:20,8
of the file and not all of it.
147
0:7:20,8 --> 0:7:24,96
And so we were working with this
underlying unreliable model
148
0:7:24,96 --> 0:7:28,84
and all models are unreliable to
various degrees, but with enough
149
0:7:28,84 --> 0:7:33,66
quality controls and layering on
top of it, we're able to get
150
0:7:34,3 --> 0:7:38,52
all of Postgres ported, and we're
able to get the Postgres regression
151
0:7:38,52 --> 0:7:41,6
suite to pass, but that's actually
a really low bar.
152
0:7:42,1 --> 0:7:46,44
The Postgres regression suite on
Postgres only has about 0.667
153
0:7:46,44 --> 0:7:47,22
code coverage.
154
0:7:47,68 --> 0:7:52,02
And so you can pass the regression
suite, but there's still 0.333
155
0:7:52,02 --> 0:7:54,02
of Postgres that isn't even being
tested.
156
0:7:54,52 --> 0:7:58,46
And on top of that, the regression
suites are largely more like
157
0:7:58,46 --> 0:7:59,62
functionality tests.
158
0:7:59,68 --> 0:8:2,72
That every feature in Postgres,
there is a regression suite for
159
0:8:2,72 --> 0:8:5,86
it that's like, there's 1 for hash
joins, there's 1 for lateral
160
0:8:5,86 --> 0:8:10,88
joins, and there's 50,000 of these
that basically just go through
161
0:8:10,88 --> 0:8:13,48
every single Postgres feature and
make sure it works to some
162
0:8:13,48 --> 0:8:13,98
degree.
163
0:8:14,14 --> 0:8:18,28
But they actually just do a couple
tests for each feature and
164
0:8:18,28 --> 0:8:19,34
then move on.
165
0:8:19,34 --> 0:8:22,28
And so the tests are more about
just making sure this feature
166
0:8:22,28 --> 0:8:27,34
is there and it exists and works
well, but doesn't actually really
167
0:8:27,34 --> 0:8:31,5
like you have a perfect implementation
of this feature.
168
0:8:32,46 --> 0:8:36,82
Nik: So you, there is an idea to
improve regression test suite,
169
0:8:36,82 --> 0:8:37,32
right?
170
0:8:37,82 --> 0:8:41,82
And this is already, this should
go to upstream, maybe, no?
171
0:8:42,72 --> 0:8:43,22
Michael M.: Yeah.
172
0:8:43,44 --> 0:8:47,3
So what we've been doing, the difference
between 0.1 and 0.2
173
0:8:47,3 --> 0:8:50,28
is 0.2 was run by Fable, which
we have a more reliable model
174
0:8:50,28 --> 0:8:54,6
now that's, is more consistent
about, okay, you ask it to port
175
0:8:54,6 --> 0:8:56,78
a file, it'll actually rewrite
the file completely.
176
0:8:56,92 --> 0:9:0,06
And so we do end up with something
that is just baseline, more
177
0:9:0,06 --> 0:9:1,02
similar to Postgres.
178
0:9:1,5 --> 0:9:5,5
And then before we released 0.2,
we started doing a little, a
179
0:9:5,5 --> 0:9:8,1
tiny bit of differential testing
and fuzz testing, and I can
180
0:9:8,1 --> 0:9:9,5
talk a lot more about that.
181
0:9:9,52 --> 0:9:14,18
But for 0.3, we're actually like
going to have a lot more of
182
0:9:14,18 --> 0:9:14,68
that.
183
0:9:15,42 --> 0:9:19,58
What we've thought of doing is
we could add more regression tests,
184
0:9:20,08 --> 0:9:24,24
but that, again, every single regression
test is only testing
185
0:9:24,24 --> 0:9:26,6
a small part of the functionality.
186
0:9:27,1 --> 0:9:30,48
And what we really want is not
just, hey, I can choose 100 tests
187
0:9:30,48 --> 0:9:31,8
and they'll all pass.
188
0:9:31,96 --> 0:9:35,24
I want to know is this thing actually
identical to Postgres?
189
0:9:35,84 --> 0:9:39,06
And some of the things we've been
doing, and I was very surprised
190
0:9:39,06 --> 0:9:40,52
by this, that it's even worked.
191
0:9:40,52 --> 0:9:41,74
I would have never thought this.
192
0:9:41,84 --> 0:9:46,22
We found this library called Kani,
which can take Rust code and
193
0:9:46,22 --> 0:9:50,2
C code and do formal verification
over the code.
194
0:9:50,66 --> 0:9:51,76
Nik: Without any execution?
195
0:9:52,2 --> 0:9:53,86
Michael M.: So it does symbolic
execution.
196
0:9:54,4 --> 0:9:57,74
So it'll run your code, but it'll
run it in a special way that
197
0:9:57,74 --> 0:10:0,3
it is more just picking out that,
hey, this is an if statement
198
0:10:0,3 --> 0:10:2,98
that is checking this condition,
as opposed to actually having
199
0:10:2,98 --> 0:10:4,34
variables go through it.
200
0:10:4,34 --> 0:10:6,98
And then I'll be able to convert
that into a format that you
201
0:10:6,98 --> 0:10:10,08
can then do formal verification
over that has the Rust, I'll
202
0:10:10,08 --> 0:10:12,92
have an expression that represents
all the Rust, and I'll have
203
0:10:12,92 --> 0:10:14,34
the same for the C code.
204
0:10:14,44 --> 0:10:17,54
And then we can do across all inputs,
are these 2 things gonna
205
0:10:17,54 --> 0:10:19,52
produce the exact same output?
206
0:10:19,9 --> 0:10:25,3
And so Postgres has 3000 different
user facing functions, everything
207
0:10:25,32 --> 0:10:28,7
from substring to regular expression
matching to there's like
208
0:10:28,7 --> 0:10:32,9
the, like you can calculate the
gamma function, there's a function
209
0:10:32,9 --> 0:10:33,58
for that.
210
0:10:33,68 --> 0:10:36,94
And so for about 1000 of these
functions, we were actually
211
0:10:36,94 --> 0:10:40,72
able to do actual formal verification
that the Rust code and
212
0:10:40,72 --> 0:10:41,86
C code are the same.
213
0:10:42,28 --> 0:10:45,98
But then beyond that, what we've
started, like formal verification
214
0:10:46,06 --> 0:10:47,12
doesn't work in all cases.
215
0:10:47,12 --> 0:10:48,3
It's actually narrow.
216
0:10:48,52 --> 0:10:51,94
And so what we've been doing for
the rest of the code base has
217
0:10:51,94 --> 0:10:55,2
been, we'll take the Rust code
and the C code and put them side
218
0:10:55,2 --> 0:10:59,74
by side, and then have a fuzzer,
like, code coverage guided fuzzer,
219
0:11:0,18 --> 0:11:4,04
generate millions of inputs across
the 2 and make sure that they
220
0:11:4,2 --> 0:11:5,74
behave exactly the same.
221
0:11:6,4 --> 0:11:10,52
And in the process of doing this,
not only did we like find over
222
0:11:10,52 --> 0:11:14,44
100 bugs in like pgrust, but we
actually found at this point
223
0:11:14,44 --> 0:11:17,34
a little bit over 20 bugs in Postgres
itself.
224
0:11:17,52 --> 0:11:17,78
Nik: I see.
225
0:11:17,78 --> 0:11:18,58
That's interesting.
226
0:11:18,74 --> 0:11:19,98
Have you reported them?
227
0:11:20,38 --> 0:11:20,74
Michael M.: Yeah.
228
0:11:20,74 --> 0:11:24,52
If you look, I think like end of
July, like 1st week of August
229
0:11:24,52 --> 0:11:29,28
on the like pgsql or like pgsql
bugs mailing list, like you'll
230
0:11:29,28 --> 0:11:31,98
see a lot of the form submissions
are actually from me.
231
0:11:32,24 --> 0:11:32,8
Nik: That's cool.
232
0:11:32,8 --> 0:11:33,44
I missed it.
233
0:11:33,44 --> 0:11:34,18
That's cool.
234
0:11:34,44 --> 0:11:35,52
Congrats, actually.
235
0:11:35,54 --> 0:11:36,1
That's good.
236
0:11:36,1 --> 0:11:36,6
Yeah.
237
0:11:37,12 --> 0:11:38,2
Michael M.: None of the bugs.
238
0:11:38,52 --> 0:11:43,2
I found 3 serious bugs, but all
of them had been found already
239
0:11:43,2 --> 0:11:43,82
by other people.
240
0:11:43,82 --> 0:11:48,66
I'd been testing on 18.3, 18.4,
and in 19 people had found them.
241
0:11:48,84 --> 0:11:53,16
There were a lot of not serious
bugs that people had not found.
242
0:11:53,76 --> 0:11:57,68
1 of my favorites is Postgres has
a quadtree implementation,
243
0:11:59,34 --> 0:12:2,32
and it will use floating points
to represent the positions in
244
0:12:2,32 --> 0:12:2,94
the quadtree.
245
0:12:3,48 --> 0:12:6,94
And there was a bug where, because
of floating point rounding
246
0:12:7,06 --> 0:12:10,38
and the arithmetic that Postgres
was doing, it was possible for
247
0:12:10,38 --> 0:12:13,94
a point to not be in any of the
4 quadrants.
248
0:12:14,44 --> 0:12:17,68
That the comparison would say,
this isn't to the left, this isn't
249
0:12:17,68 --> 0:12:19,5
to the right, and this isn't in
the center.
250
0:12:20,22 --> 0:12:22,22
And we're definitely doing like
very aggressive.
251
0:12:23,56 --> 0:12:25,86
Nik: This is in the SP-GiST, right?
252
0:12:26,18 --> 0:12:29,08
Michael M.: I think it was in,
I don't know exactly.
253
0:12:30,06 --> 0:12:37,12
I want to say it was in GiST, yeah,
but basically the amount
254
0:12:37,12 --> 0:12:39,72
of testing we're doing is so thorough
that we're actually finding
255
0:12:39,72 --> 0:12:43,66
these extremely hard to find bugs
and we're finding quite a few
256
0:12:43,66 --> 0:12:45,0
bugs in Postgres itself.
257
0:12:45,24 --> 0:12:48,18
Nik: This is very important because
it means that with AI you
258
0:12:48,18 --> 0:12:52,42
can, you basically have some mechanism
which is finding bugs
259
0:12:52,42 --> 0:12:52,8
Michael M.: all the
260
0:12:52,8 --> 0:12:53,58
Nik: time, right?
261
0:12:53,8 --> 0:12:57,54
And it can be maybe like, what
could it mean?
262
0:12:57,74 --> 0:13:2,6
Probably there's Postgres Buildfarm
for testing, which is used for
263
0:13:2,6 --> 0:13:3,3
all releases.
264
0:13:4,3 --> 0:13:8,68
With this work, it maybe should
be extended to involve more AI
265
0:13:8,68 --> 0:13:11,14
and find more bugs at scale, right?
266
0:13:11,2 --> 0:13:13,08
Michael M.: Yeah, there's a couple
things I think are really
267
0:13:13,08 --> 0:13:13,58
interesting.
268
0:13:14,06 --> 0:13:18,02
1 is that with Mythos, this whole
thing about Mythos can pose
269
0:13:18,2 --> 0:13:19,5
a security risk.
270
0:13:19,82 --> 0:13:22,36
The interesting thing about the
models is like, the security,
271
0:13:22,36 --> 0:13:26,0
the way that they pose a security
risk isn't that like, the model
272
0:13:26,0 --> 0:13:29,34
looks at your code and can find
a bug in the code.
273
0:13:29,44 --> 0:13:33,34
It's actually that the model can
build tools that can then find
274
0:13:33,34 --> 0:13:34,34
bugs in your code.
275
0:13:34,4 --> 0:13:36,98
Or like, I've been using Fable
and a lot of times it'll fall
276
0:13:36,98 --> 0:13:41,76
back to Opus 5 because it's, oh,
you found a memory bug, that
277
0:13:42,1 --> 0:13:43,5
seems like a security issue.
278
0:13:43,7 --> 0:13:46,8
Nik: I think, just From my experience,
I also found a couple
279
0:13:46,8 --> 0:13:47,94
of bugs with AI.
280
0:13:49,12 --> 0:13:50,78
A few were security-related.
281
0:13:51,34 --> 0:13:55,28
1 was officially registered and
was already patched.
282
0:13:56,04 --> 0:13:58,4
That 1 was just looking at the
code.
283
0:13:59,28 --> 0:14:1,8
I didn't use Mythos, even not Fable.
284
0:14:1,8 --> 0:14:5,78
It was like Opus 4.5 back then.
285
0:14:6,04 --> 0:14:8,76
And I just was looking at the code,
it was very old code, and
286
0:14:8,76 --> 0:14:12,34
just looking at it, it found SQL
injection, which is there already
287
0:14:12,34 --> 0:14:13,3
18 years.
288
0:14:13,9 --> 0:14:19,5
So in some contrib module, which
is not used directly, so nobody
289
0:14:19,5 --> 0:14:22,78
cared, but it was actually serious
because it's used as an example.
290
0:14:23,56 --> 0:14:28,02
So anyway, it was interesting that
just looking at code, it also
291
0:14:28,02 --> 0:14:28,66
can find bugs.
292
0:14:28,66 --> 0:14:31,7
But you're right, building tools
is even more powerful because
293
0:14:32,18 --> 0:14:36,26
just looking at the code, you can
miss complex relationship between
294
0:14:36,26 --> 0:14:37,78
various code pieces, right?
295
0:14:38,0 --> 0:14:38,4
Michael M.: Yeah.
296
0:14:38,4 --> 0:14:41,26
So with the models, like what you
can do is you can basically
297
0:14:41,26 --> 0:14:44,48
point them to be like, hey, I have
100 functions in this code
298
0:14:44,48 --> 0:14:48,0
base, write really thorough tests
for all of these different
299
0:14:48,28 --> 0:14:49,4
pieces of code.
300
0:14:49,7 --> 0:14:52,7
And so the way I like to think
about it with these models is
301
0:14:52,7 --> 0:14:56,88
you can, if you can get them to
do 1 task well, you now have
302
0:14:56,88 --> 0:15:0,56
a repeatable way to do that 1 task
100 times.
303
0:15:0,72 --> 0:15:3,24
And so if you can get a model to
be able to test 1 function,
304
0:15:3,24 --> 0:15:5,84
there isn't a reason you can't
get it to test 100 or 1000 or
305
0:15:5,84 --> 0:15:6,72
10000 functions.
306
0:15:7,68 --> 0:15:8,18
Nik: Yeah.
307
0:15:8,3 --> 0:15:10,9
How much of AI capacity have you
already used?
308
0:15:10,9 --> 0:15:14,28
Is it like just 1 $200 account
or no?
309
0:15:14,6 --> 0:15:16,66
I'm just curious, very curious.
310
0:15:17,5 --> 0:15:19,5
Asking for a friend, so to speak.
311
0:15:20,32 --> 0:15:23,94
Michael M.: The 1st version of
pgrust, the 0.1, which was done
312
0:15:23,94 --> 0:15:29,44
with Opus, which that spanned 5
different attempts, that cost
313
0:15:29,44 --> 0:15:30,52
about 100 grand.
314
0:15:31,04 --> 0:15:36,96
And then so far, with the new version,
like 0.2 and will become
315
0:15:36,96 --> 0:15:40,24
0.3 in total, we've put in probably
like another like 300 or
316
0:15:40,24 --> 0:15:41,14
400 grand.
317
0:15:41,76 --> 0:15:42,6
Where the...
318
0:15:44,44 --> 0:15:47,56
We're burning tokens so fast that
you can burn through a 5 hour
319
0:15:47,56 --> 0:15:51,18
quota on a $200 a month subscription
on the, in the order of
320
0:15:51,18 --> 0:15:52,08
like 15 minutes.
321
0:15:54,38 --> 0:15:57,26
It's because for a while, like
what we'll do is like, we'll have
322
0:15:57,26 --> 0:15:59,96
a 40 different Fable instances
like running in parallel, each
323
0:15:59,96 --> 0:16:1,82
like working on individual parts
of Postgres.
324
0:16:2,64 --> 0:16:6,02
And that just burns through credits
super fast.
325
0:16:7,74 --> 0:16:8,82
Nik: Yeah, yeah, that's impressive.
326
0:16:9,44 --> 0:16:11,28
So, yeah.
327
0:16:11,28 --> 0:16:13,1
And it takes also a lot of time,
right?
328
0:16:13,1 --> 0:16:16,64
Because like, it's just they are
like, you just need to wait
329
0:16:16,64 --> 0:16:20,68
sometimes a lot right what was
your the longest run or loop
330
0:16:20,68 --> 0:16:24,12
fully autonomous like days weeks
or something
331
0:16:25,16 --> 0:16:30,96
Michael M.: I usually well I've
had stuff like run overnight
332
0:16:31,16 --> 0:16:36,14
that worked but I think I try to
keep things on the order of,
333
0:16:36,14 --> 0:16:39,22
I try to break things down to like
tasks on the order of like,
334
0:16:39,6 --> 0:16:42,56
ideally like an hour, cause any
more than that, I find that like
335
0:16:42,56 --> 0:16:45,2
the chance of the model going off
and doing its own thing and
336
0:16:45,2 --> 0:16:45,8
like getting.
337
0:16:45,8 --> 0:16:46,3
Nik: Nonsensical.
338
0:16:46,5 --> 0:16:47,0
Yeah.
339
0:16:47,2 --> 0:16:47,7
Michael M.: Yeah.
340
0:16:47,9 --> 0:16:50,7
I found that like models, they
work really well.
341
0:16:50,98 --> 0:16:52,2
They enter 2 states.
342
0:16:52,2 --> 0:16:54,72
Like 1 is they like are just like
working really well and making
343
0:16:54,72 --> 0:16:55,54
forward progress.
344
0:16:56,0 --> 0:16:58,68
And then other times they'll just
start doing their own thing.
345
0:16:58,68 --> 0:17:1,16
They'll just start making like
a bunch of random edits to like
346
0:17:1,16 --> 0:17:3,24
code everywhere and not really
go anywhere.
347
0:17:3,24 --> 0:17:6,3
And like the more you can keep
models on like the productive
348
0:17:6,3 --> 0:17:9,44
side and not on the going in circle
side is better.
349
0:17:9,44 --> 0:17:11,96
Michael C.: I'm curious on the,
because that's a lot of money
350
0:17:11,96 --> 0:17:13,52
to spend on v0.2.
351
0:17:14,44 --> 0:17:18,04
Are you being public with how you're
funding it or like if you
352
0:17:18,04 --> 0:17:18,9
do have investors.
353
0:17:19,86 --> 0:17:20,28
Michael M.: Yeah.
354
0:17:20,28 --> 0:17:25,6
So far it's actually been entirely
funded by me where I had a
355
0:17:25,6 --> 0:17:28,86
previous startup, Freshpaint,
which did like pretty well.
356
0:17:28,86 --> 0:17:31,88
And I had the chance to sell some
of my equity and so far it's
357
0:17:31,88 --> 0:17:33,06
mostly been funded by me.
358
0:17:33,06 --> 0:17:35,52
We're hitting the point where like
it's starting to become like
359
0:17:35,74 --> 0:17:38,76
a bit unreasonable and we're starting
to look at other ways to
360
0:17:38,76 --> 0:17:42,16
like fund it and can we like
be more token efficient and things
361
0:17:42,16 --> 0:17:42,84
like that.
362
0:17:43,32 --> 0:17:44,66
Michael C.: That makes a ton of
sense.
363
0:17:44,8 --> 0:17:47,36
So yeah going back to the technical
stuff I was going to
364
0:17:47,36 --> 0:17:51,14
the test suite is really interesting
to me, and almost as a what
365
0:17:51,14 --> 0:17:54,0
can we learn from a Postgres perspective,
as well as how could
366
0:17:54,0 --> 0:17:55,3
pgrust become trustworthy.
367
0:17:55,58 --> 0:18:1,24
So, on testing the functions, I
can imagine a huge amount of
368
0:18:1,24 --> 0:18:5,92
the kind of user surface area is
going to be covered nicely,
369
0:18:5,92 --> 0:18:9,74
like especially for single user,
single query correctness.
370
0:18:10,64 --> 0:18:15,42
But I can't help but feel nervous
about like concurrency stuff
371
0:18:15,42 --> 0:18:19,2
and even Postgres and some other
databases that have had some
372
0:18:19,2 --> 0:18:22,36
interesting bugs come up when Jepsen
have got involved and run
373
0:18:22,36 --> 0:18:26,92
some like interesting tests along
the like isolation level side
374
0:18:26,92 --> 0:18:27,54
of things.
375
0:18:28,28 --> 0:18:31,5
Do you, does your testing approach
cover that yet?
376
0:18:31,56 --> 0:18:34,34
Or do you, are you going to have
needed like a slightly different
377
0:18:34,34 --> 0:18:35,54
approach for some of those things?
378
0:18:35,54 --> 0:18:36,98
How are you thinking about that?
379
0:18:37,58 --> 0:18:40,6
Michael M.: Yeah, we're basically
taking what is the modern and
380
0:18:40,6 --> 0:18:43,8
latest approach to testing and
trying to incorporate all of that
381
0:18:43,92 --> 0:18:45,36
into how we do things.
382
0:18:45,48 --> 0:18:47,68
And the 1st thing we've been doing
is we've started with the
383
0:18:47,68 --> 0:18:49,44
easy part of, okay, here's all
the pure functions.
384
0:18:49,44 --> 0:18:50,58
Here's the ones that we can verify.
385
0:18:50,58 --> 0:18:52,16
Here's the ones that we can just
fuzz.
386
0:18:52,44 --> 0:18:56,32
And then most recently I've been
working on how can we actually
387
0:18:56,32 --> 0:19:0,52
get full coverage of the database
where there's a lot of very
388
0:19:0,52 --> 0:19:5,58
stateful things that will only
trigger bugs in very rare circumstances.
389
0:19:5,68 --> 0:19:7,9
And if you actually look at the
issues people are reporting for
390
0:19:7,9 --> 0:19:11,04
pgrust, it's usually the combination
of 2 features of, oh, if
391
0:19:11,04 --> 0:19:15,04
I use a non-default collation
with certain string functions,
392
0:19:15,06 --> 0:19:16,52
it'll cause a bug.
393
0:19:16,56 --> 0:19:18,96
And so what we're trying to do
now is actually fuzz until we
394
0:19:18,96 --> 0:19:20,7
get to 100% code coverage.
395
0:19:21,14 --> 0:19:24,4
And then for the concurrency stuff,
once we have a fuzzer that
396
0:19:24,4 --> 0:19:29,16
can actually properly explore the
entire code base, we already
397
0:19:29,16 --> 0:19:34,7
purchased this tool called Antithesis,
which does Jepsen-style
398
0:19:35,32 --> 0:19:36,1
fault testing.
399
0:19:36,2 --> 0:19:37,82
It's actually a super cool product.
400
0:19:38,14 --> 0:19:44,04
They run your software inside of
a VM and then are able to inject
401
0:19:44,24 --> 0:19:46,68
faults into your running code.
402
0:19:46,68 --> 0:19:49,3
And so that can be everything from
the network is unreliable
403
0:19:49,3 --> 0:19:51,38
and your database can't talk to
its replica.
404
0:19:51,58 --> 0:19:55,08
Or it can be like, hey, your server
crashed and now is your data
405
0:19:55,08 --> 0:19:56,46
corrupted on disk or not?
406
0:19:56,46 --> 0:19:59,44
And then they actually do stuff
of they can, they have full control
407
0:19:59,44 --> 0:20:0,14
of the threading.
408
0:20:0,14 --> 0:20:1,16
It's fully deterministic.
409
0:20:1,62 --> 0:20:5,42
And so they can create all these
weird out of order sequences
410
0:20:5,66 --> 0:20:9,4
for your threads to try to just
throw the craziest stuff at your
411
0:20:9,4 --> 0:20:9,9
software.
412
0:20:10,2 --> 0:20:12,68
And right now we're working on
the fuzzer to actually explore
413
0:20:12,9 --> 0:20:13,58
the search space.
414
0:20:13,58 --> 0:20:15,98
And then once we have that, we're
going to give that to Antithesis
415
0:20:16,0 --> 0:20:18,9
to then try to come up with all
these crazy ways of running the
416
0:20:18,9 --> 0:20:21,6
code together to try to break
pgrust.
417
0:20:21,6 --> 0:20:24,14
And so I think that is, we've actually
been talking with a lot
418
0:20:24,14 --> 0:20:26,64
of database people through this
and basically they're all using
419
0:20:26,64 --> 0:20:29,48
Antithesis for their testing at
this point.
420
0:20:29,96 --> 0:20:30,3
Nik: Yeah.
421
0:20:30,3 --> 0:20:34,0
Speaking of testing, I remember
I noticed on Hacker News comments
422
0:20:34,0 --> 0:20:39,18
that on the thread that people
asked questions about was fsync
423
0:20:39,4 --> 0:20:43,44
on and did you do writes in those
tests and so on or is just
424
0:20:43,44 --> 0:20:47,42
only from memory, data fits in
buffer pool or something and
425
0:20:47,42 --> 0:20:48,9
nothing touches the disk.
426
0:20:49,2 --> 0:20:49,4
Michael M.: Yeah.
427
0:20:49,4 --> 0:20:52,0
So we have, if you look at the
code base, we actually have a
428
0:20:52,0 --> 0:20:55,44
crash simulator in pgrust that
mocks out the file system and
429
0:20:55,44 --> 0:20:57,8
then we'll try to just crash the
database at different points.
430
0:20:57,8 --> 0:20:59,44
And we've done a good amount of
testing.
431
0:20:59,6 --> 0:21:3,06
The thing is that even Postgres
does get these things wrong sometimes.
432
0:21:3,08 --> 0:21:6,76
That there was like a fsyncgate
almost like a decade ago now,
433
0:21:6,76 --> 0:21:10,32
I think, of like, yeah, fsync can
actually fail and Postgres
434
0:21:10,32 --> 0:21:11,78
actually didn't handle that case.
435
0:21:11,82 --> 0:21:15,22
Which be fair to Postgres, that's
a case that never ever happens.
436
0:21:15,58 --> 0:21:20,2
Nik: Honestly, I also think to
me going to Jepsen tests and multi-node
437
0:21:20,28 --> 0:21:24,66
basically network testing and so
on is maybe not the biggest
438
0:21:24,66 --> 0:21:26,08
priority for this maybe.
439
0:21:26,26 --> 0:21:31,16
In my opinion, I recently learned
that Postgres own tests are not
440
0:21:31,16 --> 0:21:34,26
checking properly disk outages,
for example.
441
0:21:35,24 --> 0:21:39,72
So this is like, what happens if
disk just has some problems?
442
0:21:40,46 --> 0:21:44,94
And there are some tools to improve
that in Postgres on codebase.
443
0:21:45,9 --> 0:21:46,94
So I'm very curious.
444
0:21:47,08 --> 0:21:49,64
This is purely a vibe-coded thing.
445
0:21:49,9 --> 0:21:54,86
You obviously created a good harness
to mock and so on, but still
446
0:21:54,86 --> 0:21:56,42
people reported some segfaults.
447
0:21:56,54 --> 0:21:58,16
It's obvious it's early stage.
448
0:21:58,42 --> 0:22:1,36
Will it reach some point when it's
very reliable so it can be
449
0:22:1,36 --> 0:22:2,2
put to production.
450
0:22:2,2 --> 0:22:5,68
So obviously you say yes, but what's
your like, why do you think
451
0:22:5,68 --> 0:22:5,9
so?
452
0:22:5,9 --> 0:22:7,0
Why do you think yes?
453
0:22:7,1 --> 0:22:9,56
Because I'm also very AI positive,
so to speak.
454
0:22:9,56 --> 0:22:13,34
I think actually in a few months
we will be able to revisit with
455
0:22:13,34 --> 0:22:15,0701
new models and polish it and so
on.
456
0:22:15,0701 --> 0:22:20,16
And It gives me so joy to work
with AI together, like human plus
457
0:22:20,16 --> 0:22:22,12
AI, we can achieve a lot.
458
0:22:22,36 --> 0:22:25,22
But still, there are so many doubts
around.
459
0:22:26,0 --> 0:22:30,56
And people think, for example,
it's impossible to achieve a point
460
0:22:30,66 --> 0:22:32,38
where it will be reliable.
461
0:22:32,96 --> 0:22:35,28
You started with reliability, right?
462
0:22:35,74 --> 0:22:40,16
Then you said hackers are too conservative,
but there's also
463
0:22:40,16 --> 0:22:43,7
they are too conservative to protect
that reliability thing,
464
0:22:43,7 --> 0:22:44,2
right?
465
0:22:44,28 --> 0:22:48,08
Now with vibe code, the thing fully
rewritten, won't it take
466
0:22:48,08 --> 0:22:51,44
10 years to achieve a very reliable
state?
467
0:22:51,82 --> 0:22:54,62
What makes you think it's possible
to achieve it much faster
468
0:22:54,62 --> 0:22:55,5
than 10 years?
469
0:22:56,74 --> 0:22:58,86
Michael M.: So I think for the
issues that people are reporting,
470
0:22:58,86 --> 0:23:2,8
I think 1 thing to clarify is that
so far for version 0.2, we
471
0:23:2,8 --> 0:23:6,22
had, or actually I think for version
0.2, that was actually before
472
0:23:6,22 --> 0:23:10,76
we had done even like most
of the fuzz testing I was talking
473
0:23:10,76 --> 0:23:12,84
about, where like we actually sorted
out those issues.
474
0:23:13,04 --> 0:23:16,06
The correctness work has like largely
followed version 0.2 and
475
0:23:16,06 --> 0:23:18,8
so far we've only covered like
15 or 20 percent of the codebase
476
0:23:18,8 --> 0:23:20,42
and like there's still a lot more
to cover.
477
0:23:20,46 --> 0:23:23,04
And so I'm like I'm totally not
surprised that people found issues
478
0:23:23,04 --> 0:23:25,08
in the version we released just
because like we hadn't actually
479
0:23:25,08 --> 0:23:26,62
hardened it that much.
480
0:23:26,84 --> 0:23:29,34
And to answer your question of
how we can actually get this to
481
0:23:29,34 --> 0:23:32,92
a place where people can trust
this, Back to what I said earlier
482
0:23:32,92 --> 0:23:36,66
about, it's not like the models
themselves are like reviewing
483
0:23:36,66 --> 0:23:38,66
all the code and like making sure
it's correct.
484
0:23:38,74 --> 0:23:42,1
What we're doing is like we're
building tools to actually make
485
0:23:42,1 --> 0:23:43,4
it, make sure it's correct.
486
0:23:43,48 --> 0:23:46,84
And we're doing it such a degree
that like we're able to do millions
487
0:23:46,84 --> 0:23:50,24
of inputs for individual functions
to the point that Postgres,
488
0:23:50,66 --> 0:23:53,86
this 30-year-old battle-tested
codebase, we're finding bugs in
489
0:23:53,86 --> 0:23:56,52
it that no 1 had found before,
even though people have been using
490
0:23:56,52 --> 0:23:57,94
it all this time.
491
0:23:58,04 --> 0:24:1,28
Nik: You find bugs in the old code,
but Maybe you have bugs which
492
0:24:1,28 --> 0:24:3,46
you haven't explored in new code
yet.
493
0:24:4,06 --> 0:24:5,88
So this is a really tricky question.
494
0:24:6,38 --> 0:24:8,86
What's your plan to prove it in
production?
495
0:24:9,02 --> 0:24:13,0
Just wait for people who will try
it in production on replicas,
496
0:24:13,0 --> 0:24:16,56
for example, because I always think
This thing can work as a
497
0:24:16,56 --> 0:24:17,46
physical standby.
498
0:24:18,22 --> 0:24:22,2
Is this a way or use some mirroring,
for example, PgDog has or
499
0:24:22,2 --> 0:24:26,28
other, maybe some others have,
to mirror traffic and see that
500
0:24:26,28 --> 0:24:27,18
it works well?
501
0:24:27,18 --> 0:24:30,48
Or what's your plan to prove with
production that it works?
502
0:24:30,84 --> 0:24:31,12
Michael M.: Yeah.
503
0:24:31,12 --> 0:24:34,54
So for proving it like actually
in production, our plan is to
504
0:24:34,54 --> 0:24:37,2
do exactly what you said and replicate
off of people's existing
505
0:24:37,2 --> 0:24:40,16
databases and be like read-only
standby.
506
0:24:40,68 --> 0:24:43,58
That gives people a lot of the
benefits of like what we're building
507
0:24:43,58 --> 0:24:47,9
in terms of this like ultra fast
columnar workload such that
508
0:24:47,9 --> 0:24:50,66
like people who are trying to do
analytics in Postgres and are
509
0:24:50,66 --> 0:24:53,72
struggling to do it, they can stand
up pgrust and then move
510
0:24:53,72 --> 0:24:55,12
to analytics over to that.
511
0:24:55,12 --> 0:24:58,24
And the really nice thing about
analytics is they tend to be
512
0:24:58,38 --> 0:25:2,32
like semi-production workloads
where either like it's internal
513
0:25:2,32 --> 0:25:5,5
usage and okay your database goes
down, okay my internal dashboards
514
0:25:5,5 --> 0:25:8,6
are broken for a little bit, it's
not the end of the world.
515
0:25:8,94 --> 0:25:12,52
And then over time as like we start
to get like battle tested
516
0:25:12,52 --> 0:25:16,86
in these not tier 1 database use
cases, that's I think, will
517
0:25:16,86 --> 0:25:18,96
give people more confidence that,
hey, this is actually something
518
0:25:18,96 --> 0:25:20,64
that, like, works well.
519
0:25:21,02 --> 0:25:24,7
Nik: So what I like in this idea
is that if it's, if pgrust
520
0:25:24,86 --> 0:25:29,48
is powering a physical standby,
and then it crashed, for example,
521
0:25:30,14 --> 0:25:32,22
but primary won't be affected,
right?
522
0:25:32,22 --> 0:25:34,7
Because, okay, some slot is not
used.
523
0:25:35,66 --> 0:25:40,24
And if it's working and you run
very long running query on it,
524
0:25:40,24 --> 0:25:44,76
usually it's a bad idea on regular
standbys because of hot_standby_feedback
525
0:25:44,76 --> 0:25:45,86
dilemma.
526
0:25:46,38 --> 0:25:49,64
If it's on, it will affect vacuum
on the primary.
527
0:25:49,64 --> 0:25:52,8
If it's off, it makes your node
basically single user.
528
0:25:52,8 --> 0:25:53,9
It will start lagging.
529
0:25:54,64 --> 0:25:58,94
So both choices are not okay if
you want analytics.
530
0:25:59,54 --> 0:26:2,86
But It's only true if those queries
last hours.
531
0:26:3,94 --> 0:26:8,54
If it's already, as your blog post
suggests, 300 times faster,
532
0:26:9,4 --> 0:26:14,14
then xmin horizon is not blocked
by hot_standby_feedback for long.
533
0:26:14,14 --> 0:26:15,7
It's blocked for a short time.
534
0:26:16,1 --> 0:26:17,36
This is very interesting.
535
0:26:17,78 --> 0:26:19,52
This looks promising.
536
0:26:20,64 --> 0:26:24,56
We have very fast standby, which
behaves exactly like Postgres.
537
0:26:25,6 --> 0:26:28,16
Like, primary thinks it's like
Postgres, okay?
538
0:26:28,74 --> 0:26:34,24
And we have ability to run some
aggregates and so on very fast.
539
0:26:34,74 --> 0:26:35,58
Michael M.: That's interesting.
540
0:26:35,92 --> 0:26:39,08
Actually, the thing I find really
interesting about pgrust is
541
0:26:39,08 --> 0:26:43,18
there's actually a whole bunch
of unexplored design space, new
542
0:26:43,18 --> 0:26:45,52
ways that you can do things.
543
0:26:45,9 --> 0:26:49,56
Where you mentioned, for instance,
when you have a replica, due
544
0:26:49,56 --> 0:26:54,44
to the way the WAL replay works,
there's all these like complexities
545
0:26:54,52 --> 0:26:57,74
about how did the vacuums work
and how does like the replica
546
0:26:57,74 --> 0:26:58,66
like fall behind.
547
0:26:58,66 --> 0:27:0,4
But I think there are options out
there.
548
0:27:0,4 --> 0:27:2,64
I haven't explored them thoroughly,
but there's options out there.
549
0:27:2,64 --> 0:27:5,64
If we change the file format and
the way the WAL replay works,
550
0:27:5,74 --> 0:27:8,2
maybe there's ways to actually
get around these problems.
551
0:27:8,74 --> 0:27:11,16
I think this is like a lot of what's
really exciting about pgrust
552
0:27:11,16 --> 0:27:14,24
is like, there's a lot of
like unexplored design area.
553
0:27:14,64 --> 0:27:17,84
And because it's this like experimental
new thing, there's actually
554
0:27:17,84 --> 0:27:21,24
lots of opportunity to explore
the design space.
555
0:27:22,48 --> 0:27:24,18
Nik: Right, but that's for sure.
556
0:27:24,52 --> 0:27:27,38
So experimental, you can go very
far.
557
0:27:27,74 --> 0:27:32,06
But grounding is when you put it
to production.
558
0:27:33,66 --> 0:27:36,84
This is the most interesting practical
piece here for me.
559
0:27:37,12 --> 0:27:40,9
Sounds like it's relatively low
risk to try it.
560
0:27:41,0 --> 0:27:42,08
There's a way, right?
561
0:27:42,44 --> 0:27:43,22
That's interesting.
562
0:27:44,12 --> 0:27:45,92
So, Yeah, that's cool.
563
0:27:45,92 --> 0:27:46,7
What else?
564
0:27:46,92 --> 0:27:49,68
I saw your blog post, I think
today, right?
565
0:27:49,68 --> 0:27:51,9
About JIT just in time.
566
0:27:52,04 --> 0:27:53,68
What attracted your attention?
567
0:27:53,68 --> 0:27:57,38
It was just switched off by default
finally, recently.
568
0:27:58,26 --> 0:27:59,62
Was this like, yeah.
569
0:28:0,1 --> 0:28:0,6
Michael M.: Yeah.
570
0:28:1,12 --> 0:28:4,2
As part of the Columnar stuff,
we looked at a bunch of the recent
571
0:28:4,2 --> 0:28:7,86
database literature and incorporated
a ton of stuff into pgrust.
572
0:28:8,36 --> 0:28:12,56
A lot of it actually coming from
the research into Umbra, which
573
0:28:12,56 --> 0:28:16,08
is this very experimental database
being developed by this team
574
0:28:16,08 --> 0:28:16,64
in Munich.
575
0:28:16,64 --> 0:28:19,2
And they've been publishing database
research for the last decade
576
0:28:19,2 --> 0:28:21,26
or more, and a lot of it comes
from them.
577
0:28:21,74 --> 0:28:25,44
But for JIT compilation, and this
is a lot of what I talked about
578
0:28:25,44 --> 0:28:28,18
in the post that came out this
morning, if you look at all the
579
0:28:28,18 --> 0:28:32,52
databases that have JIT compilation
in it, they either use LLVM,
580
0:28:33,14 --> 0:28:37,36
or they'll write C and C++ code
and then compile that.
581
0:28:37,36 --> 0:28:40,28
And the big downside of these approaches
is that there's actually
582
0:28:40,28 --> 0:28:44,72
a huge amount of latency overhead
to compile this code that for
583
0:28:44,72 --> 0:28:49,26
C and C++ or for LLVM, I think it's
50 milliseconds compile time.
584
0:28:49,4 --> 0:28:52,8
And so you're very limited in the
cases that you can use it.
585
0:28:52,8 --> 0:28:56,26
And you have this risk of if you're
the planner estimates the
586
0:28:56,26 --> 0:28:59,4
query wrong, you now have this extra
50 milliseconds of latency.
587
0:29:0,06 --> 0:29:4,84
But there was this approach to
JIT compilation called copy and
588
0:29:4,84 --> 0:29:7,64
I think it's like copy and fix
or something along those lines
589
0:29:7,64 --> 0:29:11,88
that was released in someone published
a paper on it in 2021
590
0:29:11,92 --> 0:29:13,94
that makes copy and patch.
591
0:29:13,94 --> 0:29:14,84
Yeah, that's it.
592
0:29:14,86 --> 0:29:16,72
Then it's much easier to write.
593
0:29:16,72 --> 0:29:19,28
Then you just have templates of
assembly code and then you just
594
0:29:19,28 --> 0:29:20,8
fill in the bits as needed.
595
0:29:20,8 --> 0:29:24,38
You can combine these templates
together to get the code you
596
0:29:24,38 --> 0:29:24,88
want.
597
0:29:24,96 --> 0:29:28,62
And so the pgrust JIT compiler
actually doesn't use something
598
0:29:28,62 --> 0:29:29,26
like LLVM.
599
0:29:29,26 --> 0:29:32,06
It actually just directly generates
the assembly.
600
0:29:32,42 --> 0:29:37,26
And so because of this, the compile
times are like 5 microseconds
601
0:29:37,84 --> 0:29:42,18
as opposed to 50 milliseconds for
LLVM.
602
0:29:42,56 --> 0:29:47,36
And this gives us a lot more, Like
we can JIT compile a lot more
603
0:29:47,36 --> 0:29:51,34
of the query that we actually,
for our executor, we JIT compile
604
0:29:51,34 --> 0:29:55,26
large parts of the query, not just
expressions, which Postgres
605
0:29:55,26 --> 0:29:59,44
will only JIT compile expressions
and tuple deforming, but we
606
0:29:59,44 --> 0:30:1,78
actually do a lot more than that.
607
0:30:1,88 --> 0:30:5,28
And so we get measurable speed
ups in many different places.
608
0:30:6,68 --> 0:30:10,56
Michael C.: Do you see any reason
Postgres couldn't use a similar
609
0:30:10,56 --> 0:30:11,06
approach?
610
0:30:12,26 --> 0:30:16,56
Michael M.: So 1 of the big, 1
of the challenges I see with Postgres
611
0:30:16,56 --> 0:30:22,0
adopting this approach is I Specifically
am targeting only the Graviton
612
0:30:22,2 --> 0:30:27,56
instruction set Whereas pgrust
is being built in a very different
613
0:30:27,56 --> 0:30:30,34
world where today most people are
running their databases in
614
0:30:30,34 --> 0:30:34,04
the cloud And for instance, there's
no Windows support in pgrust
615
0:30:34,04 --> 0:30:34,96
yet.
616
0:30:35,34 --> 0:30:38,42
And so I'm able to, because I know,
I'm just assuming that people
617
0:30:38,42 --> 0:30:42,04
are going to run this primarily
on AWS and GCP, I'm able to target
618
0:30:42,04 --> 0:30:46,14
1 specific CPU design and 1 specific
instruction set and focus
619
0:30:46,26 --> 0:30:48,76
just on that and make that work
really well.
620
0:30:50,28 --> 0:30:52,4
And for Postgres, I think it's
still possible.
621
0:30:52,44 --> 0:30:54,92
You could have the JIT compiler
only work in certain instruction
622
0:30:54,92 --> 0:31:0,26
sets, or you can have different
backends for different instruction
623
0:31:0,28 --> 0:31:0,6
sets.
624
0:31:0,6 --> 0:31:4,2
But it's an order of magnitude
more work to get that to work
625
0:31:4,2 --> 0:31:5,96
well than what I'm doing.
626
0:31:6,06 --> 0:31:9,38
Nik: And if, even if it's not in
cloud, they will run it on Mac
627
0:31:9,38 --> 0:31:11,14
Minis they have now, right?
628
0:31:14,72 --> 0:31:16,16
Should work anyway there, Yeah.
629
0:31:16,16 --> 0:31:19,2
Michael M.: I've seen multiple
people with like rooms of like,
630
0:31:19,2 --> 0:31:19,92
they'll have
631
0:31:20,8 --> 0:31:21,54
Nik: a bunch of...
632
0:31:21,54 --> 0:31:22,04
Yeah.
633
0:31:22,2067 --> 0:31:23,04
In the corner.
634
0:31:23,48 --> 0:31:23,8
Could be a good Postgres cluster.
635
0:31:23,8 --> 0:31:23,82
Yeah.
636
0:31:23,82 --> 0:31:24,84
So that's interesting.
637
0:31:25,46 --> 0:31:28,44
And you don't use GPT like Codex,
no?
638
0:31:28,44 --> 0:31:30,58
Like distributing workloads within?
639
0:31:31,52 --> 0:31:34,84
Michael M.: So in my experience,
actually like the, a lot of
640
0:31:34,84 --> 0:31:39,48
the early work we were doing with
Codex, like Codex 5.4 and Codex
641
0:31:39,48 --> 0:31:44,44
5.5, the thing that made me switch
to Claude is they released
642
0:31:44,44 --> 0:31:47,96
this feature called dynamic workflows,
which this is what was
643
0:31:47,96 --> 0:31:49,86
used for the Bun Rust rewrite.
644
0:31:50,14 --> 0:31:56,1
And dynamic workflows are, you
can basically, it will write code
645
0:31:56,14 --> 0:31:58,62
that orchestrates a bunch of different
agents.
646
0:31:58,94 --> 0:32:1,48
And so you can be like, Hey, I
have these 10 files.
647
0:32:1,48 --> 0:32:3,06
Can you rewrite all of them?
648
0:32:3,18 --> 0:32:7,2
And it'll write some JavaScript,
they'll then spin up 10 agents,
649
0:32:7,2 --> 0:32:10,58
they'll then rewrite each file
and then review it and fix any
650
0:32:10,58 --> 0:32:12,34
issues and then merge it.
651
0:32:12,34 --> 0:32:16,02
And so that level of orchestration
made it way easier to use
652
0:32:16,02 --> 0:32:16,52
Claude.
653
0:32:16,98 --> 0:32:21,8
And then in my experience, like
Fable just works, it's like way
654
0:32:21,8 --> 0:32:23,74
more reliable and trustworthy.
655
0:32:24,38 --> 0:32:28,22
1 of the actually like biggest
challenges I had with the rewrite,
656
0:32:28,48 --> 0:32:31,34
or like 1 of the things that's
actually really hard, was the
657
0:32:31,34 --> 0:32:33,06
Postgres regular expression engine.
658
0:32:33,4 --> 0:32:35,42
I don't know what it is about it.
659
0:32:35,8 --> 0:32:38,04
I think it's 10,000 lines of code.
660
0:32:38,04 --> 0:32:40,82
It's a decent amount of code, but
not too big.
661
0:32:41,0 --> 0:32:45,1
And I was doing this as a benchmark,
and I threw Sol at it on
662
0:32:45,1 --> 0:32:48,02
medium-level thinking, not the
highest thinking, but on a decent...
663
0:32:48,14 --> 0:32:52,68
Sol's a pretty good model, And
Sol wasn't able to rewrite it
664
0:32:52,68 --> 0:32:53,22
in Rust.
665
0:32:53,22 --> 0:32:54,44
I was like very surprised by that.
666
0:32:54,44 --> 0:32:57,98
I was like fully expecting that
given how smart Sol is, like
667
0:32:57,98 --> 0:32:59,94
it should just be able to do that.
668
0:33:0,06 --> 0:33:2,64
Whereas Fable was like a whole
level above that, where it was
669
0:33:2,64 --> 0:33:5,58
able to rewrite in Rust and then
it was able to do like performance
670
0:33:5,58 --> 0:33:9,62
optimization on it to make it a
tiny bit faster than the Postgres
671
0:33:9,62 --> 0:33:10,12
version.
672
0:33:10,13 --> 0:33:15,84
And so like, I think like it probably
is possible to use like
673
0:33:15,84 --> 0:33:19,28
Sol, but like I've just found the
difference in autonomy between
674
0:33:19,28 --> 0:33:23,42
the 2 models is like not worth
the difference.
675
0:33:24,92 --> 0:33:27,76
Michael C.: I think we need to
talk about licensing.
676
0:33:28,44 --> 0:33:29,06
I noticed
677
0:33:29,06 --> 0:33:29,62
Nik: you picked
678
0:33:29,62 --> 0:33:30,74
Michael C.: the AGPL.
679
0:33:31,58 --> 0:33:35,28
That's always controversial, but
at least open source, right?
680
0:33:35,28 --> 0:33:36,38
What, why pick that 1?
681
0:33:36,38 --> 0:33:37,98
I think I know why, but why?
682
0:33:38,8 --> 0:33:41,64
Michael M.: Yeah, we were looking
at the different options we
683
0:33:41,64 --> 0:33:42,6
had for licensing.
684
0:33:42,98 --> 0:33:44,6
And there's like 4 different options.
685
0:33:44,68 --> 0:33:46,92
There's you have MIT or like Postgres.
686
0:33:47,26 --> 0:33:51,66
You have like AGPL or like GPL,
where it's like the copyleft.
687
0:33:51,94 --> 0:33:54,96
You have what they call like source
available, like you have
688
0:33:54,96 --> 0:33:58,22
SSPL and like BSL, and you have
like closed source.
689
0:33:58,68 --> 0:34:2,16
And We knew we wanted to do something
open source, and it's now
690
0:34:2,16 --> 0:34:6,6
limited to the permissive or the
AGPL.
691
0:34:8,04 --> 0:34:13,54
And the thing was, we wanted something
that a third party couldn't
692
0:34:13,54 --> 0:34:17,64
just take and sell themselves to
the detriment of the work we're
693
0:34:17,64 --> 0:34:18,14
doing.
694
0:34:19,62 --> 0:34:23,26
Where there's lots of closed source
forks of Postgres.
695
0:34:23,26 --> 0:34:28,12
You have AlloyDB, you have Aurora,
now Neon is closed source.
696
0:34:28,14 --> 0:34:31,92
And none of that money that those
platforms make actually goes
697
0:34:31,92 --> 0:34:35,46
back, or like very, very tiny amounts
of it goes back to the
698
0:34:35,46 --> 0:34:37,06
actual Postgres core projects.
699
0:34:37,22 --> 0:34:40,12
And that was just something that
like I wanted to avoid with
700
0:34:40,12 --> 0:34:40,84
pgrust.
701
0:34:40,84 --> 0:34:44,06
Nik: But AGPL
doesn't protect you from selling.
702
0:34:44,06 --> 0:34:46,9
Anyone can still sell if they don't
modify code.
703
0:34:47,24 --> 0:34:50,28
And even if they modify code, they
just need to publish it.
704
0:34:50,28 --> 0:34:51,0
That's it.
705
0:34:51,74 --> 0:34:52,24
Michael M.: Yeah.
706
0:34:53,1 --> 0:34:53,32
Yeah.
707
0:34:53,32 --> 0:34:55,64
So at least the history for AGPL
is it is like extremely, basically
708
0:34:55,64 --> 0:34:56,88
no 1 has really done that.
709
0:34:56,88 --> 0:35:0,06
But like just the cost of needing
to open source your modifications
710
0:35:0,06 --> 0:35:5,78
to it has deterred like Amazon
and Google from offering any AGPL
711
0:35:5,78 --> 0:35:8,04
software on their platforms, at
least so far.
712
0:35:8,14 --> 0:35:8,76
Nik: Makes sense.
713
0:35:8,76 --> 0:35:8,92
So
714
0:35:8,92 --> 0:35:10,62
Michael M.: that seemed like the
right trade-off.
715
0:35:11,54 --> 0:35:12,04
Nik: Good.
716
0:35:12,44 --> 0:35:15,04
I know we're like almost out of
time.
717
0:35:15,1 --> 0:35:16,5
It was an interesting discussion.
718
0:35:16,5 --> 0:35:17,58
Thank you for coming, Michael.
719
0:35:17,58 --> 0:35:18,82
I enjoyed it very well.
720
0:35:18,82 --> 0:35:19,84
Good luck with your project.
721
0:35:19,84 --> 0:35:21,86
We will be keeping an eye on it.
722
0:35:22,06 --> 0:35:22,94
Michael M.: Yeah, of course.
723
0:35:23,0 --> 0:35:23,86
Thanks so much.
724
0:35:24,48 --> 0:35:24,84
Nik: Thank you.
0:0:0,06 --> 0:0:2,06
Nik: Hello, hello, this is Postgre.FM
2
0:0:2,08 --> 0:0:5,8199997
My name is Nik, PostgresAI, and
as usual with me, Michael,
3
0:0:6,66 --> 0:0:7,16
pgMustard.
4
0:0:7,44 --> 0:0:8,18
Hi, Michael.
5
0:0:8,74 --> 0:0:9,5199995
Michael C.: Hi, Nik.
6
0:0:10,08 --> 0:0:13,5
Nik: And we have a very, very interesting
guest today, Michael Malis,
7
0:0:13,5 --> 0:0:18,7
who created pgrust, which has already,
I think, 5,000 stars
8
0:0:18,7 --> 0:0:22,38
on GitHub and a lot of noise around,
like a lot of buzz.
9
0:0:22,7 --> 0:0:24,439999
Hi, Michael, thank you for coming.
10
0:0:24,96 --> 0:0:25,76
Michael M.: Yeah, of course.
11
0:0:25,76 --> 0:0:27,04
Thank you for having me.
12
0:0:28,08 --> 0:0:32,56
Nik: So, of course, I think the
1st question should be how it
13
0:0:32,56 --> 0:0:33,78
all started, why?
14
0:0:33,84 --> 0:0:35,22
Tell us the story, please.
15
0:0:36,98 --> 0:0:40,52
Michael M.: So Jason and I were
looking for projects to work
16
0:0:40,52 --> 0:0:45,56
on, and we wanted to do something
that we knew super well.
17
0:0:45,86 --> 0:0:50,38
And initially, what we were focused
on was reliability and how
18
0:0:50,38 --> 0:0:53,14
can we help people make their websites
more reliable.
19
0:0:54,14 --> 0:0:56,7
And we were working with a bunch
of people.
20
0:0:57,1 --> 0:1:1,3
And the common pattern that emerged
was that A lot of the problems
21
0:1:1,36 --> 0:1:5,16
that caused reliability issues,
a lot of them stemmed from their
22
0:1:5,16 --> 0:1:8,22
database and how they were using
their database, whether it was
23
0:1:8,22 --> 0:1:10,54
like Postgres or Redis or some
other system.
24
0:1:11,12 --> 0:1:14,54
And so we started looking at what
are ways that we can solve
25
0:1:14,54 --> 0:1:15,26
this problem.
26
0:1:15,94 --> 0:1:19,64
And we were throwing around a couple
of things, and we ended
27
0:1:19,64 --> 0:1:23,66
up figuring out that AI is actually
pretty good at rewriting
28
0:1:23,66 --> 0:1:24,78
software at this point.
29
0:1:24,86 --> 0:1:27,1
This is back in April.
30
0:1:27,88 --> 0:1:31,84
And so we're like, hey, why don't
we actually try fixing these
31
0:1:31,84 --> 0:1:35,28
problems at the source and actually
trying to modify Postgres
32
0:1:35,98 --> 0:1:39,56
and fix a lot of the things that
cause people to have different
33
0:1:39,56 --> 0:1:43,2
reliability issues, whether it's
connections or it's like certain
34
0:1:43,2 --> 0:1:46,5
queries just take a really long
time or like 1 long-running query
35
0:1:46,5 --> 0:1:49,62
can take down your database or
vacuums as we're all familiar
36
0:1:49,64 --> 0:1:50,14
with.
37
0:1:50,28 --> 0:1:55,76
And so the idea behind pgrust,
the name's a bit misleading because
38
0:1:55,76 --> 0:1:58,14
it's not actually really about
Rust.
39
0:1:58,14 --> 0:2:0,52
I think that's actually the least
interesting part about it.
40
0:2:0,52 --> 0:2:5,7
It's actually more about how do
we leverage AI to re-architect
41
0:2:6,16 --> 0:2:10,14
Postgres and just build a much
better database.
42
0:2:11,38 --> 0:2:11,88
Nik: Okay.
43
0:2:12,12 --> 0:2:14,08
And when you say better, what do
you mean?
44
0:2:14,9 --> 0:2:15,4
Michael M.: Yeah.
45
0:2:15,66 --> 0:2:21,44
So there's a combination of like
different challenges that like
46
0:2:21,44 --> 0:2:23,94
I've just seen people repeatedly
have with Postgres and I actually
47
0:2:23,94 --> 0:2:27,78
wrote a blog post called it like
the 4 horsemen about a lot of
48
0:2:27,78 --> 0:2:29,24
the challenges that people have.
49
0:2:29,32 --> 0:2:32,64
And some of the most common ones
are like, you misconfigure your
50
0:2:32,64 --> 0:2:35,54
connection limit and like all of
a sudden like nothing can connect
51
0:2:35,54 --> 0:2:36,32
to your database.
52
0:2:36,74 --> 0:2:38,64
You have like JSON support.
53
0:2:38,64 --> 0:2:42,08
Tons and tons of people use JSON
but Postgres doesn't have statistics
54
0:2:42,54 --> 0:2:44,02
for how to actually query JSON.
55
0:2:44,02 --> 0:2:46,32
So when you try to do it, you're
going to get really bad query
56
0:2:46,32 --> 0:2:48,46
plans and everything's just going
to be really slow.
57
0:2:48,68 --> 0:2:52,66
There's like the wraparound vacuum
and you have more than several
58
0:2:52,66 --> 0:2:55,84
billion transactions and like all
of a sudden if your vacuum
59
0:2:55,84 --> 0:2:58,74
can't keep up your database falls
over and there's just like
60
0:2:58,74 --> 0:3:3,92
all these problems that have been
around for such a long time.
61
0:3:3,92 --> 0:3:7,94
And like Postgres is a great product
and like it is a really
62
0:3:7,94 --> 0:3:12,98
great system, but the way the Postgres
core team approaches things
63
0:3:12,98 --> 0:3:16,44
is they approach it in terms of
stability And how do we keep
64
0:3:16,44 --> 0:3:17,56
what we have today?
65
0:3:17,56 --> 0:3:19,14
Like, how do we keep it working?
66
0:3:19,2 --> 0:3:21,26
And how do we make sure we don't
break it?
67
0:3:21,42 --> 0:3:24,44
Because like, there's like billions
of Postgres instances out
68
0:3:24,44 --> 0:3:24,84
there.
69
0:3:24,84 --> 0:3:28,64
And so number 1 priority is just
how do we not break the existing
70
0:3:28,64 --> 0:3:29,14
stuff?
71
0:3:29,54 --> 0:3:32,98
Versus how do we actually fix the
things that are not working.
72
0:3:33,34 --> 0:3:36,84
And so pgrust, because it's a
new project, we can take a little
73
0:3:36,84 --> 0:3:40,64
bit of a different approach of
let's try to actually fix the
74
0:3:40,64 --> 0:3:42,08
things that aren't working.
75
0:3:42,44 --> 0:3:45,54
And maybe some of the existing
stuff, it's not going to be quite
76
0:3:45,54 --> 0:3:48,16
as reliable, at least upfront,
as Postgres is.
77
0:3:48,16 --> 0:3:52,86
But at least we'll be able to fix
some of the long-standing architectural
78
0:3:52,96 --> 0:3:54,82
issues that Postgres has.
79
0:3:56,48 --> 0:3:57,6
Nik: Yeah, it makes sense.
80
0:3:57,84 --> 0:3:58,94
How did you do it?
81
0:3:58,94 --> 0:3:59,76
It's a lot of...
82
0:3:59,76 --> 0:4:1,5
Obviously, it's with Claude, right?
83
0:4:1,72 --> 0:4:4,74
Michael M.: Yeah, it took a couple
of attempts to actually figure
84
0:4:4,74 --> 0:4:6,52
out what is the best way to do
this.
85
0:4:6,82 --> 0:4:12,72
And what ended up working really
well was with Opus, we were
86
0:4:12,72 --> 0:4:19,06
able to take each file of Postgres
and basically transpile it
87
0:4:19,06 --> 0:4:19,74
to Rust.
88
0:4:19,82 --> 0:4:22,12
And so if you look at our code
and look at the Postgres code
89
0:4:22,12 --> 0:4:24,06
side by side, it actually looks
very similar.
90
0:4:24,16 --> 0:4:27,24
Where like the functions are the
same, some of the details may
91
0:4:27,24 --> 0:4:30,24
be a little bit different, but
overall it's actually really close
92
0:4:30,24 --> 0:4:32,5
to just a straight rewrite of Postgres.
93
0:4:32,8 --> 0:4:35,42
And so we did this across all the
files.
94
0:4:35,9 --> 0:4:40,28
There were some things that had
to be changed to actually work
95
0:4:40,28 --> 0:4:40,92
in Rust.
96
0:4:40,92 --> 0:4:44,28
1 of the big ones was how memory
management is done, because
97
0:4:44,28 --> 0:4:47,38
Rust is very particular about how
you allocate memory.
98
0:4:47,86 --> 0:4:51,04
But this approach of going through
all the files, rewriting them,
99
0:4:51,04 --> 0:4:54,44
we had a bunch of Claude agents
like going at this in parallel.
100
0:4:54,72 --> 0:4:57,24
There were like some conflicts
between different files, but we
101
0:4:57,24 --> 0:5:0,62
were able to resolve those and
then get pgrust to actually work.
102
0:5:0,62 --> 0:5:4,08
And then from there, we focused
on the Postgres test suite, getting
103
0:5:4,08 --> 0:5:5,14
all that to pass.
104
0:5:6,04 --> 0:5:10,78
And we have something that actually
looks a lot like Postgres,
105
0:5:11,04 --> 0:5:12,08
but is in Rust.
106
0:5:12,8 --> 0:5:15,6
Michael C.: I have potentially
a minor question on the source
107
0:5:15,6 --> 0:5:16,1
code.
108
0:5:16,3 --> 0:5:18,7
Really interesting that you went
file by file and that worked
109
0:5:18,7 --> 0:5:19,2
well.
110
0:5:19,36 --> 0:5:22,74
1 of my favorite things in the
source code is all the is the
111
0:5:22,74 --> 0:5:27,34
comments and they describe in quite
a lot of detail how things
112
0:5:27,34 --> 0:5:32,62
work and I wonder if you've noticed
how it rewrote those.
113
0:5:32,78 --> 0:5:37,92
I can imagine the code being somewhat
reliable in terms of it
114
0:5:37,92 --> 0:5:41,8
being translated, but I can imagine
the comments actually being
115
0:5:41,8 --> 0:5:43,76
harder in some way to be trustworthy.
116
0:5:44,18 --> 0:5:45,74
What have you found on that front?
117
0:5:47,08 --> 0:5:49,9
Michael M.: Yeah, for the comments,
like Claude has been like,
118
0:5:49,9 --> 0:5:52,36
Claude is very verbose when it
comes to comments.
119
0:5:53,26 --> 0:5:56,48
And like, every file have a whole
like big comment about like,
120
0:5:56,48 --> 0:5:59,84
what's in the file that I've actually
had to give it feedback,
121
0:5:59,84 --> 0:6:2,7
like I had to give it feedback
to comment less.
122
0:6:2,72 --> 0:6:5,84
And I think there's a linter that says
files should not have more
123
0:6:5,84 --> 0:6:7,86
than 5% of the line shouldn't be
comments.
124
0:6:9,96 --> 0:6:13,7
I haven't paid quite close attention
to the actual content of
125
0:6:13,7 --> 0:6:18,28
the comments, but I do know Claude
is making a lot of comments
126
0:6:18,28 --> 0:6:19,2
in the new code.
127
0:6:19,64 --> 0:6:21,3
Nik: It does, yeah.
128
0:6:21,58 --> 0:6:24,78
So I remember, now it's version
0.2, right?
129
0:6:24,96 --> 0:6:25,46
Michael M.: Yep.
130
0:6:25,58 --> 0:6:29,72
Nik: Yeah, and I remember version
0.1, I think there was a claim
131
0:6:29,72 --> 0:6:36,0
that all tests pass, but when I
wrote some silly stuff, like
132
0:6:36,14 --> 0:6:39,86
obviously not SQL syntax, it accepted
it without any errors.
133
0:6:40,02 --> 0:6:40,94
Now it's not so.
134
0:6:40,94 --> 0:6:46,94
In 0.2 it works fine, so it looks
like Postgres syntax is implemented
135
0:6:47,12 --> 0:6:47,88
very well.
136
0:6:48,06 --> 0:6:52,86
Was it about some lack of tests
in Postgres tests for checking
137
0:6:52,86 --> 0:6:53,46
negative stuff?
138
0:6:53,46 --> 0:6:54,3
I don't know.
139
0:6:55,08 --> 0:6:58,42
Or it just wasn't complete implementation
before?
140
0:6:58,44 --> 0:7:1,56
What changed between 0.1 and 0.2?
141
0:7:2,58 --> 0:7:7,86
Michael M.: Yeah, so pgrust 0.1 was
largely rewritten by Opus.
142
0:7:8,44 --> 0:7:13,34
And Opus I found to be a fine model,
but it's a bit difficult
143
0:7:13,34 --> 0:7:13,94
to work with.
144
0:7:13,94 --> 0:7:16,86
They'll mislead you a bit about
how much work is actually completed,
145
0:7:16,86 --> 0:7:19,28
we would ask it to rewrite this
file and it would rewrite 0.5
146
0:7:19,28 --> 0:7:20,8
of the file and not all of it.
147
0:7:20,8 --> 0:7:24,96
And so we were working with this
underlying unreliable model
148
0:7:24,96 --> 0:7:28,84
and all models are unreliable to
various degrees, but with enough
149
0:7:28,84 --> 0:7:33,66
quality controls and layering on
top of it, we're able to get
150
0:7:34,3 --> 0:7:38,52
all of Postgres ported, and we're
able to get the Postgres regression
151
0:7:38,52 --> 0:7:41,6
suite to pass, but that's actually
a really low bar.
152
0:7:42,1 --> 0:7:46,44
The Postgres regression suite on
Postgres only has about 0.667
153
0:7:46,44 --> 0:7:47,22
code coverage.
154
0:7:47,68 --> 0:7:52,02
And so you can pass the regression
suite, but there's still 0.333
155
0:7:52,02 --> 0:7:54,02
of Postgres that isn't even being
tested.
156
0:7:54,52 --> 0:7:58,46
And on top of that, the regression
suites are largely more like
157
0:7:58,46 --> 0:7:59,62
functionality tests.
158
0:7:59,68 --> 0:8:2,72
That every feature in Postgres,
there is a regression suite for
159
0:8:2,72 --> 0:8:5,86
it that's like, there's 1 for hash
joins, there's 1 for lateral
160
0:8:5,86 --> 0:8:10,88
joins, and there's 50,000 of these
that basically just go through
161
0:8:10,88 --> 0:8:13,48
every single Postgres feature and
make sure it works to some
162
0:8:13,48 --> 0:8:13,98
degree.
163
0:8:14,14 --> 0:8:18,28
But they actually just do a couple
tests for each feature and
164
0:8:18,28 --> 0:8:19,34
then move on.
165
0:8:19,34 --> 0:8:22,28
And so the tests are more about
just making sure this feature
166
0:8:22,28 --> 0:8:27,34
is there and it exists and works
well, but doesn't actually really
167
0:8:27,34 --> 0:8:31,5
like you have a perfect implementation
of this feature.
168
0:8:32,46 --> 0:8:36,82
Nik: So you, there is an idea to
improve regression test suite,
169
0:8:36,82 --> 0:8:37,32
right?
170
0:8:37,82 --> 0:8:41,82
And this is already, this should
go to upstream, maybe, no?
171
0:8:42,72 --> 0:8:43,22
Michael M.: Yeah.
172
0:8:43,44 --> 0:8:47,3
So what we've been doing, the difference
between 0.1 and 0.2
173
0:8:47,3 --> 0:8:50,28
is 0.2 was run by Fable, which
we have a more reliable model
174
0:8:50,28 --> 0:8:54,6
now that's, is more consistent
about, okay, you ask it to port
175
0:8:54,6 --> 0:8:56,78
a file, it'll actually rewrite
the file completely.
176
0:8:56,92 --> 0:9:0,06
And so we do end up with something
that is just baseline, more
177
0:9:0,06 --> 0:9:1,02
similar to Postgres.
178
0:9:1,5 --> 0:9:5,5
And then before we released 0.2,
we started doing a little, a
179
0:9:5,5 --> 0:9:8,1
tiny bit of differential testing
and fuzz testing, and I can
180
0:9:8,1 --> 0:9:9,5
talk a lot more about that.
181
0:9:9,52 --> 0:9:14,18
But for 0.3, we're actually like
going to have a lot more of
182
0:9:14,18 --> 0:9:14,68
that.
183
0:9:15,42 --> 0:9:19,58
What we've thought of doing is
we could add more regression tests,
184
0:9:20,08 --> 0:9:24,24
but that, again, every single regression
test is only testing
185
0:9:24,24 --> 0:9:26,6
a small part of the functionality.
186
0:9:27,1 --> 0:9:30,48
And what we really want is not
just, hey, I can choose 100 tests
187
0:9:30,48 --> 0:9:31,8
and they'll all pass.
188
0:9:31,96 --> 0:9:35,24
I want to know is this thing actually
identical to Postgres?
189
0:9:35,84 --> 0:9:39,06
And some of the things we've been
doing, and I was very surprised
190
0:9:39,06 --> 0:9:40,52
by this, that it's even worked.
191
0:9:40,52 --> 0:9:41,74
I would have never thought this.
192
0:9:41,84 --> 0:9:46,22
We found this library called Kani,
which can take Rust code and
193
0:9:46,22 --> 0:9:50,2
C code and do formal verification
over the code.
194
0:9:50,66 --> 0:9:51,76
Nik: Without any execution?
195
0:9:52,2 --> 0:9:53,86
Michael M.: So it does symbolic
execution.
196
0:9:54,4 --> 0:9:57,74
So it'll run your code, but it'll
run it in a special way that
197
0:9:57,74 --> 0:10:0,3
it is more just picking out that,
hey, this is an if statement
198
0:10:0,3 --> 0:10:2,98
that is checking this condition,
as opposed to actually having
199
0:10:2,98 --> 0:10:4,34
variables go through it.
200
0:10:4,34 --> 0:10:6,98
And then I'll be able to convert
that into a format that you
201
0:10:6,98 --> 0:10:10,08
can then do formal verification
over that has the Rust, I'll
202
0:10:10,08 --> 0:10:12,92
have an expression that represents
all the Rust, and I'll have
203
0:10:12,92 --> 0:10:14,34
the same for the C code.
204
0:10:14,44 --> 0:10:17,54
And then we can do across all inputs,
are these 2 things gonna
205
0:10:17,54 --> 0:10:19,52
produce the exact same output?
206
0:10:19,9 --> 0:10:25,3
And so Postgres has 3000 different
user facing functions, everything
207
0:10:25,32 --> 0:10:28,7
from substring to regular expression
matching to there's like
208
0:10:28,7 --> 0:10:32,9
the, like you can calculate the
gamma function, there's a function
209
0:10:32,9 --> 0:10:33,58
for that.
210
0:10:33,68 --> 0:10:36,94
And so for about 1000 of these
functions, we were actually
211
0:10:36,94 --> 0:10:40,72
able to do actual formal verification
that the Rust code and
212
0:10:40,72 --> 0:10:41,86
C code are the same.
213
0:10:42,28 --> 0:10:45,98
But then beyond that, what we've
started, like formal verification
214
0:10:46,06 --> 0:10:47,12
doesn't work in all cases.
215
0:10:47,12 --> 0:10:48,3
It's actually narrow.
216
0:10:48,52 --> 0:10:51,94
And so what we've been doing for
the rest of the code base has
217
0:10:51,94 --> 0:10:55,2
been, we'll take the Rust code
and the C code and put them side
218
0:10:55,2 --> 0:10:59,74
by side, and then have a fuzzer,
like, code coverage guided fuzzer,
219
0:11:0,18 --> 0:11:4,04
generate millions of inputs across
the 2 and make sure that they
220
0:11:4,2 --> 0:11:5,74
behave exactly the same.
221
0:11:6,4 --> 0:11:10,52
And in the process of doing this,
not only did we like find over
222
0:11:10,52 --> 0:11:14,44
100 bugs in like pgrust, but we
actually found at this point
223
0:11:14,44 --> 0:11:17,34
a little bit over 20 bugs in Postgres
itself.
224
0:11:17,52 --> 0:11:17,78
Nik: I see.
225
0:11:17,78 --> 0:11:18,58
That's interesting.
226
0:11:18,74 --> 0:11:19,98
Have you reported them?
227
0:11:20,38 --> 0:11:20,74
Michael M.: Yeah.
228
0:11:20,74 --> 0:11:24,52
If you look, I think like end of
July, like 1st week of August
229
0:11:24,52 --> 0:11:29,28
on the like pgsql or like pgsql
bugs mailing list, like you'll
230
0:11:29,28 --> 0:11:31,98
see a lot of the form submissions
are actually from me.
231
0:11:32,24 --> 0:11:32,8
Nik: That's cool.
232
0:11:32,8 --> 0:11:33,44
I missed it.
233
0:11:33,44 --> 0:11:34,18
That's cool.
234
0:11:34,44 --> 0:11:35,52
Congrats, actually.
235
0:11:35,54 --> 0:11:36,1
That's good.
236
0:11:36,1 --> 0:11:36,6
Yeah.
237
0:11:37,12 --> 0:11:38,2
Michael M.: None of the bugs.
238
0:11:38,52 --> 0:11:43,2
I found 3 serious bugs, but all
of them had been found already
239
0:11:43,2 --> 0:11:43,82
by other people.
240
0:11:43,82 --> 0:11:48,66
I'd been testing on 18.3, 18.4,
and in 19 people had found them.
241
0:11:48,84 --> 0:11:53,16
There were a lot of not serious
bugs that people had not found.
242
0:11:53,76 --> 0:11:57,68
1 of my favorites is Postgres has
a quadtree implementation,
243
0:11:59,34 --> 0:12:2,32
and it will use floating points
to represent the positions in
244
0:12:2,32 --> 0:12:2,94
the quadtree.
245
0:12:3,48 --> 0:12:6,94
And there was a bug where, because
of floating point rounding
246
0:12:7,06 --> 0:12:10,38
and the arithmetic that Postgres
was doing, it was possible for
247
0:12:10,38 --> 0:12:13,94
a point to not be in any of the
4 quadrants.
248
0:12:14,44 --> 0:12:17,68
That the comparison would say,
this isn't to the left, this isn't
249
0:12:17,68 --> 0:12:19,5
to the right, and this isn't in
the center.
250
0:12:20,22 --> 0:12:22,22
And we're definitely doing like
very aggressive.
251
0:12:23,56 --> 0:12:25,86
Nik: This is in the SP-GiST, right?
252
0:12:26,18 --> 0:12:29,08
Michael M.: I think it was in,
I don't know exactly.
253
0:12:30,06 --> 0:12:37,12
I want to say it was in GiST, yeah,
but basically the amount
254
0:12:37,12 --> 0:12:39,72
of testing we're doing is so thorough
that we're actually finding
255
0:12:39,72 --> 0:12:43,66
these extremely hard to find bugs
and we're finding quite a few
256
0:12:43,66 --> 0:12:45,0
bugs in Postgres itself.
257
0:12:45,24 --> 0:12:48,18
Nik: This is very important because
it means that with AI you
258
0:12:48,18 --> 0:12:52,42
can, you basically have some mechanism
which is finding bugs
259
0:12:52,42 --> 0:12:52,8
Michael M.: all the
260
0:12:52,8 --> 0:12:53,58
Nik: time, right?
261
0:12:53,8 --> 0:12:57,54
And it can be maybe like, what
could it mean?
262
0:12:57,74 --> 0:13:2,6
Probably there's Postgres Buildfarm
for testing, which is used for
263
0:13:2,6 --> 0:13:3,3
all releases.
264
0:13:4,3 --> 0:13:8,68
With this work, it maybe should
be extended to involve more AI
265
0:13:8,68 --> 0:13:11,14
and find more bugs at scale, right?
266
0:13:11,2 --> 0:13:13,08
Michael M.: Yeah, there's a couple
things I think are really
267
0:13:13,08 --> 0:13:13,58
interesting.
268
0:13:14,06 --> 0:13:18,02
1 is that with Mythos, this whole
thing about Mythos can pose
269
0:13:18,2 --> 0:13:19,5
a security risk.
270
0:13:19,82 --> 0:13:22,36
The interesting thing about the
models is like, the security,
271
0:13:22,36 --> 0:13:26,0
the way that they pose a security
risk isn't that like, the model
272
0:13:26,0 --> 0:13:29,34
looks at your code and can find
a bug in the code.
273
0:13:29,44 --> 0:13:33,34
It's actually that the model can
build tools that can then find
274
0:13:33,34 --> 0:13:34,34
bugs in your code.
275
0:13:34,4 --> 0:13:36,98
Or like, I've been using Fable
and a lot of times it'll fall
276
0:13:36,98 --> 0:13:41,76
back to Opus 5 because it's, oh,
you found a memory bug, that
277
0:13:42,1 --> 0:13:43,5
seems like a security issue.
278
0:13:43,7 --> 0:13:46,8
Nik: I think, just From my experience,
I also found a couple
279
0:13:46,8 --> 0:13:47,94
of bugs with AI.
280
0:13:49,12 --> 0:13:50,78
A few were security-related.
281
0:13:51,34 --> 0:13:55,28
1 was officially registered and
was already patched.
282
0:13:56,04 --> 0:13:58,4
That 1 was just looking at the
code.
283
0:13:59,28 --> 0:14:1,8
I didn't use Mythos, even not Fable.
284
0:14:1,8 --> 0:14:5,78
It was like Opus 4.5 back then.
285
0:14:6,04 --> 0:14:8,76
And I just was looking at the code,
it was very old code, and
286
0:14:8,76 --> 0:14:12,34
just looking at it, it found SQL
injection, which is there already
287
0:14:12,34 --> 0:14:13,3
18 years.
288
0:14:13,9 --> 0:14:19,5
So in some contrib module, which
is not used directly, so nobody
289
0:14:19,5 --> 0:14:22,78
cared, but it was actually serious
because it's used as an example.
290
0:14:23,56 --> 0:14:28,02
So anyway, it was interesting that
just looking at code, it also
291
0:14:28,02 --> 0:14:28,66
can find bugs.
292
0:14:28,66 --> 0:14:31,7
But you're right, building tools
is even more powerful because
293
0:14:32,18 --> 0:14:36,26
just looking at the code, you can
miss complex relationship between
294
0:14:36,26 --> 0:14:37,78
various code pieces, right?
295
0:14:38,0 --> 0:14:38,4
Michael M.: Yeah.
296
0:14:38,4 --> 0:14:41,26
So with the models, like what you
can do is you can basically
297
0:14:41,26 --> 0:14:44,48
point them to be like, hey, I have
100 functions in this code
298
0:14:44,48 --> 0:14:48,0
base, write really thorough tests
for all of these different
299
0:14:48,28 --> 0:14:49,4
pieces of code.
300
0:14:49,7 --> 0:14:52,7
And so the way I like to think
about it with these models is
301
0:14:52,7 --> 0:14:56,88
you can, if you can get them to
do 1 task well, you now have
302
0:14:56,88 --> 0:15:0,56
a repeatable way to do that 1 task
100 times.
303
0:15:0,72 --> 0:15:3,24
And so if you can get a model to
be able to test 1 function,
304
0:15:3,24 --> 0:15:5,84
there isn't a reason you can't
get it to test 100 or 1000 or
305
0:15:5,84 --> 0:15:6,72
10000 functions.
306
0:15:7,68 --> 0:15:8,18
Nik: Yeah.
307
0:15:8,3 --> 0:15:10,9
How much of AI capacity have you
already used?
308
0:15:10,9 --> 0:15:14,28
Is it like just 1 $200 account
or no?
309
0:15:14,6 --> 0:15:16,66
I'm just curious, very curious.
310
0:15:17,5 --> 0:15:19,5
Asking for a friend, so to speak.
311
0:15:20,32 --> 0:15:23,94
Michael M.: The 1st version of
pgrust, the 0.1, which was done
312
0:15:23,94 --> 0:15:29,44
with Opus, which that spanned 5
different attempts, that cost
313
0:15:29,44 --> 0:15:30,52
about 100 grand.
314
0:15:31,04 --> 0:15:36,96
And then so far, with the new version,
like 0.2 and will become
315
0:15:36,96 --> 0:15:40,24
0.3 in total, we've put in probably
like another like 300 or
316
0:15:40,24 --> 0:15:41,14
400 grand.
317
0:15:41,76 --> 0:15:42,6
Where the...
318
0:15:44,44 --> 0:15:47,56
We're burning tokens so fast that
you can burn through a 5 hour
319
0:15:47,56 --> 0:15:51,18
quota on a $200 a month subscription
on the, in the order of
320
0:15:51,18 --> 0:15:52,08
like 15 minutes.
321
0:15:54,38 --> 0:15:57,26
It's because for a while, like
what we'll do is like, we'll have
322
0:15:57,26 --> 0:15:59,96
a 40 different Fable instances
like running in parallel, each
323
0:15:59,96 --> 0:16:1,82
like working on individual parts
of Postgres.
324
0:16:2,64 --> 0:16:6,02
And that just burns through credits
super fast.
325
0:16:7,74 --> 0:16:8,82
Nik: Yeah, yeah, that's impressive.
326
0:16:9,44 --> 0:16:11,28
So, yeah.
327
0:16:11,28 --> 0:16:13,1
And it takes also a lot of time,
right?
328
0:16:13,1 --> 0:16:16,64
Because like, it's just they are
like, you just need to wait
329
0:16:16,64 --> 0:16:20,68
sometimes a lot right what was
your the longest run or loop
330
0:16:20,68 --> 0:16:24,12
fully autonomous like days weeks
or something
331
0:16:25,16 --> 0:16:30,96
Michael M.: I usually well I've
had stuff like run overnight
332
0:16:31,16 --> 0:16:36,14
that worked but I think I try to
keep things on the order of,
333
0:16:36,14 --> 0:16:39,22
I try to break things down to like
tasks on the order of like,
334
0:16:39,6 --> 0:16:42,56
ideally like an hour, cause any
more than that, I find that like
335
0:16:42,56 --> 0:16:45,2
the chance of the model going off
and doing its own thing and
336
0:16:45,2 --> 0:16:45,8
like getting.
337
0:16:45,8 --> 0:16:46,3
Nik: Nonsensical.
338
0:16:46,5 --> 0:16:47,0
Yeah.
339
0:16:47,2 --> 0:16:47,7
Michael M.: Yeah.
340
0:16:47,9 --> 0:16:50,7
I found that like models, they
work really well.
341
0:16:50,98 --> 0:16:52,2
They enter 2 states.
342
0:16:52,2 --> 0:16:54,72
Like 1 is they like are just like
working really well and making
343
0:16:54,72 --> 0:16:55,54
forward progress.
344
0:16:56,0 --> 0:16:58,68
And then other times they'll just
start doing their own thing.
345
0:16:58,68 --> 0:17:1,16
They'll just start making like
a bunch of random edits to like
346
0:17:1,16 --> 0:17:3,24
code everywhere and not really
go anywhere.
347
0:17:3,24 --> 0:17:6,3
And like the more you can keep
models on like the productive
348
0:17:6,3 --> 0:17:9,44
side and not on the going in circle
side is better.
349
0:17:9,44 --> 0:17:11,96
Michael C.: I'm curious on the,
because that's a lot of money
350
0:17:11,96 --> 0:17:13,52
to spend on v0.2.
351
0:17:14,44 --> 0:17:18,04
Are you being public with how you're
funding it or like if you
352
0:17:18,04 --> 0:17:18,9
do have investors.
353
0:17:19,86 --> 0:17:20,28
Michael M.: Yeah.
354
0:17:20,28 --> 0:17:25,6
So far it's actually been entirely
funded by me where I had a
355
0:17:25,6 --> 0:17:28,86
previous startup, Freshpaint,
which did like pretty well.
356
0:17:28,86 --> 0:17:31,88
And I had the chance to sell some
of my equity and so far it's
357
0:17:31,88 --> 0:17:33,06
mostly been funded by me.
358
0:17:33,06 --> 0:17:35,52
We're hitting the point where like
it's starting to become like
359
0:17:35,74 --> 0:17:38,76
a bit unreasonable and we're starting
to look at other ways to
360
0:17:38,76 --> 0:17:42,16
like fund it and can we like
be more token efficient and things
361
0:17:42,16 --> 0:17:42,84
like that.
362
0:17:43,32 --> 0:17:44,66
Michael C.: That makes a ton of
sense.
363
0:17:44,8 --> 0:17:47,36
So yeah going back to the technical
stuff I was going to
364
0:17:47,36 --> 0:17:51,14
the test suite is really interesting
to me, and almost as a what
365
0:17:51,14 --> 0:17:54,0
can we learn from a Postgres perspective,
as well as how could
366
0:17:54,0 --> 0:17:55,3
pgrust become trustworthy.
367
0:17:55,58 --> 0:18:1,24
So, on testing the functions, I
can imagine a huge amount of
368
0:18:1,24 --> 0:18:5,92
the kind of user surface area is
going to be covered nicely,
369
0:18:5,92 --> 0:18:9,74
like especially for single user,
single query correctness.
370
0:18:10,64 --> 0:18:15,42
But I can't help but feel nervous
about like concurrency stuff
371
0:18:15,42 --> 0:18:19,2
and even Postgres and some other
databases that have had some
372
0:18:19,2 --> 0:18:22,36
interesting bugs come up when Jepsen
have got involved and run
373
0:18:22,36 --> 0:18:26,92
some like interesting tests along
the like isolation level side
374
0:18:26,92 --> 0:18:27,54
of things.
375
0:18:28,28 --> 0:18:31,5
Do you, does your testing approach
cover that yet?
376
0:18:31,56 --> 0:18:34,34
Or do you, are you going to have
needed like a slightly different
377
0:18:34,34 --> 0:18:35,54
approach for some of those things?
378
0:18:35,54 --> 0:18:36,98
How are you thinking about that?
379
0:18:37,58 --> 0:18:40,6
Michael M.: Yeah, we're basically
taking what is the modern and
380
0:18:40,6 --> 0:18:43,8
latest approach to testing and
trying to incorporate all of that
381
0:18:43,92 --> 0:18:45,36
into how we do things.
382
0:18:45,48 --> 0:18:47,68
And the 1st thing we've been doing
is we've started with the
383
0:18:47,68 --> 0:18:49,44
easy part of, okay, here's all
the pure functions.
384
0:18:49,44 --> 0:18:50,58
Here's the ones that we can verify.
385
0:18:50,58 --> 0:18:52,16
Here's the ones that we can just
fuzz.
386
0:18:52,44 --> 0:18:56,32
And then most recently I've been
working on how can we actually
387
0:18:56,32 --> 0:19:0,52
get full coverage of the database
where there's a lot of very
388
0:19:0,52 --> 0:19:5,58
stateful things that will only
trigger bugs in very rare circumstances.
389
0:19:5,68 --> 0:19:7,9
And if you actually look at the
issues people are reporting for
390
0:19:7,9 --> 0:19:11,04
pgrust, it's usually the combination
of 2 features of, oh, if
391
0:19:11,04 --> 0:19:15,04
I use a non-default collation
with certain string functions,
392
0:19:15,06 --> 0:19:16,52
it'll cause a bug.
393
0:19:16,56 --> 0:19:18,96
And so what we're trying to do
now is actually fuzz until we
394
0:19:18,96 --> 0:19:20,7
get to 100% code coverage.
395
0:19:21,14 --> 0:19:24,4
And then for the concurrency stuff,
once we have a fuzzer that
396
0:19:24,4 --> 0:19:29,16
can actually properly explore the
entire code base, we already
397
0:19:29,16 --> 0:19:34,7
purchased this tool called Antithesis,
which does Jepsen-style
398
0:19:35,32 --> 0:19:36,1
fault testing.
399
0:19:36,2 --> 0:19:37,82
It's actually a super cool product.
400
0:19:38,14 --> 0:19:44,04
They run your software inside of
a VM and then are able to inject
401
0:19:44,24 --> 0:19:46,68
faults into your running code.
402
0:19:46,68 --> 0:19:49,3
And so that can be everything from
the network is unreliable
403
0:19:49,3 --> 0:19:51,38
and your database can't talk to
its replica.
404
0:19:51,58 --> 0:19:55,08
Or it can be like, hey, your server
crashed and now is your data
405
0:19:55,08 --> 0:19:56,46
corrupted on disk or not?
406
0:19:56,46 --> 0:19:59,44
And then they actually do stuff
of they can, they have full control
407
0:19:59,44 --> 0:20:0,14
of the threading.
408
0:20:0,14 --> 0:20:1,16
It's fully deterministic.
409
0:20:1,62 --> 0:20:5,42
And so they can create all these
weird out of order sequences
410
0:20:5,66 --> 0:20:9,4
for your threads to try to just
throw the craziest stuff at your
411
0:20:9,4 --> 0:20:9,9
software.
412
0:20:10,2 --> 0:20:12,68
And right now we're working on
the fuzzer to actually explore
413
0:20:12,9 --> 0:20:13,58
the search space.
414
0:20:13,58 --> 0:20:15,98
And then once we have that, we're
going to give that to Antithesis
415
0:20:16,0 --> 0:20:18,9
to then try to come up with all
these crazy ways of running the
416
0:20:18,9 --> 0:20:21,6
code together to try to break
pgrust.
417
0:20:21,6 --> 0:20:24,14
And so I think that is, we've actually
been talking with a lot
418
0:20:24,14 --> 0:20:26,64
of database people through this
and basically they're all using
419
0:20:26,64 --> 0:20:29,48
Antithesis for their testing at
this point.
420
0:20:29,96 --> 0:20:30,3
Nik: Yeah.
421
0:20:30,3 --> 0:20:34,0
Speaking of testing, I remember
I noticed on Hacker News comments
422
0:20:34,0 --> 0:20:39,18
that on the thread that people
asked questions about was fsync
423
0:20:39,4 --> 0:20:43,44
on and did you do writes in those
tests and so on or is just
424
0:20:43,44 --> 0:20:47,42
only from memory, data fits in
buffer pool or something and
425
0:20:47,42 --> 0:20:48,9
nothing touches the disk.
426
0:20:49,2 --> 0:20:49,4
Michael M.: Yeah.
427
0:20:49,4 --> 0:20:52,0
So we have, if you look at the
code base, we actually have a
428
0:20:52,0 --> 0:20:55,44
crash simulator in pgrust that
mocks out the file system and
429
0:20:55,44 --> 0:20:57,8
then we'll try to just crash the
database at different points.
430
0:20:57,8 --> 0:20:59,44
And we've done a good amount of
testing.
431
0:20:59,6 --> 0:21:3,06
The thing is that even Postgres
does get these things wrong sometimes.
432
0:21:3,08 --> 0:21:6,76
That there was like a fsyncgate
almost like a decade ago now,
433
0:21:6,76 --> 0:21:10,32
I think, of like, yeah, fsync can
actually fail and Postgres
434
0:21:10,32 --> 0:21:11,78
actually didn't handle that case.
435
0:21:11,82 --> 0:21:15,22
Which be fair to Postgres, that's
a case that never ever happens.
436
0:21:15,58 --> 0:21:20,2
Nik: Honestly, I also think to
me going to Jepsen tests and multi-node
437
0:21:20,28 --> 0:21:24,66
basically network testing and so
on is maybe not the biggest
438
0:21:24,66 --> 0:21:26,08
priority for this maybe.
439
0:21:26,26 --> 0:21:31,16
In my opinion, I recently learned
that Postgres own tests are not
440
0:21:31,16 --> 0:21:34,26
checking properly disk outages,
for example.
441
0:21:35,24 --> 0:21:39,72
So this is like, what happens if
disk just has some problems?
442
0:21:40,46 --> 0:21:44,94
And there are some tools to improve
that in Postgres on codebase.
443
0:21:45,9 --> 0:21:46,94
So I'm very curious.
444
0:21:47,08 --> 0:21:49,64
This is purely a vibe-coded thing.
445
0:21:49,9 --> 0:21:54,86
You obviously created a good harness
to mock and so on, but still
446
0:21:54,86 --> 0:21:56,42
people reported some segfaults.
447
0:21:56,54 --> 0:21:58,16
It's obvious it's early stage.
448
0:21:58,42 --> 0:22:1,36
Will it reach some point when it's
very reliable so it can be
449
0:22:1,36 --> 0:22:2,2
put to production.
450
0:22:2,2 --> 0:22:5,68
So obviously you say yes, but what's
your like, why do you think
451
0:22:5,68 --> 0:22:5,9
so?
452
0:22:5,9 --> 0:22:7,0
Why do you think yes?
453
0:22:7,1 --> 0:22:9,56
Because I'm also very AI positive,
so to speak.
454
0:22:9,56 --> 0:22:13,34
I think actually in a few months
we will be able to revisit with
455
0:22:13,34 --> 0:22:15,0701
new models and polish it and so
on.
456
0:22:15,0701 --> 0:22:20,16
And It gives me so joy to work
with AI together, like human plus
457
0:22:20,16 --> 0:22:22,12
AI, we can achieve a lot.
458
0:22:22,36 --> 0:22:25,22
But still, there are so many doubts
around.
459
0:22:26,0 --> 0:22:30,56
And people think, for example,
it's impossible to achieve a point
460
0:22:30,66 --> 0:22:32,38
where it will be reliable.
461
0:22:32,96 --> 0:22:35,28
You started with reliability, right?
462
0:22:35,74 --> 0:22:40,16
Then you said hackers are too conservative,
but there's also
463
0:22:40,16 --> 0:22:43,7
they are too conservative to protect
that reliability thing,
464
0:22:43,7 --> 0:22:44,2
right?
465
0:22:44,28 --> 0:22:48,08
Now with vibe code, the thing fully
rewritten, won't it take
466
0:22:48,08 --> 0:22:51,44
10 years to achieve a very reliable
state?
467
0:22:51,82 --> 0:22:54,62
What makes you think it's possible
to achieve it much faster
468
0:22:54,62 --> 0:22:55,5
than 10 years?
469
0:22:56,74 --> 0:22:58,86
Michael M.: So I think for the
issues that people are reporting,
470
0:22:58,86 --> 0:23:2,8
I think 1 thing to clarify is that
so far for version 0.2, we
471
0:23:2,8 --> 0:23:6,22
had, or actually I think for version
0.2, that was actually before
472
0:23:6,22 --> 0:23:10,76
we had done even like most
of the fuzz testing I was talking
473
0:23:10,76 --> 0:23:12,84
about, where like we actually sorted
out those issues.
474
0:23:13,04 --> 0:23:16,06
The correctness work has like largely
followed version 0.2 and
475
0:23:16,06 --> 0:23:18,8
so far we've only covered like
15 or 20 percent of the codebase
476
0:23:18,8 --> 0:23:20,42
and like there's still a lot more
to cover.
477
0:23:20,46 --> 0:23:23,04
And so I'm like I'm totally not
surprised that people found issues
478
0:23:23,04 --> 0:23:25,08
in the version we released just
because like we hadn't actually
479
0:23:25,08 --> 0:23:26,62
hardened it that much.
480
0:23:26,84 --> 0:23:29,34
And to answer your question of
how we can actually get this to
481
0:23:29,34 --> 0:23:32,92
a place where people can trust
this, Back to what I said earlier
482
0:23:32,92 --> 0:23:36,66
about, it's not like the models
themselves are like reviewing
483
0:23:36,66 --> 0:23:38,66
all the code and like making sure
it's correct.
484
0:23:38,74 --> 0:23:42,1
What we're doing is like we're
building tools to actually make
485
0:23:42,1 --> 0:23:43,4
it, make sure it's correct.
486
0:23:43,48 --> 0:23:46,84
And we're doing it such a degree
that like we're able to do millions
487
0:23:46,84 --> 0:23:50,24
of inputs for individual functions
to the point that Postgres,
488
0:23:50,66 --> 0:23:53,86
this 30-year-old battle-tested
codebase, we're finding bugs in
489
0:23:53,86 --> 0:23:56,52
it that no 1 had found before,
even though people have been using
490
0:23:56,52 --> 0:23:57,94
it all this time.
491
0:23:58,04 --> 0:24:1,28
Nik: You find bugs in the old code,
but Maybe you have bugs which
492
0:24:1,28 --> 0:24:3,46
you haven't explored in new code
yet.
493
0:24:4,06 --> 0:24:5,88
So this is a really tricky question.
494
0:24:6,38 --> 0:24:8,86
What's your plan to prove it in
production?
495
0:24:9,02 --> 0:24:13,0
Just wait for people who will try
it in production on replicas,
496
0:24:13,0 --> 0:24:16,56
for example, because I always think
This thing can work as a
497
0:24:16,56 --> 0:24:17,46
physical standby.
498
0:24:18,22 --> 0:24:22,2
Is this a way or use some mirroring,
for example, PgDog has or
499
0:24:22,2 --> 0:24:26,28
other, maybe some others have,
to mirror traffic and see that
500
0:24:26,28 --> 0:24:27,18
it works well?
501
0:24:27,18 --> 0:24:30,48
Or what's your plan to prove with
production that it works?
502
0:24:30,84 --> 0:24:31,12
Michael M.: Yeah.
503
0:24:31,12 --> 0:24:34,54
So for proving it like actually
in production, our plan is to
504
0:24:34,54 --> 0:24:37,2
do exactly what you said and replicate
off of people's existing
505
0:24:37,2 --> 0:24:40,16
databases and be like read-only
standby.
506
0:24:40,68 --> 0:24:43,58
That gives people a lot of the
benefits of like what we're building
507
0:24:43,58 --> 0:24:47,9
in terms of this like ultra fast
columnar workload such that
508
0:24:47,9 --> 0:24:50,66
like people who are trying to do
analytics in Postgres and are
509
0:24:50,66 --> 0:24:53,72
struggling to do it, they can stand
up pgrust and then move
510
0:24:53,72 --> 0:24:55,12
to analytics over to that.
511
0:24:55,12 --> 0:24:58,24
And the really nice thing about
analytics is they tend to be
512
0:24:58,38 --> 0:25:2,32
like semi-production workloads
where either like it's internal
513
0:25:2,32 --> 0:25:5,5
usage and okay your database goes
down, okay my internal dashboards
514
0:25:5,5 --> 0:25:8,6
are broken for a little bit, it's
not the end of the world.
515
0:25:8,94 --> 0:25:12,52
And then over time as like we start
to get like battle tested
516
0:25:12,52 --> 0:25:16,86
in these not tier 1 database use
cases, that's I think, will
517
0:25:16,86 --> 0:25:18,96
give people more confidence that,
hey, this is actually something
518
0:25:18,96 --> 0:25:20,64
that, like, works well.
519
0:25:21,02 --> 0:25:24,7
Nik: So what I like in this idea
is that if it's, if pgrust
520
0:25:24,86 --> 0:25:29,48
is powering a physical standby,
and then it crashed, for example,
521
0:25:30,14 --> 0:25:32,22
but primary won't be affected,
right?
522
0:25:32,22 --> 0:25:34,7
Because, okay, some slot is not
used.
523
0:25:35,66 --> 0:25:40,24
And if it's working and you run
very long running query on it,
524
0:25:40,24 --> 0:25:44,76
usually it's a bad idea on regular
standbys because of hot_standby_feedback
525
0:25:44,76 --> 0:25:45,86
dilemma.
526
0:25:46,38 --> 0:25:49,64
If it's on, it will affect vacuum
on the primary.
527
0:25:49,64 --> 0:25:52,8
If it's off, it makes your node
basically single user.
528
0:25:52,8 --> 0:25:53,9
It will start lagging.
529
0:25:54,64 --> 0:25:58,94
So both choices are not okay if
you want analytics.
530
0:25:59,54 --> 0:26:2,86
But It's only true if those queries
last hours.
531
0:26:3,94 --> 0:26:8,54
If it's already, as your blog post
suggests, 300 times faster,
532
0:26:9,4 --> 0:26:14,14
then xmin horizon is not blocked
by hot_standby_feedback for long.
533
0:26:14,14 --> 0:26:15,7
It's blocked for a short time.
534
0:26:16,1 --> 0:26:17,36
This is very interesting.
535
0:26:17,78 --> 0:26:19,52
This looks promising.
536
0:26:20,64 --> 0:26:24,56
We have very fast standby, which
behaves exactly like Postgres.
537
0:26:25,6 --> 0:26:28,16
Like, primary thinks it's like
Postgres, okay?
538
0:26:28,74 --> 0:26:34,24
And we have ability to run some
aggregates and so on very fast.
539
0:26:34,74 --> 0:26:35,58
Michael M.: That's interesting.
540
0:26:35,92 --> 0:26:39,08
Actually, the thing I find really
interesting about pgrust is
541
0:26:39,08 --> 0:26:43,18
there's actually a whole bunch
of unexplored design space, new
542
0:26:43,18 --> 0:26:45,52
ways that you can do things.
543
0:26:45,9 --> 0:26:49,56
Where you mentioned, for instance,
when you have a replica, due
544
0:26:49,56 --> 0:26:54,44
to the way the WAL replay works,
there's all these like complexities
545
0:26:54,52 --> 0:26:57,74
about how did the vacuums work
and how does like the replica
546
0:26:57,74 --> 0:26:58,66
like fall behind.
547
0:26:58,66 --> 0:27:0,4
But I think there are options out
there.
548
0:27:0,4 --> 0:27:2,64
I haven't explored them thoroughly,
but there's options out there.
549
0:27:2,64 --> 0:27:5,64
If we change the file format and
the way the WAL replay works,
550
0:27:5,74 --> 0:27:8,2
maybe there's ways to actually
get around these problems.
551
0:27:8,74 --> 0:27:11,16
I think this is like a lot of what's
really exciting about pgrust
552
0:27:11,16 --> 0:27:14,24
is like, there's a lot of
like unexplored design area.
553
0:27:14,64 --> 0:27:17,84
And because it's this like experimental
new thing, there's actually
554
0:27:17,84 --> 0:27:21,24
lots of opportunity to explore
the design space.
555
0:27:22,48 --> 0:27:24,18
Nik: Right, but that's for sure.
556
0:27:24,52 --> 0:27:27,38
So experimental, you can go very
far.
557
0:27:27,74 --> 0:27:32,06
But grounding is when you put it
to production.
558
0:27:33,66 --> 0:27:36,84
This is the most interesting practical
piece here for me.
559
0:27:37,12 --> 0:27:40,9
Sounds like it's relatively low
risk to try it.
560
0:27:41,0 --> 0:27:42,08
There's a way, right?
561
0:27:42,44 --> 0:27:43,22
That's interesting.
562
0:27:44,12 --> 0:27:45,92
So, Yeah, that's cool.
563
0:27:45,92 --> 0:27:46,7
What else?
564
0:27:46,92 --> 0:27:49,68
I saw your blog post, I think
today, right?
565
0:27:49,68 --> 0:27:51,9
About JIT just in time.
566
0:27:52,04 --> 0:27:53,68
What attracted your attention?
567
0:27:53,68 --> 0:27:57,38
It was just switched off by default
finally, recently.
568
0:27:58,26 --> 0:27:59,62
Was this like, yeah.
569
0:28:0,1 --> 0:28:0,6
Michael M.: Yeah.
570
0:28:1,12 --> 0:28:4,2
As part of the Columnar stuff,
we looked at a bunch of the recent
571
0:28:4,2 --> 0:28:7,86
database literature and incorporated
a ton of stuff into pgrust.
572
0:28:8,36 --> 0:28:12,56
A lot of it actually coming from
the research into Umbra, which
573
0:28:12,56 --> 0:28:16,08
is this very experimental database
being developed by this team
574
0:28:16,08 --> 0:28:16,64
in Munich.
575
0:28:16,64 --> 0:28:19,2
And they've been publishing database
research for the last decade
576
0:28:19,2 --> 0:28:21,26
or more, and a lot of it comes
from them.
577
0:28:21,74 --> 0:28:25,44
But for JIT compilation, and this
is a lot of what I talked about
578
0:28:25,44 --> 0:28:28,18
in the post that came out this
morning, if you look at all the
579
0:28:28,18 --> 0:28:32,52
databases that have JIT compilation
in it, they either use LLVM,
580
0:28:33,14 --> 0:28:37,36
or they'll write C and C++ code
and then compile that.
581
0:28:37,36 --> 0:28:40,28
And the big downside of these approaches
is that there's actually
582
0:28:40,28 --> 0:28:44,72
a huge amount of latency overhead
to compile this code that for
583
0:28:44,72 --> 0:28:49,26
C and C++ or for LLVM, I think it's
50 milliseconds compile time.
584
0:28:49,4 --> 0:28:52,8
And so you're very limited in the
cases that you can use it.
585
0:28:52,8 --> 0:28:56,26
And you have this risk of if you're
the planner estimates the
586
0:28:56,26 --> 0:28:59,4
query wrong, you now have this extra
50 milliseconds of latency.
587
0:29:0,06 --> 0:29:4,84
But there was this approach to
JIT compilation called copy and
588
0:29:4,84 --> 0:29:7,64
I think it's like copy and fix
or something along those lines
589
0:29:7,64 --> 0:29:11,88
that was released in someone published
a paper on it in 2021
590
0:29:11,92 --> 0:29:13,94
that makes copy and patch.
591
0:29:13,94 --> 0:29:14,84
Yeah, that's it.
592
0:29:14,86 --> 0:29:16,72
Then it's much easier to write.
593
0:29:16,72 --> 0:29:19,28
Then you just have templates of
assembly code and then you just
594
0:29:19,28 --> 0:29:20,8
fill in the bits as needed.
595
0:29:20,8 --> 0:29:24,38
You can combine these templates
together to get the code you
596
0:29:24,38 --> 0:29:24,88
want.
597
0:29:24,96 --> 0:29:28,62
And so the pgrust JIT compiler
actually doesn't use something
598
0:29:28,62 --> 0:29:29,26
like LLVM.
599
0:29:29,26 --> 0:29:32,06
It actually just directly generates
the assembly.
600
0:29:32,42 --> 0:29:37,26
And so because of this, the compile
times are like 5 microseconds
601
0:29:37,84 --> 0:29:42,18
as opposed to 50 milliseconds for
LLVM.
602
0:29:42,56 --> 0:29:47,36
And this gives us a lot more, Like
we can JIT compile a lot more
603
0:29:47,36 --> 0:29:51,34
of the query that we actually,
for our executor, we JIT compile
604
0:29:51,34 --> 0:29:55,26
large parts of the query, not just
expressions, which Postgres
605
0:29:55,26 --> 0:29:59,44
will only JIT compile expressions
and tuple deforming, but we
606
0:29:59,44 --> 0:30:1,78
actually do a lot more than that.
607
0:30:1,88 --> 0:30:5,28
And so we get measurable speed
ups in many different places.
608
0:30:6,68 --> 0:30:10,56
Michael C.: Do you see any reason
Postgres couldn't use a similar
609
0:30:10,56 --> 0:30:11,06
approach?
610
0:30:12,26 --> 0:30:16,56
Michael M.: So 1 of the big, 1
of the challenges I see with Postgres
611
0:30:16,56 --> 0:30:22,0
adopting this approach is I Specifically
am targeting only the Graviton
612
0:30:22,2 --> 0:30:27,56
instruction set Whereas pgrust
is being built in a very different
613
0:30:27,56 --> 0:30:30,34
world where today most people are
running their databases in
614
0:30:30,34 --> 0:30:34,04
the cloud And for instance, there's
no Windows support in pgrust
615
0:30:34,04 --> 0:30:34,96
yet.
616
0:30:35,34 --> 0:30:38,42
And so I'm able to, because I know,
I'm just assuming that people
617
0:30:38,42 --> 0:30:42,04
are going to run this primarily
on AWS and GCP, I'm able to target
618
0:30:42,04 --> 0:30:46,14
1 specific CPU design and 1 specific
instruction set and focus
619
0:30:46,26 --> 0:30:48,76
just on that and make that work
really well.
620
0:30:50,28 --> 0:30:52,4
And for Postgres, I think it's
still possible.
621
0:30:52,44 --> 0:30:54,92
You could have the JIT compiler
only work in certain instruction
622
0:30:54,92 --> 0:31:0,26
sets, or you can have different
backends for different instruction
623
0:31:0,28 --> 0:31:0,6
sets.
624
0:31:0,6 --> 0:31:4,2
But it's an order of magnitude
more work to get that to work
625
0:31:4,2 --> 0:31:5,96
well than what I'm doing.
626
0:31:6,06 --> 0:31:9,38
Nik: And if, even if it's not in
cloud, they will run it on Mac
627
0:31:9,38 --> 0:31:11,14
Minis they have now, right?
628
0:31:14,72 --> 0:31:16,16
Should work anyway there, Yeah.
629
0:31:16,16 --> 0:31:19,2
Michael M.: I've seen multiple
people with like rooms of like,
630
0:31:19,2 --> 0:31:19,92
they'll have
631
0:31:20,8 --> 0:31:21,54
Nik: a bunch of...
632
0:31:21,54 --> 0:31:22,04
Yeah.
633
0:31:22,2067 --> 0:31:23,04
In the corner.
634
0:31:23,48 --> 0:31:23,8
Could be a good Postgres cluster.
635
0:31:23,8 --> 0:31:23,82
Yeah.
636
0:31:23,82 --> 0:31:24,84
So that's interesting.
637
0:31:25,46 --> 0:31:28,44
And you don't use GPT like Codex,
no?
638
0:31:28,44 --> 0:31:30,58
Like distributing workloads within?
639
0:31:31,52 --> 0:31:34,84
Michael M.: So in my experience,
actually like the, a lot of
640
0:31:34,84 --> 0:31:39,48
the early work we were doing with
Codex, like Codex 5.4 and Codex
641
0:31:39,48 --> 0:31:44,44
5.5, the thing that made me switch
to Claude is they released
642
0:31:44,44 --> 0:31:47,96
this feature called dynamic workflows,
which this is what was
643
0:31:47,96 --> 0:31:49,86
used for the Bun Rust rewrite.
644
0:31:50,14 --> 0:31:56,1
And dynamic workflows are, you
can basically, it will write code
645
0:31:56,14 --> 0:31:58,62
that orchestrates a bunch of different
agents.
646
0:31:58,94 --> 0:32:1,48
And so you can be like, Hey, I
have these 10 files.
647
0:32:1,48 --> 0:32:3,06
Can you rewrite all of them?
648
0:32:3,18 --> 0:32:7,2
And it'll write some JavaScript,
they'll then spin up 10 agents,
649
0:32:7,2 --> 0:32:10,58
they'll then rewrite each file
and then review it and fix any
650
0:32:10,58 --> 0:32:12,34
issues and then merge it.
651
0:32:12,34 --> 0:32:16,02
And so that level of orchestration
made it way easier to use
652
0:32:16,02 --> 0:32:16,52
Claude.
653
0:32:16,98 --> 0:32:21,8
And then in my experience, like
Fable just works, it's like way
654
0:32:21,8 --> 0:32:23,74
more reliable and trustworthy.
655
0:32:24,38 --> 0:32:28,22
1 of the actually like biggest
challenges I had with the rewrite,
656
0:32:28,48 --> 0:32:31,34
or like 1 of the things that's
actually really hard, was the
657
0:32:31,34 --> 0:32:33,06
Postgres regular expression engine.
658
0:32:33,4 --> 0:32:35,42
I don't know what it is about it.
659
0:32:35,8 --> 0:32:38,04
I think it's 10,000 lines of code.
660
0:32:38,04 --> 0:32:40,82
It's a decent amount of code, but
not too big.
661
0:32:41,0 --> 0:32:45,1
And I was doing this as a benchmark,
and I threw Sol at it on
662
0:32:45,1 --> 0:32:48,02
medium-level thinking, not the
highest thinking, but on a decent...
663
0:32:48,14 --> 0:32:52,68
Sol's a pretty good model, And
Sol wasn't able to rewrite it
664
0:32:52,68 --> 0:32:53,22
in Rust.
665
0:32:53,22 --> 0:32:54,44
I was like very surprised by that.
666
0:32:54,44 --> 0:32:57,98
I was like fully expecting that
given how smart Sol is, like
667
0:32:57,98 --> 0:32:59,94
it should just be able to do that.
668
0:33:0,06 --> 0:33:2,64
Whereas Fable was like a whole
level above that, where it was
669
0:33:2,64 --> 0:33:5,58
able to rewrite in Rust and then
it was able to do like performance
670
0:33:5,58 --> 0:33:9,62
optimization on it to make it a
tiny bit faster than the Postgres
671
0:33:9,62 --> 0:33:10,12
version.
672
0:33:10,13 --> 0:33:15,84
And so like, I think like it probably
is possible to use like
673
0:33:15,84 --> 0:33:19,28
Sol, but like I've just found the
difference in autonomy between
674
0:33:19,28 --> 0:33:23,42
the 2 models is like not worth
the difference.
675
0:33:24,92 --> 0:33:27,76
Michael C.: I think we need to
talk about licensing.
676
0:33:28,44 --> 0:33:29,06
I noticed
677
0:33:29,06 --> 0:33:29,62
Nik: you picked
678
0:33:29,62 --> 0:33:30,74
Michael C.: the AGPL.
679
0:33:31,58 --> 0:33:35,28
That's always controversial, but
at least open source, right?
680
0:33:35,28 --> 0:33:36,38
What, why pick that 1?
681
0:33:36,38 --> 0:33:37,98
I think I know why, but why?
682
0:33:38,8 --> 0:33:41,64
Michael M.: Yeah, we were looking
at the different options we
683
0:33:41,64 --> 0:33:42,6
had for licensing.
684
0:33:42,98 --> 0:33:44,6
And there's like 4 different options.
685
0:33:44,68 --> 0:33:46,92
There's you have MIT or like Postgres.
686
0:33:47,26 --> 0:33:51,66
You have like AGPL or like GPL,
where it's like the copyleft.
687
0:33:51,94 --> 0:33:54,96
You have what they call like source
available, like you have
688
0:33:54,96 --> 0:33:58,22
SSPL and like BSL, and you have
like closed source.
689
0:33:58,68 --> 0:34:2,16
And We knew we wanted to do something
open source, and it's now
690
0:34:2,16 --> 0:34:6,6
limited to the permissive or the
AGPL.
691
0:34:8,04 --> 0:34:13,54
And the thing was, we wanted something
that a third party couldn't
692
0:34:13,54 --> 0:34:17,64
just take and sell themselves to
the detriment of the work we're
693
0:34:17,64 --> 0:34:18,14
doing.
694
0:34:19,62 --> 0:34:23,26
Where there's lots of closed source
forks of Postgres.
695
0:34:23,26 --> 0:34:28,12
You have AlloyDB, you have Aurora,
now Neon is closed source.
696
0:34:28,14 --> 0:34:31,92
And none of that money that those
platforms make actually goes
697
0:34:31,92 --> 0:34:35,46
back, or like very, very tiny amounts
of it goes back to the
698
0:34:35,46 --> 0:34:37,06
actual Postgres core projects.
699
0:34:37,22 --> 0:34:40,12
And that was just something that
like I wanted to avoid with
700
0:34:40,12 --> 0:34:40,84
pgrust.
701
0:34:40,84 --> 0:34:44,06
Nik: But AGPL
doesn't protect you from selling.
702
0:34:44,06 --> 0:34:46,9
Anyone can still sell if they don't
modify code.
703
0:34:47,24 --> 0:34:50,28
And even if they modify code, they
just need to publish it.
704
0:34:50,28 --> 0:34:51,0
That's it.
705
0:34:51,74 --> 0:34:52,24
Michael M.: Yeah.
706
0:34:53,1 --> 0:34:53,32
Yeah.
707
0:34:53,32 --> 0:34:55,64
So at least the history for AGPL
is it is like extremely, basically
708
0:34:55,64 --> 0:34:56,88
no 1 has really done that.
709
0:34:56,88 --> 0:35:0,06
But like just the cost of needing
to open source your modifications
710
0:35:0,06 --> 0:35:5,78
to it has deterred like Amazon
and Google from offering any AGPL
711
0:35:5,78 --> 0:35:8,04
software on their platforms, at
least so far.
712
0:35:8,14 --> 0:35:8,76
Nik: Makes sense.
713
0:35:8,76 --> 0:35:8,92
So
714
0:35:8,92 --> 0:35:10,62
Michael M.: that seemed like the
right trade-off.
715
0:35:11,54 --> 0:35:12,04
Nik: Good.
716
0:35:12,44 --> 0:35:15,04
I know we're like almost out of
time.
717
0:35:15,1 --> 0:35:16,5
It was an interesting discussion.
718
0:35:16,5 --> 0:35:17,58
Thank you for coming, Michael.
719
0:35:17,58 --> 0:35:18,82
I enjoyed it very well.
720
0:35:18,82 --> 0:35:19,84
Good luck with your project.
721
0:35:19,84 --> 0:35:21,86
We will be keeping an eye on it.
722
0:35:22,06 --> 0:35:22,94
Michael M.: Yeah, of course.
723
0:35:23,0 --> 0:35:23,86
Thanks so much.
724
0:35:24,48 --> 0:35:24,84
Nik: Thank you.
