DBOS

1
00:00:00,049 --> 00:00:03,739
Michael: Hello and welcome to Postgres.FM,
a weekly show about all things PostgreSQL.

2
00:00:03,799 --> 00:00:05,250
I am Michael, founder of pgMustard.

3
00:00:05,259 --> 00:00:08,389
I'm joined as always by
Nik, founder of Postgres.ai.

4
00:00:08,529 --> 00:00:08,859
Hey, Nik

5
00:00:09,658 --> 00:00:10,228
Nik: Hi, Michael

6
00:00:10,697 --> 00:00:13,417
Michael: And we have two
special guests with us today.

7
00:00:13,617 --> 00:00:19,227
We have Qian, who is CEO and
co-founder of DBOS, and Peter,

8
00:00:19,537 --> 00:00:21,387
CTO and co-founder of DBOS.

9
00:00:21,777 --> 00:00:22,297
Welcome both

10
00:00:24,267 --> 00:00:24,667
Qian: Thank you

11
00:00:25,693 --> 00:00:30,453
Michael: I wonder if one of you could
give us a bit of the backstory of DBOS.

12
00:00:30,603 --> 00:00:31,743
I think it's quite a cool story

13
00:00:32,443 --> 00:00:32,683
Qian: Yeah.

14
00:00:32,743 --> 00:00:35,723
So we started DBOS as a research project.

15
00:00:36,153 --> 00:00:42,137
I think it was started quite in the middle
of pandemic in summer 2020, when Professor

16
00:00:42,137 --> 00:00:48,957
Mike Stonebraker heard a talk from Matei
Zaharia, the profes- our advisors and

17
00:00:48,957 --> 00:00:53,727
also co-founder of Databricks, where they
were talking about how Databricks were u-

18
00:00:53,767 --> 00:00:59,117
was using Postgres to do some scheduling
work because, distributed scheduling

19
00:00:59,137 --> 00:01:01,217
and cluster management was a mess.

20
00:01:01,537 --> 00:01:04,847
And they were talking about how to
use a database to simplify that.

21
00:01:05,297 --> 00:01:10,477
And then I think during that summer we
talked and discussed, like, how we can

22
00:01:10,667 --> 00:01:17,027
u- extend that idea to make database
more helpful to build, reliable systems

23
00:01:17,407 --> 00:01:19,407
to even replace the operating system.

24
00:01:19,717 --> 00:01:23,920
So that's how we started the original
research project called DBOS.

25
00:01:24,700 --> 00:01:30,150
And then Peter and I were the co-leads PhD
students working on that research project.

26
00:01:30,200 --> 00:01:33,730
We published several
papers, did a bunch of work.

27
00:01:34,160 --> 00:01:39,490
And then what we found was that, the most
interesting thing was actually to use the

28
00:01:39,500 --> 00:01:44,550
database as a foundation for distributed
systems, to build distributed systems.

29
00:01:44,850 --> 00:01:50,710
And that's why when we both graduated
in 2023 we decided to do, to

30
00:01:50,710 --> 00:01:52,990
commercialize this idea, to do a startup.

31
00:01:53,190 --> 00:01:57,080
And now we are working on
DBOS Inc., the company.

32
00:01:57,778 --> 00:02:03,314
Nik: that's cool And I must admit
I observed queues in Postgres

33
00:02:03,474 --> 00:02:06,134
for so long, like 20 plus years.

34
00:02:06,814 --> 00:02:12,676
And somehow when DBOS DBOS was
announced, of course it it attracted

35
00:02:12,676 --> 00:02:17,508
my attention, but I didn't quite get
it like Because like w-w-workflow,

36
00:02:17,518 --> 00:02:19,158
like what i-what is it, right?

37
00:02:19,768 --> 00:02:24,118
But then I started to realize,
actually, to realize that in any

38
00:02:24,118 --> 00:02:30,588
business there are workflows, and to
keep the state and all the history and

39
00:02:30,588 --> 00:02:32,748
so on in database, it's very natural.

40
00:02:33,398 --> 00:02:34,958
So this is a very good idea.

41
00:02:35,458 --> 00:02:40,748
And I think it's, it's-- it revives
the idea that queue in that database

42
00:02:40,748 --> 00:02:43,858
is not only possible, but it's
possible in quite efficient way.

43
00:02:44,328 --> 00:02:47,868
But you need to… With Postgres,
you need to be very careful, right?

44
00:02:48,228 --> 00:02:53,388
Because of some, some things that
might happen if you implement

45
00:02:53,428 --> 00:02:55,188
queues and workflows in naive way.

46
00:02:55,478 --> 00:02:58,610
But I don't quite get it
still why operational system.

47
00:02:59,230 --> 00:03:05,090
I understand when Andrei Karpaty
talks about like AI as a CPU in

48
00:03:05,090 --> 00:03:10,140
the center, like memory, like every
p-peripheral devices, everything.

49
00:03:10,530 --> 00:03:14,660
I understand this analogy, but here
I don't quite get the analogy why

50
00:03:14,700 --> 00:03:19,470
is there is some intent or idea to
implement whole works operational

51
00:03:19,470 --> 00:03:21,580
system in, in da-database.

52
00:03:22,130 --> 00:03:23,680
Like what's your view?

53
00:03:24,264 --> 00:03:24,604
Peter: Yeah.

54
00:03:24,624 --> 00:03:30,174
So what we're doing as a company is
using the database as a foundation

55
00:03:30,184 --> 00:03:33,914
for distributed workflows, and then
for things around the distributed

56
00:03:33,964 --> 00:03:37,794
workflows like queues, like messaging,
like notifications, and so on.

57
00:03:38,461 --> 00:03:45,221
I think the core idea here is that
as your program runs, you regularly

58
00:03:45,231 --> 00:03:50,951
checkpoint the state of that program
in a Postgres database so that if the

59
00:03:50,951 --> 00:03:55,211
program ever fails, if anything happens
to it, if it crashes, if it's interrupted,

60
00:03:55,541 --> 00:03:58,751
it can reload those checkpoints to
recover from exactly where it left off.

61
00:03:59,111 --> 00:04:03,647
And that, durable workflows idea backed
by Postgres is the core idea of DBOS,

62
00:04:03,777 --> 00:04:10,397
using Postgres as an orchestrator to
make durable workflows that run really

63
00:04:10,547 --> 00:04:13,047
w- that run with high performance
and that are really simple because

64
00:04:13,047 --> 00:04:15,777
all they have is, all they have,
all they rely on is your database

65
00:04:17,344 --> 00:04:18,224
Nik: Yeah, this makes sense.

66
00:04:18,374 --> 00:04:19,414
I understand that.

67
00:04:19,804 --> 00:04:24,474
I'm just still processing the idea.

68
00:04:24,474 --> 00:04:25,904
It's like whole operational system.

69
00:04:26,682 --> 00:04:30,442
Qian: I think our goal is that when
we think about operating system,

70
00:04:30,492 --> 00:04:33,822
we should not be limited to the
operating system we're running

71
00:04:33,822 --> 00:04:36,032
our, on our laptop or desktop.

72
00:04:36,432 --> 00:04:40,052
Thinking like anything operational,
I think Kubernetes can be called

73
00:04:40,062 --> 00:04:42,282
as a distributed operating system.

74
00:04:42,712 --> 00:04:46,372
Thinking about like anything that
can run your application as like a

75
00:04:46,372 --> 00:04:49,112
meta name for like operating system.

76
00:04:49,492 --> 00:04:54,872
So I think nowadays, especially with
AI, operating system is like cool again.

77
00:04:54,902 --> 00:04:59,282
Everyone is calling themselves an AI
operating system or, like something

78
00:04:59,302 --> 00:05:03,562
AI-powered operating system for
engineers, for sales, for whatsoever.

79
00:05:03,872 --> 00:05:08,622
So I think operating system now, like
the idea is more about how to properly

80
00:05:08,902 --> 00:05:14,012
run your applications in any environment,
like in a distributed environment,

81
00:05:14,032 --> 00:05:15,832
in a highly available environment.

82
00:05:16,192 --> 00:05:20,802
It's about how to provide the
guarantee and provide kind of an

83
00:05:20,842 --> 00:05:26,002
illusion that you are ru- like normal,
like modern developers no longer

84
00:05:26,002 --> 00:05:30,672
directly write or interact with
low-level operating system primitives.

85
00:05:31,012 --> 00:05:36,342
They usually interact with some- something
like Kubernetes or cloud resources.

86
00:05:36,692 --> 00:05:40,042
So like you can think of the cloud
as an entire operating system.

87
00:05:40,052 --> 00:05:46,012
So like I think the name like DBOS right
now really focuses on how to operate

88
00:05:46,082 --> 00:05:49,062
those, applications at scale and reliably.

89
00:05:49,372 --> 00:05:54,251
So it's I would say it's eventually
at a grand vision, we hope everything

90
00:05:54,251 --> 00:05:56,851
will be backed by a database,

91
00:05:57,377 --> 00:05:57,737
Nik: Yeah.

92
00:05:58,421 --> 00:06:01,401
Qian: then that will be like
a super operating system.

93
00:06:02,431 --> 00:06:02,681
Yeah.

94
00:06:02,731 --> 00:06:06,881
Nik: Yeah this makes sense in, in
broader meaning operating system.

95
00:06:06,901 --> 00:06:07,091
Yeah.

96
00:06:07,121 --> 00:06:09,331
Not just like Linux or Windows

97
00:06:09,381 --> 00:06:10,021
or something, Right.

98
00:06:10,031 --> 00:06:10,631
I get this.

99
00:06:10,631 --> 00:06:14,163
And also like there are even broader
systems like there, there are companies

100
00:06:14,163 --> 00:06:18,137
like which are building like an
operational-- operating system for

101
00:06:18,137 --> 00:06:20,187
businesses in broader meaning, right?

102
00:06:20,647 --> 00:06:25,593
I also remember when Anthropic released
I think Ultracode with workflows.

103
00:06:26,083 --> 00:06:31,243
I remember on Hacker News
top both topics were present.

104
00:06:31,503 --> 00:06:34,965
One was about their workflows,
another was about some blog

105
00:06:34,965 --> 00:06:36,745
post from your blog, actually.

106
00:06:37,175 --> 00:06:37,765
I remember.

107
00:06:37,945 --> 00:06:40,515
It was like-- And for me, it was
like this is two different things.

108
00:06:40,815 --> 00:06:45,075
And I remember internal discussion in
our team actually, that we use these

109
00:06:45,075 --> 00:06:51,565
workflows in Claude Code, but workflows
which are like saving some state

110
00:06:51,565 --> 00:06:53,065
in database is something different.

111
00:06:53,065 --> 00:06:59,085
And then I remember an aha moment when
we realized we actually have agentic

112
00:06:59,085 --> 00:07:03,485
workflows ins-internally already
working, and we save state in Postgres.

113
00:07:04,035 --> 00:07:06,305
So I realized it's very close topics.

114
00:07:06,345 --> 00:07:11,875
I think you also use this resonating
moment when they released it and you like

115
00:07:11,895 --> 00:07:13,675
to discuss workflows in database, right?

116
00:07:14,689 --> 00:07:19,005
So I have one specific technical question
though because Peter, you mentioned

117
00:07:19,015 --> 00:07:21,685
that if something crashes, right?

118
00:07:22,425 --> 00:07:27,055
And ch- this checkpoint and database were
all good, like we ha- we have stayed safe.

119
00:07:27,135 --> 00:07:30,525
I don't like fully understand why
checkpoint is needed because we always

120
00:07:30,525 --> 00:07:32,295
can replay more after checkpoint.

121
00:07:32,315 --> 00:07:33,525
There are WALs and so on, right?

122
00:07:33,765 --> 00:07:34,355
But anyway, we

123
00:07:34,641 --> 00:07:35,181
Michael: checkpoint

124
00:07:35,745 --> 00:07:37,325
Nik: Ah, okay, a checkpoint there.

125
00:07:37,555 --> 00:07:41,525
But what wonders me, if, for example,
we se-send an email but we don't

126
00:07:41,585 --> 00:07:44,635
know the result of that yet, right?

127
00:07:44,635 --> 00:07:50,135
I'm like moving us slowly to the topic
of two-phase commit or something, right?

128
00:07:50,665 --> 00:07:51,625
How do you handle that?

129
00:07:52,557 --> 00:07:52,937
Peter: Yeah.

130
00:07:53,047 --> 00:07:58,057
So if your step is taking an action
in an external system and it fails

131
00:07:58,057 --> 00:08:01,527
in the middle of the step, then we're
going to have to retry that step, and

132
00:08:01,527 --> 00:08:05,337
that, that's unavoidable in any kind
of workflow or durability system.

133
00:08:05,707 --> 00:08:09,987
So our recommendation then is for
individual steps that take-- have

134
00:08:09,987 --> 00:08:11,817
side effects, external systems
to make those steps idempotent.

135
00:08:12,917 --> 00:08:14,537
And, we provide help for that.

136
00:08:14,547 --> 00:08:17,407
Every workflow has a unique ID
that you can use as an idempotency

137
00:08:17,407 --> 00:08:19,160
key in systems that support that.

138
00:08:19,170 --> 00:08:21,510
Many email systems do
support idempotency keys.

139
00:08:22,090 --> 00:08:24,670
Some problems are fundamental
to distributed systems.

140
00:08:24,850 --> 00:08:28,820
If you're taking a side effect
somewhere else and it fails and you

141
00:08:28,820 --> 00:08:30,460
don't know whether it succeeded,
you're gonna have to retry it.

142
00:08:30,750 --> 00:08:33,140
And so you should really try to make
sure that, that action's idempotent

143
00:08:34,749 --> 00:08:38,619
Michael: And another thing I didn't
fully appreciate till recently, I kept

144
00:08:38,619 --> 00:08:42,069
focusing on the word durable I see
durable workflows mentioned quite often.

145
00:08:42,069 --> 00:08:45,209
I'm thinking about reliability and
retries, and that all makes sense.

146
00:08:45,219 --> 00:08:48,559
The thing I didn't really
appreciate was actions that

147
00:08:48,559 --> 00:08:50,029
quite-- could be quite costly.

148
00:08:50,109 --> 00:08:55,659
If you've got a five-step workflow and
you've spent a lot of tokens or a lot

149
00:08:55,749 --> 00:09:00,539
of money, or you've sent physical goods
or something like that in steps one,

150
00:09:00,539 --> 00:09:04,469
two, and three, and it fails at step
four, you can avoid the c- the cost

151
00:09:04,469 --> 00:09:06,399
of redoing steps one, two, and three.

152
00:09:06,409 --> 00:09:11,299
Like it, the-- that hadn't really clicked
for me, that sometimes it's about other

153
00:09:11,359 --> 00:09:14,459
side effects, like might just be cost
or it might be safety or it might be

154
00:09:14,469 --> 00:09:19,212
something else that you don't have to risk
re-replaying is that a big part of it at

155
00:09:19,222 --> 00:09:22,772
the moment in terms of… or are there
parts of that might be helpful to explain?

156
00:09:24,352 --> 00:09:25,202
Qian: Yeah, definitely.

157
00:09:25,252 --> 00:09:28,502
I think especially nowadays,
everyone's building AI agents

158
00:09:28,872 --> 00:09:32,282
and those new models are becom-
becoming more and more expensive.

159
00:09:32,542 --> 00:09:36,902
So if you have already asked the
model about something, you want to

160
00:09:37,362 --> 00:09:39,632
use that output as much as possible.

161
00:09:40,072 --> 00:09:44,462
And if something crashes, you want
to resume from where it left off.

162
00:09:44,472 --> 00:09:47,572
Say, like you already asked the
model what to do next, then you

163
00:09:47,572 --> 00:09:50,662
should continue from that step
instead of asking the models again.

164
00:09:50,992 --> 00:09:55,919
Because otherwise you have to burn all the
tokens and this will quickly become too

165
00:09:55,919 --> 00:10:00,452
costly if you're having like processing
thousands of documents and then you

166
00:10:00,452 --> 00:10:02,972
want to do it in a more efficient way.

167
00:10:03,022 --> 00:10:06,652
That's where DBOS is used a
lot, like building agents and

168
00:10:06,662 --> 00:10:08,801
building AI-powered pipelines.

169
00:10:09,211 --> 00:10:12,441
I think cost saving is
definitely… It's like becoming

170
00:10:12,441 --> 00:10:13,731
more and more important, I'd say.

171
00:10:14,571 --> 00:10:14,771
Yeah.

172
00:10:15,814 --> 00:10:16,414
Michael: Makes sense.

173
00:10:16,454 --> 00:10:16,934
Interesting

174
00:10:17,092 --> 00:10:18,642
Qian: And also compute resources, right?

175
00:10:18,652 --> 00:10:22,532
Even if you don't have to pay those,
Even for some people who will self-host

176
00:10:22,532 --> 00:10:26,542
their GPUs and - self-host their model
inference, then you don't want to

177
00:10:26,552 --> 00:10:28,482
burn all the compute resources again.

178
00:10:28,492 --> 00:10:30,252
It's all about efficiency.

179
00:10:30,822 --> 00:10:36,212
I think yeah, durability itself
sounds like not, doesn't sound like

180
00:10:36,212 --> 00:10:41,512
a very useful word, but it indicates
a lot about cost savings and also

181
00:10:41,512 --> 00:10:44,728
in- indicates about after you've done
something, you don't repeat it again.

182
00:10:45,068 --> 00:10:48,448
So it's indicates about
correctness as well.

183
00:10:49,058 --> 00:10:49,198
Yeah

184
00:10:49,750 --> 00:10:53,070
Nik: I have a question about why
should it be inside Postgres?

185
00:10:53,070 --> 00:10:56,200
We know there is-- there are systems
which do the same outside, right?

186
00:10:56,240 --> 00:10:59,950
Temporal is one of the
most popular systems.

187
00:11:00,500 --> 00:11:04,160
I even from my personal experience,
I even had a moment when one of our

188
00:11:04,160 --> 00:11:09,467
clients reimplemented our Postgres
major upgrade workflow in Temporal.

189
00:11:09,587 --> 00:11:10,457
it was, It was Gadget

190
00:11:10,657 --> 00:11:12,194
Michael: They spoke
about it on the episode.

191
00:11:12,330 --> 00:11:14,720
Nik: Yeah, we had an episode
about that, and it was like we

192
00:11:14,720 --> 00:11:16,730
always used Ansible and so on.

193
00:11:17,000 --> 00:11:18,670
Of course it's very different, right?

194
00:11:18,700 --> 00:11:22,300
When you have workflow, you
save everything, it's great.

195
00:11:22,670 --> 00:11:26,220
But you move further, let's bring
the whole thing inside Postgres.

196
00:11:26,240 --> 00:11:29,736
What are pros and cons and, why do
you think this is a winning strategy?

197
00:11:30,540 --> 00:11:30,770
Peter: Yeah.

198
00:11:30,770 --> 00:11:32,800
So I'd, what I'd ask is why not Postgres?

199
00:11:33,320 --> 00:11:40,350
The temporal model is that you are
taking… is that you're outsourcing

200
00:11:40,560 --> 00:11:45,960
the control flow of your program to
an external orchestrator, and that's

201
00:11:45,960 --> 00:11:47,210
a lot of additional infrastructure.

202
00:11:47,840 --> 00:11:52,830
You can either self-host it, in which case
to run your workflows, you have to spin

203
00:11:52,830 --> 00:11:57,227
up highly available temporal cluster, a
database for it, which is Cassan- and its

204
00:11:57,227 --> 00:11:59,127
database is Cassandra and Elasticsearch.

205
00:11:59,417 --> 00:12:02,417
And now you're spinning up three
new distributed systems to run your

206
00:12:02,417 --> 00:12:06,067
workflows, and all three of those systems
are totally on your critical path.

207
00:12:06,077 --> 00:12:07,787
If anything happens to
them, Nothing will work.

208
00:12:07,947 --> 00:12:09,027
Your workflows will not function.

209
00:12:09,027 --> 00:12:10,087
Your business grinds to a halt.

210
00:12:10,557 --> 00:12:13,367
So you're putting three complex
distributed systems on your critical path.

211
00:12:13,697 --> 00:12:14,997
Or - you use their cloud service.

212
00:12:14,997 --> 00:12:17,317
Now you're putting an external
SaaS on your critical path.

213
00:12:17,967 --> 00:12:18,707
You don't need to do that.

214
00:12:19,727 --> 00:12:21,007
You just need a database.

215
00:12:21,967 --> 00:12:26,131
And what we're trying to do in DBOS is
totally replace all of those distributed

216
00:12:26,131 --> 00:12:30,561
systems, all those external orchestrators
with your Postgres database so that you

217
00:12:30,571 --> 00:12:32,541
can use your database as an orchestrator.

218
00:12:32,761 --> 00:12:34,261
It's already on your critical path.

219
00:12:34,271 --> 00:12:36,541
It's already something that you're
making sure is highly available.

220
00:12:36,551 --> 00:12:38,331
It's already critical for
your application to work.

221
00:12:38,861 --> 00:12:42,641
Why not also use it as your workflow
orchestrator so that you have

222
00:12:42,641 --> 00:12:45,691
less infrastructure, so that your
operations are simpler, so that

223
00:12:45,691 --> 00:12:48,451
there are fewer things that can break
and bring down everything you have

224
00:12:49,647 --> 00:12:50,477
Nik: Two things here.

225
00:12:51,007 --> 00:12:56,767
One is who will-- like Postgres
doesn't have scheduler, right?

226
00:12:57,287 --> 00:13:00,357
You can use pg_cron or something,
or you need to still need

227
00:13:00,357 --> 00:13:02,237
external help to tick, right?

228
00:13:02,287 --> 00:13:03,047
To move forward.

229
00:13:03,117 --> 00:13:07,597
And another thing is that if it's all
running on the primary and primary CPU,

230
00:13:07,597 --> 00:13:10,177
first of all, is not infinite, right?

231
00:13:10,177 --> 00:13:15,377
So you need to avoid heavy things
and have them outside still, right?

232
00:13:16,135 --> 00:13:17,465
Peter: Yeah, and that's
the reason Postgres is

233
00:13:19,805 --> 00:13:19,855
the orchestrator.

234
00:13:19,855 --> 00:13:20,665
The actual business logic
of your workflows is still

235
00:13:20,665 --> 00:13:21,585
running in your application.

236
00:13:21,595 --> 00:13:24,165
The scheduler is running in your
application, for example, and

237
00:13:24,165 --> 00:13:25,465
then Postgres is the orchestrator.

238
00:13:25,845 --> 00:13:29,065
So your applications are running
a scheduler, and your applications

239
00:13:29,065 --> 00:13:32,785
will enqueue when the sched- when
the cron fires, your, one of your

240
00:13:32,785 --> 00:13:36,775
application servers will enqueue a
scheduled workflow into Postgres.

241
00:13:37,205 --> 00:13:38,985
And then Postgres there
does orchestration.

242
00:13:39,165 --> 00:13:43,245
It coordinates multiple servers to
make sure that only one of those

243
00:13:43,245 --> 00:13:46,605
servers actually executes the scheduled
workflow when its cron tick fires.

244
00:13:47,165 --> 00:13:50,275
So your applications your regular
servers are doing the work, and

245
00:13:50,275 --> 00:13:52,865
Postgres is orchestrating them to
make sure that every, all the work

246
00:13:52,865 --> 00:13:56,095
is being done totally reliably, that
everything happens exactly once.

247
00:13:56,705 --> 00:13:57,365
Nik: Makes sense.

248
00:13:57,695 --> 00:14:02,495
So this is when you say critical
path, you don't include things

249
00:14:04,085 --> 00:14:07,555
that like sending email can be done
from multiple servers and they are

250
00:14:07,555 --> 00:14:10,225
replaceable like stateless and so on.

251
00:14:10,245 --> 00:14:11,585
Yeah, I get that.

252
00:14:11,845 --> 00:14:19,388
It feels slightly awkward for
me to try to criticize the idea.

253
00:14:19,428 --> 00:14:22,508
I'm a big fan of myself, like
maybe Michael should join

254
00:14:22,508 --> 00:14:24,498
and find some other questions

255
00:14:25,177 --> 00:14:27,717
Michael: I was actually gonna
ask a question that probably

256
00:14:27,717 --> 00:14:29,647
goes even more positive.

257
00:14:29,697 --> 00:14:33,327
M- or maybe not, but I would expect
there to be performance benefits to

258
00:14:33,327 --> 00:14:39,237
using your already h- used database for
this rather than an external service.

259
00:14:39,527 --> 00:14:44,058
So I was only gonna say What are, what are
the performance trade-offs and what do you

260
00:14:44,058 --> 00:14:48,038
do to minimize the impact you're having
on primary CPU and that kind of thing?

261
00:14:49,898 --> 00:14:50,708
Peter: Yeah, absolutely.

262
00:14:50,708 --> 00:14:55,058
So latency in particular gets a lot better
because now the latency of executing

263
00:14:55,058 --> 00:14:59,928
a step is just one write to your local
Postgres as opposed to a async dispatch

264
00:14:59,938 --> 00:15:01,448
from somebody else's cloud service.

265
00:15:02,938 --> 00:15:05,098
So definitely latency improves.

266
00:15:05,648 --> 00:15:07,438
Now throughput then
becomes database bounded.

267
00:15:07,768 --> 00:15:11,508
You can process as many
workflows per second as, as your

268
00:15:11,508 --> 00:15:12,568
Postgres allows if you want.

269
00:15:12,618 --> 00:15:15,882
And if you need to scale beyond
that, you need to shard across

270
00:15:15,882 --> 00:15:17,242
multiple Postgres databases.

271
00:15:17,742 --> 00:15:20,172
The good news is that these
operations are fairly light.

272
00:15:20,592 --> 00:15:25,712
So our rule of thumb is that four Postgres
cores can handle 1,000 actions per second.

273
00:15:26,322 --> 00:15:30,362
So for a large database, that means
you can vertically scale for a large

274
00:15:30,382 --> 00:15:33,902
Postgres database up to… we've
benchmarked around 40,000 actions per

275
00:15:33,902 --> 00:15:35,632
second on a single Postgres database.

276
00:15:36,092 --> 00:15:38,152
And then of course, you need more
than that, you can shard or you can

277
00:15:38,162 --> 00:15:39,512
use multiple Postgres databases.

278
00:15:39,662 --> 00:15:42,322
So your throughput does become
bounded by your database, but the

279
00:15:42,322 --> 00:15:44,742
good news is that your database
can handle a lot of throughput

280
00:15:46,250 --> 00:15:46,530
Nik: Yeah.

281
00:15:46,710 --> 00:15:47,050
Yeah.

282
00:15:47,364 --> 00:15:47,654
Michael: and we're

283
00:15:47,654 --> 00:15:51,228
talking… so let's say we've
got a workflow with six steps.

284
00:15:51,658 --> 00:15:56,458
We've got one write at the beginning, one
write per step, and one write at the end.

285
00:15:56,458 --> 00:16:01,198
So we're still only talking eight writes
total for a si- and this workflow's having

286
00:16:01,198 --> 00:16:04,848
to do things in the meantime, it's not
gonna be running through them instantly.

287
00:16:05,178 --> 00:16:06,458
So yeah, great.

288
00:16:06,488 --> 00:16:07,238
Okay, cool.

289
00:16:07,459 --> 00:16:10,769
Peter: And you can handle 40,000 of
those actions per second on one database

290
00:16:11,313 --> 00:16:12,173
Nik: Yeah, I'm curious.

291
00:16:12,193 --> 00:16:15,833
I researched a little bit and I noticed
y-that you started with repeatable

292
00:16:15,893 --> 00:16:21,333
read isolation l-level and then reduced
it to default read committed, right?

293
00:16:21,373 --> 00:16:23,423
I'm very curious what was behind that.

294
00:16:24,293 --> 00:16:26,603
Is it the throughput
question or anything else?

295
00:16:27,969 --> 00:16:30,689
Peter: So we use isolation levels
there's a lot of different parts

296
00:16:30,689 --> 00:16:32,729
of our system that use isolation
levels for different reason.

297
00:16:33,406 --> 00:16:39,446
we've designed the system so that
almost everything is a single statement.

298
00:16:40,376 --> 00:16:43,396
So it can just be a, it can just
execute a recommitted, and that

299
00:16:43,396 --> 00:16:44,666
makes things far more efficient.

300
00:16:44,986 --> 00:16:47,456
As many things as possible execute
as a single Postgres statement,

301
00:16:47,466 --> 00:16:49,936
so they can just be atomic and you
don't have to worry about isolation.

302
00:16:49,946 --> 00:16:51,116
You set it to default recommitted.

303
00:16:52,126 --> 00:16:56,626
The one thing that we can't do
that for is the main DQ query.

304
00:16:57,326 --> 00:17:00,406
this is probably the single most
important s- query in the entire system,

305
00:17:00,866 --> 00:17:07,856
and it's the query that every worker
runs periodically on every queue to

306
00:17:08,156 --> 00:17:13,160
dequeue the latest work off the queue
And that's a very complex query we've

307
00:17:13,160 --> 00:17:14,960
put a ton of effort in optimizing.

308
00:17:16,590 --> 00:17:21,110
That query has a, actually has a
conditional isolation level, s- depending

309
00:17:21,110 --> 00:17:23,440
on what, how you've configured your queue.

310
00:17:24,490 --> 00:17:30,680
So if your queue uses global flow control
primitives, if your queue, for example,

311
00:17:30,680 --> 00:17:35,180
has a global concurrency limit or a global
rate limit, so you say, for example,

312
00:17:35,180 --> 00:17:40,440
that this queue can only execute one
operation at a time across every worker,

313
00:17:40,960 --> 00:17:45,230
then that requires global coordination,
which means we have to use a high

314
00:17:45,230 --> 00:17:47,040
isolation level like repeatable read.

315
00:17:48,640 --> 00:17:54,170
And that has coordination costs, but
if you have those global limits, you're

316
00:17:54,170 --> 00:17:57,080
probably fine paying the cost because
you're only executing one workflow

317
00:17:57,080 --> 00:17:58,490
at a time across all your servers.

318
00:17:58,785 --> 00:18:02,030
th- your throughput was limited
anyways in practice though, what we

319
00:18:02,030 --> 00:18:05,359
see is that high throughput queues
usually only use local limits.

320
00:18:05,629 --> 00:18:08,619
They might say only execute 10
workflows at a time per worker.

321
00:18:11,459 --> 00:18:13,629
And those local limits don't
require global coordination.

322
00:18:14,319 --> 00:18:19,389
And that means that it's perfectly safe
to run the queues query at a very low

323
00:18:19,389 --> 00:18:21,039
isolation level at at read committed.

324
00:18:21,599 --> 00:18:26,409
And that way you can have this query
running across hundreds of servers at the

325
00:18:26,409 --> 00:18:28,919
same time, or thousands of servers at the
same time without them conflicting because

326
00:18:28,919 --> 00:18:29,829
they're all running read committed.

327
00:18:30,309 --> 00:18:35,449
And that lets you get this thirty, forty
thousand dequeues, dequeued workflows per

328
00:18:35,449 --> 00:18:37,129
second, actions per second throughput.

329
00:18:37,469 --> 00:18:41,359
And it really caps out at thirty
thousand dequeues per second on a single

330
00:18:41,359 --> 00:18:44,139
Postgres across thousands of workers
because we've optimized this query

331
00:18:45,169 --> 00:18:48,299
Nik: And in case of sharding
solution re-repeat-repeatable read

332
00:18:48,319 --> 00:18:49,729
wouldn't be available anyway, right?

333
00:18:50,259 --> 00:18:53,919
So because like global
limits it's… Yeah.

334
00:18:53,949 --> 00:18:54,149
You

335
00:18:54,457 --> 00:18:57,997
Peter: But in practice, global limits tend
to go with relatively low throughputs.

336
00:18:58,326 --> 00:18:59,686
Nik: And it's flexible now, right?

337
00:18:59,686 --> 00:19:03,146
So u-u-user can control it
and decide what, what's in.

338
00:19:03,156 --> 00:19:03,676
That's great.

339
00:19:04,316 --> 00:19:07,386
And I also noticed listen to
file was recently added, right?

340
00:19:07,556 --> 00:19:09,756
Or not recently, maybe I'm wrong.

341
00:19:09,814 --> 00:19:14,774
Qian: We've been having listen notify,
but recently we've been optimizing it

342
00:19:15,306 --> 00:19:15,696
Nik: yeah.

343
00:19:15,966 --> 00:19:20,736
We ha-- Yeah, I think it was last year,
Recall AI they discovered this awful

344
00:19:20,806 --> 00:19:23,806
global lock on commit for Notifier, right?

345
00:19:23,836 --> 00:19:26,076
What's your deal with this right now?

346
00:19:26,816 --> 00:19:31,386
Peter: Listen/notify is really powerful
but has multiple performance issues.

347
00:19:32,016 --> 00:19:33,706
There was that issue that… I forget.

348
00:19:33,716 --> 00:19:35,626
There was that issue that reached
top of Hacker News last year.

349
00:19:35,966 --> 00:19:37,876
That was actually not
the issue we ran into.

350
00:19:37,896 --> 00:19:39,386
That issue was solved in Postgres 19.

351
00:19:39,386 --> 00:19:40,446
That is not the issue we ran into.

352
00:19:40,778 --> 00:19:44,048
that issue is that notify had
terrible perform- notify's broadcast

353
00:19:44,048 --> 00:19:44,478
mode had terrible performance for
large numbers of sparse readers.

354
00:19:44,478 --> 00:19:44,488
Uh,

355
00:19:48,668 --> 00:19:49,678
that's actually an unrelated issue.

356
00:19:49,678 --> 00:19:50,558
We do not have that.

357
00:19:50,558 --> 00:19:51,458
We have dense readers.

358
00:19:51,458 --> 00:19:52,378
We don't have sparse readers.

359
00:19:53,011 --> 00:19:55,571
Nik: Can you explain please the
sparse readers versus dense readers?

360
00:19:55,831 --> 00:19:56,191
Peter: Oh, sorry.

361
00:19:56,991 --> 00:20:00,091
The problem that everybody was talking
about last year was that if you're

362
00:20:00,091 --> 00:20:05,081
broadcasting to a huge number of
readers, each of which is only reading

363
00:20:05,091 --> 00:20:10,091
a small fraction of all messages,
that had terrible performance.

364
00:20:11,751 --> 00:20:14,389
So that's actually not
the issue we ran into.

365
00:20:14,509 --> 00:20:17,609
The way DBOS is set up, pretty much
all workers are listening to all

366
00:20:17,619 --> 00:20:20,749
messages and that performs fine.

367
00:20:21,149 --> 00:20:23,399
The issue we ran into was
the global lock at notify.

368
00:20:24,549 --> 00:20:30,429
When you commit a transaction that
performs a notify, it has to acquire a

369
00:20:30,429 --> 00:20:34,629
global lock for the entire duration of
the commit, and that includes the fsync.

370
00:20:35,949 --> 00:20:40,089
And what that means is that these
transactions can't be group committed.

371
00:20:40,449 --> 00:20:45,719
They have to commit sequentially, which
essentially bounds the throughput of

372
00:20:46,759 --> 00:20:48,759
any operations that perform notify.

373
00:20:48,759 --> 00:20:50,869
You can't perform more than a
certain number of them per second

374
00:20:50,869 --> 00:20:52,009
because you can't group commit.

375
00:20:52,239 --> 00:20:54,299
So you're stuck waiting out the
entire duration of the fsync.

376
00:20:55,456 --> 00:20:58,286
the optimization that we settled on
there to make listen notify, because

377
00:20:58,286 --> 00:20:59,416
we really want to use listen notify.

378
00:20:59,436 --> 00:21:02,886
It provides incredibly low
latency, and when it works, it

379
00:21:02,886 --> 00:21:03,866
provides incredibly low latency.

380
00:21:04,226 --> 00:21:07,386
So w- the optimization we
settled on was notify batching.

381
00:21:07,956 --> 00:21:14,639
So instead of sending a notify,
originally, we were sending notifies

382
00:21:14,649 --> 00:21:17,949
in a trigger every single time you,
for example, wrote to a stream.

383
00:21:19,689 --> 00:21:23,419
And the optimization was instead it
was removing the trigger and instead

384
00:21:23,419 --> 00:21:26,239
buffering stream writes in a background
thread and then sending all the

385
00:21:26,239 --> 00:21:29,999
notifies in the batch and send that
batch e-every ten milliseconds or so.

386
00:21:30,729 --> 00:21:32,989
A huge batch of notifies is
actually very fast because you

387
00:21:32,999 --> 00:21:35,069
only have to hold the global lock
for the duration of that commit.

388
00:21:35,344 --> 00:21:36,054
that's fast.

389
00:21:36,314 --> 00:21:41,054
And what this does is this allows us to
use listen notify without re- running

390
00:21:41,054 --> 00:21:43,114
it, worrying about the global lock
because we're batching commits and

391
00:21:43,114 --> 00:21:46,574
the global lock's only required for
commits, and still getting the benefits

392
00:21:46,574 --> 00:21:49,995
of low latency because we can still
emit these every, every few milliseconds

393
00:21:51,329 --> 00:21:55,179
Qian: I think, the overall observation is
that we don't really need that global o-

394
00:21:55,269 --> 00:21:59,919
ordering when we're doing the application
level streaming, like streaming,

395
00:22:00,019 --> 00:22:02,039
for example, results to the client.

396
00:22:02,509 --> 00:22:07,889
Then what do you-- we're leveraging the
listen/notify to do is to notify like,

397
00:22:07,899 --> 00:22:09,619
"Hey, we have some results available."

398
00:22:09,679 --> 00:22:11,689
So we don't really need
that strict ordering.

399
00:22:12,059 --> 00:22:16,019
And I think that was the main aha moment
for us that, okay, we don't need to

400
00:22:16,019 --> 00:22:18,399
limit ourselves to that exclusive locks.

401
00:22:18,409 --> 00:22:19,489
We can do batching.

402
00:22:19,969 --> 00:22:20,149
Yeah.

403
00:22:22,123 --> 00:22:25,353
Nik: And yeah, and this is… okay.

404
00:22:25,353 --> 00:22:27,983
So and also the problem is happening.

405
00:22:28,073 --> 00:22:32,853
You, Peter mentioned that it was
fixed in 19, and it's happening when

406
00:22:33,783 --> 00:22:35,763
commit has a lot to write, right?

407
00:22:35,763 --> 00:22:37,443
So like it's this fsync.

408
00:22:38,903 --> 00:22:42,643
Basically, transaction is significant in
terms of how much it's writing, right?

409
00:22:42,643 --> 00:22:44,923
So th-this increases the problem.

410
00:22:44,933 --> 00:22:49,643
But in your case, you have only
like brief writes in transaction.

411
00:22:50,363 --> 00:22:51,203
Not a lot, right?

412
00:22:51,253 --> 00:22:56,273
Not like a lot of JSONs to be
written and fsynced during commit,

413
00:22:56,716 --> 00:23:00,936
So the problem shouldn't be huge
a-already if I'm not mistaken.

414
00:23:01,308 --> 00:23:02,368
and you decided to group it.

415
00:23:02,368 --> 00:23:03,678
This is interesting optimization.

416
00:23:04,328 --> 00:23:06,398
I, I wanted to raise
this question, of course.

417
00:23:06,458 --> 00:23:12,458
You use skip locked updates and
what's happening if we have some

418
00:23:12,468 --> 00:23:17,368
xmin horizon blockers, so long
transactions or lagging logical slots?

419
00:23:18,048 --> 00:23:21,308
Obviously, a lot of dead tuples
should be accumulated, right?

420
00:23:22,688 --> 00:23:28,758
Is it a requirement if you use DBOS
at, like at high rates, at a lot

421
00:23:28,758 --> 00:23:30,278
of events per second and so on?

422
00:23:30,278 --> 00:23:35,578
Is it a requirement I need to watch
closely what is my xmin horizon?

423
00:23:35,708 --> 00:23:40,358
Don't allow long-running transactions,
don't allow logical… lagging

424
00:23:40,358 --> 00:23:44,684
logical replication slots long
transactions, so on standbys with

425
00:23:44,864 --> 00:23:46,564
hot standby feedback on and so on.

426
00:23:46,624 --> 00:23:49,594
Is it a requirement or you solve
this somehow I don't know, like

427
00:23:49,594 --> 00:23:53,794
maybe mitigate-mitigating bloat with
repacking, reindexing and so on?

428
00:23:54,834 --> 00:23:55,194
Peter: Got it.

429
00:23:55,204 --> 00:24:00,764
So just to be clear, we're only using
skip locked on our own workflow status

430
00:24:00,784 --> 00:24:06,644
table, and that's typically isolated
within its own Postgres database.

431
00:24:07,504 --> 00:24:14,906
So our recommendation is Don't take
out ultra long-running transactions

432
00:24:15,526 --> 00:24:19,086
inside the DBOS database, and you
probably weren't doing that anyways

433
00:24:19,116 --> 00:24:20,696
because DBOS certainly doesn't do that.

434
00:24:21,016 --> 00:24:22,406
And then the rest should be fine.

435
00:24:23,396 --> 00:24:25,046
The DBOS operations are isolated here

436
00:24:25,558 --> 00:24:29,998
Nik: So you don't use skip locked
for like events t- tables where

437
00:24:29,998 --> 00:24:32,648
events or workflow data is stored?

438
00:24:33,260 --> 00:24:37,600
Peter: It's only used for deques off of
the main workflow status table, and that's

439
00:24:37,600 --> 00:24:39,440
usually pretty isolated from user data

440
00:24:40,282 --> 00:24:44,612
Nik: But still, it's you need a s- single
update on every status change, right?

441
00:24:45,996 --> 00:24:49,986
Peter: Yes, and that does lead
to dead tuple accumulation.

442
00:24:50,440 --> 00:24:53,960
but that's a separate issue that
y-yes, at ma- at massive, at

443
00:24:56,340 --> 00:24:59,890
really high throughputs, like when
we were doing these benchmarks at,

444
00:25:00,090 --> 00:25:08,224
40,000 workflows per second, we
saw high CPU usage due to dead

445
00:25:08,234 --> 00:25:11,704
tuple cleanup and autovacuum because
we're turning on these indexes.

446
00:25:12,034 --> 00:25:16,024
And we've done a lot of work
to optimize indexes, so most,

447
00:25:16,294 --> 00:25:18,094
almost every index is partial.

448
00:25:18,384 --> 00:25:22,144
So most indexes are only being
maintained either for a small

449
00:25:22,144 --> 00:25:25,314
subset of workflows or only for a
small subset of workflow lifetime.

450
00:25:26,224 --> 00:25:33,684
But there's still a fair amount of index
churn on the order of… And this, of

451
00:25:33,684 --> 00:25:37,873
course, not as hot a dead tuple, and it's
going to create a bunch of dead index.

452
00:25:37,873 --> 00:25:40,733
It's going to create two or
three dead index entries that are

453
00:25:40,733 --> 00:25:42,193
going to have to be cleaned up.

454
00:25:43,373 --> 00:25:47,773
At really high throughputs, this can
become the bottleneck but that sets in

455
00:25:47,783 --> 00:25:52,653
around 20 or 30,000 actions per second
is when we start seeing the tuple

456
00:25:52,653 --> 00:25:55,213
accumulation really becoming a problem,
because there really aren't that…

457
00:25:55,343 --> 00:25:57,373
Tuple accumulation become
a problem, because we have

458
00:25:57,373 --> 00:25:58,433
optimized this as much as we can.

459
00:25:59,238 --> 00:26:00,808
Nik: Yeah, I see it very differently.

460
00:26:01,668 --> 00:26:08,098
I see it like when you say 20,
30,000 per second, it means for me it

461
00:26:08,098 --> 00:26:14,108
becomes 2,000, 3,000 or 200, 300 per
second if we talk about really long

462
00:26:14,228 --> 00:26:19,008
transaction happening in parallel
because they block vacuum, uh, work.

463
00:26:19,558 --> 00:26:25,058
We cannot delete the tuples, and at
very high throughput with shorter

464
00:26:25,138 --> 00:26:29,798
transactions will be-become the same
if you have much lower throughput,

465
00:26:30,578 --> 00:26:33,168
Peter: Yeah, which is why
our recommendation is isolate

466
00:26:33,248 --> 00:26:35,048
the DBOS system database

467
00:26:35,466 --> 00:26:38,286
from the database we're using, we're
using long-running transactions.

468
00:26:38,536 --> 00:26:42,026
These don't have to be in the same, in
the… These don't have to be in the

469
00:26:42,026 --> 00:26:47,380
same Postgres database, and they probably
shouldn't be, especially for long-running

470
00:26:47,380 --> 00:26:48,920
transactions that could block Autovacuum

471
00:26:49,354 --> 00:26:52,174
Nik: Yeah, this is what my… Yeah,
th-this is exactly what I was asking.

472
00:26:52,224 --> 00:26:55,154
It's not like like requirement, but
it definitely if you have long-running

473
00:26:55,344 --> 00:26:59,664
transactions or any other explain
horizon blockers performance of

474
00:26:59,664 --> 00:27:02,244
DB-DBOS will, will suffer eventually.

475
00:27:02,674 --> 00:27:05,174
Peter: Yeah, but these don't, these
can be isolated from each other.

476
00:27:05,684 --> 00:27:09,034
They don't have to run in the same
place, and they probably shouldn't run

477
00:27:09,034 --> 00:27:11,234
in the same place, especially at scale

478
00:27:12,218 --> 00:27:13,098
Nik: This is interesting.

479
00:27:13,108 --> 00:27:14,518
We have a lot of common things.

480
00:27:14,568 --> 00:27:18,138
I agree with putting everything
to Postgres open source.

481
00:27:18,138 --> 00:27:21,628
I w- didn't mention that the thing
you're building is open source, which

482
00:27:21,628 --> 00:27:23,398
is great, and like this is super cool.

483
00:27:24,328 --> 00:27:28,978
And and like we don't need
extra things on critical path.

484
00:27:29,618 --> 00:27:30,298
That's great.

485
00:27:30,338 --> 00:27:33,298
Also I guess you have self-served
option as well, right?

486
00:27:33,348 --> 00:27:36,078
If it is open source, you
can self-host yourself.

487
00:27:36,188 --> 00:27:36,978
Like freedom, right?

488
00:27:36,978 --> 00:27:37,438
It's great.

489
00:27:38,108 --> 00:27:42,588
And you run also y-you
can run DBOS in, on RDS.

490
00:27:42,638 --> 00:27:43,478
Absolute fr-freedom.

491
00:27:43,798 --> 00:27:47,958
But when you say let's split it,
it's against the first principle.

492
00:27:47,958 --> 00:27:50,988
Like first principle, let's
have it everything in one place.

493
00:27:51,338 --> 00:27:56,511
Because if we have, this thing
and we have other tables, we can

494
00:27:56,631 --> 00:27:58,571
organize code right inside database.

495
00:27:58,971 --> 00:28:02,871
Sometimes it's a bad idea if
it's very long, if CPU-consuming.

496
00:28:03,601 --> 00:28:06,821
But if it's something simple, I
want to update everything inside,

497
00:28:06,831 --> 00:28:09,471
like using triggers, PostgreSQL
code functions, and so on.

498
00:28:09,471 --> 00:28:10,411
I don't want to split.

499
00:28:10,991 --> 00:28:15,951
But once I have database, say terabyte in
size, and I start some analytical queries

500
00:28:15,951 --> 00:28:20,161
right there because I'm not I'm not grown
enough to have ClickHouse or something.

501
00:28:20,161 --> 00:28:21,971
I have only Postgres, right?

502
00:28:22,221 --> 00:28:26,451
Those long-running queries
will take time, right?

503
00:28:26,981 --> 00:28:32,901
And in this case updates and data, MVCC
in Postgres actually is not a good thing

504
00:28:32,921 --> 00:28:35,281
to deal with, as we all know, right?

505
00:28:35,291 --> 00:28:38,911
This is I'm j- I'm asking because, like
you m- you know that I'm trying to revive

506
00:28:38,921 --> 00:28:43,011
this PGQ from Skype, which is bloat-free.

507
00:28:43,641 --> 00:28:45,481
No, almost no updates.

508
00:28:45,881 --> 00:28:48,361
For event table, there
are no updates there.

509
00:28:48,691 --> 00:28:52,511
But I don't see how it will… I
don't see clearly how it will be

510
00:28:52,511 --> 00:28:54,171
used to implement workflows yet.

511
00:28:54,261 --> 00:28:59,551
But for many tasks, it's great, and we
don't have problems with bloat there.

512
00:29:00,461 --> 00:29:04,471
And I'm trying to understand
how our operate properly.

513
00:29:04,471 --> 00:29:09,621
Like in, in case of updates, we
need to establish some rules.

514
00:29:09,701 --> 00:29:13,301
Like for example, we should stop
allowing long-running transactions.

515
00:29:13,871 --> 00:29:16,781
I'm I'm glad I participated in
building transaction timeout.

516
00:29:16,781 --> 00:29:20,031
So we set transaction
timeout, say like 10 minutes.

517
00:29:20,091 --> 00:29:20,591
That's it.

518
00:29:21,311 --> 00:29:26,171
And I'm very curious, like maybe
there is a way to define some

519
00:29:26,881 --> 00:29:32,401
boundaries and rules when it's time
to move away and to like to split to

520
00:29:32,811 --> 00:29:35,761
different Postgres cluster, right?

521
00:29:35,811 --> 00:29:36,881
To split these workloads.

522
00:29:37,261 --> 00:29:39,311
M- And there are no good rules yet.

523
00:29:39,351 --> 00:29:39,861
Th-This is a problem

524
00:29:40,839 --> 00:29:45,119
Peter: One is that you don't have to split
multiple Postgres clusters or servers.

525
00:29:45,129 --> 00:29:48,779
You can, if I remember how Autovacuum
works correctly, it is isolated across

526
00:29:48,779 --> 00:29:50,369
multiple databases on the same cluster.

527
00:29:50,441 --> 00:29:51,471
Nik: Logical databases.

528
00:29:51,709 --> 00:29:52,339
Peter: Yes.

529
00:29:52,689 --> 00:29:54,689
So like the xmin horizon
is set per database.

530
00:29:54,699 --> 00:29:57,526
So so you don't have to split
your physical infrastructure.

531
00:29:57,816 --> 00:29:59,166
You can share physical infrastructure

532
00:30:00,670 --> 00:30:01,720
Nik: Yeah that's a good point.

533
00:30:01,933 --> 00:30:02,383
I agree.

534
00:30:03,393 --> 00:30:07,253
Still, I feel like it would be great
to have some benchmarks or something to

535
00:30:07,253 --> 00:30:11,383
define when it's time, when it's time
to split, when it's fine, and which

536
00:30:11,383 --> 00:30:13,063
transactions should be disallowed.

537
00:30:13,923 --> 00:30:15,493
I think it-- you're very right.

538
00:30:15,533 --> 00:30:20,653
If you have 30,000 workflows per
second it-- your requirements become

539
00:30:20,663 --> 00:30:24,043
much stricter in terms of what means
long-running transaction, right?

540
00:30:24,423 --> 00:30:26,023
How much xmin horizon is lagging.

541
00:30:26,723 --> 00:30:29,413
Anyway, thank you for
diving into this topic.

542
00:30:29,413 --> 00:30:31,913
I'm also trying to learn this myself

543
00:30:32,629 --> 00:30:32,819
Qian: Yeah.

544
00:30:33,639 --> 00:30:37,129
I think that's actually a good, really
good point because you are right that

545
00:30:37,159 --> 00:30:42,429
if you put… Y- there are benefits
of co-locating the workflow tables

546
00:30:42,459 --> 00:30:44,369
with your actual application data.

547
00:30:44,789 --> 00:30:48,599
One thing we were talking about
in some of our blog posts is this

548
00:30:48,609 --> 00:30:50,639
transactional outbox pattern.

549
00:30:50,699 --> 00:30:55,749
If you say someone, like we updated
the customer data, now we want to

550
00:30:55,749 --> 00:30:59,709
kickstart some of the workflows,
then by co-locating them in the same

551
00:30:59,709 --> 00:31:01,749
database, you can do it in a trigger.

552
00:31:01,759 --> 00:31:05,699
Say, whenever you update this customer
data, automatically start or enqueue

553
00:31:05,739 --> 00:31:09,969
a DBOS workflow in the same database,
and that's really convenient and

554
00:31:09,969 --> 00:31:14,039
that, that's a way that you can
guarantee whenever there's updates

555
00:31:14,049 --> 00:31:17,699
in this table, you'll kickstart a
workflow that is durable, that will,

556
00:31:17,919 --> 00:31:19,719
that DBOS will guarantee will finish.

557
00:31:20,119 --> 00:31:23,209
And that's a pattern that is
really difficult to do with,

558
00:31:23,919 --> 00:31:27,159
say, like external orchestrator
because there will always be a gap.

559
00:31:27,539 --> 00:31:30,999
And by co-locating in the same database,
there's a way to close this gap.

560
00:31:31,489 --> 00:31:33,149
But everything has a trade-off.

561
00:31:33,229 --> 00:31:36,649
Like you said, if you put in the
same database, you have to be really

562
00:31:36,649 --> 00:31:40,409
careful about what you, what queries
you put in that database, like how

563
00:31:40,409 --> 00:31:44,089
to avoid head of line blocking, how
to avoid bloating in that database.

564
00:31:44,569 --> 00:31:48,809
So I think that's a topic we're
continuously, benchmarking and

565
00:31:48,809 --> 00:31:52,489
also by talking to users and see
what kind of workflows and what

566
00:31:52,489 --> 00:31:54,289
kind of workloads they're running.

567
00:31:54,589 --> 00:31:58,649
And we're, we are also trying to
categorize like a more detailed,

568
00:31:58,699 --> 00:32:00,009
guidelines for people as well.

569
00:32:00,807 --> 00:32:01,767
Nik: Yeah, interesting.

570
00:32:01,767 --> 00:32:05,597
But any-anyway I'm glad that this, this
topic is becoming more and more popular

571
00:32:05,597 --> 00:32:09,327
and other companies started to join
t- like M-Microsoft recently, right?

572
00:32:09,537 --> 00:32:10,347
With pgDurable.

573
00:32:10,347 --> 00:32:11,807
What do you think about that system?

574
00:32:12,720 --> 00:32:13,050
Qian: Yeah.

575
00:32:13,050 --> 00:32:14,180
So this is great.

576
00:32:14,270 --> 00:32:18,450
I think pgDurable is mostly
for pure SQLs, right?

577
00:32:18,490 --> 00:32:22,000
It's, it doesn't really work for
if you want to send a email or

578
00:32:22,000 --> 00:32:23,710
interact with external systems.

579
00:32:24,330 --> 00:32:28,673
So we'll definitely, take a look and
see if there are any ways to collaborate

580
00:32:28,673 --> 00:32:30,893
or borrow ideas from pgDurable.

581
00:32:31,143 --> 00:32:35,213
But I think in general it's
another validation of the Postgres

582
00:32:35,849 --> 00:32:36,169
Nik: yeah.

583
00:32:36,169 --> 00:32:36,389
yeah.

584
00:32:36,389 --> 00:32:39,109
Everything more and more people
start looking in the same direction.

585
00:32:39,109 --> 00:32:41,199
I think yeah, the topic is warming up.

586
00:32:41,199 --> 00:32:42,109
I agree with you here.

587
00:32:42,329 --> 00:32:45,939
Also, there was a system called
Absurd, which got my attention.

588
00:32:46,409 --> 00:32:47,719
Single SQL file as well.

589
00:32:47,719 --> 00:32:49,999
It… I think it was
released last November.

590
00:32:50,809 --> 00:32:51,519
And yeah.

591
00:32:51,935 --> 00:32:57,335
I noticed in pgdurable is that the
tendency to have a SQL PL, PostgreSQL

592
00:32:57,335 --> 00:32:59,495
code quite ef-ephemeral way.

593
00:33:00,225 --> 00:33:03,355
And this is w- like you're just
passing it as a parameter, like

594
00:33:03,355 --> 00:33:04,465
to lump the functions, right?

595
00:33:04,935 --> 00:33:09,555
And this concerns me because I want
my code still to be in Git, right?

596
00:33:09,595 --> 00:33:13,315
To be versionable, fix,
reviewable, and so on, right?

597
00:33:14,345 --> 00:33:20,315
So this leads me to question, what
are the best use cases, case studies

598
00:33:20,315 --> 00:33:25,270
you ha- you had already, you where
you observed debor… I'm sorry,

599
00:33:25,280 --> 00:33:27,260
like I pronounce it DBOS for myself.

600
00:33:27,260 --> 00:33:27,820
I don't know why.

601
00:33:28,260 --> 00:33:31,660
What are the best use case studies,
use cases you have, like where

602
00:33:31,660 --> 00:33:36,160
it shines in terms of how like,
like it suits really well, right?

603
00:33:36,620 --> 00:33:41,760
Sending emails or like processing orders
in e-commerce or something like this.

604
00:33:41,790 --> 00:33:43,220
What are the best cases?

605
00:33:45,223 --> 00:33:45,703
Peter: Yeah.

606
00:33:45,866 --> 00:33:48,356
the thing about workflows is that
they're really powerful f- they're

607
00:33:48,356 --> 00:33:49,536
really powerful for a lot of things.

608
00:33:49,546 --> 00:33:54,430
So of course, the, the really
hype new use case we see is AI

609
00:33:54,430 --> 00:33:55,860
agents and operations around them.

610
00:33:56,340 --> 00:34:01,360
So people using workflows both to
make their actual agents durable.

611
00:34:01,400 --> 00:34:04,690
This is particularly important for
long-running interactive agents that

612
00:34:04,690 --> 00:34:08,690
might, talk to a person or do multiple
rounds of conversation with a person

613
00:34:08,690 --> 00:34:12,230
or require approvals for a person
over the course of hours or days.

614
00:34:13,050 --> 00:34:17,320
Similarly, data pipelines like document
processing pipelines for agents often

615
00:34:17,320 --> 00:34:21,411
require durability, and, you know,
workflows can really help there.

616
00:34:21,914 --> 00:34:23,994
So those are kind of
the, the new use cases.

617
00:34:24,234 --> 00:34:28,834
And then there's this whole host of
traditional business workflows that

618
00:34:28,844 --> 00:34:32,184
have always needed durability and
where Postgres backed durability

619
00:34:32,184 --> 00:34:33,594
just makes it easier to provide it.

620
00:34:34,004 --> 00:34:36,054
And that's things like logistics.

621
00:34:36,224 --> 00:34:38,384
It's things like, like
a- like lab automation.

622
00:34:38,906 --> 00:34:42,693
It's things like new user registration,
making sure new users are properly

623
00:34:42,703 --> 00:34:45,923
updated in every single one of your
cor- of your corporation's databases.

624
00:34:47,263 --> 00:34:51,835
So we see this really broad spectrum
of use cases for workflows because

625
00:34:52,675 --> 00:34:59,819
really any important operation that
touches customer data, touches important

626
00:34:59,819 --> 00:35:03,699
data, it probably should be a workflow
if you're really operating at scale

627
00:35:03,699 --> 00:35:04,989
and you really care about reliability

628
00:35:06,020 --> 00:35:10,340
Qian: I think in summary it's we
really view Postgres as the central

629
00:35:10,360 --> 00:35:12,670
orchestrator to connect things together.

630
00:35:13,220 --> 00:35:16,840
You can connect external systems
to your local database, or you

631
00:35:16,840 --> 00:35:22,600
can do event processing to connect
multiple systems like your other

632
00:35:22,610 --> 00:35:26,960
databases or your email server,
payment system, all sorts of things.

633
00:35:27,190 --> 00:35:31,820
So we re- try to make sure your
Postgres is the glue for everything.

634
00:35:33,130 --> 00:35:33,300
Yeah

635
00:35:33,784 --> 00:35:36,994
Nik: And speaking of Postgres
maybe last question from me.

636
00:35:37,424 --> 00:35:41,191
So now Postgres 19 beta 2
just released, Postgres 20

637
00:35:41,201 --> 00:35:42,511
development has started already.

638
00:35:42,641 --> 00:35:48,551
What what do you think should be changed
in Postgres or brought to Postgres to make

639
00:35:48,651 --> 00:35:51,131
the product you are building even better?

640
00:35:52,548 --> 00:35:55,718
Peter: I think we do have to go…
I would like to go back to notify.

641
00:35:55,738 --> 00:36:03,958
Removing that global lock is not easy,
and I'm not sure if it's doable with the

642
00:36:03,958 --> 00:36:09,808
way notify is currently implemented, but
that area of the code hasn't really been

643
00:36:09,818 --> 00:36:14,418
worked on much since Postgres 9 and,
maybe there's something that could be done

644
00:36:14,418 --> 00:36:18,108
there to remove the footguns from notify.

645
00:36:19,048 --> 00:36:23,415
And even if you don't remove them,
I think notify's really complicated

646
00:36:23,415 --> 00:36:25,945
performance characteristics should
probably be, should probably be actually

647
00:36:25,945 --> 00:36:29,022
documented instead of having to figure
this out like through by… I mean,

648
00:36:29,192 --> 00:36:31,472
we figured this out basically by,
by reading the Postgres source code.

649
00:36:31,982 --> 00:36:33,002
That, that's one thing.

650
00:36:33,692 --> 00:36:36,912
Qian: Yeah, I think another thing
is about, like, how to ho- better

651
00:36:36,922 --> 00:36:42,062
s- even better support for larger
larger blobs or larger columns.

652
00:36:42,850 --> 00:36:45,880
when you-- when people are using DBOS,
sometimes they want to checkpoint, say, a

653
00:36:45,880 --> 00:36:48,254
PDF or something as part of the output.

654
00:36:48,714 --> 00:36:53,404
And still I know Postgres has some good
support for larger blobs right now.

655
00:36:53,404 --> 00:36:58,324
But I think most of the time, we still
recommend people to say "Upload your

656
00:36:58,784 --> 00:37:04,874
large documents to S3, then store a
pointer in the database." But that still

657
00:37:04,874 --> 00:37:09,734
leaves, a little bit gap in the durable
checkpointing, because there could be

658
00:37:09,734 --> 00:37:14,664
the case where you upload something
but you don't checkpoint or vice versa.

659
00:37:14,684 --> 00:37:19,254
It'll be great if we can really put
more, like, larger data into Postgres.

660
00:37:20,948 --> 00:37:24,318
Michael: There is a nice change,
I think, coming in 19 around that

661
00:37:24,478 --> 00:37:28,918
in that the, I think the default
TOAST compression is changing from

662
00:37:29,508 --> 00:37:34,198
like a, an older PGLZ, like a
Postgres specific compression

663
00:37:34,248 --> 00:37:36,438
algorithm to, I think it's LZ4.

664
00:37:37,050 --> 00:37:39,770
but yeah only minor on, on
the grand scheme of things.

665
00:37:40,148 --> 00:37:40,568
nice one

666
00:37:42,088 --> 00:37:42,108
Nik: Yeah.

667
00:37:42,388 --> 00:37:49,028
And speaking of notify, I think, yes,
global lock is still there, not removed.

668
00:37:49,028 --> 00:37:50,898
I remember some work
last year in PostgreSQL

669
00:37:51,826 --> 00:37:55,846
Peter: There was work in 19 on
improving the performance of

670
00:37:56,316 --> 00:38:01,476
broadcast, of the notify broadcast
when not all listeners are listening

671
00:38:01,506 --> 00:38:03,766
for all messages, which is great.

672
00:38:03,956 --> 00:38:05,076
That's a real performance improvement.

673
00:38:05,836 --> 00:38:08,516
The global lock is much harder to get rid
of, and I'm not sure if it's possible.

674
00:38:08,596 --> 00:38:09,306
But that is…

675
00:38:09,472 --> 00:38:11,302
Nik: work on removing that lock.

676
00:38:11,302 --> 00:38:12,592
I remember that, that work.

677
00:38:12,592 --> 00:38:17,502
I also remember I just founded my
email to document, as you said,

678
00:38:17,612 --> 00:38:22,052
the limitations from July 10, 2025.

679
00:38:22,072 --> 00:38:23,722
Nobody answered my email.

680
00:38:24,802 --> 00:38:26,072
So there is a patch proposed.

681
00:38:26,122 --> 00:38:27,212
Let's document it at least.

682
00:38:27,222 --> 00:38:27,522
Yeah,

683
00:38:27,634 --> 00:38:30,394
Peter: Yeah, the I, certainly 19 doesn't
include any work on the global lock.

684
00:38:30,414 --> 00:38:32,014
Maybe there's some in
development for 20, I'm not

685
00:38:32,084 --> 00:38:32,434
Nik: Yeah.

686
00:38:32,674 --> 00:38:33,064
Yeah.

687
00:38:33,124 --> 00:38:33,504
Yeah.

688
00:38:33,634 --> 00:38:34,414
It's worth checking.

689
00:38:34,744 --> 00:38:36,534
And that documentation should be changed.

690
00:38:36,534 --> 00:38:37,184
I agree with you.

691
00:38:37,724 --> 00:38:42,144
So maybe someone listening to us will
go and comment on that proposal, and

692
00:38:42,144 --> 00:38:43,764
we will get it documented at least.

693
00:38:44,454 --> 00:38:44,794
Great.

694
00:38:44,814 --> 00:38:45,714
Thank you so much.

695
00:38:45,764 --> 00:38:46,824
I enjoyed it a lot.

696
00:38:47,184 --> 00:38:50,234
I wish you all the best
with this wonderful product.

697
00:38:50,527 --> 00:38:54,147
I'm going to try it in a couple
of places, and I encourage

698
00:38:54,207 --> 00:38:55,577
our listeners to try it out.

699
00:38:55,807 --> 00:38:58,207
Especially it's open source, so
it's so easy to start, right?

700
00:39:00,653 --> 00:39:00,983
Thank you

701
00:39:01,851 --> 00:39:02,361
Michael: Nice one.

702
00:39:02,441 --> 00:39:03,471
Thanks very much, both of you

703
00:39:04,857 --> 00:39:05,617
Peter: Yeah, thanks a lot

704
00:39:06,163 --> 00:39:07,223
Nik: Yeah, have a great week.

705
00:39:08,179 --> 00:39:08,269
Qian: You

706
00:39:08,269 --> 00:39:08,539
too.

707
00:39:08,783 --> 00:39:08,913
Nik: Bye-bye