1
00:00:00,000 --> 00:00:04,160
Currently, banking stage does not keep up if we are receiving that much traffic.
2
00:00:04,160 --> 00:00:09,440
We will basically only receive into the scheduler some fraction of those packets
3
00:00:09,440 --> 00:00:14,800
and then have an incomplete review of all the transactions that we've received.
4
00:00:14,800 --> 00:00:20,560
We can only schedule off of those ones that we've deserialized and inspected to see which ones are the highest priority.
5
00:00:20,560 --> 00:00:27,200
And one of the main bottlenecks there is just the deserialization we use for our transaction type.
6
00:00:27,200 --> 00:00:31,520
ScrapingBits is brought to you by the following sponsors.
7
00:00:31,520 --> 00:00:39,520
MEV Protocol, maximize your ETH staking value with MEVETH, exclusively on MEV.io.
8
00:00:39,520 --> 00:00:45,040
And Composable, execute any intent on any chain coming soon to Mantis.app.
9
00:00:45,040 --> 00:00:48,640
That's MANTIS.APP.
10
00:00:48,640 --> 00:00:57,440
And Fastline Labs, the only MEV and intent-centric team that has a daily deodorant application rate of over 68%.
11
00:01:00,960 --> 00:01:04,960
GMGM everyone, my name is DaGachi, the host of ScrapingBits.
12
00:01:04,960 --> 00:01:10,480
And today, we have another ANSA participant, Andrew Fitzgerald.
13
00:01:10,480 --> 00:01:12,000
GM, sir, how's it going?
14
00:01:12,000 --> 00:01:14,560
Good, yeah, thanks for having me. Excited to be here.
15
00:01:14,560 --> 00:01:18,720
Excited to have you on. I'm sure a lot of people are going to enjoy this episode.
16
00:01:18,720 --> 00:01:24,560
Highly anticipated. Just for the people that aren't familiar with you, who are you and what do you do?
17
00:01:24,560 --> 00:01:29,280
So I'm an employee at HAN and my main focus is working on banking.
18
00:01:29,280 --> 00:01:34,800
Basically all things banking stage. My big pet project has been the transaction scheduler.
19
00:01:34,800 --> 00:01:36,240
The new one, not the old one.
20
00:01:36,240 --> 00:01:36,960
1.18?
21
00:01:36,960 --> 00:01:45,520
Yeah, it's becoming an option in 1.18. It'll be the default in 1.19 or 2.0, whatever we end up calling it.
22
00:01:45,520 --> 00:01:51,520
Yeah, super excited for that to get out to mainnet because there is a lot of very nice properties.
23
00:01:51,520 --> 00:01:56,480
Very excited to talk about this. And just before we do, how did you get into this position?
24
00:01:56,480 --> 00:02:00,080
What were you doing before? Give me your lore, your history.
25
00:02:00,080 --> 00:02:05,600
Yeah, sure. So I got my PhD in nuclear engineering at the University of Michigan.
26
00:02:05,600 --> 00:02:09,680
And obviously not a very traditional background for people doing crypto stuff.
27
00:02:09,680 --> 00:02:13,520
I was working on an HPC Neutronix simulation tool.
28
00:02:13,520 --> 00:02:18,080
And so a lot of the stuff I was doing was just optimizing a code for performance.
29
00:02:18,080 --> 00:02:25,680
And about half the work I was doing was coming up with new methods for simulation and then half implementing and optimizing.
30
00:02:25,680 --> 00:02:28,960
I found myself kind of enjoying that second part a bit more.
31
00:02:28,960 --> 00:02:34,400
And so when I finished my PhD, I just said, screw it. I'll go into finance, make more money,
32
00:02:34,400 --> 00:02:39,920
and do just be a software dev. So I worked at a trading firm for almost three years.
33
00:02:39,920 --> 00:02:44,000
And then a recruiter reached out to me about a position at Solana Labs.
34
00:02:44,000 --> 00:02:47,920
It sounded interesting. I was already somewhat crypto interested.
35
00:02:47,920 --> 00:02:52,800
I hadn't done any actual blockchain dev or was trading meme coins back then.
36
00:02:52,800 --> 00:02:58,720
I joined labs. Yeah, been there since until we moved on back in December.
37
00:02:58,720 --> 00:03:07,120
What a crazy career switch from nuclear to finance and then to crypto, straight DGN.
38
00:03:07,120 --> 00:03:14,240
Yeah, it's actually surprising how many of the nuclear PhDs and STEM PhDs in general go into finance.
39
00:03:14,240 --> 00:03:18,240
Yeah, I think usually the sciences just don't make that much money.
40
00:03:19,440 --> 00:03:25,840
And finance is just where it's at money-wise. So it just makes sense to switch over.
41
00:03:25,840 --> 00:03:30,160
Yeah, I was offered a postdoc position for like $35,000.
42
00:03:30,160 --> 00:03:33,200
I was like, maybe this isn't the right path for me.
43
00:03:34,080 --> 00:03:39,280
I mean, yeah, very interesting to do that stuff, but you need to join the dark side to make the real money.
44
00:03:39,280 --> 00:03:43,360
Yeah, it was definitely super interesting work. I do miss it at times, but yeah,
45
00:03:43,360 --> 00:03:46,320
it didn't make sense lifestyle-wise. Yeah, definitely.
46
00:03:46,320 --> 00:03:50,240
I feel like you got to mix science with engineering to at least make it
47
00:03:50,880 --> 00:03:55,200
a startup in some sense to actually make money through that cool stuff.
48
00:03:55,200 --> 00:04:00,800
Well, like a research law, I don't even know, really. Anyway, so you joined labs, right?
49
00:04:00,800 --> 00:04:04,880
How did you get into actually doing the scheduler and the banking stage?
50
00:04:04,880 --> 00:04:11,120
Why did you choose these specifically and why not consensus or the voting areas? Why this?
51
00:04:12,080 --> 00:04:17,200
It's actually kind of funny. One of the engineers who interviewed me basically gave me
52
00:04:17,200 --> 00:04:21,760
the scheduler design problem as an interview question. So when we were talking about it,
53
00:04:21,760 --> 00:04:25,680
I was very interested in it. It just seemed like an interesting problem with NP hard.
54
00:04:25,680 --> 00:04:31,280
So it's like, there's no set way to do this. It's very open to using heuristics to optimize.
55
00:04:31,280 --> 00:04:35,840
Everything's always changing. It's always super interesting to work on that sort of problem.
56
00:04:35,840 --> 00:04:39,600
And then when I joined, I realized like, oh, wait, they're actually working on this.
57
00:04:39,600 --> 00:04:43,200
So I just asked them, like, hey, can I do that? And yeah, I ended up
58
00:04:43,200 --> 00:04:47,600
just basically taking that on as a project after working at labs for a few months because
59
00:04:47,600 --> 00:04:51,200
I needed to get somewhat familiar with how the rest of the system worked first.
60
00:04:51,200 --> 00:04:57,840
Right. Interesting. So you're like, oh, this is my interview question. Let me just continue doing
61
00:04:57,840 --> 00:05:00,000
that interview question. Basically.
62
00:05:00,000 --> 00:05:05,520
Okay. So you have to rewrite the entire scheduler. So let's break it down. What actually is the
63
00:05:05,520 --> 00:05:08,960
scheduler and the banking stage before we get deep into it?
64
00:05:08,960 --> 00:05:15,120
Sure. So yeah, people have always referred to what's currently there as a scheduler. And I kind of
65
00:05:15,120 --> 00:05:19,360
don't like that because I don't think it is a scheduler. Right now, we basically have like
66
00:05:19,360 --> 00:05:24,960
four threads that just independently pull from a channel coming from like the SIG verify stage.
67
00:05:24,960 --> 00:05:29,600
And they're all just sort of. What's the SIG verify stage as well, by the way?
68
00:05:29,600 --> 00:05:34,480
Sorry. Oh, yeah, sure. It's like packets come into the network through like quick, and then
69
00:05:35,040 --> 00:05:41,200
we send those packets onto a stage called SIG verify. And it basically does some sanity checks
70
00:05:41,200 --> 00:05:46,000
that the packets are transactions. And then once it knows that like, hey, this is probably a
71
00:05:46,000 --> 00:05:52,480
transaction, then it will check all the signatures, like validly signing that message.
72
00:05:53,120 --> 00:05:56,240
Right. That way, by the time they get to banking stage, we know like, hey,
73
00:05:56,800 --> 00:06:00,800
these, you know, we don't have to verify the signatures there because it's a fairly expensive
74
00:06:00,800 --> 00:06:06,400
operation. No excellence. Yeah. SIG verify uses like a bunch of threads to do that in parallel.
75
00:06:06,400 --> 00:06:11,520
It's relatively easily parallelizable. Okay. Okay. And then what happens after that?
76
00:06:11,520 --> 00:06:17,120
You go to the banking stage. Yeah. So banking stage will then, depending on what time it is.
77
00:06:17,120 --> 00:06:21,440
So like, if you're not the leader, you will probably be just like buffering those packets
78
00:06:21,440 --> 00:06:26,960
and forwarding them to who is the leader. And if you are the leader, you will basically try to
79
00:06:26,960 --> 00:06:32,640
choose which ones you want to try to execute. And at least how it currently works, we have four
80
00:06:33,200 --> 00:06:38,480
banking stage threads, and they each just pull off that channel of SIG verify, put them in a buffer,
81
00:06:38,480 --> 00:06:44,320
sort them by priority. And they just try to grab like the first 64 non-conflicting transactions,
82
00:06:44,880 --> 00:06:49,680
non-conflicting, I mean like the read, write blocks. Like you can't have two transactions,
83
00:06:49,680 --> 00:06:53,760
like writing the same account in the same batch. Then they just will go through some checks,
84
00:06:53,760 --> 00:06:57,440
make sure no other thread is touching that account, grab the lock of that account,
85
00:06:57,440 --> 00:07:02,560
make sure we can fit those transactions into like the block space, actually execute them.
86
00:07:02,560 --> 00:07:08,960
And then we record them into the POH stream and then commit them. And then when we commit them,
87
00:07:08,960 --> 00:07:15,040
I think they get sent out over broadcast stage. Okay. Which just uses turbine to propagate those
88
00:07:15,040 --> 00:07:22,320
to the rest of the network. And you say sort by priority and how do you classify priority?
89
00:07:22,320 --> 00:07:29,040
Yeah, that's a good question. So it used to be just like a FIFO queue. And then sometime around
90
00:07:29,040 --> 00:07:35,280
when I joined, there's this concept of priority fees and there's this CU price, which is just like,
91
00:07:35,280 --> 00:07:41,600
Hey, how many micro-lamp courts per CU are you willing to pay to get this transaction executed?
92
00:07:41,600 --> 00:07:46,960
And so the old banking stage just takes that number, which is like specified in instruction
93
00:07:46,960 --> 00:07:51,440
on the transaction. Oh, okay. So if you want it to be number one, you could just max that out.
94
00:07:51,440 --> 00:07:56,640
Yep. Yeah. And you would have to pay more for that. Yeah, true. But I don't imagine anyone else
95
00:07:56,640 --> 00:08:00,960
would be maxing that out. Well, maybe now they would. But there was also an interesting thing.
96
00:08:00,960 --> 00:08:07,520
I think I spoke to someone else about MEV, basically MEVBOTS doing, not paying any priority
97
00:08:07,520 --> 00:08:13,920
fee at all. And they were still winning. I'm not sure why, but does that ring a bell by any chance?
98
00:08:13,920 --> 00:08:20,560
It depends. There's a lot of edge cases in the current banking stage where we have these more
99
00:08:20,560 --> 00:08:25,760
competing threads that can just let the lower priority transactions jump the queue if they
100
00:08:25,760 --> 00:08:33,040
just so happen to be the top priority in that thread. Right now each thread basically doesn't
101
00:08:33,040 --> 00:08:38,160
have a view of what other threads have in terms of transactions. So it might be at the front of
102
00:08:38,160 --> 00:08:44,960
its own local queue. So it's like a race condition. Yep. Yeah. And that's sort of the main motivation
103
00:08:44,960 --> 00:08:50,400
for having a separate scheduling thread. All the scheduling thread does is just have a global view
104
00:08:50,400 --> 00:08:56,800
of what that priority list should be. And then I send work to the worker threads, obviously.
105
00:08:56,800 --> 00:09:03,920
And you said the first 64 transactions, are they all being pushed into different threads themselves?
106
00:09:03,920 --> 00:09:10,320
Or is it like each 64 transactions is pushed into a new thread or is it shared between them all?
107
00:09:10,320 --> 00:09:16,880
It's like each thread has its local buffer. It will grab the first 64 non-quickly transactions
108
00:09:16,880 --> 00:09:23,200
and then it will execute those 64 transactions up to 64 transactions in serial. Okay. Okay.
109
00:09:24,320 --> 00:09:31,280
How does it check if it's non-conflicting? They're not being shared at all. So it would be like 64
110
00:09:31,280 --> 00:09:38,160
and then the 128 and then continuing. Well, 64, then the next 64, then the next 64,
111
00:09:38,160 --> 00:09:43,920
and they're all split into different threads. Oh, no. We've got like four threads. This is where
112
00:09:43,920 --> 00:09:48,720
drawing helps. We've got like four threads. Each of them have their own buffer. Okay.
113
00:09:48,720 --> 00:09:54,720
And then we use this thing called, we basically have like multiple iterators, 64 of them,
114
00:09:54,720 --> 00:10:01,600
in to each buffer. And they start at the front and then march forward to find, basically to create
115
00:10:01,600 --> 00:10:08,080
batches up to 64 transactions. And then each batch is just executed on that thread. They're not going
116
00:10:08,080 --> 00:10:14,800
to another thread for execution. Couldn't one batch actually be executed faster than other ones?
117
00:10:14,800 --> 00:10:19,840
Yep. Yeah. They're all just running as fast as they can. So like executing a batch on thread one
118
00:10:19,840 --> 00:10:24,240
is not going to wait for like the execution to finish on thread two. So what would actually
119
00:10:25,040 --> 00:10:31,280
cause one thread to execute faster than another? Just like simpler transactions, right? Like
120
00:10:31,280 --> 00:10:36,800
something like a simple transfer does not take as much time as a, like a swap on Jupiter.
121
00:10:36,800 --> 00:10:39,680
Rude. Okay. Cause it's going for the emulator, right?
122
00:10:40,240 --> 00:10:47,040
Yeah. Like the Jupiter stuff would have to like load that program and then, sorry, like build up
123
00:10:47,040 --> 00:10:52,720
the VM and then send it through there. There's like a lot of overhead in terms of like executing
124
00:10:52,720 --> 00:10:58,400
BPF stuff. Not a ton, but it's more than just native programs, which are not doing that.
125
00:10:58,400 --> 00:11:02,480
Yeah, definitely. Have you ever heard of like the mutex lock game where people are just locking
126
00:11:02,480 --> 00:11:08,720
each other's programs? I guess if a thread is looking for non-conflicting transactions,
127
00:11:09,440 --> 00:11:14,480
then if it's already in one thread, it just wouldn't, another thread wouldn't grab it
128
00:11:14,480 --> 00:11:19,120
because it's already being used by one. Is that correct? Yeah. Okay. That's how it works in like
129
00:11:19,120 --> 00:11:25,360
the current banking stage. Right. I see. Okay. That makes sense. Is that not in the new one at all?
130
00:11:25,360 --> 00:11:31,360
Yeah. It's like the new one basically keeps track of which threads have which locks. And so it's
131
00:11:31,360 --> 00:11:38,160
only ever going to send work to a thread if that work doesn't conflict with any other thread.
132
00:11:38,160 --> 00:11:46,320
Okay. So if I was an MEV bot and I see my competitor use, you know, N accounts and like
133
00:11:46,320 --> 00:11:52,480
one of the elements I actually just use as a lock in mine, it doesn't do anything. It's just locking
134
00:11:52,480 --> 00:11:58,240
it and I'm processing mine because I'm before them, then their transaction wouldn't actually be
135
00:11:58,240 --> 00:12:03,600
processed because I've held that lock. Right. Yeah. I think right now that's a big issue
136
00:12:03,600 --> 00:12:08,080
like the old banking stage because you don't necessarily have to pay more than them.
137
00:12:10,240 --> 00:12:14,000
Right. You can get lucky and just like grab that lock and then that prevents them from doing
138
00:12:14,000 --> 00:12:19,120
anything. Damning that. So they never have to get it. Right. That makes sense. In the new banking
139
00:12:19,120 --> 00:12:24,560
stage, as long as like those packets have arrived at the scheduler, then like whoever is paying more
140
00:12:24,560 --> 00:12:30,000
priority fees, like, you know, going to get that. That makes sense. Okay. I guess the fees aren't
141
00:12:30,000 --> 00:12:36,240
even that expensive either. So it's, if they both send the same amount of priority fee, who gets it?
142
00:12:36,240 --> 00:12:40,720
Is it just a race condition again at that point? I think it would be at that point, like default to
143
00:12:40,720 --> 00:12:46,000
whoever like arrived first, like a time race rather than, I actually don't remember how it's
144
00:12:46,000 --> 00:12:52,000
going to depend. I don't remember how that's resolved exactly. First in, first out probably.
145
00:12:52,000 --> 00:12:57,120
But if it's sorted by that, I guess it would be sorted by maybe the transaction hash, like the
146
00:12:57,120 --> 00:13:03,120
size is bigger than another one. I'm not sure. Or like, yeah, like time of inbound. That would make
147
00:13:03,120 --> 00:13:08,800
sense. Yeah. I think it's just time. I'd have to look, we use this min-nock for like sorting the
148
00:13:08,800 --> 00:13:16,000
priority order. I just don't remember how that handles like things exactly match. Okay. So what
149
00:13:16,000 --> 00:13:22,880
were the major flaws in the current bank, which, and how did that architecture change? What did you
150
00:13:22,880 --> 00:13:29,520
change for the new scheduler and banking stage? What was the old one? What were the problems? How
151
00:13:29,520 --> 00:13:35,600
did you fix it? Are there any more problems? Yeah. So the old banking stage, as I mentioned, had
152
00:13:35,600 --> 00:13:40,480
sort of these four independent threads. None of them knew what the other ones were doing.
153
00:13:40,480 --> 00:13:46,720
And so immediately we have this sort of locking problem. If we've got a bunch of, let's say there's
154
00:13:46,720 --> 00:13:51,200
like an NFT mint going on, everyone's trying to get like the highest, or sorry, trying to get that
155
00:13:51,200 --> 00:13:57,200
NFT first. So all of like the highest priority transactions in those four threads buffers are
156
00:13:57,200 --> 00:14:02,160
all going to be touching the same state. And then it's just like a race condition on who wins that.
157
00:14:02,160 --> 00:14:06,960
And so we have like a lot of account in use errors where basically we're not executing
158
00:14:06,960 --> 00:14:12,160
these transactions because the account is being used by another thread. It's just pretty inefficient.
159
00:14:12,160 --> 00:14:17,200
And then it's not clear like, you know, this was the next highest paying priority one, but when do
160
00:14:17,200 --> 00:14:22,560
I retry it? And that's difficult without having some way of knowing what the other threads have.
161
00:14:22,560 --> 00:14:27,600
And that's like a big issue with just the account locking. And then the other one is just there's
162
00:14:27,600 --> 00:14:32,720
no global view of priority. So no, none of the threads know what the actual highest paying
163
00:14:32,720 --> 00:14:37,280
transaction, like the most valuable transaction to pack is. So we could end up in a situation,
164
00:14:37,280 --> 00:14:42,240
you know, where just by chance, all of the very high priority transactions end up in one thread
165
00:14:42,240 --> 00:14:46,880
and a bunch of low priority ones end up in another thread. Thread with the low priority transactions,
166
00:14:46,880 --> 00:14:50,880
it's still going to go through those and pack a bunch of, you know, basically non-valuable
167
00:14:50,880 --> 00:14:55,360
transactions. And that could fill up our block. And then, you know, we're not selling to the
168
00:14:55,360 --> 00:15:00,000
people who are paying us the most. So inefficient from the leader's perspective.
169
00:15:00,000 --> 00:15:05,520
When a friend grabs a lock, what happened to that transaction that was using that same account?
170
00:15:05,520 --> 00:15:08,320
Does it just pushed all the way back to the bottom or what happens?
171
00:15:08,320 --> 00:15:11,520
Do you mean if it failed to grab the lock? Yeah, exactly.
172
00:15:12,080 --> 00:15:17,920
Yeah. It basically, an old scheduler will basically just retry that transaction the next time we do
173
00:15:17,920 --> 00:15:23,520
an iteration. It won't work again until we've gone through like the entire rest of the buffer.
174
00:15:23,520 --> 00:15:29,520
Would it still be at the top of it or would it be pushing other transactions in front of it though?
175
00:15:29,520 --> 00:15:34,400
Yeah, I guess like logically it would then go sort of back to the bottom of the buffer.
176
00:15:34,400 --> 00:15:36,400
Oh God. Yeah, it's not great.
177
00:15:37,040 --> 00:15:41,120
Yeah. Okay. So what were the solutions that you did with the new scheduler?
178
00:15:41,120 --> 00:15:47,120
Yeah, the new scheduler will basically, it has a global view of the priority because there's only
179
00:15:47,120 --> 00:15:52,160
a single priority queue. So we always know like, this is the most valuable transaction for me to
180
00:15:52,160 --> 00:15:57,200
pack in terms of reward per see you. Okay. So that's immediately better. But then we don't have this
181
00:15:57,200 --> 00:16:02,240
locking problem because we're keeping track of which of our threads have which locks. So we're
182
00:16:02,240 --> 00:16:09,200
only going to send things if they're not conflicting with other things. And then if there's a conflict,
183
00:16:09,200 --> 00:16:15,360
say like I sent some transaction one that touched account A and B to thread one and another that
184
00:16:15,360 --> 00:16:22,640
touched C and B to thread two. At that point, I can't schedule some third transaction that touches
185
00:16:22,640 --> 00:16:28,960
B and C. We're conflicting like across threads. Right. And at that point, we will basically block
186
00:16:28,960 --> 00:16:34,240
on that transaction and not schedule anything that conflicts with it until we can schedule that one.
187
00:16:34,240 --> 00:16:39,680
If that's like the next highest paying. Okay. So sort of in that way, it's guaranteeing that
188
00:16:39,680 --> 00:16:46,160
we're respecting like the priority order in our buffer, never executing things out of order.
189
00:16:46,160 --> 00:16:51,760
Okay. Never is a strong word there because we like queue up work and, you know, we might have
190
00:16:51,760 --> 00:16:57,760
sent things to be worked on and then a new higher paying transaction comes in. Obviously, because
191
00:16:57,760 --> 00:17:02,160
we've already scheduled that work, we can't unschedule it. Okay. Let's say you did a
192
00:17:02,160 --> 00:17:07,840
transaction with a priority fee of like 10. It's in one of the threads now and I send a transaction
193
00:17:07,840 --> 00:17:13,520
that's 11. Will your one be processed before mine? You know mine is, it's like still in the thread,
194
00:17:13,520 --> 00:17:18,400
but your one will still be processed, right? Yeah. Basically what once it's sent by the scheduler,
195
00:17:18,400 --> 00:17:22,960
it's, you know, in flight, we're not going to go back and stop that. We could definitely optimize
196
00:17:22,960 --> 00:17:27,200
there. I think there's some work that needs to be done there right now. That's how it works.
197
00:17:27,200 --> 00:17:30,640
And then when it's processed, what does it actually go? It's building the block, right?
198
00:17:31,200 --> 00:17:37,760
Yeah. So we have like a separate scheduling thread and that's just sending these transactions to
199
00:17:37,760 --> 00:17:42,480
worker threads, which there are like four or more. And then those worker threads basically do what
200
00:17:42,480 --> 00:17:47,040
the old bank of the stage was doing. They, you know, grab the account locks, just make sure that
201
00:17:47,040 --> 00:17:51,440
I didn't screw anything up on the scheduler. Run the normal checks, technical loads and executes
202
00:17:51,440 --> 00:17:56,160
and long records of commits. And then recording goes into like the actual POH stream and then,
203
00:17:56,720 --> 00:18:03,120
you know, broadcast those transactions. Does it record in like an iterator or is it all at once?
204
00:18:03,120 --> 00:18:07,760
Let me clarify that actually. So as you're processing transactions and they're being put
205
00:18:07,760 --> 00:18:13,520
into like, let's say this vector is, are you waiting for the vector to be finished and no
206
00:18:13,520 --> 00:18:20,880
more being sent for you to finish the block or is it continuously adding? So let's say Fred A spits
207
00:18:20,880 --> 00:18:27,120
out a new one, Fred B spits out a new one. It processes Fred A's transaction as an iterator and
208
00:18:27,120 --> 00:18:33,680
then Fred B's transaction right after, or is it like stacking them and then putting it all in?
209
00:18:33,680 --> 00:18:39,520
And is there any way that someone could actually change the order after the process and the
210
00:18:39,520 --> 00:18:45,440
scheduler? So once it's been recorded, you can't change the order. Okay. Or it would be difficult
211
00:18:45,440 --> 00:18:52,000
because we broadcast that out already potentially. We can't like say pixie faxies, but yeah, in terms
212
00:18:52,000 --> 00:18:58,880
of like, you know, recording immediately versus stacking up, it's sort of a mix. So the scheduler
213
00:18:58,880 --> 00:19:04,800
doesn't schedule individual transactions. It schedules batches of transactions. Okay. And
214
00:19:04,800 --> 00:19:10,400
these batches of transactions, we like will load and execute all of them and then record them as
215
00:19:10,400 --> 00:19:16,320
like a single mix and hash into the POH. Basically the reason we, I don't know the whole context,
216
00:19:16,320 --> 00:19:22,640
but one of the reasons is that we basically have like this lock around the POH thread. And so we
217
00:19:22,640 --> 00:19:27,680
don't want to be grabbing that lock every time we had a process transaction overhead and doing so.
218
00:19:28,240 --> 00:19:33,920
So it's more efficient overall. I think when I benched, this is like 30% faster to like batches
219
00:19:33,920 --> 00:19:40,000
versus individual transactions. Right. Got you. Okay. Since you're doing batches, let's do the
220
00:19:40,000 --> 00:19:45,360
priority thing again. When you have a priority of 10, I have a priority of 11, your one's in the
221
00:19:45,360 --> 00:19:51,280
thread before mine. My one comes right after. Your one's been put into this batch. Does mine
222
00:19:51,840 --> 00:19:57,280
go below yours, even though my priority is higher? Or is the priority really just for those threads
223
00:19:57,280 --> 00:20:03,440
to basically select what to go first? It's not like actually in order of priority for...
224
00:20:04,000 --> 00:20:12,640
Yeah. So we are not like, I think Anatoly had this idea of basically enforcing priority on order
225
00:20:12,640 --> 00:20:17,600
on the block and like sort of wait until all of the block is completed and then reorder things.
226
00:20:17,600 --> 00:20:24,880
Right. Yeah. That is not what we're doing right now. The priority is strictly for the leader to
227
00:20:24,880 --> 00:20:30,160
attempt to like choose the most valuable transactions. It's still free to order them however they want.
228
00:20:30,160 --> 00:20:36,160
In your case right now, because that first transaction was already sent off to be executed,
229
00:20:36,160 --> 00:20:41,120
it might've already been completed and the scheduler was just not aware. Your transaction would still
230
00:20:41,120 --> 00:20:45,280
go after that. It would just go to the next batch. Okay. Got you. What are the challenges you kind of
231
00:20:45,280 --> 00:20:52,800
see or saw, or maybe you are seeing still, with actually building something that is efficient at
232
00:20:52,800 --> 00:20:59,280
like sorting transactions, processing them, especially with spam in the equation. So what
233
00:20:59,280 --> 00:21:04,800
are the challenges you kind of face with the spam? How do you think to deal with it or try to deal
234
00:21:04,800 --> 00:21:10,960
with it? Maybe it hasn't worked, maybe it has worked, just that kind of stuff. Yeah. So one of
235
00:21:10,960 --> 00:21:15,920
the things that we'd noticed some time ago, we noticed this because of the banking trace data
236
00:21:15,920 --> 00:21:21,840
that we added in, which lets us inspect which packets were in banking stage that ended up not
237
00:21:21,840 --> 00:21:26,800
getting into the block. That's been super useful for like going back and seeing, you know, why
238
00:21:26,800 --> 00:21:32,320
wasn't this blocked back there efficiently? And after we added that, we were seeing on the internet
239
00:21:32,320 --> 00:21:38,160
that there were these malicious spammers that were sending out a transaction with high priority fee
240
00:21:38,160 --> 00:21:43,360
that locked a bunch of accounts basically to block their competitors from accessing that state. Yeah.
241
00:21:43,360 --> 00:21:48,640
And they had like an invalid fee there. They couldn't pay the fee. And the old banking stage
242
00:21:48,640 --> 00:21:53,760
at that time was still like scheduling it first. And so it like basically let them take the locks
243
00:21:53,760 --> 00:21:59,360
and block their competitor. So we added in a bunch of like checks before we actually
244
00:22:00,400 --> 00:22:05,520
send things off to be processed. So we check before we ever take locks, but like, hey, this can
245
00:22:05,520 --> 00:22:11,280
pay the fees. We basically started doing a bunch of checks earlier on on these transactions. And
246
00:22:11,280 --> 00:22:16,160
often we can do that, you know, before we'd come leader. So like, if you can't pay the fee now,
247
00:22:16,160 --> 00:22:20,400
I'm just going to drop you. You know, there's a high likelihood that you're still not going to
248
00:22:20,400 --> 00:22:24,080
be able to pay the fee when I'm due to come leader. There's an off chance that maybe you've had a
249
00:22:24,080 --> 00:22:29,520
funding transaction in there, but from my perspective, that's not worth the risk. You know,
250
00:22:29,520 --> 00:22:35,120
keep more transactions that I know are more valuable. That can make sense. Yeah. Cause
251
00:22:35,120 --> 00:22:39,920
people do funding transactions. I don't even know that. I don't know of anybody doing that. It's
252
00:22:40,720 --> 00:22:43,840
probably doesn't work. It probably doesn't work very well anymore. I probably,
253
00:22:43,840 --> 00:22:47,440
anybody was, they're not doing it anymore. Cause I would have screwed them over.
254
00:22:47,440 --> 00:22:52,080
Yeah. I would imagine it working at all. If you just see like a standalone transaction
255
00:22:52,080 --> 00:22:58,560
that isn't paying anything, I don't know. It's, yeah, it's a bit weird. I'm just saying. Yeah.
256
00:22:58,560 --> 00:23:04,560
The biggest problem is spam right now, the congestion issue, spam, of course. And then also
257
00:23:04,560 --> 00:23:12,240
like malicious adversarial congestion, which is the major, like the major issue at the moment.
258
00:23:12,240 --> 00:23:16,480
Do you think that's like a network issue, like a sorting of transactions issue,
259
00:23:17,040 --> 00:23:19,680
or maybe even the whole pipeline? What are your thoughts on that?
260
00:23:19,680 --> 00:23:24,480
Yeah, I think it's a problem sort of in the whole pipeline. Actually, it's really interesting. The
261
00:23:24,480 --> 00:23:29,360
things I've been working on this week, a lot of it has been networking issues. The networking
262
00:23:29,360 --> 00:23:34,960
stack had a bunch of really inefficient spots. It's my understanding. I'm not super familiar.
263
00:23:34,960 --> 00:23:41,040
And especially with relations like the fake weighted QoS, there was bugs in there that made
264
00:23:41,040 --> 00:23:46,480
it not work as well. People were spanning like to open up new connections that let them just get in
265
00:23:46,480 --> 00:23:52,160
their transactions and they would spam a bunch of transactions to basically try to get first in one
266
00:23:52,160 --> 00:23:58,480
of these four independent banking threads. Right. Yeah. So even just moving, it's like other people
267
00:23:58,480 --> 00:24:03,520
have been working on these networking fixes, which helps reduce that on the network networking side.
268
00:24:03,520 --> 00:24:07,840
But until we move to the new scheduler, there's sort of still this incentive to spam because
269
00:24:07,840 --> 00:24:12,480
you might get lucky, end up on a thread where you're not conflicting with anything high priority.
270
00:24:12,480 --> 00:24:17,280
So you get first and you win the race. Doesn't happen in the new schedulers, but really reduces
271
00:24:17,280 --> 00:24:23,440
the incentive to spam in terms of banking stage. I think there's also like pretty significant issues
272
00:24:23,440 --> 00:24:31,040
in terms of just our sort of pipeline throughput. The networking stack used to be,
273
00:24:31,600 --> 00:24:37,280
my understanding, used to be the bottleneck or the major bottleneck. And with the recent changes
274
00:24:37,280 --> 00:24:43,840
that is becoming less the case. And now we have bottlenecks in SIGTHERIFY and that can only do
275
00:24:43,840 --> 00:24:48,960
like, I forget it's like 200k TPS is what someone told me recently. All those packets are feeding
276
00:24:48,960 --> 00:24:54,240
into banking stage and actually currently banking stage does not keep up if we are receiving that
277
00:24:54,240 --> 00:25:00,240
much traffic. We will basically only receive into the scheduler some fraction of those packets.
278
00:25:00,240 --> 00:25:06,080
And then we basically have an incomplete review of all the transactions that we've received.
279
00:25:06,080 --> 00:25:11,200
We can only schedule off of those ones that we've deserialized and inspected to see which ones are
280
00:25:11,200 --> 00:25:17,040
the highest priority. And I've been working a lot with another Hans Ledeb Alphandre who
281
00:25:17,840 --> 00:25:22,880
done a bunch of like profiling work. And one of the main bottlenecks there is just the
282
00:25:22,880 --> 00:25:29,120
deserialization we use for our transaction type. It does a bunch of like internal allocations.
283
00:25:29,120 --> 00:25:34,720
So I've been working on this this week for a bunch and it's quite interesting. There's this
284
00:25:34,720 --> 00:25:40,400
version transaction type and it has like five, at least five internal VECs that it has to like
285
00:25:40,400 --> 00:25:45,200
allocate on the heap. And every time you do like a heap allocation, you're basically risking this
286
00:25:45,200 --> 00:25:51,520
really long call or has to maybe do some like bookkeeping work in whatever allocator you're using.
287
00:25:51,520 --> 00:25:57,040
And most of the time that's fast, it's just going to like some thread local cache of memory that
288
00:25:57,040 --> 00:26:03,440
I think Jmalloc uses. But occasionally, in particular when like the memory pattern doesn't
289
00:26:03,440 --> 00:26:09,120
fit in with like recent accesses and allocations, then you're sort of risking these long calls.
290
00:26:09,120 --> 00:26:15,120
I just take like 10 milliseconds sometimes to like a few hundred transactions and it's just
291
00:26:15,120 --> 00:26:20,160
ridiculously slow. Okay, yeah, most of the time it's fast. Basically, yeah, I've been working
292
00:26:20,160 --> 00:26:26,320
this week and like that's like a huge bottleneck in banking stage. And we're spending like 30 or
293
00:26:26,320 --> 00:26:32,240
40% of the time just deserializing these transactions. Oh, wow. Yeah, 10 milliseconds.
294
00:26:32,240 --> 00:26:40,640
So a long time when the block is for exactly. And yeah, you know, I've been working on like
295
00:26:40,640 --> 00:26:45,040
this tech project and I don't know if other people will go for it because it uses unsafe rust. But
296
00:26:45,040 --> 00:26:51,200
like all that super unnecessary, we have this like serialized packet, we can just calculate offsets
297
00:26:51,200 --> 00:26:58,080
into it. That is extremely fast. I can do it in, you know, 20, 30 nanoseconds per packet.
298
00:26:58,080 --> 00:27:04,400
And so basically what used to be taking for two, 300 milliseconds to deserialize like 90,000
299
00:27:04,400 --> 00:27:08,960
transactions, I can do that in two milliseconds now. What was the solution for that? Basically,
300
00:27:08,960 --> 00:27:13,840
just creating like a new transaction type that doesn't do these internal allocations.
301
00:27:13,840 --> 00:27:19,040
Oh, okay. Got you. So it's kind of like a big change because we'd have to change our transaction
302
00:27:19,040 --> 00:27:22,400
type everywhere. So I don't know if other people are going to go along with this quite yet. Oh,
303
00:27:22,400 --> 00:27:27,120
yeah, you have to change the type. Like a super recent development. Yeah. Interesting. But yeah,
304
00:27:27,120 --> 00:27:31,040
I think that that's like a huge problem is basically if banking stage isn't keeping up
305
00:27:31,040 --> 00:27:36,400
with the packet signal if I sending it, then banking stage has this sort of incomplete view
306
00:27:36,400 --> 00:27:41,120
on what transactions it should even schedule. And there might be, you know, a really juicy
307
00:27:41,120 --> 00:27:45,600
transaction in there and that paid me a ton of fees. And I'm just not saying it might be
308
00:27:45,600 --> 00:27:51,040
serialization is so slow. I feel like you don't even have to do full, full transaction checks,
309
00:27:51,040 --> 00:27:56,240
either. I mean, I could be wrong, but you can just like simulating half of it or partially
310
00:27:56,240 --> 00:28:01,120
to some degree. Maybe, maybe there are heuristics that. Oh, yeah, we don't simulate them. We don't
311
00:28:01,120 --> 00:28:05,760
simulate them like when we're receiving. Oh, okay. It's like not being like the whole transaction
312
00:28:05,760 --> 00:28:10,320
check. What do you simulate them in the banking stage? So we just execute them in the banking
313
00:28:10,320 --> 00:28:15,520
stage because I guess the same process of simulation, except at the end, we like commit
314
00:28:15,520 --> 00:28:21,520
those results back to the accounts database. Oh, okay. So when do you simulate in the pipeline?
315
00:28:21,520 --> 00:28:26,720
The only place we simulate in the pipeline is like on our PCs. Like we don't actually do that in the
316
00:28:26,720 --> 00:28:33,760
validator. Oh, okay. Yeah. I think, I think Gido does simulation in like their block engine, but
317
00:28:33,760 --> 00:28:40,000
that's not like part of the yeah. Yeah. I did the one with seg the other day. Okay. Cool. So I was
318
00:28:40,000 --> 00:28:44,880
talking about all about that. If you were, do you think the execution is actually slow at all?
319
00:28:44,880 --> 00:28:51,360
Yeah, execution is definitely slow in some aspects and I think we can improve things there. I'm not
320
00:28:51,360 --> 00:28:56,800
that familiar with like the VLU stuff. Some of it comes down to just like app developers. You're not
321
00:28:56,800 --> 00:29:02,080
writing a smart contracts that efficient, that is efficient. And there's no way I can execute that
322
00:29:02,080 --> 00:29:07,360
efficiently. Yeah, that makes sense. But at the same time, like Percy, I think that we can probably
323
00:29:07,360 --> 00:29:13,120
do things faster than we are currently. When I was speaking to Joe, one of your co-workers,
324
00:29:13,120 --> 00:29:20,640
he was working on runtime V2 for, for Agave. And we talked all about that. And I think it does seem
325
00:29:20,640 --> 00:29:26,320
very inefficient because you're all spinning up, up and down a new VM each time. I think for every
326
00:29:26,320 --> 00:29:30,800
instruction, not even every transaction, it's for every instruction in the transaction. And yeah,
327
00:29:30,800 --> 00:29:35,680
as you said, like if something is super convoluted and there's just like a lot of shit happening
328
00:29:35,680 --> 00:29:40,960
and you can probably just like rewrite it. Maybe if you want to be super fast, maybe you have
329
00:29:40,960 --> 00:29:47,200
something that caches that program, but actually optimizes it automatically. So it's faster. If it
330
00:29:47,200 --> 00:29:53,600
like, let's say like a radium program, if that's just used all the time, that shitty program,
331
00:29:53,600 --> 00:29:59,840
I'm not saying it's shitty, but like, let's say it is for, for this purpose. Yeah. I think I get what
332
00:29:59,840 --> 00:30:04,800
you're saying. Yeah. Then you optimize it and cache the optimized version. So it runs faster.
333
00:30:04,800 --> 00:30:09,840
Yeah. I don't know if that's like technically possible. I know we do like some sort of like
334
00:30:09,840 --> 00:30:16,320
compilation and like stories compiled like programs, which I think is effectively that. I
335
00:30:16,320 --> 00:30:19,840
don't think it's going to do like much optimization though. I don't think it's going to like change
336
00:30:19,840 --> 00:30:25,040
the actual like core logic, the BPA instructions we're saying to do. Well, I guess if it's, if you
337
00:30:25,040 --> 00:30:30,800
like customize the bytecode as much as possible and there's less text. I think the issue, the issue
338
00:30:30,800 --> 00:30:35,920
with doing something like that, I think it's technically possible. Like you're just saying
339
00:30:35,920 --> 00:30:40,800
like, Hey, everyone's using radium. What's right. Are there any like radium version that's more
340
00:30:40,800 --> 00:30:45,680
efficient? Exactly. Just optimize radium itself. It does the exact same thing. It's just less,
341
00:30:45,680 --> 00:30:50,400
you know, less shit to actually execute. Yeah. I think the main issue with like in the pro trick
342
00:30:50,400 --> 00:30:55,040
that one, it's like a ton of work, but two is if it's using like a different amount of CUs,
343
00:30:55,600 --> 00:31:01,840
then that can lead to like consensus differences. So unless like all of the clients, so like class
344
00:31:01,840 --> 00:31:07,840
and firedance or you know, we're all using like the same optimized version, then like we're going
345
00:31:07,840 --> 00:31:12,240
to end up with something that fails consensus. Maybe an optimized version would be good for
346
00:31:12,240 --> 00:31:20,640
simulations. Yeah. Yeah. Like, but is that really the only thing? Yeah. I don't know. I think if we
347
00:31:20,640 --> 00:31:25,920
want programs that are faster, it's sort of down to dev a little bit to optimize their own stuff.
348
00:31:26,480 --> 00:31:32,160
You know, we can make the VM faster in general, but we shouldn't be too thinking winners. No.
349
00:31:32,160 --> 00:31:37,040
Yeah. That makes sense. Okay. Hmm. Yeah. I'm trying to think. And actually, yeah, an interesting
350
00:31:37,040 --> 00:31:42,640
aspect of like the new scheduler compared to the old one, the old one, the old banking stage just
351
00:31:42,640 --> 00:31:49,440
went off of that, that CU price in terms of like priority, but the new scheduler will, it's sort of
352
00:31:49,440 --> 00:31:56,240
by like reward over transaction costs. Like inherently it's sort of prioritizing people who
353
00:31:56,240 --> 00:32:02,880
request fewer CUs, per use fewer CUs. Oh, really? So like optimization is actually mattering now.
354
00:32:02,880 --> 00:32:07,200
Yeah. It's like actually incentivizing people to optimize their app. Oh, heavy. They optimize their
355
00:32:07,200 --> 00:32:14,000
apps. They can like pay the same fee, but get much higher priority. So the less CU, the actually
356
00:32:14,000 --> 00:32:20,800
higher priority in the new scheduler. Yep. Okay. Bring in the optimizers. Yeah. I'm excited to like
357
00:32:22,560 --> 00:32:26,640
it's not going to like change overnight as we like roll out the scheduler, but it'll be interesting
358
00:32:26,640 --> 00:32:32,400
to see maybe how that evolves over time. I was thinking of this before because coming from EFE
359
00:32:32,400 --> 00:32:38,000
as an optimizer, like riding real bike code, it's the benefit obviously with gas. But if you think
360
00:32:38,000 --> 00:32:43,680
about it, if there's a limited amount of shit to store and also simulation wise and execution,
361
00:32:44,320 --> 00:32:50,400
it's always just better to have less because it's just less overhead to actually compute. So this is
362
00:32:50,400 --> 00:32:57,040
like another benefit incentivizing a bit more to actually nerd snipe some optimizers like myself.
363
00:32:57,040 --> 00:33:03,680
Yeah. It actually has not marginal benefit, but like major benefit now. So that's fantastic. Great
364
00:33:03,680 --> 00:33:10,400
change. Yeah. Give him a promotion guys. Come on. But I think like assembly in Rust is actually such
365
00:33:10,400 --> 00:33:15,200
a great skill as well. So like people that will learn this will get so good at Rust and like
366
00:33:15,200 --> 00:33:20,640
understand a lot better. So I think it's like a massive W honestly, not just for Solana, but
367
00:33:20,640 --> 00:33:26,320
for actually onboarding devs into low level Rust engineering as well. I haven't written like any
368
00:33:26,320 --> 00:33:31,920
smart contract code myself or like nothing, not like a simplistic example. Like I don't know too
369
00:33:31,920 --> 00:33:36,880
much about like how to optimize it to be honest. Oh, okay. That's like my entire background. Okay.
370
00:33:36,880 --> 00:33:42,560
Gotcha. Yeah. Optimizing security, fuzzing, all that stuff. That's kind of my bread and butter.
371
00:33:42,560 --> 00:33:47,760
Yeah. It seemed like fun work. I had a two billion dollar shit. I love, you know, virtual machines as
372
00:33:47,760 --> 00:33:54,640
well, emulating them, simulating stuff. I have here, I feel it was very easy to do. Solana seems
373
00:33:54,640 --> 00:33:58,960
a lot more complex because it's registry based, because it's Rust, but still fun. None of the
374
00:33:58,960 --> 00:34:03,840
less, right? That's sick though. Is there any other like giant optimizations you did that people
375
00:34:03,840 --> 00:34:10,480
should be aware of for this upcoming schedule? Not that I can really think of off the top of my head.
376
00:34:10,480 --> 00:34:17,680
Yeah. Okay. So there's like the threads changed, this CU's changed. That's really cool. I like that.
377
00:34:17,680 --> 00:34:23,600
Is there anything you're seeing like other validators do that differ from, you know,
378
00:34:23,600 --> 00:34:29,920
GAVE or the JITALabs one? What are they really prioritizing in these that you see? I guess like
379
00:34:29,920 --> 00:34:35,840
a lot of validators are trying to do like MEV or like voting MEV, I guess, optimizing the voting
380
00:34:35,840 --> 00:34:41,440
part. But what do you see like the most efficient, low hanging fruits for the validator are?
381
00:34:41,440 --> 00:34:48,240
That's a good question. I think that there is like a lot of vote optimization stuff that could be done.
382
00:34:48,240 --> 00:34:55,200
I don't know too much about how we vote currently, but it seems like we've seen people have been
383
00:34:55,200 --> 00:35:00,320
making like changes to how their validators voting to get more like vote credits. Yeah.
384
00:35:00,320 --> 00:35:07,360
Some of that's sort of dubious at best and that sort of then touches like timely vote credit
385
00:35:07,360 --> 00:35:14,080
proposal to incentivize people to like keep consensus alive. Yeah. Well, yeah, everyone
386
00:35:14,080 --> 00:35:19,520
like delays their voting and just doesn't vote that they're waiting for the port to be selected.
387
00:35:19,520 --> 00:35:24,320
Yeah. Then we're just kind of going to kill ourselves. See, I think that there's like
388
00:35:24,320 --> 00:35:30,080
proposals like that, which can help address this. But yeah, I think there's like this sort of some
389
00:35:30,080 --> 00:35:36,240
issues with like when we choose to fork when we're leader versus not. I think there's probably too
390
00:35:36,240 --> 00:35:42,960
much waiting right now. And yeah, again, I'm not super familiar with this. I know another engineer,
391
00:35:42,960 --> 00:35:49,280
Justin Starry, is working on like improving some of this logic. And I know that he has got a few
392
00:35:49,840 --> 00:35:56,160
work in progress things to sort of improve the skip rate here of the validator improved by
393
00:35:56,160 --> 00:36:02,160
in reducing that time sensitive time waited as well. So the faster you do it, the more
394
00:36:02,160 --> 00:36:07,520
the better it is rather than waiting for other people like other votes and all that other shit.
395
00:36:07,520 --> 00:36:12,640
That's I think that's pretty important. Yeah, I think the scheduler was made that. Oh,
396
00:36:12,640 --> 00:36:18,960
that's where all the kind of shit happens. Because all the the mutex locks. Is there like mutex? I
397
00:36:18,960 --> 00:36:25,760
mean, programming mutex is quite difficult anyway, at least doing it efficiently in a blockchain like
398
00:36:25,760 --> 00:36:32,160
this word, you know, the lock matters, like, it actually changes whether you land or if you don't
399
00:36:32,160 --> 00:36:39,040
land. What are you seeing like any other kind of adversarial games or anything interesting
400
00:36:39,040 --> 00:36:46,080
in the current scheduler banking stage? And whether that's might happen in the new one that you've
401
00:36:46,080 --> 00:36:51,760
built? So we have seen some things around like abusing this the fact that like the old scheduler
402
00:36:51,760 --> 00:36:57,120
just uses raw CU price. But again, that's not really an issue in a new one, because it has this
403
00:36:57,120 --> 00:37:04,480
reward over cost priority or prioritization in it. But yeah, other than that, not not too many like
404
00:37:04,480 --> 00:37:10,320
not too many things that I can think of on top of my head. Okay. And let's let's say people wanted
405
00:37:10,320 --> 00:37:15,840
to continue optimizing. Is there anything you just didn't optimize in the scheduler banking stage
406
00:37:15,840 --> 00:37:22,080
that you would like to future wise? Anything that kind of sparked your interest, you just didn't have
407
00:37:22,080 --> 00:37:27,680
a chance to anything that seemed important? Yeah, I mean, it's definitely like still a work in
408
00:37:27,680 --> 00:37:34,080
progress. There have been like recent PRs. So an issue sort of going back in time a little bit.
409
00:37:34,080 --> 00:37:39,120
People don't make accurate CU requests. And generally, we've seen like that people will
410
00:37:39,120 --> 00:37:45,840
request like 70 times more than they need. Oh, ridiculous. Like it's insane in some cases. And
411
00:37:45,840 --> 00:37:50,960
so the old scheduler, you know, we're basically there's no trust in like what's the use you're
412
00:37:50,960 --> 00:37:55,360
requesting. And when I wrote the new scheduler, we basically were operating off that assumption
413
00:37:55,360 --> 00:38:00,560
that these were not accurate at all. And why are we not accurate? Just because people request
414
00:38:00,560 --> 00:38:06,320
like 70 times more in some cases than they need. Oh, okay. So basically, it's like I can't really
415
00:38:06,320 --> 00:38:11,840
trust these requests and sees and the initial scheduler was written with that sort of assumption
416
00:38:11,840 --> 00:38:17,840
in mind. But then as I've like made this change to do the reward over cost prioritization, we've seen
417
00:38:17,840 --> 00:38:22,480
that curve ratio come down and the values that we're seeing in like the transactions that I'm
418
00:38:22,480 --> 00:38:27,920
not making into the block are much lower. You know, worst case people are average like requesting like
419
00:38:27,920 --> 00:38:35,200
saying like five times more use the rainy, which isn't so bad. And so like recently been doing some
420
00:38:35,200 --> 00:38:40,480
load balancing stuff using the CUs rather than just like transaction counts, which means that
421
00:38:40,480 --> 00:38:46,160
like basically work is spread more evenly across the threads. We can like sort of recent development
422
00:38:46,160 --> 00:38:51,840
because until recently, we didn't know that we could rely on the CUs at all. Why does having
423
00:38:51,840 --> 00:38:58,240
seven X, you matter, see, see use that you really need actually matter? I don't have any idea of
424
00:38:58,240 --> 00:39:04,720
like how long you're actually going to take. You can take anywhere from zero to a whole amount of
425
00:39:04,720 --> 00:39:11,120
those CUs. So it's like, that's not useful to me to load balances. Right? If I'm trying to schedule
426
00:39:11,120 --> 00:39:17,840
like similar amounts of work to each thread, it's not useful if you know, like my range of times for
427
00:39:17,840 --> 00:39:23,680
each transaction is like zero to five milliseconds, too big of a range to like make a useful
428
00:39:23,680 --> 00:39:30,240
explanation. Awesome. Right. Got you. Okay. Okay. Cause like, I guess there is a time associated
429
00:39:30,240 --> 00:39:37,360
with execution with the amount of CUs. What was kind of like your estimation of what's like the
430
00:39:37,360 --> 00:39:44,640
average amount of CUs like, I think it's like 30 CUs per micro is like the cluster average. Okay.
431
00:39:44,640 --> 00:39:51,040
30 CUs. I think that's like an out of date value, even with the new stuff in the new scheduler.
432
00:39:51,040 --> 00:39:56,640
I'm trying, I'm not like using that value. I'm not going to like associate time estimate with like
433
00:39:56,640 --> 00:40:02,400
the CU that could be inaccurate still, but more just trying to load balance, like how many CUs I
434
00:40:02,400 --> 00:40:06,480
send to each thread. Okay. So you've got like maybe a max amount for each thread and you're like,
435
00:40:06,480 --> 00:40:11,600
okay, this is kind of similar. Let's spoush them. Yeah. It's basically like, Hey, I don't want to
436
00:40:11,600 --> 00:40:16,960
create a block that like, Hey, I accidentally sent, you know, 30 million CUs to one thread.
437
00:40:16,960 --> 00:40:22,800
Let's, and then it's just stuck there. Keep it somewhat parallel. Way longer. Yeah. Okay.
438
00:40:22,800 --> 00:40:30,160
Interesting. So yeah. And just also just excited about like some of the longer term SIMDs that we
439
00:40:30,160 --> 00:40:35,920
can optimize the scheduler after. And what are you looking at there? I talked about this a little
440
00:40:35,920 --> 00:40:40,720
bit earlier. Each of these batches that we send to the workers, there's like a constraint in the
441
00:40:40,720 --> 00:40:46,000
protocol right now that each of them, like, you know, it's up, it's up to, you know, let's say 64
442
00:40:46,000 --> 00:40:51,680
transactions, but all of those transactions can't touch like any of the same accounts. If they did
443
00:40:51,680 --> 00:40:56,000
and you like put that into your blocks and it would be like a protocol error. And there's no
444
00:40:56,000 --> 00:41:01,920
real reason for that. So like we recently had SIMD 83 accepted removes this constraint and just says,
445
00:41:01,920 --> 00:41:08,480
yeah, put whatever transactions you want in an entry. And then if they do conflict, they need to
446
00:41:08,480 --> 00:41:14,640
need to just be executed in that order. Yes. That makes like a lot more sense. It also can like
447
00:41:14,640 --> 00:41:21,840
help the scheduler be more efficient. So if the top 10 paying transactions are all conflicting,
448
00:41:21,840 --> 00:41:25,920
put them in a batch, send them all there. They'll be executed in that order. I don't have to like
449
00:41:25,920 --> 00:41:31,360
send them in 10 separate boxes and potentially slow them down with other work between them.
450
00:41:31,360 --> 00:41:38,160
I just say, what do you, okay. And it's like final question. Where do you actually see Solana going?
451
00:41:38,160 --> 00:41:43,520
If it's something bad, how do we redirect it into the right direction? Yeah. And I guess like also
452
00:41:43,520 --> 00:41:48,320
any of you on Solana, what do you see that going? Yeah. I don't really have like big concerns about
453
00:41:48,320 --> 00:41:53,920
the direction Solana is going right now as like a whole. I think, you know, the general direction
454
00:41:53,920 --> 00:42:00,720
is good right now. In terms of mev, I think it's hard to say. There's like different kinds of mev
455
00:42:00,720 --> 00:42:08,400
and I think some of it is like a net negative and some of it's fine. What kind is a net negative?
456
00:42:09,280 --> 00:42:14,960
Obviously the locks one and the spam, but is there anything else? Yes. I mean, I think we saw this
457
00:42:14,960 --> 00:42:21,200
recently with Jito. It was like users were getting sandwiched because of the like mempool
458
00:42:21,200 --> 00:42:27,120
and circars would just like sandwich user transactions. Oh yeah. I think that that's like,
459
00:42:27,120 --> 00:42:31,600
you know, I think Jito made the right call on turning that off, but you know, there's, you know,
460
00:42:32,160 --> 00:42:37,920
probably entities out there who are, I guess, less respectable than Jito, let's just say,
461
00:42:37,920 --> 00:42:44,800
and willing to do that and offer that service to their circars. Oh, that's okay. So just because
462
00:42:44,800 --> 00:42:50,000
Jito has disabled it doesn't mean that someone else might call on that. You know, personally,
463
00:42:50,000 --> 00:42:55,520
I think that that's like a net negative for the ecosystem. If users are sort of afraid of getting
464
00:42:55,520 --> 00:43:01,040
sandwiched every time they send a defi transaction, they're just not going to. Yeah, definitely. Or
465
00:43:01,040 --> 00:43:04,880
they just won't send it to that validator and then they'll just be a skipped block basically.
466
00:43:04,880 --> 00:43:12,400
Yeah. They might like avoid certain validators. Yeah. I think that at that point is a good way to
467
00:43:13,120 --> 00:43:18,320
sort of disincentivize people from doing this sort of thing. Like, you know, if you're aware of
468
00:43:18,320 --> 00:43:24,400
validators doing this sort of shit and like name and shame them. Yeah. There are other types of
469
00:43:24,400 --> 00:43:30,080
like, I think there are other types of Mav that are, you know, fine. If there's like an arbitrage
470
00:43:30,080 --> 00:43:36,000
opportunity, Jito sort of like auctions off that space to people. And, you know, it's not necessarily
471
00:43:36,000 --> 00:43:42,640
in terms of block space or priority fees. They just like have this Mav tip where they say, yeah,
472
00:43:42,640 --> 00:43:49,360
give me some percentage of profit you get from like this arm. I wonder actually if it's possible
473
00:43:49,360 --> 00:43:56,480
to sandwich people without a mempool. Cause if you're maybe, I guess there's no other way of
474
00:43:57,120 --> 00:44:03,040
knowing whether something's being locked outside of being the validator, but maybe you could like
475
00:44:03,040 --> 00:44:11,280
take a gamble. Just like, this is just like a naive thing. You could like lock and then you
476
00:44:11,280 --> 00:44:16,960
can guess someone's being locked and then like back run that locked one. Yeah. I don't know if
477
00:44:16,960 --> 00:44:23,200
you need to like hear too much about the lock in like, cause mostly the schedule is just like,
478
00:44:23,200 --> 00:44:29,280
am I in the correct priority order to sort of get these things? Oh yeah. That'd be very bad order.
479
00:44:29,280 --> 00:44:34,320
That'd be a super complex system. Yeah. You always, you don't know like, hey, what else is in that queue
480
00:44:34,320 --> 00:44:39,360
that I don't know about. Probability stuff. But yeah, yeah, it'll be so hard to. I mean,
481
00:44:39,360 --> 00:44:45,040
there's like definitely ways I can think of to sandwich users right now. I kind of will not say
482
00:44:45,040 --> 00:44:50,800
those publicly. Okay. Cause you're already doing it. No, okay. No, I'm not just like there's,
483
00:44:50,800 --> 00:44:56,800
there are some ways to like see transactions without being the validator. Yeah. Really? Yeah.
484
00:44:56,800 --> 00:45:03,600
I guess, yeah. Yeah. I'll just refrain from saying any more here. Okay. No problem. I think anyone
485
00:45:03,600 --> 00:45:08,960
who is actually like motivated to search would know what I'm talking about, but yeah. Yeah. It's
486
00:45:08,960 --> 00:45:16,160
not worth like exposing something like that in my opinion. 100%. Well, man, Andrew, it's been a,
487
00:45:16,160 --> 00:45:20,480
it's been a pleasure learning about this new scheduler that you've built. Very informative
488
00:45:21,040 --> 00:45:26,240
episode as predicted at the start. And I'm sure a lot of people would find a lot of value in this.
489
00:45:26,240 --> 00:45:30,480
I know I did, but man, it's been a pleasure. Thank you so much for spending your time and
490
00:45:30,480 --> 00:45:37,200
sharing your knowledge with us. Thanks for having me. And I'm looking forward to using your scheduler.
491
00:45:37,200 --> 00:45:41,440
Me too. All right, man. Have a good day and thank you everyone for listening.
00:00:00,000 --> 00:00:04,160
Currently, banking stage does not keep up if we are receiving that much traffic.
2
00:00:04,160 --> 00:00:09,440
We will basically only receive into the scheduler some fraction of those packets
3
00:00:09,440 --> 00:00:14,800
and then have an incomplete review of all the transactions that we've received.
4
00:00:14,800 --> 00:00:20,560
We can only schedule off of those ones that we've deserialized and inspected to see which ones are the highest priority.
5
00:00:20,560 --> 00:00:27,200
And one of the main bottlenecks there is just the deserialization we use for our transaction type.
6
00:00:27,200 --> 00:00:31,520
ScrapingBits is brought to you by the following sponsors.
7
00:00:31,520 --> 00:00:39,520
MEV Protocol, maximize your ETH staking value with MEVETH, exclusively on MEV.io.
8
00:00:39,520 --> 00:00:45,040
And Composable, execute any intent on any chain coming soon to Mantis.app.
9
00:00:45,040 --> 00:00:48,640
That's MANTIS.APP.
10
00:00:48,640 --> 00:00:57,440
And Fastline Labs, the only MEV and intent-centric team that has a daily deodorant application rate of over 68%.
11
00:01:00,960 --> 00:01:04,960
GMGM everyone, my name is DaGachi, the host of ScrapingBits.
12
00:01:04,960 --> 00:01:10,480
And today, we have another ANSA participant, Andrew Fitzgerald.
13
00:01:10,480 --> 00:01:12,000
GM, sir, how's it going?
14
00:01:12,000 --> 00:01:14,560
Good, yeah, thanks for having me. Excited to be here.
15
00:01:14,560 --> 00:01:18,720
Excited to have you on. I'm sure a lot of people are going to enjoy this episode.
16
00:01:18,720 --> 00:01:24,560
Highly anticipated. Just for the people that aren't familiar with you, who are you and what do you do?
17
00:01:24,560 --> 00:01:29,280
So I'm an employee at HAN and my main focus is working on banking.
18
00:01:29,280 --> 00:01:34,800
Basically all things banking stage. My big pet project has been the transaction scheduler.
19
00:01:34,800 --> 00:01:36,240
The new one, not the old one.
20
00:01:36,240 --> 00:01:36,960
1.18?
21
00:01:36,960 --> 00:01:45,520
Yeah, it's becoming an option in 1.18. It'll be the default in 1.19 or 2.0, whatever we end up calling it.
22
00:01:45,520 --> 00:01:51,520
Yeah, super excited for that to get out to mainnet because there is a lot of very nice properties.
23
00:01:51,520 --> 00:01:56,480
Very excited to talk about this. And just before we do, how did you get into this position?
24
00:01:56,480 --> 00:02:00,080
What were you doing before? Give me your lore, your history.
25
00:02:00,080 --> 00:02:05,600
Yeah, sure. So I got my PhD in nuclear engineering at the University of Michigan.
26
00:02:05,600 --> 00:02:09,680
And obviously not a very traditional background for people doing crypto stuff.
27
00:02:09,680 --> 00:02:13,520
I was working on an HPC Neutronix simulation tool.
28
00:02:13,520 --> 00:02:18,080
And so a lot of the stuff I was doing was just optimizing a code for performance.
29
00:02:18,080 --> 00:02:25,680
And about half the work I was doing was coming up with new methods for simulation and then half implementing and optimizing.
30
00:02:25,680 --> 00:02:28,960
I found myself kind of enjoying that second part a bit more.
31
00:02:28,960 --> 00:02:34,400
And so when I finished my PhD, I just said, screw it. I'll go into finance, make more money,
32
00:02:34,400 --> 00:02:39,920
and do just be a software dev. So I worked at a trading firm for almost three years.
33
00:02:39,920 --> 00:02:44,000
And then a recruiter reached out to me about a position at Solana Labs.
34
00:02:44,000 --> 00:02:47,920
It sounded interesting. I was already somewhat crypto interested.
35
00:02:47,920 --> 00:02:52,800
I hadn't done any actual blockchain dev or was trading meme coins back then.
36
00:02:52,800 --> 00:02:58,720
I joined labs. Yeah, been there since until we moved on back in December.
37
00:02:58,720 --> 00:03:07,120
What a crazy career switch from nuclear to finance and then to crypto, straight DGN.
38
00:03:07,120 --> 00:03:14,240
Yeah, it's actually surprising how many of the nuclear PhDs and STEM PhDs in general go into finance.
39
00:03:14,240 --> 00:03:18,240
Yeah, I think usually the sciences just don't make that much money.
40
00:03:19,440 --> 00:03:25,840
And finance is just where it's at money-wise. So it just makes sense to switch over.
41
00:03:25,840 --> 00:03:30,160
Yeah, I was offered a postdoc position for like $35,000.
42
00:03:30,160 --> 00:03:33,200
I was like, maybe this isn't the right path for me.
43
00:03:34,080 --> 00:03:39,280
I mean, yeah, very interesting to do that stuff, but you need to join the dark side to make the real money.
44
00:03:39,280 --> 00:03:43,360
Yeah, it was definitely super interesting work. I do miss it at times, but yeah,
45
00:03:43,360 --> 00:03:46,320
it didn't make sense lifestyle-wise. Yeah, definitely.
46
00:03:46,320 --> 00:03:50,240
I feel like you got to mix science with engineering to at least make it
47
00:03:50,880 --> 00:03:55,200
a startup in some sense to actually make money through that cool stuff.
48
00:03:55,200 --> 00:04:00,800
Well, like a research law, I don't even know, really. Anyway, so you joined labs, right?
49
00:04:00,800 --> 00:04:04,880
How did you get into actually doing the scheduler and the banking stage?
50
00:04:04,880 --> 00:04:11,120
Why did you choose these specifically and why not consensus or the voting areas? Why this?
51
00:04:12,080 --> 00:04:17,200
It's actually kind of funny. One of the engineers who interviewed me basically gave me
52
00:04:17,200 --> 00:04:21,760
the scheduler design problem as an interview question. So when we were talking about it,
53
00:04:21,760 --> 00:04:25,680
I was very interested in it. It just seemed like an interesting problem with NP hard.
54
00:04:25,680 --> 00:04:31,280
So it's like, there's no set way to do this. It's very open to using heuristics to optimize.
55
00:04:31,280 --> 00:04:35,840
Everything's always changing. It's always super interesting to work on that sort of problem.
56
00:04:35,840 --> 00:04:39,600
And then when I joined, I realized like, oh, wait, they're actually working on this.
57
00:04:39,600 --> 00:04:43,200
So I just asked them, like, hey, can I do that? And yeah, I ended up
58
00:04:43,200 --> 00:04:47,600
just basically taking that on as a project after working at labs for a few months because
59
00:04:47,600 --> 00:04:51,200
I needed to get somewhat familiar with how the rest of the system worked first.
60
00:04:51,200 --> 00:04:57,840
Right. Interesting. So you're like, oh, this is my interview question. Let me just continue doing
61
00:04:57,840 --> 00:05:00,000
that interview question. Basically.
62
00:05:00,000 --> 00:05:05,520
Okay. So you have to rewrite the entire scheduler. So let's break it down. What actually is the
63
00:05:05,520 --> 00:05:08,960
scheduler and the banking stage before we get deep into it?
64
00:05:08,960 --> 00:05:15,120
Sure. So yeah, people have always referred to what's currently there as a scheduler. And I kind of
65
00:05:15,120 --> 00:05:19,360
don't like that because I don't think it is a scheduler. Right now, we basically have like
66
00:05:19,360 --> 00:05:24,960
four threads that just independently pull from a channel coming from like the SIG verify stage.
67
00:05:24,960 --> 00:05:29,600
And they're all just sort of. What's the SIG verify stage as well, by the way?
68
00:05:29,600 --> 00:05:34,480
Sorry. Oh, yeah, sure. It's like packets come into the network through like quick, and then
69
00:05:35,040 --> 00:05:41,200
we send those packets onto a stage called SIG verify. And it basically does some sanity checks
70
00:05:41,200 --> 00:05:46,000
that the packets are transactions. And then once it knows that like, hey, this is probably a
71
00:05:46,000 --> 00:05:52,480
transaction, then it will check all the signatures, like validly signing that message.
72
00:05:53,120 --> 00:05:56,240
Right. That way, by the time they get to banking stage, we know like, hey,
73
00:05:56,800 --> 00:06:00,800
these, you know, we don't have to verify the signatures there because it's a fairly expensive
74
00:06:00,800 --> 00:06:06,400
operation. No excellence. Yeah. SIG verify uses like a bunch of threads to do that in parallel.
75
00:06:06,400 --> 00:06:11,520
It's relatively easily parallelizable. Okay. Okay. And then what happens after that?
76
00:06:11,520 --> 00:06:17,120
You go to the banking stage. Yeah. So banking stage will then, depending on what time it is.
77
00:06:17,120 --> 00:06:21,440
So like, if you're not the leader, you will probably be just like buffering those packets
78
00:06:21,440 --> 00:06:26,960
and forwarding them to who is the leader. And if you are the leader, you will basically try to
79
00:06:26,960 --> 00:06:32,640
choose which ones you want to try to execute. And at least how it currently works, we have four
80
00:06:33,200 --> 00:06:38,480
banking stage threads, and they each just pull off that channel of SIG verify, put them in a buffer,
81
00:06:38,480 --> 00:06:44,320
sort them by priority. And they just try to grab like the first 64 non-conflicting transactions,
82
00:06:44,880 --> 00:06:49,680
non-conflicting, I mean like the read, write blocks. Like you can't have two transactions,
83
00:06:49,680 --> 00:06:53,760
like writing the same account in the same batch. Then they just will go through some checks,
84
00:06:53,760 --> 00:06:57,440
make sure no other thread is touching that account, grab the lock of that account,
85
00:06:57,440 --> 00:07:02,560
make sure we can fit those transactions into like the block space, actually execute them.
86
00:07:02,560 --> 00:07:08,960
And then we record them into the POH stream and then commit them. And then when we commit them,
87
00:07:08,960 --> 00:07:15,040
I think they get sent out over broadcast stage. Okay. Which just uses turbine to propagate those
88
00:07:15,040 --> 00:07:22,320
to the rest of the network. And you say sort by priority and how do you classify priority?
89
00:07:22,320 --> 00:07:29,040
Yeah, that's a good question. So it used to be just like a FIFO queue. And then sometime around
90
00:07:29,040 --> 00:07:35,280
when I joined, there's this concept of priority fees and there's this CU price, which is just like,
91
00:07:35,280 --> 00:07:41,600
Hey, how many micro-lamp courts per CU are you willing to pay to get this transaction executed?
92
00:07:41,600 --> 00:07:46,960
And so the old banking stage just takes that number, which is like specified in instruction
93
00:07:46,960 --> 00:07:51,440
on the transaction. Oh, okay. So if you want it to be number one, you could just max that out.
94
00:07:51,440 --> 00:07:56,640
Yep. Yeah. And you would have to pay more for that. Yeah, true. But I don't imagine anyone else
95
00:07:56,640 --> 00:08:00,960
would be maxing that out. Well, maybe now they would. But there was also an interesting thing.
96
00:08:00,960 --> 00:08:07,520
I think I spoke to someone else about MEV, basically MEVBOTS doing, not paying any priority
97
00:08:07,520 --> 00:08:13,920
fee at all. And they were still winning. I'm not sure why, but does that ring a bell by any chance?
98
00:08:13,920 --> 00:08:20,560
It depends. There's a lot of edge cases in the current banking stage where we have these more
99
00:08:20,560 --> 00:08:25,760
competing threads that can just let the lower priority transactions jump the queue if they
100
00:08:25,760 --> 00:08:33,040
just so happen to be the top priority in that thread. Right now each thread basically doesn't
101
00:08:33,040 --> 00:08:38,160
have a view of what other threads have in terms of transactions. So it might be at the front of
102
00:08:38,160 --> 00:08:44,960
its own local queue. So it's like a race condition. Yep. Yeah. And that's sort of the main motivation
103
00:08:44,960 --> 00:08:50,400
for having a separate scheduling thread. All the scheduling thread does is just have a global view
104
00:08:50,400 --> 00:08:56,800
of what that priority list should be. And then I send work to the worker threads, obviously.
105
00:08:56,800 --> 00:09:03,920
And you said the first 64 transactions, are they all being pushed into different threads themselves?
106
00:09:03,920 --> 00:09:10,320
Or is it like each 64 transactions is pushed into a new thread or is it shared between them all?
107
00:09:10,320 --> 00:09:16,880
It's like each thread has its local buffer. It will grab the first 64 non-quickly transactions
108
00:09:16,880 --> 00:09:23,200
and then it will execute those 64 transactions up to 64 transactions in serial. Okay. Okay.
109
00:09:24,320 --> 00:09:31,280
How does it check if it's non-conflicting? They're not being shared at all. So it would be like 64
110
00:09:31,280 --> 00:09:38,160
and then the 128 and then continuing. Well, 64, then the next 64, then the next 64,
111
00:09:38,160 --> 00:09:43,920
and they're all split into different threads. Oh, no. We've got like four threads. This is where
112
00:09:43,920 --> 00:09:48,720
drawing helps. We've got like four threads. Each of them have their own buffer. Okay.
113
00:09:48,720 --> 00:09:54,720
And then we use this thing called, we basically have like multiple iterators, 64 of them,
114
00:09:54,720 --> 00:10:01,600
in to each buffer. And they start at the front and then march forward to find, basically to create
115
00:10:01,600 --> 00:10:08,080
batches up to 64 transactions. And then each batch is just executed on that thread. They're not going
116
00:10:08,080 --> 00:10:14,800
to another thread for execution. Couldn't one batch actually be executed faster than other ones?
117
00:10:14,800 --> 00:10:19,840
Yep. Yeah. They're all just running as fast as they can. So like executing a batch on thread one
118
00:10:19,840 --> 00:10:24,240
is not going to wait for like the execution to finish on thread two. So what would actually
119
00:10:25,040 --> 00:10:31,280
cause one thread to execute faster than another? Just like simpler transactions, right? Like
120
00:10:31,280 --> 00:10:36,800
something like a simple transfer does not take as much time as a, like a swap on Jupiter.
121
00:10:36,800 --> 00:10:39,680
Rude. Okay. Cause it's going for the emulator, right?
122
00:10:40,240 --> 00:10:47,040
Yeah. Like the Jupiter stuff would have to like load that program and then, sorry, like build up
123
00:10:47,040 --> 00:10:52,720
the VM and then send it through there. There's like a lot of overhead in terms of like executing
124
00:10:52,720 --> 00:10:58,400
BPF stuff. Not a ton, but it's more than just native programs, which are not doing that.
125
00:10:58,400 --> 00:11:02,480
Yeah, definitely. Have you ever heard of like the mutex lock game where people are just locking
126
00:11:02,480 --> 00:11:08,720
each other's programs? I guess if a thread is looking for non-conflicting transactions,
127
00:11:09,440 --> 00:11:14,480
then if it's already in one thread, it just wouldn't, another thread wouldn't grab it
128
00:11:14,480 --> 00:11:19,120
because it's already being used by one. Is that correct? Yeah. Okay. That's how it works in like
129
00:11:19,120 --> 00:11:25,360
the current banking stage. Right. I see. Okay. That makes sense. Is that not in the new one at all?
130
00:11:25,360 --> 00:11:31,360
Yeah. It's like the new one basically keeps track of which threads have which locks. And so it's
131
00:11:31,360 --> 00:11:38,160
only ever going to send work to a thread if that work doesn't conflict with any other thread.
132
00:11:38,160 --> 00:11:46,320
Okay. So if I was an MEV bot and I see my competitor use, you know, N accounts and like
133
00:11:46,320 --> 00:11:52,480
one of the elements I actually just use as a lock in mine, it doesn't do anything. It's just locking
134
00:11:52,480 --> 00:11:58,240
it and I'm processing mine because I'm before them, then their transaction wouldn't actually be
135
00:11:58,240 --> 00:12:03,600
processed because I've held that lock. Right. Yeah. I think right now that's a big issue
136
00:12:03,600 --> 00:12:08,080
like the old banking stage because you don't necessarily have to pay more than them.
137
00:12:10,240 --> 00:12:14,000
Right. You can get lucky and just like grab that lock and then that prevents them from doing
138
00:12:14,000 --> 00:12:19,120
anything. Damning that. So they never have to get it. Right. That makes sense. In the new banking
139
00:12:19,120 --> 00:12:24,560
stage, as long as like those packets have arrived at the scheduler, then like whoever is paying more
140
00:12:24,560 --> 00:12:30,000
priority fees, like, you know, going to get that. That makes sense. Okay. I guess the fees aren't
141
00:12:30,000 --> 00:12:36,240
even that expensive either. So it's, if they both send the same amount of priority fee, who gets it?
142
00:12:36,240 --> 00:12:40,720
Is it just a race condition again at that point? I think it would be at that point, like default to
143
00:12:40,720 --> 00:12:46,000
whoever like arrived first, like a time race rather than, I actually don't remember how it's
144
00:12:46,000 --> 00:12:52,000
going to depend. I don't remember how that's resolved exactly. First in, first out probably.
145
00:12:52,000 --> 00:12:57,120
But if it's sorted by that, I guess it would be sorted by maybe the transaction hash, like the
146
00:12:57,120 --> 00:13:03,120
size is bigger than another one. I'm not sure. Or like, yeah, like time of inbound. That would make
147
00:13:03,120 --> 00:13:08,800
sense. Yeah. I think it's just time. I'd have to look, we use this min-nock for like sorting the
148
00:13:08,800 --> 00:13:16,000
priority order. I just don't remember how that handles like things exactly match. Okay. So what
149
00:13:16,000 --> 00:13:22,880
were the major flaws in the current bank, which, and how did that architecture change? What did you
150
00:13:22,880 --> 00:13:29,520
change for the new scheduler and banking stage? What was the old one? What were the problems? How
151
00:13:29,520 --> 00:13:35,600
did you fix it? Are there any more problems? Yeah. So the old banking stage, as I mentioned, had
152
00:13:35,600 --> 00:13:40,480
sort of these four independent threads. None of them knew what the other ones were doing.
153
00:13:40,480 --> 00:13:46,720
And so immediately we have this sort of locking problem. If we've got a bunch of, let's say there's
154
00:13:46,720 --> 00:13:51,200
like an NFT mint going on, everyone's trying to get like the highest, or sorry, trying to get that
155
00:13:51,200 --> 00:13:57,200
NFT first. So all of like the highest priority transactions in those four threads buffers are
156
00:13:57,200 --> 00:14:02,160
all going to be touching the same state. And then it's just like a race condition on who wins that.
157
00:14:02,160 --> 00:14:06,960
And so we have like a lot of account in use errors where basically we're not executing
158
00:14:06,960 --> 00:14:12,160
these transactions because the account is being used by another thread. It's just pretty inefficient.
159
00:14:12,160 --> 00:14:17,200
And then it's not clear like, you know, this was the next highest paying priority one, but when do
160
00:14:17,200 --> 00:14:22,560
I retry it? And that's difficult without having some way of knowing what the other threads have.
161
00:14:22,560 --> 00:14:27,600
And that's like a big issue with just the account locking. And then the other one is just there's
162
00:14:27,600 --> 00:14:32,720
no global view of priority. So no, none of the threads know what the actual highest paying
163
00:14:32,720 --> 00:14:37,280
transaction, like the most valuable transaction to pack is. So we could end up in a situation,
164
00:14:37,280 --> 00:14:42,240
you know, where just by chance, all of the very high priority transactions end up in one thread
165
00:14:42,240 --> 00:14:46,880
and a bunch of low priority ones end up in another thread. Thread with the low priority transactions,
166
00:14:46,880 --> 00:14:50,880
it's still going to go through those and pack a bunch of, you know, basically non-valuable
167
00:14:50,880 --> 00:14:55,360
transactions. And that could fill up our block. And then, you know, we're not selling to the
168
00:14:55,360 --> 00:15:00,000
people who are paying us the most. So inefficient from the leader's perspective.
169
00:15:00,000 --> 00:15:05,520
When a friend grabs a lock, what happened to that transaction that was using that same account?
170
00:15:05,520 --> 00:15:08,320
Does it just pushed all the way back to the bottom or what happens?
171
00:15:08,320 --> 00:15:11,520
Do you mean if it failed to grab the lock? Yeah, exactly.
172
00:15:12,080 --> 00:15:17,920
Yeah. It basically, an old scheduler will basically just retry that transaction the next time we do
173
00:15:17,920 --> 00:15:23,520
an iteration. It won't work again until we've gone through like the entire rest of the buffer.
174
00:15:23,520 --> 00:15:29,520
Would it still be at the top of it or would it be pushing other transactions in front of it though?
175
00:15:29,520 --> 00:15:34,400
Yeah, I guess like logically it would then go sort of back to the bottom of the buffer.
176
00:15:34,400 --> 00:15:36,400
Oh God. Yeah, it's not great.
177
00:15:37,040 --> 00:15:41,120
Yeah. Okay. So what were the solutions that you did with the new scheduler?
178
00:15:41,120 --> 00:15:47,120
Yeah, the new scheduler will basically, it has a global view of the priority because there's only
179
00:15:47,120 --> 00:15:52,160
a single priority queue. So we always know like, this is the most valuable transaction for me to
180
00:15:52,160 --> 00:15:57,200
pack in terms of reward per see you. Okay. So that's immediately better. But then we don't have this
181
00:15:57,200 --> 00:16:02,240
locking problem because we're keeping track of which of our threads have which locks. So we're
182
00:16:02,240 --> 00:16:09,200
only going to send things if they're not conflicting with other things. And then if there's a conflict,
183
00:16:09,200 --> 00:16:15,360
say like I sent some transaction one that touched account A and B to thread one and another that
184
00:16:15,360 --> 00:16:22,640
touched C and B to thread two. At that point, I can't schedule some third transaction that touches
185
00:16:22,640 --> 00:16:28,960
B and C. We're conflicting like across threads. Right. And at that point, we will basically block
186
00:16:28,960 --> 00:16:34,240
on that transaction and not schedule anything that conflicts with it until we can schedule that one.
187
00:16:34,240 --> 00:16:39,680
If that's like the next highest paying. Okay. So sort of in that way, it's guaranteeing that
188
00:16:39,680 --> 00:16:46,160
we're respecting like the priority order in our buffer, never executing things out of order.
189
00:16:46,160 --> 00:16:51,760
Okay. Never is a strong word there because we like queue up work and, you know, we might have
190
00:16:51,760 --> 00:16:57,760
sent things to be worked on and then a new higher paying transaction comes in. Obviously, because
191
00:16:57,760 --> 00:17:02,160
we've already scheduled that work, we can't unschedule it. Okay. Let's say you did a
192
00:17:02,160 --> 00:17:07,840
transaction with a priority fee of like 10. It's in one of the threads now and I send a transaction
193
00:17:07,840 --> 00:17:13,520
that's 11. Will your one be processed before mine? You know mine is, it's like still in the thread,
194
00:17:13,520 --> 00:17:18,400
but your one will still be processed, right? Yeah. Basically what once it's sent by the scheduler,
195
00:17:18,400 --> 00:17:22,960
it's, you know, in flight, we're not going to go back and stop that. We could definitely optimize
196
00:17:22,960 --> 00:17:27,200
there. I think there's some work that needs to be done there right now. That's how it works.
197
00:17:27,200 --> 00:17:30,640
And then when it's processed, what does it actually go? It's building the block, right?
198
00:17:31,200 --> 00:17:37,760
Yeah. So we have like a separate scheduling thread and that's just sending these transactions to
199
00:17:37,760 --> 00:17:42,480
worker threads, which there are like four or more. And then those worker threads basically do what
200
00:17:42,480 --> 00:17:47,040
the old bank of the stage was doing. They, you know, grab the account locks, just make sure that
201
00:17:47,040 --> 00:17:51,440
I didn't screw anything up on the scheduler. Run the normal checks, technical loads and executes
202
00:17:51,440 --> 00:17:56,160
and long records of commits. And then recording goes into like the actual POH stream and then,
203
00:17:56,720 --> 00:18:03,120
you know, broadcast those transactions. Does it record in like an iterator or is it all at once?
204
00:18:03,120 --> 00:18:07,760
Let me clarify that actually. So as you're processing transactions and they're being put
205
00:18:07,760 --> 00:18:13,520
into like, let's say this vector is, are you waiting for the vector to be finished and no
206
00:18:13,520 --> 00:18:20,880
more being sent for you to finish the block or is it continuously adding? So let's say Fred A spits
207
00:18:20,880 --> 00:18:27,120
out a new one, Fred B spits out a new one. It processes Fred A's transaction as an iterator and
208
00:18:27,120 --> 00:18:33,680
then Fred B's transaction right after, or is it like stacking them and then putting it all in?
209
00:18:33,680 --> 00:18:39,520
And is there any way that someone could actually change the order after the process and the
210
00:18:39,520 --> 00:18:45,440
scheduler? So once it's been recorded, you can't change the order. Okay. Or it would be difficult
211
00:18:45,440 --> 00:18:52,000
because we broadcast that out already potentially. We can't like say pixie faxies, but yeah, in terms
212
00:18:52,000 --> 00:18:58,880
of like, you know, recording immediately versus stacking up, it's sort of a mix. So the scheduler
213
00:18:58,880 --> 00:19:04,800
doesn't schedule individual transactions. It schedules batches of transactions. Okay. And
214
00:19:04,800 --> 00:19:10,400
these batches of transactions, we like will load and execute all of them and then record them as
215
00:19:10,400 --> 00:19:16,320
like a single mix and hash into the POH. Basically the reason we, I don't know the whole context,
216
00:19:16,320 --> 00:19:22,640
but one of the reasons is that we basically have like this lock around the POH thread. And so we
217
00:19:22,640 --> 00:19:27,680
don't want to be grabbing that lock every time we had a process transaction overhead and doing so.
218
00:19:28,240 --> 00:19:33,920
So it's more efficient overall. I think when I benched, this is like 30% faster to like batches
219
00:19:33,920 --> 00:19:40,000
versus individual transactions. Right. Got you. Okay. Since you're doing batches, let's do the
220
00:19:40,000 --> 00:19:45,360
priority thing again. When you have a priority of 10, I have a priority of 11, your one's in the
221
00:19:45,360 --> 00:19:51,280
thread before mine. My one comes right after. Your one's been put into this batch. Does mine
222
00:19:51,840 --> 00:19:57,280
go below yours, even though my priority is higher? Or is the priority really just for those threads
223
00:19:57,280 --> 00:20:03,440
to basically select what to go first? It's not like actually in order of priority for...
224
00:20:04,000 --> 00:20:12,640
Yeah. So we are not like, I think Anatoly had this idea of basically enforcing priority on order
225
00:20:12,640 --> 00:20:17,600
on the block and like sort of wait until all of the block is completed and then reorder things.
226
00:20:17,600 --> 00:20:24,880
Right. Yeah. That is not what we're doing right now. The priority is strictly for the leader to
227
00:20:24,880 --> 00:20:30,160
attempt to like choose the most valuable transactions. It's still free to order them however they want.
228
00:20:30,160 --> 00:20:36,160
In your case right now, because that first transaction was already sent off to be executed,
229
00:20:36,160 --> 00:20:41,120
it might've already been completed and the scheduler was just not aware. Your transaction would still
230
00:20:41,120 --> 00:20:45,280
go after that. It would just go to the next batch. Okay. Got you. What are the challenges you kind of
231
00:20:45,280 --> 00:20:52,800
see or saw, or maybe you are seeing still, with actually building something that is efficient at
232
00:20:52,800 --> 00:20:59,280
like sorting transactions, processing them, especially with spam in the equation. So what
233
00:20:59,280 --> 00:21:04,800
are the challenges you kind of face with the spam? How do you think to deal with it or try to deal
234
00:21:04,800 --> 00:21:10,960
with it? Maybe it hasn't worked, maybe it has worked, just that kind of stuff. Yeah. So one of
235
00:21:10,960 --> 00:21:15,920
the things that we'd noticed some time ago, we noticed this because of the banking trace data
236
00:21:15,920 --> 00:21:21,840
that we added in, which lets us inspect which packets were in banking stage that ended up not
237
00:21:21,840 --> 00:21:26,800
getting into the block. That's been super useful for like going back and seeing, you know, why
238
00:21:26,800 --> 00:21:32,320
wasn't this blocked back there efficiently? And after we added that, we were seeing on the internet
239
00:21:32,320 --> 00:21:38,160
that there were these malicious spammers that were sending out a transaction with high priority fee
240
00:21:38,160 --> 00:21:43,360
that locked a bunch of accounts basically to block their competitors from accessing that state. Yeah.
241
00:21:43,360 --> 00:21:48,640
And they had like an invalid fee there. They couldn't pay the fee. And the old banking stage
242
00:21:48,640 --> 00:21:53,760
at that time was still like scheduling it first. And so it like basically let them take the locks
243
00:21:53,760 --> 00:21:59,360
and block their competitor. So we added in a bunch of like checks before we actually
244
00:22:00,400 --> 00:22:05,520
send things off to be processed. So we check before we ever take locks, but like, hey, this can
245
00:22:05,520 --> 00:22:11,280
pay the fees. We basically started doing a bunch of checks earlier on on these transactions. And
246
00:22:11,280 --> 00:22:16,160
often we can do that, you know, before we'd come leader. So like, if you can't pay the fee now,
247
00:22:16,160 --> 00:22:20,400
I'm just going to drop you. You know, there's a high likelihood that you're still not going to
248
00:22:20,400 --> 00:22:24,080
be able to pay the fee when I'm due to come leader. There's an off chance that maybe you've had a
249
00:22:24,080 --> 00:22:29,520
funding transaction in there, but from my perspective, that's not worth the risk. You know,
250
00:22:29,520 --> 00:22:35,120
keep more transactions that I know are more valuable. That can make sense. Yeah. Cause
251
00:22:35,120 --> 00:22:39,920
people do funding transactions. I don't even know that. I don't know of anybody doing that. It's
252
00:22:40,720 --> 00:22:43,840
probably doesn't work. It probably doesn't work very well anymore. I probably,
253
00:22:43,840 --> 00:22:47,440
anybody was, they're not doing it anymore. Cause I would have screwed them over.
254
00:22:47,440 --> 00:22:52,080
Yeah. I would imagine it working at all. If you just see like a standalone transaction
255
00:22:52,080 --> 00:22:58,560
that isn't paying anything, I don't know. It's, yeah, it's a bit weird. I'm just saying. Yeah.
256
00:22:58,560 --> 00:23:04,560
The biggest problem is spam right now, the congestion issue, spam, of course. And then also
257
00:23:04,560 --> 00:23:12,240
like malicious adversarial congestion, which is the major, like the major issue at the moment.
258
00:23:12,240 --> 00:23:16,480
Do you think that's like a network issue, like a sorting of transactions issue,
259
00:23:17,040 --> 00:23:19,680
or maybe even the whole pipeline? What are your thoughts on that?
260
00:23:19,680 --> 00:23:24,480
Yeah, I think it's a problem sort of in the whole pipeline. Actually, it's really interesting. The
261
00:23:24,480 --> 00:23:29,360
things I've been working on this week, a lot of it has been networking issues. The networking
262
00:23:29,360 --> 00:23:34,960
stack had a bunch of really inefficient spots. It's my understanding. I'm not super familiar.
263
00:23:34,960 --> 00:23:41,040
And especially with relations like the fake weighted QoS, there was bugs in there that made
264
00:23:41,040 --> 00:23:46,480
it not work as well. People were spanning like to open up new connections that let them just get in
265
00:23:46,480 --> 00:23:52,160
their transactions and they would spam a bunch of transactions to basically try to get first in one
266
00:23:52,160 --> 00:23:58,480
of these four independent banking threads. Right. Yeah. So even just moving, it's like other people
267
00:23:58,480 --> 00:24:03,520
have been working on these networking fixes, which helps reduce that on the network networking side.
268
00:24:03,520 --> 00:24:07,840
But until we move to the new scheduler, there's sort of still this incentive to spam because
269
00:24:07,840 --> 00:24:12,480
you might get lucky, end up on a thread where you're not conflicting with anything high priority.
270
00:24:12,480 --> 00:24:17,280
So you get first and you win the race. Doesn't happen in the new schedulers, but really reduces
271
00:24:17,280 --> 00:24:23,440
the incentive to spam in terms of banking stage. I think there's also like pretty significant issues
272
00:24:23,440 --> 00:24:31,040
in terms of just our sort of pipeline throughput. The networking stack used to be,
273
00:24:31,600 --> 00:24:37,280
my understanding, used to be the bottleneck or the major bottleneck. And with the recent changes
274
00:24:37,280 --> 00:24:43,840
that is becoming less the case. And now we have bottlenecks in SIGTHERIFY and that can only do
275
00:24:43,840 --> 00:24:48,960
like, I forget it's like 200k TPS is what someone told me recently. All those packets are feeding
276
00:24:48,960 --> 00:24:54,240
into banking stage and actually currently banking stage does not keep up if we are receiving that
277
00:24:54,240 --> 00:25:00,240
much traffic. We will basically only receive into the scheduler some fraction of those packets.
278
00:25:00,240 --> 00:25:06,080
And then we basically have an incomplete review of all the transactions that we've received.
279
00:25:06,080 --> 00:25:11,200
We can only schedule off of those ones that we've deserialized and inspected to see which ones are
280
00:25:11,200 --> 00:25:17,040
the highest priority. And I've been working a lot with another Hans Ledeb Alphandre who
281
00:25:17,840 --> 00:25:22,880
done a bunch of like profiling work. And one of the main bottlenecks there is just the
282
00:25:22,880 --> 00:25:29,120
deserialization we use for our transaction type. It does a bunch of like internal allocations.
283
00:25:29,120 --> 00:25:34,720
So I've been working on this this week for a bunch and it's quite interesting. There's this
284
00:25:34,720 --> 00:25:40,400
version transaction type and it has like five, at least five internal VECs that it has to like
285
00:25:40,400 --> 00:25:45,200
allocate on the heap. And every time you do like a heap allocation, you're basically risking this
286
00:25:45,200 --> 00:25:51,520
really long call or has to maybe do some like bookkeeping work in whatever allocator you're using.
287
00:25:51,520 --> 00:25:57,040
And most of the time that's fast, it's just going to like some thread local cache of memory that
288
00:25:57,040 --> 00:26:03,440
I think Jmalloc uses. But occasionally, in particular when like the memory pattern doesn't
289
00:26:03,440 --> 00:26:09,120
fit in with like recent accesses and allocations, then you're sort of risking these long calls.
290
00:26:09,120 --> 00:26:15,120
I just take like 10 milliseconds sometimes to like a few hundred transactions and it's just
291
00:26:15,120 --> 00:26:20,160
ridiculously slow. Okay, yeah, most of the time it's fast. Basically, yeah, I've been working
292
00:26:20,160 --> 00:26:26,320
this week and like that's like a huge bottleneck in banking stage. And we're spending like 30 or
293
00:26:26,320 --> 00:26:32,240
40% of the time just deserializing these transactions. Oh, wow. Yeah, 10 milliseconds.
294
00:26:32,240 --> 00:26:40,640
So a long time when the block is for exactly. And yeah, you know, I've been working on like
295
00:26:40,640 --> 00:26:45,040
this tech project and I don't know if other people will go for it because it uses unsafe rust. But
296
00:26:45,040 --> 00:26:51,200
like all that super unnecessary, we have this like serialized packet, we can just calculate offsets
297
00:26:51,200 --> 00:26:58,080
into it. That is extremely fast. I can do it in, you know, 20, 30 nanoseconds per packet.
298
00:26:58,080 --> 00:27:04,400
And so basically what used to be taking for two, 300 milliseconds to deserialize like 90,000
299
00:27:04,400 --> 00:27:08,960
transactions, I can do that in two milliseconds now. What was the solution for that? Basically,
300
00:27:08,960 --> 00:27:13,840
just creating like a new transaction type that doesn't do these internal allocations.
301
00:27:13,840 --> 00:27:19,040
Oh, okay. Got you. So it's kind of like a big change because we'd have to change our transaction
302
00:27:19,040 --> 00:27:22,400
type everywhere. So I don't know if other people are going to go along with this quite yet. Oh,
303
00:27:22,400 --> 00:27:27,120
yeah, you have to change the type. Like a super recent development. Yeah. Interesting. But yeah,
304
00:27:27,120 --> 00:27:31,040
I think that that's like a huge problem is basically if banking stage isn't keeping up
305
00:27:31,040 --> 00:27:36,400
with the packet signal if I sending it, then banking stage has this sort of incomplete view
306
00:27:36,400 --> 00:27:41,120
on what transactions it should even schedule. And there might be, you know, a really juicy
307
00:27:41,120 --> 00:27:45,600
transaction in there and that paid me a ton of fees. And I'm just not saying it might be
308
00:27:45,600 --> 00:27:51,040
serialization is so slow. I feel like you don't even have to do full, full transaction checks,
309
00:27:51,040 --> 00:27:56,240
either. I mean, I could be wrong, but you can just like simulating half of it or partially
310
00:27:56,240 --> 00:28:01,120
to some degree. Maybe, maybe there are heuristics that. Oh, yeah, we don't simulate them. We don't
311
00:28:01,120 --> 00:28:05,760
simulate them like when we're receiving. Oh, okay. It's like not being like the whole transaction
312
00:28:05,760 --> 00:28:10,320
check. What do you simulate them in the banking stage? So we just execute them in the banking
313
00:28:10,320 --> 00:28:15,520
stage because I guess the same process of simulation, except at the end, we like commit
314
00:28:15,520 --> 00:28:21,520
those results back to the accounts database. Oh, okay. So when do you simulate in the pipeline?
315
00:28:21,520 --> 00:28:26,720
The only place we simulate in the pipeline is like on our PCs. Like we don't actually do that in the
316
00:28:26,720 --> 00:28:33,760
validator. Oh, okay. Yeah. I think, I think Gido does simulation in like their block engine, but
317
00:28:33,760 --> 00:28:40,000
that's not like part of the yeah. Yeah. I did the one with seg the other day. Okay. Cool. So I was
318
00:28:40,000 --> 00:28:44,880
talking about all about that. If you were, do you think the execution is actually slow at all?
319
00:28:44,880 --> 00:28:51,360
Yeah, execution is definitely slow in some aspects and I think we can improve things there. I'm not
320
00:28:51,360 --> 00:28:56,800
that familiar with like the VLU stuff. Some of it comes down to just like app developers. You're not
321
00:28:56,800 --> 00:29:02,080
writing a smart contracts that efficient, that is efficient. And there's no way I can execute that
322
00:29:02,080 --> 00:29:07,360
efficiently. Yeah, that makes sense. But at the same time, like Percy, I think that we can probably
323
00:29:07,360 --> 00:29:13,120
do things faster than we are currently. When I was speaking to Joe, one of your co-workers,
324
00:29:13,120 --> 00:29:20,640
he was working on runtime V2 for, for Agave. And we talked all about that. And I think it does seem
325
00:29:20,640 --> 00:29:26,320
very inefficient because you're all spinning up, up and down a new VM each time. I think for every
326
00:29:26,320 --> 00:29:30,800
instruction, not even every transaction, it's for every instruction in the transaction. And yeah,
327
00:29:30,800 --> 00:29:35,680
as you said, like if something is super convoluted and there's just like a lot of shit happening
328
00:29:35,680 --> 00:29:40,960
and you can probably just like rewrite it. Maybe if you want to be super fast, maybe you have
329
00:29:40,960 --> 00:29:47,200
something that caches that program, but actually optimizes it automatically. So it's faster. If it
330
00:29:47,200 --> 00:29:53,600
like, let's say like a radium program, if that's just used all the time, that shitty program,
331
00:29:53,600 --> 00:29:59,840
I'm not saying it's shitty, but like, let's say it is for, for this purpose. Yeah. I think I get what
332
00:29:59,840 --> 00:30:04,800
you're saying. Yeah. Then you optimize it and cache the optimized version. So it runs faster.
333
00:30:04,800 --> 00:30:09,840
Yeah. I don't know if that's like technically possible. I know we do like some sort of like
334
00:30:09,840 --> 00:30:16,320
compilation and like stories compiled like programs, which I think is effectively that. I
335
00:30:16,320 --> 00:30:19,840
don't think it's going to do like much optimization though. I don't think it's going to like change
336
00:30:19,840 --> 00:30:25,040
the actual like core logic, the BPA instructions we're saying to do. Well, I guess if it's, if you
337
00:30:25,040 --> 00:30:30,800
like customize the bytecode as much as possible and there's less text. I think the issue, the issue
338
00:30:30,800 --> 00:30:35,920
with doing something like that, I think it's technically possible. Like you're just saying
339
00:30:35,920 --> 00:30:40,800
like, Hey, everyone's using radium. What's right. Are there any like radium version that's more
340
00:30:40,800 --> 00:30:45,680
efficient? Exactly. Just optimize radium itself. It does the exact same thing. It's just less,
341
00:30:45,680 --> 00:30:50,400
you know, less shit to actually execute. Yeah. I think the main issue with like in the pro trick
342
00:30:50,400 --> 00:30:55,040
that one, it's like a ton of work, but two is if it's using like a different amount of CUs,
343
00:30:55,600 --> 00:31:01,840
then that can lead to like consensus differences. So unless like all of the clients, so like class
344
00:31:01,840 --> 00:31:07,840
and firedance or you know, we're all using like the same optimized version, then like we're going
345
00:31:07,840 --> 00:31:12,240
to end up with something that fails consensus. Maybe an optimized version would be good for
346
00:31:12,240 --> 00:31:20,640
simulations. Yeah. Yeah. Like, but is that really the only thing? Yeah. I don't know. I think if we
347
00:31:20,640 --> 00:31:25,920
want programs that are faster, it's sort of down to dev a little bit to optimize their own stuff.
348
00:31:26,480 --> 00:31:32,160
You know, we can make the VM faster in general, but we shouldn't be too thinking winners. No.
349
00:31:32,160 --> 00:31:37,040
Yeah. That makes sense. Okay. Hmm. Yeah. I'm trying to think. And actually, yeah, an interesting
350
00:31:37,040 --> 00:31:42,640
aspect of like the new scheduler compared to the old one, the old one, the old banking stage just
351
00:31:42,640 --> 00:31:49,440
went off of that, that CU price in terms of like priority, but the new scheduler will, it's sort of
352
00:31:49,440 --> 00:31:56,240
by like reward over transaction costs. Like inherently it's sort of prioritizing people who
353
00:31:56,240 --> 00:32:02,880
request fewer CUs, per use fewer CUs. Oh, really? So like optimization is actually mattering now.
354
00:32:02,880 --> 00:32:07,200
Yeah. It's like actually incentivizing people to optimize their app. Oh, heavy. They optimize their
355
00:32:07,200 --> 00:32:14,000
apps. They can like pay the same fee, but get much higher priority. So the less CU, the actually
356
00:32:14,000 --> 00:32:20,800
higher priority in the new scheduler. Yep. Okay. Bring in the optimizers. Yeah. I'm excited to like
357
00:32:22,560 --> 00:32:26,640
it's not going to like change overnight as we like roll out the scheduler, but it'll be interesting
358
00:32:26,640 --> 00:32:32,400
to see maybe how that evolves over time. I was thinking of this before because coming from EFE
359
00:32:32,400 --> 00:32:38,000
as an optimizer, like riding real bike code, it's the benefit obviously with gas. But if you think
360
00:32:38,000 --> 00:32:43,680
about it, if there's a limited amount of shit to store and also simulation wise and execution,
361
00:32:44,320 --> 00:32:50,400
it's always just better to have less because it's just less overhead to actually compute. So this is
362
00:32:50,400 --> 00:32:57,040
like another benefit incentivizing a bit more to actually nerd snipe some optimizers like myself.
363
00:32:57,040 --> 00:33:03,680
Yeah. It actually has not marginal benefit, but like major benefit now. So that's fantastic. Great
364
00:33:03,680 --> 00:33:10,400
change. Yeah. Give him a promotion guys. Come on. But I think like assembly in Rust is actually such
365
00:33:10,400 --> 00:33:15,200
a great skill as well. So like people that will learn this will get so good at Rust and like
366
00:33:15,200 --> 00:33:20,640
understand a lot better. So I think it's like a massive W honestly, not just for Solana, but
367
00:33:20,640 --> 00:33:26,320
for actually onboarding devs into low level Rust engineering as well. I haven't written like any
368
00:33:26,320 --> 00:33:31,920
smart contract code myself or like nothing, not like a simplistic example. Like I don't know too
369
00:33:31,920 --> 00:33:36,880
much about like how to optimize it to be honest. Oh, okay. That's like my entire background. Okay.
370
00:33:36,880 --> 00:33:42,560
Gotcha. Yeah. Optimizing security, fuzzing, all that stuff. That's kind of my bread and butter.
371
00:33:42,560 --> 00:33:47,760
Yeah. It seemed like fun work. I had a two billion dollar shit. I love, you know, virtual machines as
372
00:33:47,760 --> 00:33:54,640
well, emulating them, simulating stuff. I have here, I feel it was very easy to do. Solana seems
373
00:33:54,640 --> 00:33:58,960
a lot more complex because it's registry based, because it's Rust, but still fun. None of the
374
00:33:58,960 --> 00:34:03,840
less, right? That's sick though. Is there any other like giant optimizations you did that people
375
00:34:03,840 --> 00:34:10,480
should be aware of for this upcoming schedule? Not that I can really think of off the top of my head.
376
00:34:10,480 --> 00:34:17,680
Yeah. Okay. So there's like the threads changed, this CU's changed. That's really cool. I like that.
377
00:34:17,680 --> 00:34:23,600
Is there anything you're seeing like other validators do that differ from, you know,
378
00:34:23,600 --> 00:34:29,920
GAVE or the JITALabs one? What are they really prioritizing in these that you see? I guess like
379
00:34:29,920 --> 00:34:35,840
a lot of validators are trying to do like MEV or like voting MEV, I guess, optimizing the voting
380
00:34:35,840 --> 00:34:41,440
part. But what do you see like the most efficient, low hanging fruits for the validator are?
381
00:34:41,440 --> 00:34:48,240
That's a good question. I think that there is like a lot of vote optimization stuff that could be done.
382
00:34:48,240 --> 00:34:55,200
I don't know too much about how we vote currently, but it seems like we've seen people have been
383
00:34:55,200 --> 00:35:00,320
making like changes to how their validators voting to get more like vote credits. Yeah.
384
00:35:00,320 --> 00:35:07,360
Some of that's sort of dubious at best and that sort of then touches like timely vote credit
385
00:35:07,360 --> 00:35:14,080
proposal to incentivize people to like keep consensus alive. Yeah. Well, yeah, everyone
386
00:35:14,080 --> 00:35:19,520
like delays their voting and just doesn't vote that they're waiting for the port to be selected.
387
00:35:19,520 --> 00:35:24,320
Yeah. Then we're just kind of going to kill ourselves. See, I think that there's like
388
00:35:24,320 --> 00:35:30,080
proposals like that, which can help address this. But yeah, I think there's like this sort of some
389
00:35:30,080 --> 00:35:36,240
issues with like when we choose to fork when we're leader versus not. I think there's probably too
390
00:35:36,240 --> 00:35:42,960
much waiting right now. And yeah, again, I'm not super familiar with this. I know another engineer,
391
00:35:42,960 --> 00:35:49,280
Justin Starry, is working on like improving some of this logic. And I know that he has got a few
392
00:35:49,840 --> 00:35:56,160
work in progress things to sort of improve the skip rate here of the validator improved by
393
00:35:56,160 --> 00:36:02,160
in reducing that time sensitive time waited as well. So the faster you do it, the more
394
00:36:02,160 --> 00:36:07,520
the better it is rather than waiting for other people like other votes and all that other shit.
395
00:36:07,520 --> 00:36:12,640
That's I think that's pretty important. Yeah, I think the scheduler was made that. Oh,
396
00:36:12,640 --> 00:36:18,960
that's where all the kind of shit happens. Because all the the mutex locks. Is there like mutex? I
397
00:36:18,960 --> 00:36:25,760
mean, programming mutex is quite difficult anyway, at least doing it efficiently in a blockchain like
398
00:36:25,760 --> 00:36:32,160
this word, you know, the lock matters, like, it actually changes whether you land or if you don't
399
00:36:32,160 --> 00:36:39,040
land. What are you seeing like any other kind of adversarial games or anything interesting
400
00:36:39,040 --> 00:36:46,080
in the current scheduler banking stage? And whether that's might happen in the new one that you've
401
00:36:46,080 --> 00:36:51,760
built? So we have seen some things around like abusing this the fact that like the old scheduler
402
00:36:51,760 --> 00:36:57,120
just uses raw CU price. But again, that's not really an issue in a new one, because it has this
403
00:36:57,120 --> 00:37:04,480
reward over cost priority or prioritization in it. But yeah, other than that, not not too many like
404
00:37:04,480 --> 00:37:10,320
not too many things that I can think of on top of my head. Okay. And let's let's say people wanted
405
00:37:10,320 --> 00:37:15,840
to continue optimizing. Is there anything you just didn't optimize in the scheduler banking stage
406
00:37:15,840 --> 00:37:22,080
that you would like to future wise? Anything that kind of sparked your interest, you just didn't have
407
00:37:22,080 --> 00:37:27,680
a chance to anything that seemed important? Yeah, I mean, it's definitely like still a work in
408
00:37:27,680 --> 00:37:34,080
progress. There have been like recent PRs. So an issue sort of going back in time a little bit.
409
00:37:34,080 --> 00:37:39,120
People don't make accurate CU requests. And generally, we've seen like that people will
410
00:37:39,120 --> 00:37:45,840
request like 70 times more than they need. Oh, ridiculous. Like it's insane in some cases. And
411
00:37:45,840 --> 00:37:50,960
so the old scheduler, you know, we're basically there's no trust in like what's the use you're
412
00:37:50,960 --> 00:37:55,360
requesting. And when I wrote the new scheduler, we basically were operating off that assumption
413
00:37:55,360 --> 00:38:00,560
that these were not accurate at all. And why are we not accurate? Just because people request
414
00:38:00,560 --> 00:38:06,320
like 70 times more in some cases than they need. Oh, okay. So basically, it's like I can't really
415
00:38:06,320 --> 00:38:11,840
trust these requests and sees and the initial scheduler was written with that sort of assumption
416
00:38:11,840 --> 00:38:17,840
in mind. But then as I've like made this change to do the reward over cost prioritization, we've seen
417
00:38:17,840 --> 00:38:22,480
that curve ratio come down and the values that we're seeing in like the transactions that I'm
418
00:38:22,480 --> 00:38:27,920
not making into the block are much lower. You know, worst case people are average like requesting like
419
00:38:27,920 --> 00:38:35,200
saying like five times more use the rainy, which isn't so bad. And so like recently been doing some
420
00:38:35,200 --> 00:38:40,480
load balancing stuff using the CUs rather than just like transaction counts, which means that
421
00:38:40,480 --> 00:38:46,160
like basically work is spread more evenly across the threads. We can like sort of recent development
422
00:38:46,160 --> 00:38:51,840
because until recently, we didn't know that we could rely on the CUs at all. Why does having
423
00:38:51,840 --> 00:38:58,240
seven X, you matter, see, see use that you really need actually matter? I don't have any idea of
424
00:38:58,240 --> 00:39:04,720
like how long you're actually going to take. You can take anywhere from zero to a whole amount of
425
00:39:04,720 --> 00:39:11,120
those CUs. So it's like, that's not useful to me to load balances. Right? If I'm trying to schedule
426
00:39:11,120 --> 00:39:17,840
like similar amounts of work to each thread, it's not useful if you know, like my range of times for
427
00:39:17,840 --> 00:39:23,680
each transaction is like zero to five milliseconds, too big of a range to like make a useful
428
00:39:23,680 --> 00:39:30,240
explanation. Awesome. Right. Got you. Okay. Okay. Cause like, I guess there is a time associated
429
00:39:30,240 --> 00:39:37,360
with execution with the amount of CUs. What was kind of like your estimation of what's like the
430
00:39:37,360 --> 00:39:44,640
average amount of CUs like, I think it's like 30 CUs per micro is like the cluster average. Okay.
431
00:39:44,640 --> 00:39:51,040
30 CUs. I think that's like an out of date value, even with the new stuff in the new scheduler.
432
00:39:51,040 --> 00:39:56,640
I'm trying, I'm not like using that value. I'm not going to like associate time estimate with like
433
00:39:56,640 --> 00:40:02,400
the CU that could be inaccurate still, but more just trying to load balance, like how many CUs I
434
00:40:02,400 --> 00:40:06,480
send to each thread. Okay. So you've got like maybe a max amount for each thread and you're like,
435
00:40:06,480 --> 00:40:11,600
okay, this is kind of similar. Let's spoush them. Yeah. It's basically like, Hey, I don't want to
436
00:40:11,600 --> 00:40:16,960
create a block that like, Hey, I accidentally sent, you know, 30 million CUs to one thread.
437
00:40:16,960 --> 00:40:22,800
Let's, and then it's just stuck there. Keep it somewhat parallel. Way longer. Yeah. Okay.
438
00:40:22,800 --> 00:40:30,160
Interesting. So yeah. And just also just excited about like some of the longer term SIMDs that we
439
00:40:30,160 --> 00:40:35,920
can optimize the scheduler after. And what are you looking at there? I talked about this a little
440
00:40:35,920 --> 00:40:40,720
bit earlier. Each of these batches that we send to the workers, there's like a constraint in the
441
00:40:40,720 --> 00:40:46,000
protocol right now that each of them, like, you know, it's up, it's up to, you know, let's say 64
442
00:40:46,000 --> 00:40:51,680
transactions, but all of those transactions can't touch like any of the same accounts. If they did
443
00:40:51,680 --> 00:40:56,000
and you like put that into your blocks and it would be like a protocol error. And there's no
444
00:40:56,000 --> 00:41:01,920
real reason for that. So like we recently had SIMD 83 accepted removes this constraint and just says,
445
00:41:01,920 --> 00:41:08,480
yeah, put whatever transactions you want in an entry. And then if they do conflict, they need to
446
00:41:08,480 --> 00:41:14,640
need to just be executed in that order. Yes. That makes like a lot more sense. It also can like
447
00:41:14,640 --> 00:41:21,840
help the scheduler be more efficient. So if the top 10 paying transactions are all conflicting,
448
00:41:21,840 --> 00:41:25,920
put them in a batch, send them all there. They'll be executed in that order. I don't have to like
449
00:41:25,920 --> 00:41:31,360
send them in 10 separate boxes and potentially slow them down with other work between them.
450
00:41:31,360 --> 00:41:38,160
I just say, what do you, okay. And it's like final question. Where do you actually see Solana going?
451
00:41:38,160 --> 00:41:43,520
If it's something bad, how do we redirect it into the right direction? Yeah. And I guess like also
452
00:41:43,520 --> 00:41:48,320
any of you on Solana, what do you see that going? Yeah. I don't really have like big concerns about
453
00:41:48,320 --> 00:41:53,920
the direction Solana is going right now as like a whole. I think, you know, the general direction
454
00:41:53,920 --> 00:42:00,720
is good right now. In terms of mev, I think it's hard to say. There's like different kinds of mev
455
00:42:00,720 --> 00:42:08,400
and I think some of it is like a net negative and some of it's fine. What kind is a net negative?
456
00:42:09,280 --> 00:42:14,960
Obviously the locks one and the spam, but is there anything else? Yes. I mean, I think we saw this
457
00:42:14,960 --> 00:42:21,200
recently with Jito. It was like users were getting sandwiched because of the like mempool
458
00:42:21,200 --> 00:42:27,120
and circars would just like sandwich user transactions. Oh yeah. I think that that's like,
459
00:42:27,120 --> 00:42:31,600
you know, I think Jito made the right call on turning that off, but you know, there's, you know,
460
00:42:32,160 --> 00:42:37,920
probably entities out there who are, I guess, less respectable than Jito, let's just say,
461
00:42:37,920 --> 00:42:44,800
and willing to do that and offer that service to their circars. Oh, that's okay. So just because
462
00:42:44,800 --> 00:42:50,000
Jito has disabled it doesn't mean that someone else might call on that. You know, personally,
463
00:42:50,000 --> 00:42:55,520
I think that that's like a net negative for the ecosystem. If users are sort of afraid of getting
464
00:42:55,520 --> 00:43:01,040
sandwiched every time they send a defi transaction, they're just not going to. Yeah, definitely. Or
465
00:43:01,040 --> 00:43:04,880
they just won't send it to that validator and then they'll just be a skipped block basically.
466
00:43:04,880 --> 00:43:12,400
Yeah. They might like avoid certain validators. Yeah. I think that at that point is a good way to
467
00:43:13,120 --> 00:43:18,320
sort of disincentivize people from doing this sort of thing. Like, you know, if you're aware of
468
00:43:18,320 --> 00:43:24,400
validators doing this sort of shit and like name and shame them. Yeah. There are other types of
469
00:43:24,400 --> 00:43:30,080
like, I think there are other types of Mav that are, you know, fine. If there's like an arbitrage
470
00:43:30,080 --> 00:43:36,000
opportunity, Jito sort of like auctions off that space to people. And, you know, it's not necessarily
471
00:43:36,000 --> 00:43:42,640
in terms of block space or priority fees. They just like have this Mav tip where they say, yeah,
472
00:43:42,640 --> 00:43:49,360
give me some percentage of profit you get from like this arm. I wonder actually if it's possible
473
00:43:49,360 --> 00:43:56,480
to sandwich people without a mempool. Cause if you're maybe, I guess there's no other way of
474
00:43:57,120 --> 00:44:03,040
knowing whether something's being locked outside of being the validator, but maybe you could like
475
00:44:03,040 --> 00:44:11,280
take a gamble. Just like, this is just like a naive thing. You could like lock and then you
476
00:44:11,280 --> 00:44:16,960
can guess someone's being locked and then like back run that locked one. Yeah. I don't know if
477
00:44:16,960 --> 00:44:23,200
you need to like hear too much about the lock in like, cause mostly the schedule is just like,
478
00:44:23,200 --> 00:44:29,280
am I in the correct priority order to sort of get these things? Oh yeah. That'd be very bad order.
479
00:44:29,280 --> 00:44:34,320
That'd be a super complex system. Yeah. You always, you don't know like, hey, what else is in that queue
480
00:44:34,320 --> 00:44:39,360
that I don't know about. Probability stuff. But yeah, yeah, it'll be so hard to. I mean,
481
00:44:39,360 --> 00:44:45,040
there's like definitely ways I can think of to sandwich users right now. I kind of will not say
482
00:44:45,040 --> 00:44:50,800
those publicly. Okay. Cause you're already doing it. No, okay. No, I'm not just like there's,
483
00:44:50,800 --> 00:44:56,800
there are some ways to like see transactions without being the validator. Yeah. Really? Yeah.
484
00:44:56,800 --> 00:45:03,600
I guess, yeah. Yeah. I'll just refrain from saying any more here. Okay. No problem. I think anyone
485
00:45:03,600 --> 00:45:08,960
who is actually like motivated to search would know what I'm talking about, but yeah. Yeah. It's
486
00:45:08,960 --> 00:45:16,160
not worth like exposing something like that in my opinion. 100%. Well, man, Andrew, it's been a,
487
00:45:16,160 --> 00:45:20,480
it's been a pleasure learning about this new scheduler that you've built. Very informative
488
00:45:21,040 --> 00:45:26,240
episode as predicted at the start. And I'm sure a lot of people would find a lot of value in this.
489
00:45:26,240 --> 00:45:30,480
I know I did, but man, it's been a pleasure. Thank you so much for spending your time and
490
00:45:30,480 --> 00:45:37,200
sharing your knowledge with us. Thanks for having me. And I'm looking forward to using your scheduler.
491
00:45:37,200 --> 00:45:41,440
Me too. All right, man. Have a good day and thank you everyone for listening.
