Job Snijders 0:00
And I knew the rsync protocol from my system administration
experience, and I have used rsync in many contexts to make
backups, to transfer files, and I like this utility so it feeling
like it's not performing as well as it should be, or could be, in
relationship to the data. Yeah, at some point I was questioning
myself, but so let's dive into why I think they are somewhat
inefficient. And in the case of rsync, what is very efficient is
that it transfers the difference on the block level between now
and, I don't know, like 30 minutes ago or an hour ago, but what is
very inefficient is figuring out what changed.
George Michaelson 0:52
You're listening to ping, a podcast by APNIC, discussing all
things related to measuring the Internet. I'm your host, George
Michaelson, this time on ping, I'm talking to Job Snijders. We
caught up at IETF 125 held in Shenzhen, job presented to the
SIDROPS Working Group on a new protocol he's been designing called
Erik the Erik protocol was named in honor of Erik Bais, who died
in May 2024 Erik was a stalwart of the ripe routing community. He
was chair of the address policy Working Group and active in the
Dutch cloud community and the Data Center Association RPKI, the
principal mechanism for determining secure inter domain routing
intent. Hence, SIDR depends on every relying party validating the
data and collecting all the signed statements from all the
publication points worldwide. This is a time consuming process
which inherently serializes as a sequence of bytes fetched to form
a given repository state at a given publication point and how the
protocol works out what has changed since the last fetch by this
user and what to send causes this serialization. It's not very
efficient, and it's not scaling as well as we'd like, given the
amount of data rising and the number of validators or RPs who are
now fetching the data. Job's. Erik protocol is designed to improve
significantly on the two mechanisms defined at present, the rsync
protocol, originally designed in the mid 1990s for file system
synchronization and RRDP, a SIDR specific "delta" protocol, which
was designed to improve on rsync, using experience gained from
implementing the NRTM mechanism, use the copy data in the RIR
Whois databases. Job, welcome back to ping. It's lovely to talk to
you again.
Job Snijders 2:54
Thank you so much, George,
George Michaelson 2:55
So you are very active at the moment in RPKI. You've been doing
some really interesting work in the mechanisms we're using there,
which kind of does relate to activity you've also been measuring.
This is, if you like constructs coming out of the measurements
you've been doing on the behavior of the system. Do you think
that's fair?
Job Snijders 3:14
Yeah. Or phrased differently, I develop RPKI, software that is
used by many ISPs around the world, and part of that development
process is observing how does my software behave in the wild with
the real RPKI distributed database that exists on the Internet.
And this basically means that every day, I wake up and I run my
programs, and I look at the outputs and I observe what is the
program doing? Now, at some points, I had a feeling creep on me,
that whereas, like it is, it is taking so long for the process to
complete.
George Michaelson 3:51
Yeah. So just to give people who may not be deeply embedded in
RPKI a sense of this, this is a public key infrastructure data
model where all of the people who are involved in signing over
their routing intent are making signed objects called ROAs that
declare their intention for origin-as in BGP routing, and a new
emerging structure called ASPA we might talk about later, but the
point is to perform the cryptographic validation in this
information space, you have to get everything. And alongside this,
people are free to choose where they publish. And alongside this,
for many, many people, the process of fetching data has become
essentially serialized. I go and look at place A and then I go and
look at place B. And so you in the software development space,
you've been observing the validator that you've written, RPKI
client, getting slower and slower and slower to arrive at the end
point, haven't you? And you've been kind of going back in saying
what's going on here.
Job Snijders 4:58
Well, slower and slower and. It depends on the day. It was almost
like observing the weather, like, how fast is the system today?
And as I observed that, yeah, over time, the database is growing
so
George Michaelson 5:09
so that's like a given. It gets slower because it gets larger,
Job Snijders 5:12
right? So if we compare the size of the database between now and,
say, five years ago, the database is now five times larger, give
or take like, depending on how you look at it, but it's
substantially bigger than it was a few years ago. And the slowness
of the downloads, it surprised me, because the protocols that we
use today are advertised as efficient "change only" protocols that
should perform well,
George Michaelson 5:40
yeah, so these protocols are actually quite interesting. One of
them is almost an undocumented historical artifact of a past time.
It's this protocol called rsync, which was designed by a lovely
guy in Canberra in Australia, Andrew Tridgel, and it was based on
the idea of examining Unix files, block by block, and only sending
the blocks in files you could detect had changed. And if you
consider constructs like log files and other append file
structures, it was really beautiful for this, because it would
basically just do tiny checksum exchanges, ignore the front part,
and then send the blocks that were the growth of the file. But
that mechanism is an archaic model of code that's been brought
into the lifetime of everyone here since about 1990 the newer
protocol that we use, RRDP, it was designed, actually, in part
because of work people like you had done designing this protocol
called NRTM, which is used to propagate changes in Whois it was
designed to be a "delta" protocol, a mechanism to explicitly say,
if you know what you've got and you tell me when I got it, I'll
give you the things that have changed since you got it. So you're
absolutely right Job, both of these protocols are designed to be
efficient in air quotes that you can't see on radio in that data
model, but please continue you're observing they're not as
efficient as we think,
Job Snijders 7:09
right? And I knew the rsync protocol from my system administration
experience, and I have used rsync in many contexts to make
backups, to transfer files, and I like this utility so it feeling
like it's not performing as well as it should be, or could be, in
relationship to the data? Yeah, at some point I was questioning
myself, but so let's dive into why I think they are somewhat
inefficient. And in the case of rsync, what is very efficient is
that it transfers the difference on the block level between now
and, I don't know, like 30 minutes ago or an hour ago, but what is
very inefficient is figuring out what's changed.
George Michaelson 7:51
Yes, there's a pre state in the protocol, right? You tell it the
base of the object set you want to get, typically, a file system
anchor point, a mount point in the file system. And it has to
descend the entire tree of state and build a map. Here are all the
things in this state. And the client has to say, got it, got it.
Got it, got Oh, not that one got it, got it, got it.
Job Snijders 8:15
And, you know, the data is these days, host on memory file system
or SSD, so the IOPS part is, at least on the client side, is not a
prohibitive aspect. You can walk a file tree very fast, but the
RPKI nowadays is roughly 520,000 tiny objects.
George Michaelson 8:33
Oh, wow. That's huge
Job Snijders 8:34
with a median file size of two kilobytes. And some of you may
recall that if you have a large directory with many, many, many
tiny files, moving that around on your computer becomes tedious.
George Michaelson 8:47
That actually incurs costs in the design of the operating systems,
model of how much it has to hold in state in memory against the
catalog of files in a directory you exhaust that buffer in kernel
walking into directories that are large spaces. And rsync is a
beautiful protocol for doing block differences, but it has to
interrogate the file to find the block differences. And if all
your files are tiny, this is a hugely inefficient cost. It's
basically asking tiny objects have you changed? Which begs the
question, How often do tiny objects in the RPKI ecology change?
And I have a feeling Job re-signing means that quite a lot of
these objects change all the time.
Job Snijders 9:33
If you look on a day to day basis, about 180,000 of the objects
change somewhere in a 24 hour period.
George Michaelson 9:42
And these don't form a nice, regular sub pyramid of this scheme.
They're distributed throughout the 500,000 object
Job Snijders 9:50
set every second, two objects change somewhere on the planet, and
we don't know where they'll change on the planet. But even if we
ignore the descent on the file systems. Let's ignore the entire
file system. Just look at the rsync protocol. Sending you the list
of the files has become prohibitively expensive, because a list of
files in the rsync protocol to cover, say more than 500,000 files
is around four megabytes,
George Michaelson 10:20
right.
Job Snijders 10:20
So that means that every time I connect to the servers and nothing
changed, I am still receiving four megabytes of data. [George:
right], the most expensive API call imaginable.
George Michaelson 10:32
and it's not a single packet round trip to be told, four Meg, this
is sent over higher protocols that involve SYN ACK sequences.
They're inherently a serialized stream from that source. So you're
doing four megabytes to fetch that data every time, possibly to be
told, nothing to see here. Move along.
Job Snijders 10:53
So I connect to the rsync server. I'm like, Hey, George, what do
you got for me today?
George Michaelson 10:57
Same stuff I had yesterday?
Job Snijders 10:59
No, I wish you would tell me that what you're telling me is the
full inventory of everything you have.
George Michaelson 11:05
I got sandwiches, I got soup without sandwich, I got coffee, I got
fruit, soup and fruit, or the option of the coffee in the soup,
Job Snijders 11:13
right? And I'll be like, Oh George, I have just eaten all of the
but I'll be back in five minutes
George Michaelson 11:19
and asking me exactly the same relationship
Job Snijders 11:22
over and over and over. So my realization with rsync was the
transfer is highly efficient. You know, it's hard to beat Sure.
You know, it's great, but figuring out what changed that is the
expensive part, and it is becoming a blocking factor to scale the
RPKI to the dimensions that we need for future generations.
George Michaelson 11:43
So RRDP, the Delta protocol, informed by NRTM and change
Job Snijders 11:51
IXFR
George Michaelson 11:52
And DNS Ixfr was an intercept to a more efficient transfer style,
where, if you think about things like MPEG encoding, where you
have I blocks and P blocks, the index blocks and the changes. This
model assumes you periodically may incur a cost of a massive
synchronization, but you're then given a set of deltas that you
can apply. And I want to believe we did an okay job designing
that. And I want to believe that should be more efficient, but I'm
looking at you thinking you're about to tell me it doesn't do a
good a job as we want.
Job Snijders 12:28
Well, it does something that I'm not sure that was fully
understood in terms of consequences at the time of design. So
let's walk through why RRDP appears to be efficient. The idea is
that the publication engine writes into a journal, and you
download the journal in segments, the deltas and on paper, this
sounds great, because between the last time that I synchronized
and this time that I'm synchronized, the server will tell me,
Well, three new segments or three new deltas appeared, and I
download the three deltas, and we're good to go. And these deltas
contain the binary objects base 64 encoders, wrapped in XML. And
originally I thought these layers of encapsulation are what makes
it inefficient.
George Michaelson 13:16
It does sound like over encoding of the fundamental data, so it's
a reasonable thing. That's why it's slow. But it
Job Snijders 13:23
It wasn't because the moment you run this through any compression,
gzip for instance, it compresses down to levels where it's like
the encapsulation is non existent,
George Michaelson 13:35
right? Because the framing components, textually of this stuff
appear as level as if compressed
George Michaelson 13:41
the tokens.
Job Snijders 13:42
That was a red herring, right? Okay, but why does it feel so my
realization with RDP is that RDP was designed for full history
replication, and on paper, this sounds great. You have a validator
you want to synchronize to the database so you replicate the full
history of the database, [George: sure], but you don't need the
full history. The RPKI is churning so much that in the span of 30
to 60 minutes, a lot of events have taken place that have
overtaken previous events. So the inefficiency in RRDP is that
you're always downloading data that has become irrelevant.
George Michaelson 14:22
Yeah. So there's a component of mathematics, the concept of
partially ordered sets and sequences in things like MQTT queuing
systems, this kind of model of message bus, where, if you consider
an object A and it's created at time t and then at some future
intervals, t plus 1t, plus 2t, plus three, you add a field to A,
you double the size of A, you regret it, and you shrink the size
of A. And the very last thing you do is you delete A that
partially ordered set of A, modify, modify, modify, delete, if
you're asking what. It's the current state. The current state is
delete. You don't need to know that prior history if all you want
to know is it doesn't exist. And it feels like you're saying this
protocol had no concept for any object, of constructing the
partially ordered set and reducing it to the minimally necessary
components to arrive at its final state. It rehearses all the
history to get there.
Job Snijders 15:28
Yep. So if we make a weird analogy, you want today's newspaper. So
you're like, hey, Job, can you get me today's newspaper? And I'll
be like, Sure, George, but you have to accept yesterday's and
yesterday's yesterdays newspaper as well. And us clients are like,
but I have no interest in yesterday's news.
George Michaelson 15:48
Yeah, that's under the cat food. I don't need that,
Job Snijders 15:51
right? And I'll be like, I am so sorry, but you have to download
yesterday's newspaper. There is no way to not download it and
arrive at today's newspaper.
George Michaelson 16:01
So we have a protocol designed to be more efficient than rsync, a
protocol designed around the models of Delta. It has an encoding
overhead that you've been able to demonstrate from analyzing its
behavior disappears under compression, and we're left with it's
sending the maximal set of change history when all it needs is the
final state
Job Snijders 16:21
exactly!
George Michaelson 16:22
Well, it would be lovely to imagine we could re enter both of
these protocols and optimize them to achieve a better outcome, but
I have a feeling the rsync community don't particularly feel this
need because their needs are met in rsync and equivalents specific
to synchronizing file systems, because that's what rsync is
designed to do. And RRDP was designed as the better protocol, but
it's manifestly not fit for purpose. So my gut feel is we start
again and try and invent something a little better. So tell me,
Job, did you come up with a better way of doing this?
Job Snijders 17:00
Yes, sir, I did. And not only do I think that the new approach,
which is called Erik synchronization, is better in terms of the
downsides that we discussed just now, but it also addresses an
issue that rsync and RRDP share with each other in the way that
the repository distribution was arranged in the RPKI data
structure. Each publication point has a monopoly on access to its
data. So this means I download APNIC data from the APNIC RRDP or
rsync service, and I cannot connect to the ripe server to get
APNIC data,
George Michaelson 17:41
you know...
Job Snijders 17:42
and this
George Michaelson 17:42
we did talk about this in hypothesis during the design phases. I
should say at this point that although my current life is to be a
talking head in a previous life, I had a role to play in the
design of these kinds of systems. And we talked about the idea it
would be nice if some neutral point could hold and publish all the
data we want.
Job Snijders 18:04
Yeah,
George Michaelson 18:04
but we've encoded in these cryptographically signed objects
reference points to this is where you go to get me. And we didn't
think about, how do you map from that canonical reference point to
an intermediate agent who could probably supply it to you. Now,
intermediates will raise a lot of hackles, but the thing is,
remember, everything in this system is cryptographically signed,
so there is no real downside to being told it by a third party.
You are required to check it before we move on Job. Am I right in
thinking you call it Erik as a memento mori of someone in the
Dutch research and routing community.
Job Snijders 18:44
Yep, the protocol is named after my friend Erik Bais, who
unfortunately passed away in 2024
George Michaelson 18:51
he was a quite significant member of the community, wasn't he? He
took lots of interesting initiatives.
Job Snijders 18:57
He was a big fan of routing security and a promoter of RPKI, he
hosted the very first RPKI nodes to help distribute RPKI data. So
at some point it's clicked. I was like, if I'm making a new
protocol, it must have a name. I'm not going to go for a weird
background name. I will just ... reference my friends.
George Michaelson 19:17
So we have a problem. The architecture requires a fetch from the
originator, and we have no conscious understanding of use of
aggregating intermediaries or any structure that might permit us
to understand the overall shape of the information space. So you
put this on the table as one of the goals you wanted to fix
Job Snijders 19:38
Right The goal is we solve some problems common to rsync and RDP
we solve and we take the best parts from rsync and the best parts
from RDP. And in my mind, if I look at the attractive parts of
RDP, it runs over HTTPS. It's the lingua franca of this community.
George Michaelson 19:57
It's the new waste in the hourglass for. Working protocol
development.
Job Snijders 20:01
So we want something over HTTPS. We want something that allows you
to jump beyond the historic events that no longer matter. So that
is the component from rsync. We want something that doesn't
require an active process on the server side, so that it's all
static data, which is and property of RRDP
George Michaelson 20:22
Yeah, there were efforts to modify rsync to include pre computed
catalog models of data fetch that was actually really important
back in the usenet UUCP days. But the thing is, they're always
adjunct services, and if you're designing a protocol to replace
it, you should build the behavior in day one.
Job Snijders 20:40
Yep, I consider an unwelcome complication in RRDP, the fact that
an XML container was used because it meant that all the RPKI
validators had to link in an additional software library
dependency to be able to interact with the protocol. So I tried to
make the protocol reuse components that we're using anyway in this
ecosystem. So the messaging layer in the Erik synchronization
protocol is ASN.1 or distinguished encoding rules DER, because
that's what validators already talk
George Michaelson 21:15
gotta say, this is my least favorite protocol. But if you're a
telecoms engineer, ASN.1 is a piece of IBM era, historical
dependency of bit efficient on the wire encoding in a context. So
in ASN.1, you receive two bits. You know their meaning because you
were told in a schema what to expect in this two bit sequence,
given things that have happened before, and it is tiny on the
wire. It's efficient.
Job Snijders 21:44
It is not only efficient, but it gives you canonical encoding
forms and canonicalizing What the on the wire, shape of the bytes
is, means you have item potency, and that opens your door up to
distributed computing solutions where you have multiple writers,
multiple readers, all doing work concurrently, and you can build
redundancy into the production and consumption of the
George Michaelson 22:09
data, because the data stream is absolutely predictable based on
the condition it was in at the time it was generated, and
expectations of behavior anyone else can do a reproduction, they
know exactly how to perform the encoding to match
Job Snijders 22:24
right. And the reproduction in the Erik protocol, it happens on
both the clients and the server side, and both encode towards a
Merkle tree shape, [George: right] And then you can do the
comparison to figure out, what is it that changed?
George Michaelson 22:40
Now I don't want to do a deep dive down structural behaviors, but
Merkle trees are interesting. They're the product of someone who I
think we can say is genius class.
Job Snijders 22:51
Oh, absolutely,
George Michaelson 22:52
in information management, cryptography, design of systems, and
they've become very important in a space I hate, which is Bitcoin
and that world of crypto currency, but it's important to
distinguish the information structure from the horrible behaviors
in Bitcoin and crypto currency. The information structure is
fascinating,
Job Snijders 23:15
yeah, allow me to quickly summarize how a Merkle tree works.
Imagine you have 10 files on offer, and I want to have those 10
files now, the most naive way to replicate the data would be to
just ask you, George, send me the 10 files, and then I have
downloaded 10 files. Now, if one of them changed, how do we
efficiently figure out which one of the files changed? And a
possible approach is you hash a SHA256, message digest over the 10
files, and you have 10 files, so you end up with 10 hashes.
George Michaelson 23:50
So each file gets a hash.
George Michaelson 23:52
Yeah, you sent me the 10 hashes. I calculate the hashes on the 10
files I have in my match
George Michaelson 23:58
and if one of them matches and one of them doesn't you've
identified the changed file,
Job Snijders 24:03
right? But you sending me 10 hashes, still is a little bit
inefficient. So what if you create a hash over those 10 hashes,
and you sent me that one hash,
George Michaelson 24:14
right? And so if you now scale from 10 files to 100 files, but
imagine i hash every 10 files, I can only send you meta hashes,
and you get to detect one of the block of 10 is to be considered.
And you can decide the size of that meta footprint and how
effective the hashing calculation is as a function of your sense
of risk of change inside this framework
Job Snijders 24:37
Right! So in the Erik synchronization protocol, the Merkle tree is
three phases deep, and what it boils down to is I can ask the
intermediate the relay for what is the latest hash over the top of
it, and I get one hash. And if it's the same as what I computed
over the data, I have locally, then we're done. Then my
exchanging just a few bytes, I have figured out that we are
synchronized.
George Michaelson 25:04
So a point here is the hash is a calculation that is valid against
no change, which means if you compute the hash on change and store
it, when you're asked, What is your hash, you don't have to
calculate it. And so both this intermediate and you as a client,
if you compute your hash in detection of change the minute you
decide to do the game again, you can fling the hash and check the
hash. Nobody has to do sums just because a client comes to the
door.
Job Snijders 25:35
And if the hash is different, I will fetch the hash that you gave
me. And the trick in the Erik synchronization protocol is that you
you don't fetch the objects by name. You fetch objects by hash. So
I asked the web server give me the object that belongs to this
hash, and that means that I have integrity verification on the
object contents, because the hash of the object is the URL, so to
speak, and then I fetch the data that changed, and I can walk the
tree to figure out where in the tree are the objects that change.
So this is very, very different from both RRDP and rsync, because
the client knows a-priori what it wants to fetch from the server,
whereas in RRDP, it's a little bit more like a surprise.
George Michaelson 26:23
Yeah, tell I know something's changed. Just give me the changes.
Give me where's this model? You have a 3d hashing model that
directs down to the smallest set you'll wear the cost of getting
to then find the specific change. And you have moved from a
namespace of canonical fetch points which are encoded in objects
in RPKI, to hashes which are constant length, objects constant
size, constant cost. You've taken variable cost because it's
potentially pointing at indirect places that require you to
connect elsewhere. So you've actually done two things, Job: you've
constructed an information model that is indexed on hashing, and
you have intruded the concept of an intermediary that is
potentially able to tell you this across many back end sources.
Job Snijders 27:13
Yep. And why this all matters if we look at ROA propagation, the
ROA is a very powerful utility, right? It is a very high impact
facility that we use for good to improve routing security. But if
you create a ROA and you mistype the origin ASN, you might disrupt
service for the customer whose prefix is encompassed in the ROA.
So we're sort of moving away from the consequences of a
typographic error in the BGP routing protocol to there being
consequences if you make a typographic error in the creation of
your ROA.
George Michaelson 27:52
And it should be said that, again, reaching back into the past,
we'd imagine cryptography was about making the longest possible
signing assertions to reduce the burden of signing. And what we
forgot is an object is valid for the lifetime of its certificate,
and there are risks in long lived cryptographic signed objects.
Some of them, yeah, it's beneficial to live a long time down at
the edge where things have to change and you may make mistakes of
consequence. You actually want short lifetimes, which means you
want the ability to say, I need that to go away and have people be
responsive to it. People need to look at these things more often
than we thought.
Job Snijders 28:33
You're totally right. So let's, let's run through a scenario that
could happen to anyone. Your boss tells you, hey, RPKI, I heard
good stories about it. Can you make a ROA? And ROAs propagates on
the Internet in the span of between, say, 10 and 120 minutes. And
P95 is a little bit less than an hour. So you could say, you know,
on average, most of the internet will have seen your new ROA
within an hour, but it also could be as short as 10 minutes, if
you're lucky, and if you're a little bit unlucky, it could take
longer. So you create a ROA, and after you hit submit, you
realize, oh, boy, that was not optimal. I copy paste it, and I
left off the last digit, but the ROA you created that contains the
error propagates in 10 minutes, and then you quickly fix that ROA.
But that version of the ROA takes two hours to propagate, and that
means that for 110 minutes, you're kind of fiddling your thumbs,
hoping the ROA will start propagating, but it's out of your
control. And this is why I think it is critical for this community
to really engineer for optimal performance. We have consistent
performance, predictable performance. ROAs must propagate as fast
as they are created, and I think the Erik synchronization protocol
is a step towards predictable propagation times.
George Michaelson 30:01
So here we are sitting in IETF 125, in Shenzhen, and something
that's come up as a discussion item in the corridors is the idea
of code-first, standards development. And I know from other
conversations that you've been promoting the idea for work to be
taken seriously inside SIDROPS it's important we get to working
implementations beyond conceptual proof. This isn't a paper design
is this Job?
Job Snijders 30:29
I am happy to report we have running code. And it is really cool
to be thinking about ideas and what could work in practice, but it
is much, much more interesting to verify that the ideas do in the
work in practice.
George Michaelson 30:44
And so you now have some measurements using real world data to
demonstrate what it would look like fetching the world of ROA
through intermediaries, using Erik
Job Snijders 30:55
Right, my preliminary measurements suggest that, depending on the
refresh cycle of the validator. So a validator could, say,
synchronize every 15 minutes or every 60 Minutes. Like let's take
those two patterns as an example. And if you synchronize a
validator every 15 minutes using rsync, it is quite costly,
because, as we discussed, with rsync, you receive the entire menu
of everything that's on order, and then you exchange differences.
And if you compare rsync to RRDP on the 15 minute interval, RRDP
is more efficient because there's no giant file list in RRDP, but
if you synchronize every hour, because with RRDP, you're pulling
in the full history of everything that changed
George Michaelson 31:42
And across the surface of this space. Some things have gone
through multiple changes. You're pulling more data than you need,
Job Snijders 31:49
right? So, weirdly enough, if you synchronize every hour, rsync is
more efficient from a bandwidth consumption perspective. And these
results surprised me. I was like, That is counter intuitive!
George Michaelson 32:00
enough, we know a kind of range of interest, 15 minutes and an
hour, two protocols with a weird outcome. How does Erik play in
this space?
Job Snijders 32:08
My initial measurements suggest that Erik synchronization performs
better in both scenarios. And this is really good news, because
for an RP to decide which of the protocols should I be using, and
basing that on the synchronization frequency is a massive
implementation complexity. So having a protocol that, regardless
of the timing interval, performs better is a very, very desirable
thing to have, because it means that whatever schedule you're on,
Erik synchronization performs better.
George Michaelson 32:40
So you mentioned that it's HTTP is a substrate. This makes me
immediately ask, Is there potential for the CDN community to be
able to consider themselves as the intermediaries and provide us
with a huge surface of reliable, short path visibility? Because
one of the dimensions of slowness you haven't mentioned is that if
I'm a validator in Chile and I'm attempting to collect the entire
surface of RPKI, I have to have a reliable path to Indonesia and
to Mongolia and to Europe and North America and my buddies nearby
in Uruguay. I need to be able to get to all of them to complete my
act. If this was in an intermediary, and the CDN had an anycast
surface, I would be fetching at a constant distance cost to that
CDN with a better protocol. Is this a potential?
Job Snijders 33:36
Yeah, yeah. So because we're fetching by hash, not by file name,
not by FQDN. You can connect to anycasted instances and rotate
between the instances. So we've now converted the RPKI
distribution problem into something that is
George Michaelson 33:51
this is now similar to a model like 1.1 dot 1.1
Job Snijders 33:56
and 8.8, dot 8.8, and nine, dot 9.9, and whatever open resolver
you want to use, we have now constructed a system where you can
fetch RPKI data from publicly accessible endpoints that contain
data for multiple RPKI origins. So as you point out like this,
this latency factor from the client to all over the world, the
latency becomes a predictable elements. You can fetch from a close
by mirror, and you can use multiple mirrors concurrently. So if
one instance is a little bit behind or outdated for one reason or
another, it doesn't matter the client recognizes, oh, you contain
old data. Well, I'm going to try the next Erik relay.
George Michaelson 34:40
So if I consider myself in the role of a relay that front facing
service through Erik, I have the potential to use either Erik or
RRDP or rsync in my back, depending on capabilities in the real
publication point, don't I?
Job Snijders 34:56
So an Erik relay merges all the data be it received via rsync over
fee four or IPV6, received through RDP, received through Erik, it
synthesizes a final construction of the data set, presents that to
the clients in the Erik format. But it doesn't have a monopoly on
the data. The client is absolutely free to connect to multiple
relays operated by multiple organizations. So I think this is a
huge improvement in robustness.
George Michaelson 35:28
Oh, it sounds like a very positive outcome. You were presenting
this in SIDROPS this week?
Job Snijders 35:33
Yeah, I presented updates, and one of the fun things to share in
that update is that there is running code. So I have set up a
number of relays. Some of them are, you know, only in Europe. Some
of them are globally distributed. So I created a diverse set of
relays to help test what does the performance of Erik look like
from a global perspective? And Tom Harrison from APNIC did a
wonderful job creating a Erik client implementation. So now we
have all the building blocks to do extensive measurements over the
span of multiple months to with high precision, observe what is
the difference. If there is any difference between Erik, RRDP and
rsync, and my hope is that the Erik testing instances will show to
consume less data, be faster in converging but we'll see. I mean,
George Michaelson 36:27
well, we'll have you back and talk about this another time when
this emerges. Job, that's been absolutely fascinating. We'll put
pointers to this into the blog that goes with the podcast. Thank
you.
Job Snijders 36:38
And if I may, do one more call out, if you're interested in
researching these phenomenas, we are developing the Erik protocol
at this moment. So for students that are interested in this type
of problem space, like, how do you synchronize hundreds and 1000s
of tiny objects to 1000s of Computer
George Michaelson 36:58
There are Research Opportunities
Job Snijders 36:59
Exactly? And we need the help from younger students to progress
this protocol to be something that we want to use the next 10 20,
30 years.
George Michaelson 37:09
Fantastic. That's great Job -thank you.
Job Snijders 37:11
All right, George, until next time,
George Michaelson 37:14
if you've got a story or research to share here on ping, why not
get in contact by email to ping@apnic.net or via the APNIC social
media channels, also remember the measurement@apnic.net mail list
on orbit is there to discuss and share relevant collaborative
opportunities, grants and funding, opportunities, jobs and
graduate placings, or To seek feedback from the community on your
own measurement projects, be sure to check out the APNIC website
for All your resource and community needs until next time you you.
And I knew the rsync protocol from my system administration
experience, and I have used rsync in many contexts to make
backups, to transfer files, and I like this utility so it feeling
like it's not performing as well as it should be, or could be, in
relationship to the data. Yeah, at some point I was questioning
myself, but so let's dive into why I think they are somewhat
inefficient. And in the case of rsync, what is very efficient is
that it transfers the difference on the block level between now
and, I don't know, like 30 minutes ago or an hour ago, but what is
very inefficient is figuring out what changed.
George Michaelson 0:52
You're listening to ping, a podcast by APNIC, discussing all
things related to measuring the Internet. I'm your host, George
Michaelson, this time on ping, I'm talking to Job Snijders. We
caught up at IETF 125 held in Shenzhen, job presented to the
SIDROPS Working Group on a new protocol he's been designing called
Erik the Erik protocol was named in honor of Erik Bais, who died
in May 2024 Erik was a stalwart of the ripe routing community. He
was chair of the address policy Working Group and active in the
Dutch cloud community and the Data Center Association RPKI, the
principal mechanism for determining secure inter domain routing
intent. Hence, SIDR depends on every relying party validating the
data and collecting all the signed statements from all the
publication points worldwide. This is a time consuming process
which inherently serializes as a sequence of bytes fetched to form
a given repository state at a given publication point and how the
protocol works out what has changed since the last fetch by this
user and what to send causes this serialization. It's not very
efficient, and it's not scaling as well as we'd like, given the
amount of data rising and the number of validators or RPs who are
now fetching the data. Job's. Erik protocol is designed to improve
significantly on the two mechanisms defined at present, the rsync
protocol, originally designed in the mid 1990s for file system
synchronization and RRDP, a SIDR specific "delta" protocol, which
was designed to improve on rsync, using experience gained from
implementing the NRTM mechanism, use the copy data in the RIR
Whois databases. Job, welcome back to ping. It's lovely to talk to
you again.
Job Snijders 2:54
Thank you so much, George,
George Michaelson 2:55
So you are very active at the moment in RPKI. You've been doing
some really interesting work in the mechanisms we're using there,
which kind of does relate to activity you've also been measuring.
This is, if you like constructs coming out of the measurements
you've been doing on the behavior of the system. Do you think
that's fair?
Job Snijders 3:14
Yeah. Or phrased differently, I develop RPKI, software that is
used by many ISPs around the world, and part of that development
process is observing how does my software behave in the wild with
the real RPKI distributed database that exists on the Internet.
And this basically means that every day, I wake up and I run my
programs, and I look at the outputs and I observe what is the
program doing? Now, at some points, I had a feeling creep on me,
that whereas, like it is, it is taking so long for the process to
complete.
George Michaelson 3:51
Yeah. So just to give people who may not be deeply embedded in
RPKI a sense of this, this is a public key infrastructure data
model where all of the people who are involved in signing over
their routing intent are making signed objects called ROAs that
declare their intention for origin-as in BGP routing, and a new
emerging structure called ASPA we might talk about later, but the
point is to perform the cryptographic validation in this
information space, you have to get everything. And alongside this,
people are free to choose where they publish. And alongside this,
for many, many people, the process of fetching data has become
essentially serialized. I go and look at place A and then I go and
look at place B. And so you in the software development space,
you've been observing the validator that you've written, RPKI
client, getting slower and slower and slower to arrive at the end
point, haven't you? And you've been kind of going back in saying
what's going on here.
Job Snijders 4:58
Well, slower and slower and. It depends on the day. It was almost
like observing the weather, like, how fast is the system today?
And as I observed that, yeah, over time, the database is growing
so
George Michaelson 5:09
so that's like a given. It gets slower because it gets larger,
Job Snijders 5:12
right? So if we compare the size of the database between now and,
say, five years ago, the database is now five times larger, give
or take like, depending on how you look at it, but it's
substantially bigger than it was a few years ago. And the slowness
of the downloads, it surprised me, because the protocols that we
use today are advertised as efficient "change only" protocols that
should perform well,
George Michaelson 5:40
yeah, so these protocols are actually quite interesting. One of
them is almost an undocumented historical artifact of a past time.
It's this protocol called rsync, which was designed by a lovely
guy in Canberra in Australia, Andrew Tridgel, and it was based on
the idea of examining Unix files, block by block, and only sending
the blocks in files you could detect had changed. And if you
consider constructs like log files and other append file
structures, it was really beautiful for this, because it would
basically just do tiny checksum exchanges, ignore the front part,
and then send the blocks that were the growth of the file. But
that mechanism is an archaic model of code that's been brought
into the lifetime of everyone here since about 1990 the newer
protocol that we use, RRDP, it was designed, actually, in part
because of work people like you had done designing this protocol
called NRTM, which is used to propagate changes in Whois it was
designed to be a "delta" protocol, a mechanism to explicitly say,
if you know what you've got and you tell me when I got it, I'll
give you the things that have changed since you got it. So you're
absolutely right Job, both of these protocols are designed to be
efficient in air quotes that you can't see on radio in that data
model, but please continue you're observing they're not as
efficient as we think,
Job Snijders 7:09
right? And I knew the rsync protocol from my system administration
experience, and I have used rsync in many contexts to make
backups, to transfer files, and I like this utility so it feeling
like it's not performing as well as it should be, or could be, in
relationship to the data? Yeah, at some point I was questioning
myself, but so let's dive into why I think they are somewhat
inefficient. And in the case of rsync, what is very efficient is
that it transfers the difference on the block level between now
and, I don't know, like 30 minutes ago or an hour ago, but what is
very inefficient is figuring out what's changed.
George Michaelson 7:51
Yes, there's a pre state in the protocol, right? You tell it the
base of the object set you want to get, typically, a file system
anchor point, a mount point in the file system. And it has to
descend the entire tree of state and build a map. Here are all the
things in this state. And the client has to say, got it, got it.
Got it, got Oh, not that one got it, got it, got it.
Job Snijders 8:15
And, you know, the data is these days, host on memory file system
or SSD, so the IOPS part is, at least on the client side, is not a
prohibitive aspect. You can walk a file tree very fast, but the
RPKI nowadays is roughly 520,000 tiny objects.
George Michaelson 8:33
Oh, wow. That's huge
Job Snijders 8:34
with a median file size of two kilobytes. And some of you may
recall that if you have a large directory with many, many, many
tiny files, moving that around on your computer becomes tedious.
George Michaelson 8:47
That actually incurs costs in the design of the operating systems,
model of how much it has to hold in state in memory against the
catalog of files in a directory you exhaust that buffer in kernel
walking into directories that are large spaces. And rsync is a
beautiful protocol for doing block differences, but it has to
interrogate the file to find the block differences. And if all
your files are tiny, this is a hugely inefficient cost. It's
basically asking tiny objects have you changed? Which begs the
question, How often do tiny objects in the RPKI ecology change?
And I have a feeling Job re-signing means that quite a lot of
these objects change all the time.
Job Snijders 9:33
If you look on a day to day basis, about 180,000 of the objects
change somewhere in a 24 hour period.
George Michaelson 9:42
And these don't form a nice, regular sub pyramid of this scheme.
They're distributed throughout the 500,000 object
Job Snijders 9:50
set every second, two objects change somewhere on the planet, and
we don't know where they'll change on the planet. But even if we
ignore the descent on the file systems. Let's ignore the entire
file system. Just look at the rsync protocol. Sending you the list
of the files has become prohibitively expensive, because a list of
files in the rsync protocol to cover, say more than 500,000 files
is around four megabytes,
George Michaelson 10:20
right.
Job Snijders 10:20
So that means that every time I connect to the servers and nothing
changed, I am still receiving four megabytes of data. [George:
right], the most expensive API call imaginable.
George Michaelson 10:32
and it's not a single packet round trip to be told, four Meg, this
is sent over higher protocols that involve SYN ACK sequences.
They're inherently a serialized stream from that source. So you're
doing four megabytes to fetch that data every time, possibly to be
told, nothing to see here. Move along.
Job Snijders 10:53
So I connect to the rsync server. I'm like, Hey, George, what do
you got for me today?
George Michaelson 10:57
Same stuff I had yesterday?
Job Snijders 10:59
No, I wish you would tell me that what you're telling me is the
full inventory of everything you have.
George Michaelson 11:05
I got sandwiches, I got soup without sandwich, I got coffee, I got
fruit, soup and fruit, or the option of the coffee in the soup,
Job Snijders 11:13
right? And I'll be like, Oh George, I have just eaten all of the
but I'll be back in five minutes
George Michaelson 11:19
and asking me exactly the same relationship
Job Snijders 11:22
over and over and over. So my realization with rsync was the
transfer is highly efficient. You know, it's hard to beat Sure.
You know, it's great, but figuring out what changed that is the
expensive part, and it is becoming a blocking factor to scale the
RPKI to the dimensions that we need for future generations.
George Michaelson 11:43
So RRDP, the Delta protocol, informed by NRTM and change
Job Snijders 11:51
IXFR
George Michaelson 11:52
And DNS Ixfr was an intercept to a more efficient transfer style,
where, if you think about things like MPEG encoding, where you
have I blocks and P blocks, the index blocks and the changes. This
model assumes you periodically may incur a cost of a massive
synchronization, but you're then given a set of deltas that you
can apply. And I want to believe we did an okay job designing
that. And I want to believe that should be more efficient, but I'm
looking at you thinking you're about to tell me it doesn't do a
good a job as we want.
Job Snijders 12:28
Well, it does something that I'm not sure that was fully
understood in terms of consequences at the time of design. So
let's walk through why RRDP appears to be efficient. The idea is
that the publication engine writes into a journal, and you
download the journal in segments, the deltas and on paper, this
sounds great, because between the last time that I synchronized
and this time that I'm synchronized, the server will tell me,
Well, three new segments or three new deltas appeared, and I
download the three deltas, and we're good to go. And these deltas
contain the binary objects base 64 encoders, wrapped in XML. And
originally I thought these layers of encapsulation are what makes
it inefficient.
George Michaelson 13:16
It does sound like over encoding of the fundamental data, so it's
a reasonable thing. That's why it's slow. But it
Job Snijders 13:23
It wasn't because the moment you run this through any compression,
gzip for instance, it compresses down to levels where it's like
the encapsulation is non existent,
George Michaelson 13:35
right? Because the framing components, textually of this stuff
appear as level as if compressed
George Michaelson 13:41
the tokens.
Job Snijders 13:42
That was a red herring, right? Okay, but why does it feel so my
realization with RDP is that RDP was designed for full history
replication, and on paper, this sounds great. You have a validator
you want to synchronize to the database so you replicate the full
history of the database, [George: sure], but you don't need the
full history. The RPKI is churning so much that in the span of 30
to 60 minutes, a lot of events have taken place that have
overtaken previous events. So the inefficiency in RRDP is that
you're always downloading data that has become irrelevant.
George Michaelson 14:22
Yeah. So there's a component of mathematics, the concept of
partially ordered sets and sequences in things like MQTT queuing
systems, this kind of model of message bus, where, if you consider
an object A and it's created at time t and then at some future
intervals, t plus 1t, plus 2t, plus three, you add a field to A,
you double the size of A, you regret it, and you shrink the size
of A. And the very last thing you do is you delete A that
partially ordered set of A, modify, modify, modify, delete, if
you're asking what. It's the current state. The current state is
delete. You don't need to know that prior history if all you want
to know is it doesn't exist. And it feels like you're saying this
protocol had no concept for any object, of constructing the
partially ordered set and reducing it to the minimally necessary
components to arrive at its final state. It rehearses all the
history to get there.
Job Snijders 15:28
Yep. So if we make a weird analogy, you want today's newspaper. So
you're like, hey, Job, can you get me today's newspaper? And I'll
be like, Sure, George, but you have to accept yesterday's and
yesterday's yesterdays newspaper as well. And us clients are like,
but I have no interest in yesterday's news.
George Michaelson 15:48
Yeah, that's under the cat food. I don't need that,
Job Snijders 15:51
right? And I'll be like, I am so sorry, but you have to download
yesterday's newspaper. There is no way to not download it and
arrive at today's newspaper.
George Michaelson 16:01
So we have a protocol designed to be more efficient than rsync, a
protocol designed around the models of Delta. It has an encoding
overhead that you've been able to demonstrate from analyzing its
behavior disappears under compression, and we're left with it's
sending the maximal set of change history when all it needs is the
final state
Job Snijders 16:21
exactly!
George Michaelson 16:22
Well, it would be lovely to imagine we could re enter both of
these protocols and optimize them to achieve a better outcome, but
I have a feeling the rsync community don't particularly feel this
need because their needs are met in rsync and equivalents specific
to synchronizing file systems, because that's what rsync is
designed to do. And RRDP was designed as the better protocol, but
it's manifestly not fit for purpose. So my gut feel is we start
again and try and invent something a little better. So tell me,
Job, did you come up with a better way of doing this?
Job Snijders 17:00
Yes, sir, I did. And not only do I think that the new approach,
which is called Erik synchronization, is better in terms of the
downsides that we discussed just now, but it also addresses an
issue that rsync and RRDP share with each other in the way that
the repository distribution was arranged in the RPKI data
structure. Each publication point has a monopoly on access to its
data. So this means I download APNIC data from the APNIC RRDP or
rsync service, and I cannot connect to the ripe server to get
APNIC data,
George Michaelson 17:41
you know...
Job Snijders 17:42
and this
George Michaelson 17:42
we did talk about this in hypothesis during the design phases. I
should say at this point that although my current life is to be a
talking head in a previous life, I had a role to play in the
design of these kinds of systems. And we talked about the idea it
would be nice if some neutral point could hold and publish all the
data we want.
Job Snijders 18:04
Yeah,
George Michaelson 18:04
but we've encoded in these cryptographically signed objects
reference points to this is where you go to get me. And we didn't
think about, how do you map from that canonical reference point to
an intermediate agent who could probably supply it to you. Now,
intermediates will raise a lot of hackles, but the thing is,
remember, everything in this system is cryptographically signed,
so there is no real downside to being told it by a third party.
You are required to check it before we move on Job. Am I right in
thinking you call it Erik as a memento mori of someone in the
Dutch research and routing community.
Job Snijders 18:44
Yep, the protocol is named after my friend Erik Bais, who
unfortunately passed away in 2024
George Michaelson 18:51
he was a quite significant member of the community, wasn't he? He
took lots of interesting initiatives.
Job Snijders 18:57
He was a big fan of routing security and a promoter of RPKI, he
hosted the very first RPKI nodes to help distribute RPKI data. So
at some point it's clicked. I was like, if I'm making a new
protocol, it must have a name. I'm not going to go for a weird
background name. I will just ... reference my friends.
George Michaelson 19:17
So we have a problem. The architecture requires a fetch from the
originator, and we have no conscious understanding of use of
aggregating intermediaries or any structure that might permit us
to understand the overall shape of the information space. So you
put this on the table as one of the goals you wanted to fix
Job Snijders 19:38
Right The goal is we solve some problems common to rsync and RDP
we solve and we take the best parts from rsync and the best parts
from RDP. And in my mind, if I look at the attractive parts of
RDP, it runs over HTTPS. It's the lingua franca of this community.
George Michaelson 19:57
It's the new waste in the hourglass for. Working protocol
development.
Job Snijders 20:01
So we want something over HTTPS. We want something that allows you
to jump beyond the historic events that no longer matter. So that
is the component from rsync. We want something that doesn't
require an active process on the server side, so that it's all
static data, which is and property of RRDP
George Michaelson 20:22
Yeah, there were efforts to modify rsync to include pre computed
catalog models of data fetch that was actually really important
back in the usenet UUCP days. But the thing is, they're always
adjunct services, and if you're designing a protocol to replace
it, you should build the behavior in day one.
Job Snijders 20:40
Yep, I consider an unwelcome complication in RRDP, the fact that
an XML container was used because it meant that all the RPKI
validators had to link in an additional software library
dependency to be able to interact with the protocol. So I tried to
make the protocol reuse components that we're using anyway in this
ecosystem. So the messaging layer in the Erik synchronization
protocol is ASN.1 or distinguished encoding rules DER, because
that's what validators already talk
George Michaelson 21:15
gotta say, this is my least favorite protocol. But if you're a
telecoms engineer, ASN.1 is a piece of IBM era, historical
dependency of bit efficient on the wire encoding in a context. So
in ASN.1, you receive two bits. You know their meaning because you
were told in a schema what to expect in this two bit sequence,
given things that have happened before, and it is tiny on the
wire. It's efficient.
Job Snijders 21:44
It is not only efficient, but it gives you canonical encoding
forms and canonicalizing What the on the wire, shape of the bytes
is, means you have item potency, and that opens your door up to
distributed computing solutions where you have multiple writers,
multiple readers, all doing work concurrently, and you can build
redundancy into the production and consumption of the
George Michaelson 22:09
data, because the data stream is absolutely predictable based on
the condition it was in at the time it was generated, and
expectations of behavior anyone else can do a reproduction, they
know exactly how to perform the encoding to match
Job Snijders 22:24
right. And the reproduction in the Erik protocol, it happens on
both the clients and the server side, and both encode towards a
Merkle tree shape, [George: right] And then you can do the
comparison to figure out, what is it that changed?
George Michaelson 22:40
Now I don't want to do a deep dive down structural behaviors, but
Merkle trees are interesting. They're the product of someone who I
think we can say is genius class.
Job Snijders 22:51
Oh, absolutely,
George Michaelson 22:52
in information management, cryptography, design of systems, and
they've become very important in a space I hate, which is Bitcoin
and that world of crypto currency, but it's important to
distinguish the information structure from the horrible behaviors
in Bitcoin and crypto currency. The information structure is
fascinating,
Job Snijders 23:15
yeah, allow me to quickly summarize how a Merkle tree works.
Imagine you have 10 files on offer, and I want to have those 10
files now, the most naive way to replicate the data would be to
just ask you, George, send me the 10 files, and then I have
downloaded 10 files. Now, if one of them changed, how do we
efficiently figure out which one of the files changed? And a
possible approach is you hash a SHA256, message digest over the 10
files, and you have 10 files, so you end up with 10 hashes.
George Michaelson 23:50
So each file gets a hash.
George Michaelson 23:52
Yeah, you sent me the 10 hashes. I calculate the hashes on the 10
files I have in my match
George Michaelson 23:58
and if one of them matches and one of them doesn't you've
identified the changed file,
Job Snijders 24:03
right? But you sending me 10 hashes, still is a little bit
inefficient. So what if you create a hash over those 10 hashes,
and you sent me that one hash,
George Michaelson 24:14
right? And so if you now scale from 10 files to 100 files, but
imagine i hash every 10 files, I can only send you meta hashes,
and you get to detect one of the block of 10 is to be considered.
And you can decide the size of that meta footprint and how
effective the hashing calculation is as a function of your sense
of risk of change inside this framework
Job Snijders 24:37
Right! So in the Erik synchronization protocol, the Merkle tree is
three phases deep, and what it boils down to is I can ask the
intermediate the relay for what is the latest hash over the top of
it, and I get one hash. And if it's the same as what I computed
over the data, I have locally, then we're done. Then my
exchanging just a few bytes, I have figured out that we are
synchronized.
George Michaelson 25:04
So a point here is the hash is a calculation that is valid against
no change, which means if you compute the hash on change and store
it, when you're asked, What is your hash, you don't have to
calculate it. And so both this intermediate and you as a client,
if you compute your hash in detection of change the minute you
decide to do the game again, you can fling the hash and check the
hash. Nobody has to do sums just because a client comes to the
door.
Job Snijders 25:35
And if the hash is different, I will fetch the hash that you gave
me. And the trick in the Erik synchronization protocol is that you
you don't fetch the objects by name. You fetch objects by hash. So
I asked the web server give me the object that belongs to this
hash, and that means that I have integrity verification on the
object contents, because the hash of the object is the URL, so to
speak, and then I fetch the data that changed, and I can walk the
tree to figure out where in the tree are the objects that change.
So this is very, very different from both RRDP and rsync, because
the client knows a-priori what it wants to fetch from the server,
whereas in RRDP, it's a little bit more like a surprise.
George Michaelson 26:23
Yeah, tell I know something's changed. Just give me the changes.
Give me where's this model? You have a 3d hashing model that
directs down to the smallest set you'll wear the cost of getting
to then find the specific change. And you have moved from a
namespace of canonical fetch points which are encoded in objects
in RPKI, to hashes which are constant length, objects constant
size, constant cost. You've taken variable cost because it's
potentially pointing at indirect places that require you to
connect elsewhere. So you've actually done two things, Job: you've
constructed an information model that is indexed on hashing, and
you have intruded the concept of an intermediary that is
potentially able to tell you this across many back end sources.
Job Snijders 27:13
Yep. And why this all matters if we look at ROA propagation, the
ROA is a very powerful utility, right? It is a very high impact
facility that we use for good to improve routing security. But if
you create a ROA and you mistype the origin ASN, you might disrupt
service for the customer whose prefix is encompassed in the ROA.
So we're sort of moving away from the consequences of a
typographic error in the BGP routing protocol to there being
consequences if you make a typographic error in the creation of
your ROA.
George Michaelson 27:52
And it should be said that, again, reaching back into the past,
we'd imagine cryptography was about making the longest possible
signing assertions to reduce the burden of signing. And what we
forgot is an object is valid for the lifetime of its certificate,
and there are risks in long lived cryptographic signed objects.
Some of them, yeah, it's beneficial to live a long time down at
the edge where things have to change and you may make mistakes of
consequence. You actually want short lifetimes, which means you
want the ability to say, I need that to go away and have people be
responsive to it. People need to look at these things more often
than we thought.
Job Snijders 28:33
You're totally right. So let's, let's run through a scenario that
could happen to anyone. Your boss tells you, hey, RPKI, I heard
good stories about it. Can you make a ROA? And ROAs propagates on
the Internet in the span of between, say, 10 and 120 minutes. And
P95 is a little bit less than an hour. So you could say, you know,
on average, most of the internet will have seen your new ROA
within an hour, but it also could be as short as 10 minutes, if
you're lucky, and if you're a little bit unlucky, it could take
longer. So you create a ROA, and after you hit submit, you
realize, oh, boy, that was not optimal. I copy paste it, and I
left off the last digit, but the ROA you created that contains the
error propagates in 10 minutes, and then you quickly fix that ROA.
But that version of the ROA takes two hours to propagate, and that
means that for 110 minutes, you're kind of fiddling your thumbs,
hoping the ROA will start propagating, but it's out of your
control. And this is why I think it is critical for this community
to really engineer for optimal performance. We have consistent
performance, predictable performance. ROAs must propagate as fast
as they are created, and I think the Erik synchronization protocol
is a step towards predictable propagation times.
George Michaelson 30:01
So here we are sitting in IETF 125, in Shenzhen, and something
that's come up as a discussion item in the corridors is the idea
of code-first, standards development. And I know from other
conversations that you've been promoting the idea for work to be
taken seriously inside SIDROPS it's important we get to working
implementations beyond conceptual proof. This isn't a paper design
is this Job?
Job Snijders 30:29
I am happy to report we have running code. And it is really cool
to be thinking about ideas and what could work in practice, but it
is much, much more interesting to verify that the ideas do in the
work in practice.
George Michaelson 30:44
And so you now have some measurements using real world data to
demonstrate what it would look like fetching the world of ROA
through intermediaries, using Erik
Job Snijders 30:55
Right, my preliminary measurements suggest that, depending on the
refresh cycle of the validator. So a validator could, say,
synchronize every 15 minutes or every 60 Minutes. Like let's take
those two patterns as an example. And if you synchronize a
validator every 15 minutes using rsync, it is quite costly,
because, as we discussed, with rsync, you receive the entire menu
of everything that's on order, and then you exchange differences.
And if you compare rsync to RRDP on the 15 minute interval, RRDP
is more efficient because there's no giant file list in RRDP, but
if you synchronize every hour, because with RRDP, you're pulling
in the full history of everything that changed
George Michaelson 31:42
And across the surface of this space. Some things have gone
through multiple changes. You're pulling more data than you need,
Job Snijders 31:49
right? So, weirdly enough, if you synchronize every hour, rsync is
more efficient from a bandwidth consumption perspective. And these
results surprised me. I was like, That is counter intuitive!
George Michaelson 32:00
enough, we know a kind of range of interest, 15 minutes and an
hour, two protocols with a weird outcome. How does Erik play in
this space?
Job Snijders 32:08
My initial measurements suggest that Erik synchronization performs
better in both scenarios. And this is really good news, because
for an RP to decide which of the protocols should I be using, and
basing that on the synchronization frequency is a massive
implementation complexity. So having a protocol that, regardless
of the timing interval, performs better is a very, very desirable
thing to have, because it means that whatever schedule you're on,
Erik synchronization performs better.
George Michaelson 32:40
So you mentioned that it's HTTP is a substrate. This makes me
immediately ask, Is there potential for the CDN community to be
able to consider themselves as the intermediaries and provide us
with a huge surface of reliable, short path visibility? Because
one of the dimensions of slowness you haven't mentioned is that if
I'm a validator in Chile and I'm attempting to collect the entire
surface of RPKI, I have to have a reliable path to Indonesia and
to Mongolia and to Europe and North America and my buddies nearby
in Uruguay. I need to be able to get to all of them to complete my
act. If this was in an intermediary, and the CDN had an anycast
surface, I would be fetching at a constant distance cost to that
CDN with a better protocol. Is this a potential?
Job Snijders 33:36
Yeah, yeah. So because we're fetching by hash, not by file name,
not by FQDN. You can connect to anycasted instances and rotate
between the instances. So we've now converted the RPKI
distribution problem into something that is
George Michaelson 33:51
this is now similar to a model like 1.1 dot 1.1
Job Snijders 33:56
and 8.8, dot 8.8, and nine, dot 9.9, and whatever open resolver
you want to use, we have now constructed a system where you can
fetch RPKI data from publicly accessible endpoints that contain
data for multiple RPKI origins. So as you point out like this,
this latency factor from the client to all over the world, the
latency becomes a predictable elements. You can fetch from a close
by mirror, and you can use multiple mirrors concurrently. So if
one instance is a little bit behind or outdated for one reason or
another, it doesn't matter the client recognizes, oh, you contain
old data. Well, I'm going to try the next Erik relay.
George Michaelson 34:40
So if I consider myself in the role of a relay that front facing
service through Erik, I have the potential to use either Erik or
RRDP or rsync in my back, depending on capabilities in the real
publication point, don't I?
Job Snijders 34:56
So an Erik relay merges all the data be it received via rsync over
fee four or IPV6, received through RDP, received through Erik, it
synthesizes a final construction of the data set, presents that to
the clients in the Erik format. But it doesn't have a monopoly on
the data. The client is absolutely free to connect to multiple
relays operated by multiple organizations. So I think this is a
huge improvement in robustness.
George Michaelson 35:28
Oh, it sounds like a very positive outcome. You were presenting
this in SIDROPS this week?
Job Snijders 35:33
Yeah, I presented updates, and one of the fun things to share in
that update is that there is running code. So I have set up a
number of relays. Some of them are, you know, only in Europe. Some
of them are globally distributed. So I created a diverse set of
relays to help test what does the performance of Erik look like
from a global perspective? And Tom Harrison from APNIC did a
wonderful job creating a Erik client implementation. So now we
have all the building blocks to do extensive measurements over the
span of multiple months to with high precision, observe what is
the difference. If there is any difference between Erik, RRDP and
rsync, and my hope is that the Erik testing instances will show to
consume less data, be faster in converging but we'll see. I mean,
George Michaelson 36:27
well, we'll have you back and talk about this another time when
this emerges. Job, that's been absolutely fascinating. We'll put
pointers to this into the blog that goes with the podcast. Thank
you.
Job Snijders 36:38
And if I may, do one more call out, if you're interested in
researching these phenomenas, we are developing the Erik protocol
at this moment. So for students that are interested in this type
of problem space, like, how do you synchronize hundreds and 1000s
of tiny objects to 1000s of Computer
George Michaelson 36:58
There are Research Opportunities
Job Snijders 36:59
Exactly? And we need the help from younger students to progress
this protocol to be something that we want to use the next 10 20,
30 years.
George Michaelson 37:09
Fantastic. That's great Job -thank you.
Job Snijders 37:11
All right, George, until next time,
George Michaelson 37:14
if you've got a story or research to share here on ping, why not
get in contact by email to ping@apnic.net or via the APNIC social
media channels, also remember the measurement@apnic.net mail list
on orbit is there to discuss and share relevant collaborative
opportunities, grants and funding, opportunities, jobs and
graduate placings, or To seek feedback from the community on your
own measurement projects, be sure to check out the APNIC website
for All your resource and community needs until next time you you.
