Duane Wessels 0:00
There is this jump that you're talking about, the 5011 jump, 30
days after it goes from I don't know, very small percentage up to
like maybe 80% or 90 85% in just a day or two. But that remaining
10, 15% takes a while for them to get it, and those could be
resolvers or other DNS software that does not implement RFC 5011
for whatever reason, right? It could be simpler software, and so I
think we realize that you know not everything out there will
implement RFC 5011.
George Michaelson 0:37
You're listening to Ping, a podcast by APNIC discussing all things
related to measuring the Internet. I'm your host George
Michaelson. This time I'm talking to Duane Wessels again. Duane
is from Verisign, and we're talking about completion of the most
recent DNSSEC KSK roll, which started in 2024. Duane and his co-
author Roy Arends from ICANN have published a two-part blog series
at Verisign and ICANN discussing the experiences of the two
DNSSEC key signing key or KSK roll-over. I say the most recent KSK
roll started in 2024, but there's actually a twist to the story
that we discuss in this episode. Sometimes plans don't work out
how you expect, and this time there were two reasons for the role
to happen in 24: the global pandemic and a completely separate
hardware supply chain issue. It's worth understanding a bit of
history here. DNS wasn't designed with security in mind. The
cryptographic system deployed in the DNS is very similar to
website public-private certificates, except there's no
certificate. In the DNS. There are just the two keys, public and
private. A special key pair called the key signing key or KSK is
used to sign the zone signing key or ZSK. The KSK is both used and
changed less frequently than the ZSK. Signing the KSK at the very
top of the DNS is quite a big deal. It establishes what's called
the trust anchor in the public-private keying system. You want to
make sure it's distributed widely and signal change as clearly as
possible. The DNS KSK was first deployed in July 2010. There was
an intent to change it in a reasonable number of years, but for a
number of reasons, this didn't happen until the first change of
the top level key happened in July 2017, seven years was
definitely too long. To be fair, we'd never done something as
significant and visible as this before. Key roll-over aren't fast.
This transaction completed in October 2018. With experience under
our belts, the DNS community aimed to commence the next key
roll-over in 2023, improving on the seven-year wait from the first
time, and this is what Duane and Roy have been blogging about.
Duane, welcome back to Ping. It's great to see you again.
Duane Wessels 3:16
Thank you very much, George. It's very great to be here.
George Michaelson 3:18
You've recently been writing again on the Verisign blog with your
co-author Roy Arends from ICANN. I actually picked it up from
Circle ID, who I think did a republication. You're writing kind of
like a second part story on the key roll in the DNSSEC route. That's
right?
Duane Wessels 3:39
That's right. Yeah. So Roy and I wrote together a blog post
published both by ICANN and Verisign to talk about what to expect
a little bit for the upcoming KSK roll-over, which happens in in
October.
George Michaelson 3:52
We should probably roll the clock backwards a bit and kind of
cover a bit of history here. DNS, one of the oldest protocols in
the global Internet had no baked-in security framework, and there
was a long design exercise to come up with a way of doing public
private keying in order to get signatures over the state of the
DNS. That kind of took two forms. There's the long-lived key, the
KSK, the key signing key, and that signs over much more frequently
used, possibly shorter-lived keys, the zone signing keys, and the
part of the game that we're talking about here. It's the KSK part,
isn't it?
Duane Wessels 4:33
Yeah, it's the KSK part, and for the root zone, it's particularly
interesting, in my opinion, because unlike many organizations that
might deploy DNSSEC; those other organizations may operate or may
do both KSK and ZSK operations. But it's the root zone. ICANN has
the responsibility for the KSK, and Verisign as a root zone
maintainer has responsibility for the ZSK, the zone signing.
George Michaelson 4:58
We have the deployment starting. In mid year 2010, we went too
long, didn't we? I mean, we didn't get to a point of achieving our
first key roll until about seven or eight years later.
Duane Wessels 5:09
Yeah, it was definitely interesting when the root zone was
initially signed, like you said, starting in 2010. I think in most
people's minds, most people who were involved, the idea was like
after five years we'll do a key roll-over, but something happened
in 2014, which kind of made it necessary to postpone that. Which
was the IANA stewardship transition, right?
Duane Wessels 5:29
So that's one reason why that first KSK roll-over didn't really
start until well, it was scheduled for 2017.
George Michaelson 5:35
If we also kind of come to this point, we hadn't really thought
through technology to make a key roll effective and easy to do.
When you're operating in most PKI public key cryptography windows,
you kind of have an expectation. You're going to send people out
into the field with new trust anchors. You're going to do a bunch
of stuff. The DNS, we were looking for automatic mechanisms, and
we had to come up with a standard mechanism for signaling, we're
about to enter this window. So that's 5011? RFC 5011.
Duane Wessels 6:07
Yep, yep.
George Michaelson 6:08
And it imposes quite well defined time windows when you should
expect to see things happen. So in the case of that first key
roll, you actually were doing quite a lot of measurement, weren't
you? You were standing to look at the behavior of the systems and
get a handle on what was going on?
Duane Wessels 6:27
Yeah. So as you said, RFC 5011 is the one that describes how
resolvers should adopt a new trust anchor that gets published in
the DNS. I don't remember the year, unfortunately, that RSC 5011
was published off the top of my head, but something that was sort
of an afterthought, I guess, in my opinion, for operating you know
things like the root zone was there was no way for anyone to know
which trust anchor resolvers had picked up, and so myself and others -
George Michaelson 6:56
yeah, it was the outbound signal from the key authority to say I'm
going to make a change, and the missing bit here is a resolver
being able to say, "Hello, This is the one I'm looking at." We
didn't have that mechanism, did we?
Duane Wessels 7:09
Yeah. So another RFC 8145 was published in 2016 2017.
George Michaelson 7:16
Yeah, 2017.
Duane Wessels 7:18
So that was kind of a late comer to this whole process, but
we did get some data from that got deployed in resolver
software, and we started getting data from resolvers about which
trust anchors they had configured.
George Michaelson 7:31
Yeah, your first blog that you wrote after the first key roll,
reflecting on things, there was a bit of an anomalous signal
popped up in the way people were using the technology.
Duane Wessels 7:43
yeah. So this trust anchor signaling technique is -it's not
perfect. It has some complications, some things that you have to
be aware of when you're looking at the data and thinking about how
it works. One aspect of it is that it's not secure data in itself,
right? So there's nothing in the signal that can tell you that it
was an authentic signal, meaning somebody could inject false data
if they wanted to. That's one problem. Another point that
people raised was, you know, it's very common these days
that homes and offices have many devices operating behind single
IP addresses, and so you get a signal or maybe two signals or
more from a single IP address, and they could be saying
different things. They could be mixed signals, right?
George Michaelson 8:18
Yeah. Contradictory.
Duane Wessels 8:20
They can be contradictory, yeah. And so, in this protocol, there's
no real way to sort those out, and so that's something you have to
just be aware of.
George Michaelson 8:26
We had that first key deployment 2010. We have the roll delayed
because of some procedural aspects of structural governance in
ICANN. We get the roll started in 2017, and we have this defined
timer. And in the modern Internet world, people are routinely
talking about transactional change occurring in microseconds. You
switch between alternate DNSs in microseconds. You fall back to
another database server in microseconds. We do not talk in
microseconds in things like key roll, do we?
Duane Wessels 9:00
No, we don't, and I think the reason for that really is that it's
very common that DNS software, your BIND, your Unbound, those
kind of things, they get updated on pretty long life cycles,
right? Yeah, it can take quite a while for software updates to
percolate out through all the places that they need to go, and
that is one reason why things in the DNS don't happen, as you
said, on microsecond timescales,
George Michaelson 9:23
They don't even happen on day timescales because the initial
signaling window of interest is a 30 day timer to see clear signal
of a change in behavior. But the actual duration of the key roll
spanned out over more than a year, didn't it? That first roll, it
was 15, 16, months?
Duane Wessels 9:43
Well, we have to talk a little bit, I guess, about the different
phases of the key roll-over, right? So they're lettered phases A
through G. The interesting ones were phase D
George Michaelson 9:54
seven. I had to do the count on my fingers to remember the span
here.
Duane Wessels 9:59
So phase D. Was the phase when the new key first appeared in the
DNS, right? Yeah. And then phase E was the phase where the signing
switched from signing with the old key to signing with the new
key. And in the schedule for 2017, there was one calendar quarter
between those phases. So the expectation was the key gets
published in the root zone. 30 days later, the resolvers see it
and do their RFC 5011 dance, and they pick it up and they adopt
it. And then another 60 days later, you actually do the key roll.
And I think in hindsight, that was a little bit of an aggressive
schedule. The current key roll-over is much more slow than that.
George Michaelson 10:37
So we had this first roll. We had phase stages. It takes months to
complete, and we have a window in the in between period where
we're actually sending larger data because we have to have a
crossover period where we're sending in both old and new. And in
some ways, the critical time, the one that we're actually
approaching now in this second roll, is when that first signature
goes away, and what we're left with is the new signing. Right?
It's actually the drop off the other side.
Duane Wessels 11:06
Yeah. The tricky part about that, or any key roll-over, is when you
flip them and you start signing with the new key. What is the
behavior of the population of resolvers out there? You know, did
they get the new key? Are they going to have resolution failures?
George Michaelson 11:17
I really strongly recommend people listening to this podcast, go
and have a look at either the old and the new blog, the first and
second blog, or the second blog has these beautiful timing
diagrams that you collected. Now, in strict sense, you didn't
collect this data in your functional role as the operator of this
technology, did you? This was collected through a different
functional role that Verisign has?
Duane Wessels 11:43
Yeah. So the Trust Anchor signaling data, the way it works is a
resolver sends a signal in the form of a DNS query to a name
server authoritative for the zone that Trust Anchor is for. So in
this case, these queries go to the root zone, and Verisign as root
server operator receives those queries, and that's where we were
able to collect it and analyze it and make these graphs.
George Michaelson 12:05
But that's a kind of separate functional role to being the party
that's contracted from ICANN to perform operations using the key.
We really do have to understand there's a separation.
Duane Wessels 12:15
Yeah, technically those are those are different roles. Verisign is
both the Root Zone Maintainer, which is our role as the ZSK
operator, but we're also a root server operator running the actual
services for the root zone.
George Michaelson 12:34
These charts, and truly, folks, they're worth looking at. It's the
most amazing uptick signal for that 30 day window when 5011 first
kicks in. But you then had a quite long persisting tail. There
seemed to be a population of people that just didn't quite get the
signal.
Duane Wessels 12:42
Yeah, so there are. That's right. There is this jump that you're
talking about, the 5011 jump, 30 days after it goes from I don't
know, very small percentage up to like maybe 80% or 90, 85% in
just a day or two. But that remaining 10, 15% takes a while for
them to get it, and those could be resolvers or other DNS software
that does not implement RFC 5011 for whatever reason, right? Could
be simpler software, And so I think we realize that you know
not everything out there will implement RFC 5011, and there are
other ways that trust anchors could be up.
George Michaelson 13:12
Yeah, and it's kind of interesting when you think about the
community burden that Verisign's taken on providing this facility.
It's very easy for people in my role with no strong responsibility
to kind of look at 80/20 80% 85% and think "yay job done go home"
and you guys are sitting there thinking wow we have 2.5 to 3
billion users and if 15% of the global community is sitting behind
a service that hasn't seen this. That's a really astronomically
large number of people. You can't be so casual, can you?
Duane Wessels 13:46
We do take that pretty seriously. You know, one of the real
challenges we had with that roll-over in 2017 was that this trust
actor signaling technology was really pretty new, and so the
sample sizes were relatively small. You know, we're talking less
than 1000 source addresses per day. We're sending these signals,
and so there was a lot of questions about, you know, is this
representative? How many end users might be impacted by
this? As you said,
George Michaelson 14:12
Yeah. Let's come to the second key roll. Seven years was too long,
so there's kind of a vibe out there in the community. Let's do
this more often. Roll frequently, and the model settled on kind of
three years. Is that right?
Duane Wessels 14:25
Yeah, yeah. IANA did a community consultation, developed this
plan. It was a three-year cadence to the key roll-over every three
years.
George Michaelson 14:32
Yeah. So we get three years up from 2017, 2018. Right. We hit a
world event. Right. I mean, this is not the year that you want to
start undertaking some technology change. COVID happened, and part
of the behavior of this keying stuff is it's like a public
engagement. It's a public performance, a ceremony. It's called the
key ceremony, and it's inclusive of people from all around the
world. Coming in and visibly participating in that keying process,
you have a role in this, Duane, don't you?
Duane Wessels 15:06
Yeah, Verisign has a role. We go there and verify what we call the
key signing request. Yeah, normally everyone was traveling to
these meetings four times a year. and the pandemic really changed that.
George Michaelson 15:19
So schedule gets put back, and then we hit a second event, which
is a little less predictable and understood. Keys they don't just
sit on floppy disks attached to a laptop, right? Keys actually
have to live in secure structural devices called hardware security
modules (HSMs) and the way the key signing for the DNS root works,
you have two facilities, one east coast, one west coast, and they
run identical hardware. And you'd selected a device. The community
had kind of come to that device as the target to run for this
function.
Duane Wessels 15:56
Yeah, in fact, so ICANN as the KSK operator had selected HSMs, as
you said.
George Michaelson 16:02
Yeah,
Duane Wessels 16:02
Verisign also used HSMs for the ZSK in our facility. So yeah, both
parties were using those those HSMs.
George Michaelson 16:11
HSMs they have this set of levels you can operate in, and you
really do want to know the implications of the level you run in.
And I believe the KSK operates in FIPS, so Federal Information
Processing Standard, FIPS 120 Level 2, and it's a mode which
allows you to take a key and send it to another of exactly the
same generation using special keys to protect it in transit
between the two machines, and that's how you can have East Coast
and West Coast both operating. But you really cannot get those
keys out and into a new machine, can you? That's not on the table.
Duane Wessels 16:50
Yeah, same vendor, like you said, you can do key export and
import, but between vendors, it's not supported, and it may be out
of the standard of.
George Michaelson 16:57
A second thing that I've personally experienced operating HSMs is
that the company that operates the HSM has to be in a long-term
relationship with you because you have to have some certainty
about spare parts and also about hardware and software
generations. And we experienced our vendor coming to us and saying
we can't commit that the unit you currently have will be supported
if you buy our current technology. You've hung on to it too long,
so the idea is you really have to keep in lockstep through the
generations that you're running with. And I believe the signal
that came was the hardware technology you guys have got to operate
this system is not going to be available.
Duane Wessels 17:39
Yeah, it was a little bit surprising because ICANN and Verisign
had just restarted the KSK roll-over process. The key was
generated in 2023. We called it KSK 2023. It was generated on
those HSMs, and then you know just a month or so after that key
got generated, there was an announcement from that vendor that
they were going to make those end of life. They were not going to
support them. I think past 2025 was was the date they gave.
George Michaelson 18:04
So 2025 is inside the foreseeable lifetime of the keys that you
were generating. I could handle someone saying a year after
end of life we won't be there. You could kind of go "Fine. We'll
keep on this course", but with the hardware being end of life
inside the life of the key, you really had to reconsider, didn't
you?
Duane Wessels 18:22
Yeah. So at that point, there were no great options, but
basically, that KSK 2023 had to be abandoned, and both ICANN and
Verisign had to find new HSM vendors to bring in test and
evaluate and all that stuff.
George Michaelson 18:38
And these are pretty heavy weight tests too. My experience
of going to the keying ceremony is that there's absolutely no
transactional moment in that ceremony where you guys haven't sat
outside and tested it in the hardware you have available before it
actually goes live on the day. Everything has to be seen to work.
Yeah, we come to the second key roll. It's 2023, and we hit this
problem where the unit that you're using reaches its end of life,
and because of that, you have to start looking at the duration of
the keys you're likely to be creating. And you had this crop up.
Duane Wessels 19:14
Yeah, you're right. So in 2023, as you said, we had just sort of
resumed the process. Keys were generated on those existing HSMs,
and it wasn't more than a month or two after that where the vendor
for those HSMs announced that not only would the model be end of
life, but they were not going to support that model in the future.
I think they give a deadline of 2025 or so
George Michaelson 19:36
... Inside the lifetime of the keys that were being created. So -
Duane Wessels 19:40
Yes, definitely,
George Michaelson 19:41
Yeah, and you basically had to reset and start again, didn't you?
With ICANN, you guys had to go back into tender process, identify
new vendor source equipment, and then start the whole thing from
scratch because you can't take the key out of one manufacturer and
bring it into another.
Duane Wessels 19:58
That's right. So both ICANN and Verisign. We're using the same HSM
vendor for the root zone. We had to identify new vendors, new
HSMs, and just said do all the feature testing, evaluations,
and whatnot. And so that basically delayed the roll-over
for another year.
George Michaelson 20:14
Yeah, we actually are already in the time cycle, the window when
we're performing this key role. It started in 2024, didn't it?
With publication of the key visibly into the zone.
Duane Wessels 20:26
Yep, the keys were generated in 2024. You're testing my memory
about publication of the zone. It may have been in 2025,
George Michaelson 20:34
And we've arrived at the back end of those seven lettered
sequences, October of this year, we're going to see dropping of
the old key. The 2017 key is going to be removed from the signing
set.
Duane Wessels 20:50
Yes, on October 11th, the root zone will have a DNS key RRSET that
is signed by the new KSK, and the old KSK will still be post
published for a while, but it won't be used for signing.
George Michaelson 21:03
This thing that I really loved in your second blog, the piece that
you and Roy from ICANN Roy Arends wrote, is that you've done
exactly the same timing diagram, haven't you? You ran the same
charting method.
Duane Wessels 21:16
Yeah. So we've been collecting this data all this time, and every
now and then taking a look at what the trust anchor signal data
says for the blog post. We generated a graph that lines up the
publication dates of the two keys: the 2017 key and the 2024.
George Michaelson 21:30
It's like a giant time signal uptick that you can align the two
data sets on.
Duane Wessels 21:36
Almost like a step function.
George Michaelson 21:37
Yeah. So when you did the first round, you had that 15, 10, 15
percent gap at the top mark that hadn't quite got with the new
program this time round. The signal is quite a bit better, isn't
it?
Duane Wessels 21:49
Well, for one thing, we have a lot more sources of the signal now.
We have something like 160,000, which is really good. But what's
interesting to me in that graph is that, well, in 2017, there was
this sort of anomalous -
George Michaelson 22:01
Oh yeah, there was a weird dip in the middle of it.
Duane Wessels 22:04
Dip in the data, which I have, to be honest, I've never been able
to fully explain. But if you sort of put that aside,
otherwise the two graphs line up very nicely. There's this huge
step function jump, and then after that, there's kind of this
slower increase, ticking up and up a little bit day by day by day,
eventually getting up into the range where you know now we're
at like 95% or so of sources have the new key
George Michaelson 22:25
95 that's a lot better signal than last time round yeah yeah and
in October the final transitional step happens and you come back
to a single signing outcome using the new key and we're looking
at about a three-year window before we enter this cycle again.
But next time round is going to be a bit different,
Duane Wessels 22:49
But to be clear, not three years until we enter the cycle,
because actually we enter the cycle right away. It's three years
until the next roll. Both Verisign & ICANN teams have a lot
of work to do almost immediately after this roll-over, to get ready
for the next one, which, as you said, is going to be an algorithm
transition.
George Michaelson 23:06
Key roll is when you say, using this cryptographic system we've
got, I'm going to use a new key. Algorithm roll is when you say
I'm going to use a new system to design keys and to do the
encoding and decoding. What's actually going to be happening in
this algorithm change? We're going from RSA to ECDSA. Is that it?
Duane Wessels 23:30
That's the plan to go from RSA to ECDSA. Currently, for the
root zone, both the KSK and the ZSK are RSA 2048 bit keys, and
then once we're past the algorithm transition, so they will
both be ECDSA keys. But due to the way algorithm roll-over work,
in the middle you have this period where you're double signing.
Basically, you have to sign with both algorithms.
George Michaelson 23:54
Yeah, so we're going to have slightly different behavior in this
next cycle of key change because we're going to have larger
packets happening, and we're also going to be seeing slightly
different behavior with this new key algorithm. I noticed in the
recent blog that both you and Roy have been very careful to give
people a clear signal if you want to know from looking in your
local configuration that you have the right key. Here is how you
can see the key. The key actually carries like an identity number,
doesn't it? When it's sent to you.
Duane Wessels 24:29
In DNSSEC, keys have tags. We say key tags. It's a 16 bit
identifier, a number between zero and 65,000, usually in decimal,
sometimes hexadecimal. But you can find that key tag in the files
that the resolvers read and write on their system.
George Michaelson 24:45
So, if you're worried or are interested, and you'd like to
understand your own resolver, the blog actually helps you where to
find the file and where to find the key tag and what key tags you
should be looking for. And when we come to the algorithm. Roll,
you'll be able to do the same thing as well.
Duane Wessels 25:02
Yes, we should be able to do the same thing. Yeah, it should be a
very interesting project.
George Michaelson 25:06
It's going to involve slightly larger key ceremonies as well. I
think I'm probably going to be attending two ceremonies next year
instead of one because ICANN have said they'd like to get all of
the holders of fragments of the securing technology in the room to
perform the transition into the new algorithm. In some ways, this
new key signing has been a slightly more relaxed activity.
Duane Wessels 25:31
I attend the key signing ceremonies as well, as you said, and I
find them really fascinating. There's a lot that goes on more than
just even signing keys. I mean, there's all this maintenance
of HSMs, acceptance testing? as you said, there's these recovery
key shareholders. Those shares should be exercised every now
and then to make sure that they still work. and so what
originally seemed like when the root was first signed, the
ceremonies were I don't know a few hours long. They've sort of
grown to longer things that even take sometimes more than one
day. You know, they'll split the activities over a day or two.
George Michaelson 26:05
Yeah, an an interesting quirk is that when you've finished with an
HSM and with a key, you're kind of sitting there holding a a box,
a small pizza box, and you look at it. What are you going to do
with this machine? It's really not a good idea to just let it go
out into the world. So one of the things we're going to be doing
in this next coming ceremony is taking those older HSMs and very
carefully breaking them.
Duane Wessels 26:32
Yeah, taking them apart. They're designed so that when you take
them apart, they should stop working. And so that's something
that we can actually test.
George Michaelson 26:39
Yeah. So we're going to come out the other side of this with some
physical components that we have to deal with, but we've
maintained a kind of zone of security around all of the trust of
these private keys.
Duane Wessels 26:51
Well, I'm looking forward to seeing you at one of the upcoming
ceremonies.
George Michaelson 26:54
Yeah, thank you, Duane. This has been really fascinating. We'll
put links to the blogs on the Verisign website, in the article
that goes up with this podcast.
Duane Wessels 27:04
All right, thank you.
George Michaelson 27:05
Thank you.
Duane Wessels 27:06
All right, see you next time, George.
George Michaelson 27:08
If you've got a story or research to share here on Ping, why not
get in contact by email to ping@apnic.net or via the APNIC social
media channels? Also, remember the measurement@apnic.net mail.
list on orbit is there to discuss and share relevant collaborative
opportunities, grants and funding opportunities, jobs and graduate
placings, or to seek feedback from the community on your own
measurement projects. Be sure to check out the APNIC website for
all your resource and community needs. Until next time.
There is this jump that you're talking about, the 5011 jump, 30
days after it goes from I don't know, very small percentage up to
like maybe 80% or 90 85% in just a day or two. But that remaining
10, 15% takes a while for them to get it, and those could be
resolvers or other DNS software that does not implement RFC 5011
for whatever reason, right? It could be simpler software, and so I
think we realize that you know not everything out there will
implement RFC 5011.
George Michaelson 0:37
You're listening to Ping, a podcast by APNIC discussing all things
related to measuring the Internet. I'm your host George
Michaelson. This time I'm talking to Duane Wessels again. Duane
is from Verisign, and we're talking about completion of the most
recent DNSSEC KSK roll, which started in 2024. Duane and his co-
author Roy Arends from ICANN have published a two-part blog series
at Verisign and ICANN discussing the experiences of the two
DNSSEC key signing key or KSK roll-over. I say the most recent KSK
roll started in 2024, but there's actually a twist to the story
that we discuss in this episode. Sometimes plans don't work out
how you expect, and this time there were two reasons for the role
to happen in 24: the global pandemic and a completely separate
hardware supply chain issue. It's worth understanding a bit of
history here. DNS wasn't designed with security in mind. The
cryptographic system deployed in the DNS is very similar to
website public-private certificates, except there's no
certificate. In the DNS. There are just the two keys, public and
private. A special key pair called the key signing key or KSK is
used to sign the zone signing key or ZSK. The KSK is both used and
changed less frequently than the ZSK. Signing the KSK at the very
top of the DNS is quite a big deal. It establishes what's called
the trust anchor in the public-private keying system. You want to
make sure it's distributed widely and signal change as clearly as
possible. The DNS KSK was first deployed in July 2010. There was
an intent to change it in a reasonable number of years, but for a
number of reasons, this didn't happen until the first change of
the top level key happened in July 2017, seven years was
definitely too long. To be fair, we'd never done something as
significant and visible as this before. Key roll-over aren't fast.
This transaction completed in October 2018. With experience under
our belts, the DNS community aimed to commence the next key
roll-over in 2023, improving on the seven-year wait from the first
time, and this is what Duane and Roy have been blogging about.
Duane, welcome back to Ping. It's great to see you again.
Duane Wessels 3:16
Thank you very much, George. It's very great to be here.
George Michaelson 3:18
You've recently been writing again on the Verisign blog with your
co-author Roy Arends from ICANN. I actually picked it up from
Circle ID, who I think did a republication. You're writing kind of
like a second part story on the key roll in the DNSSEC route. That's
right?
Duane Wessels 3:39
That's right. Yeah. So Roy and I wrote together a blog post
published both by ICANN and Verisign to talk about what to expect
a little bit for the upcoming KSK roll-over, which happens in in
October.
George Michaelson 3:52
We should probably roll the clock backwards a bit and kind of
cover a bit of history here. DNS, one of the oldest protocols in
the global Internet had no baked-in security framework, and there
was a long design exercise to come up with a way of doing public
private keying in order to get signatures over the state of the
DNS. That kind of took two forms. There's the long-lived key, the
KSK, the key signing key, and that signs over much more frequently
used, possibly shorter-lived keys, the zone signing keys, and the
part of the game that we're talking about here. It's the KSK part,
isn't it?
Duane Wessels 4:33
Yeah, it's the KSK part, and for the root zone, it's particularly
interesting, in my opinion, because unlike many organizations that
might deploy DNSSEC; those other organizations may operate or may
do both KSK and ZSK operations. But it's the root zone. ICANN has
the responsibility for the KSK, and Verisign as a root zone
maintainer has responsibility for the ZSK, the zone signing.
George Michaelson 4:58
We have the deployment starting. In mid year 2010, we went too
long, didn't we? I mean, we didn't get to a point of achieving our
first key roll until about seven or eight years later.
Duane Wessels 5:09
Yeah, it was definitely interesting when the root zone was
initially signed, like you said, starting in 2010. I think in most
people's minds, most people who were involved, the idea was like
after five years we'll do a key roll-over, but something happened
in 2014, which kind of made it necessary to postpone that. Which
was the IANA stewardship transition, right?
Duane Wessels 5:29
So that's one reason why that first KSK roll-over didn't really
start until well, it was scheduled for 2017.
George Michaelson 5:35
If we also kind of come to this point, we hadn't really thought
through technology to make a key roll effective and easy to do.
When you're operating in most PKI public key cryptography windows,
you kind of have an expectation. You're going to send people out
into the field with new trust anchors. You're going to do a bunch
of stuff. The DNS, we were looking for automatic mechanisms, and
we had to come up with a standard mechanism for signaling, we're
about to enter this window. So that's 5011? RFC 5011.
Duane Wessels 6:07
Yep, yep.
George Michaelson 6:08
And it imposes quite well defined time windows when you should
expect to see things happen. So in the case of that first key
roll, you actually were doing quite a lot of measurement, weren't
you? You were standing to look at the behavior of the systems and
get a handle on what was going on?
Duane Wessels 6:27
Yeah. So as you said, RFC 5011 is the one that describes how
resolvers should adopt a new trust anchor that gets published in
the DNS. I don't remember the year, unfortunately, that RSC 5011
was published off the top of my head, but something that was sort
of an afterthought, I guess, in my opinion, for operating you know
things like the root zone was there was no way for anyone to know
which trust anchor resolvers had picked up, and so myself and others -
George Michaelson 6:56
yeah, it was the outbound signal from the key authority to say I'm
going to make a change, and the missing bit here is a resolver
being able to say, "Hello, This is the one I'm looking at." We
didn't have that mechanism, did we?
Duane Wessels 7:09
Yeah. So another RFC 8145 was published in 2016 2017.
George Michaelson 7:16
Yeah, 2017.
Duane Wessels 7:18
So that was kind of a late comer to this whole process, but
we did get some data from that got deployed in resolver
software, and we started getting data from resolvers about which
trust anchors they had configured.
George Michaelson 7:31
Yeah, your first blog that you wrote after the first key roll,
reflecting on things, there was a bit of an anomalous signal
popped up in the way people were using the technology.
Duane Wessels 7:43
yeah. So this trust anchor signaling technique is -it's not
perfect. It has some complications, some things that you have to
be aware of when you're looking at the data and thinking about how
it works. One aspect of it is that it's not secure data in itself,
right? So there's nothing in the signal that can tell you that it
was an authentic signal, meaning somebody could inject false data
if they wanted to. That's one problem. Another point that
people raised was, you know, it's very common these days
that homes and offices have many devices operating behind single
IP addresses, and so you get a signal or maybe two signals or
more from a single IP address, and they could be saying
different things. They could be mixed signals, right?
George Michaelson 8:18
Yeah. Contradictory.
Duane Wessels 8:20
They can be contradictory, yeah. And so, in this protocol, there's
no real way to sort those out, and so that's something you have to
just be aware of.
George Michaelson 8:26
We had that first key deployment 2010. We have the roll delayed
because of some procedural aspects of structural governance in
ICANN. We get the roll started in 2017, and we have this defined
timer. And in the modern Internet world, people are routinely
talking about transactional change occurring in microseconds. You
switch between alternate DNSs in microseconds. You fall back to
another database server in microseconds. We do not talk in
microseconds in things like key roll, do we?
Duane Wessels 9:00
No, we don't, and I think the reason for that really is that it's
very common that DNS software, your BIND, your Unbound, those
kind of things, they get updated on pretty long life cycles,
right? Yeah, it can take quite a while for software updates to
percolate out through all the places that they need to go, and
that is one reason why things in the DNS don't happen, as you
said, on microsecond timescales,
George Michaelson 9:23
They don't even happen on day timescales because the initial
signaling window of interest is a 30 day timer to see clear signal
of a change in behavior. But the actual duration of the key roll
spanned out over more than a year, didn't it? That first roll, it
was 15, 16, months?
Duane Wessels 9:43
Well, we have to talk a little bit, I guess, about the different
phases of the key roll-over, right? So they're lettered phases A
through G. The interesting ones were phase D
George Michaelson 9:54
seven. I had to do the count on my fingers to remember the span
here.
Duane Wessels 9:59
So phase D. Was the phase when the new key first appeared in the
DNS, right? Yeah. And then phase E was the phase where the signing
switched from signing with the old key to signing with the new
key. And in the schedule for 2017, there was one calendar quarter
between those phases. So the expectation was the key gets
published in the root zone. 30 days later, the resolvers see it
and do their RFC 5011 dance, and they pick it up and they adopt
it. And then another 60 days later, you actually do the key roll.
And I think in hindsight, that was a little bit of an aggressive
schedule. The current key roll-over is much more slow than that.
George Michaelson 10:37
So we had this first roll. We had phase stages. It takes months to
complete, and we have a window in the in between period where
we're actually sending larger data because we have to have a
crossover period where we're sending in both old and new. And in
some ways, the critical time, the one that we're actually
approaching now in this second roll, is when that first signature
goes away, and what we're left with is the new signing. Right?
It's actually the drop off the other side.
Duane Wessels 11:06
Yeah. The tricky part about that, or any key roll-over, is when you
flip them and you start signing with the new key. What is the
behavior of the population of resolvers out there? You know, did
they get the new key? Are they going to have resolution failures?
George Michaelson 11:17
I really strongly recommend people listening to this podcast, go
and have a look at either the old and the new blog, the first and
second blog, or the second blog has these beautiful timing
diagrams that you collected. Now, in strict sense, you didn't
collect this data in your functional role as the operator of this
technology, did you? This was collected through a different
functional role that Verisign has?
Duane Wessels 11:43
Yeah. So the Trust Anchor signaling data, the way it works is a
resolver sends a signal in the form of a DNS query to a name
server authoritative for the zone that Trust Anchor is for. So in
this case, these queries go to the root zone, and Verisign as root
server operator receives those queries, and that's where we were
able to collect it and analyze it and make these graphs.
George Michaelson 12:05
But that's a kind of separate functional role to being the party
that's contracted from ICANN to perform operations using the key.
We really do have to understand there's a separation.
Duane Wessels 12:15
Yeah, technically those are those are different roles. Verisign is
both the Root Zone Maintainer, which is our role as the ZSK
operator, but we're also a root server operator running the actual
services for the root zone.
George Michaelson 12:34
These charts, and truly, folks, they're worth looking at. It's the
most amazing uptick signal for that 30 day window when 5011 first
kicks in. But you then had a quite long persisting tail. There
seemed to be a population of people that just didn't quite get the
signal.
Duane Wessels 12:42
Yeah, so there are. That's right. There is this jump that you're
talking about, the 5011 jump, 30 days after it goes from I don't
know, very small percentage up to like maybe 80% or 90, 85% in
just a day or two. But that remaining 10, 15% takes a while for
them to get it, and those could be resolvers or other DNS software
that does not implement RFC 5011 for whatever reason, right? Could
be simpler software, And so I think we realize that you know
not everything out there will implement RFC 5011, and there are
other ways that trust anchors could be up.
George Michaelson 13:12
Yeah, and it's kind of interesting when you think about the
community burden that Verisign's taken on providing this facility.
It's very easy for people in my role with no strong responsibility
to kind of look at 80/20 80% 85% and think "yay job done go home"
and you guys are sitting there thinking wow we have 2.5 to 3
billion users and if 15% of the global community is sitting behind
a service that hasn't seen this. That's a really astronomically
large number of people. You can't be so casual, can you?
Duane Wessels 13:46
We do take that pretty seriously. You know, one of the real
challenges we had with that roll-over in 2017 was that this trust
actor signaling technology was really pretty new, and so the
sample sizes were relatively small. You know, we're talking less
than 1000 source addresses per day. We're sending these signals,
and so there was a lot of questions about, you know, is this
representative? How many end users might be impacted by
this? As you said,
George Michaelson 14:12
Yeah. Let's come to the second key roll. Seven years was too long,
so there's kind of a vibe out there in the community. Let's do
this more often. Roll frequently, and the model settled on kind of
three years. Is that right?
Duane Wessels 14:25
Yeah, yeah. IANA did a community consultation, developed this
plan. It was a three-year cadence to the key roll-over every three
years.
George Michaelson 14:32
Yeah. So we get three years up from 2017, 2018. Right. We hit a
world event. Right. I mean, this is not the year that you want to
start undertaking some technology change. COVID happened, and part
of the behavior of this keying stuff is it's like a public
engagement. It's a public performance, a ceremony. It's called the
key ceremony, and it's inclusive of people from all around the
world. Coming in and visibly participating in that keying process,
you have a role in this, Duane, don't you?
Duane Wessels 15:06
Yeah, Verisign has a role. We go there and verify what we call the
key signing request. Yeah, normally everyone was traveling to
these meetings four times a year. and the pandemic really changed that.
George Michaelson 15:19
So schedule gets put back, and then we hit a second event, which
is a little less predictable and understood. Keys they don't just
sit on floppy disks attached to a laptop, right? Keys actually
have to live in secure structural devices called hardware security
modules (HSMs) and the way the key signing for the DNS root works,
you have two facilities, one east coast, one west coast, and they
run identical hardware. And you'd selected a device. The community
had kind of come to that device as the target to run for this
function.
Duane Wessels 15:56
Yeah, in fact, so ICANN as the KSK operator had selected HSMs, as
you said.
George Michaelson 16:02
Yeah,
Duane Wessels 16:02
Verisign also used HSMs for the ZSK in our facility. So yeah, both
parties were using those those HSMs.
George Michaelson 16:11
HSMs they have this set of levels you can operate in, and you
really do want to know the implications of the level you run in.
And I believe the KSK operates in FIPS, so Federal Information
Processing Standard, FIPS 120 Level 2, and it's a mode which
allows you to take a key and send it to another of exactly the
same generation using special keys to protect it in transit
between the two machines, and that's how you can have East Coast
and West Coast both operating. But you really cannot get those
keys out and into a new machine, can you? That's not on the table.
Duane Wessels 16:50
Yeah, same vendor, like you said, you can do key export and
import, but between vendors, it's not supported, and it may be out
of the standard of.
George Michaelson 16:57
A second thing that I've personally experienced operating HSMs is
that the company that operates the HSM has to be in a long-term
relationship with you because you have to have some certainty
about spare parts and also about hardware and software
generations. And we experienced our vendor coming to us and saying
we can't commit that the unit you currently have will be supported
if you buy our current technology. You've hung on to it too long,
so the idea is you really have to keep in lockstep through the
generations that you're running with. And I believe the signal
that came was the hardware technology you guys have got to operate
this system is not going to be available.
Duane Wessels 17:39
Yeah, it was a little bit surprising because ICANN and Verisign
had just restarted the KSK roll-over process. The key was
generated in 2023. We called it KSK 2023. It was generated on
those HSMs, and then you know just a month or so after that key
got generated, there was an announcement from that vendor that
they were going to make those end of life. They were not going to
support them. I think past 2025 was was the date they gave.
George Michaelson 18:04
So 2025 is inside the foreseeable lifetime of the keys that you
were generating. I could handle someone saying a year after
end of life we won't be there. You could kind of go "Fine. We'll
keep on this course", but with the hardware being end of life
inside the life of the key, you really had to reconsider, didn't
you?
Duane Wessels 18:22
Yeah. So at that point, there were no great options, but
basically, that KSK 2023 had to be abandoned, and both ICANN and
Verisign had to find new HSM vendors to bring in test and
evaluate and all that stuff.
George Michaelson 18:38
And these are pretty heavy weight tests too. My experience
of going to the keying ceremony is that there's absolutely no
transactional moment in that ceremony where you guys haven't sat
outside and tested it in the hardware you have available before it
actually goes live on the day. Everything has to be seen to work.
Yeah, we come to the second key roll. It's 2023, and we hit this
problem where the unit that you're using reaches its end of life,
and because of that, you have to start looking at the duration of
the keys you're likely to be creating. And you had this crop up.
Duane Wessels 19:14
Yeah, you're right. So in 2023, as you said, we had just sort of
resumed the process. Keys were generated on those existing HSMs,
and it wasn't more than a month or two after that where the vendor
for those HSMs announced that not only would the model be end of
life, but they were not going to support that model in the future.
I think they give a deadline of 2025 or so
George Michaelson 19:36
... Inside the lifetime of the keys that were being created. So -
Duane Wessels 19:40
Yes, definitely,
George Michaelson 19:41
Yeah, and you basically had to reset and start again, didn't you?
With ICANN, you guys had to go back into tender process, identify
new vendor source equipment, and then start the whole thing from
scratch because you can't take the key out of one manufacturer and
bring it into another.
Duane Wessels 19:58
That's right. So both ICANN and Verisign. We're using the same HSM
vendor for the root zone. We had to identify new vendors, new
HSMs, and just said do all the feature testing, evaluations,
and whatnot. And so that basically delayed the roll-over
for another year.
George Michaelson 20:14
Yeah, we actually are already in the time cycle, the window when
we're performing this key role. It started in 2024, didn't it?
With publication of the key visibly into the zone.
Duane Wessels 20:26
Yep, the keys were generated in 2024. You're testing my memory
about publication of the zone. It may have been in 2025,
George Michaelson 20:34
And we've arrived at the back end of those seven lettered
sequences, October of this year, we're going to see dropping of
the old key. The 2017 key is going to be removed from the signing
set.
Duane Wessels 20:50
Yes, on October 11th, the root zone will have a DNS key RRSET that
is signed by the new KSK, and the old KSK will still be post
published for a while, but it won't be used for signing.
George Michaelson 21:03
This thing that I really loved in your second blog, the piece that
you and Roy from ICANN Roy Arends wrote, is that you've done
exactly the same timing diagram, haven't you? You ran the same
charting method.
Duane Wessels 21:16
Yeah. So we've been collecting this data all this time, and every
now and then taking a look at what the trust anchor signal data
says for the blog post. We generated a graph that lines up the
publication dates of the two keys: the 2017 key and the 2024.
George Michaelson 21:30
It's like a giant time signal uptick that you can align the two
data sets on.
Duane Wessels 21:36
Almost like a step function.
George Michaelson 21:37
Yeah. So when you did the first round, you had that 15, 10, 15
percent gap at the top mark that hadn't quite got with the new
program this time round. The signal is quite a bit better, isn't
it?
Duane Wessels 21:49
Well, for one thing, we have a lot more sources of the signal now.
We have something like 160,000, which is really good. But what's
interesting to me in that graph is that, well, in 2017, there was
this sort of anomalous -
George Michaelson 22:01
Oh yeah, there was a weird dip in the middle of it.
Duane Wessels 22:04
Dip in the data, which I have, to be honest, I've never been able
to fully explain. But if you sort of put that aside,
otherwise the two graphs line up very nicely. There's this huge
step function jump, and then after that, there's kind of this
slower increase, ticking up and up a little bit day by day by day,
eventually getting up into the range where you know now we're
at like 95% or so of sources have the new key
George Michaelson 22:25
95 that's a lot better signal than last time round yeah yeah and
in October the final transitional step happens and you come back
to a single signing outcome using the new key and we're looking
at about a three-year window before we enter this cycle again.
But next time round is going to be a bit different,
Duane Wessels 22:49
But to be clear, not three years until we enter the cycle,
because actually we enter the cycle right away. It's three years
until the next roll. Both Verisign & ICANN teams have a lot
of work to do almost immediately after this roll-over, to get ready
for the next one, which, as you said, is going to be an algorithm
transition.
George Michaelson 23:06
Key roll is when you say, using this cryptographic system we've
got, I'm going to use a new key. Algorithm roll is when you say
I'm going to use a new system to design keys and to do the
encoding and decoding. What's actually going to be happening in
this algorithm change? We're going from RSA to ECDSA. Is that it?
Duane Wessels 23:30
That's the plan to go from RSA to ECDSA. Currently, for the
root zone, both the KSK and the ZSK are RSA 2048 bit keys, and
then once we're past the algorithm transition, so they will
both be ECDSA keys. But due to the way algorithm roll-over work,
in the middle you have this period where you're double signing.
Basically, you have to sign with both algorithms.
George Michaelson 23:54
Yeah, so we're going to have slightly different behavior in this
next cycle of key change because we're going to have larger
packets happening, and we're also going to be seeing slightly
different behavior with this new key algorithm. I noticed in the
recent blog that both you and Roy have been very careful to give
people a clear signal if you want to know from looking in your
local configuration that you have the right key. Here is how you
can see the key. The key actually carries like an identity number,
doesn't it? When it's sent to you.
Duane Wessels 24:29
In DNSSEC, keys have tags. We say key tags. It's a 16 bit
identifier, a number between zero and 65,000, usually in decimal,
sometimes hexadecimal. But you can find that key tag in the files
that the resolvers read and write on their system.
George Michaelson 24:45
So, if you're worried or are interested, and you'd like to
understand your own resolver, the blog actually helps you where to
find the file and where to find the key tag and what key tags you
should be looking for. And when we come to the algorithm. Roll,
you'll be able to do the same thing as well.
Duane Wessels 25:02
Yes, we should be able to do the same thing. Yeah, it should be a
very interesting project.
George Michaelson 25:06
It's going to involve slightly larger key ceremonies as well. I
think I'm probably going to be attending two ceremonies next year
instead of one because ICANN have said they'd like to get all of
the holders of fragments of the securing technology in the room to
perform the transition into the new algorithm. In some ways, this
new key signing has been a slightly more relaxed activity.
Duane Wessels 25:31
I attend the key signing ceremonies as well, as you said, and I
find them really fascinating. There's a lot that goes on more than
just even signing keys. I mean, there's all this maintenance
of HSMs, acceptance testing? as you said, there's these recovery
key shareholders. Those shares should be exercised every now
and then to make sure that they still work. and so what
originally seemed like when the root was first signed, the
ceremonies were I don't know a few hours long. They've sort of
grown to longer things that even take sometimes more than one
day. You know, they'll split the activities over a day or two.
George Michaelson 26:05
Yeah, an an interesting quirk is that when you've finished with an
HSM and with a key, you're kind of sitting there holding a a box,
a small pizza box, and you look at it. What are you going to do
with this machine? It's really not a good idea to just let it go
out into the world. So one of the things we're going to be doing
in this next coming ceremony is taking those older HSMs and very
carefully breaking them.
Duane Wessels 26:32
Yeah, taking them apart. They're designed so that when you take
them apart, they should stop working. And so that's something
that we can actually test.
George Michaelson 26:39
Yeah. So we're going to come out the other side of this with some
physical components that we have to deal with, but we've
maintained a kind of zone of security around all of the trust of
these private keys.
Duane Wessels 26:51
Well, I'm looking forward to seeing you at one of the upcoming
ceremonies.
George Michaelson 26:54
Yeah, thank you, Duane. This has been really fascinating. We'll
put links to the blogs on the Verisign website, in the article
that goes up with this podcast.
Duane Wessels 27:04
All right, thank you.
George Michaelson 27:05
Thank you.
Duane Wessels 27:06
All right, see you next time, George.
George Michaelson 27:08
If you've got a story or research to share here on Ping, why not
get in contact by email to ping@apnic.net or via the APNIC social
media channels? Also, remember the measurement@apnic.net mail.
list on orbit is there to discuss and share relevant collaborative
opportunities, grants and funding opportunities, jobs and graduate
placings, or to seek feedback from the community on your own
measurement projects. Be sure to check out the APNIC website for
all your resource and community needs. Until next time.
