Geoff Huston 0:00
It's kind of okay in June, because most of us don't have hangovers
on the first of July to cope with the mess. On the other hand,
you'd think it's the worst possible time you can possibly add a
leap second is the last second of December when most of the
Earth's population is heavily imbibing and celebrating and not
worrying at all about whether their computers have just gone into
a mess because adding a second has, in the past, proved to be
remarkably disruptive for computers, because things that go sleep
one second, that means I'm in a new something or other. And the
answer is, well, sometimes you're in the same minute, and it's not
predictable. It's not every June or every December. It's only when
the Earth has slowed down. So, for example, between 2000 and 2006
there were no leap seconds, no extra seconds.
George Michaelson 1:00
You're listening to ping, a podcast by APNIC discussing all things
related to measuring the Internet. I'm your host, George
Michaelson, this time, I'm talking to Geoff Huston from APNIC labs
again in his regular monthly spot on ping, Geoff has been looking
at NTP again. NTP is the network time protocol. It's one of the
older systems we depend on, designed and implemented by Dave
Mills, who died in 2024. Dave had been working on time
synchronization from the mid 1970s Geoff has been thinking about
NTP, moves to secure NTP and our increasing dependency on the
underlying concepts of a coordinated sense of time. The dependency
in the modern world on highly synchronized clocks cannot be
overstated. It creeps into every sector of daily life, from
aircraft and space navigation to finance systems and event
scheduling of all kinds, but our model of time is based on the
rotation of the Earth and the length of the second. And
unfortunately, while we now define the length of the second to
astonishingly accurate levels, the rotation of the Earth isn't as
stable as we'd like. Our model of time has to make some
adjustments, and our model of time has been coded over the years
to varying degrees of a start date known as an epoch, and how we
represent time inside the machines. It's all coming a bit unstuck.
Geoff, welcome back to ping. What shall we talk about this time?
Geoff Huston 2:35
Well, I could start with the universe, but I'll scale it back down
a bit. I want to talk about the rotating earth. [George: what!!!]
I really do. It's kind of celestial mechanics, which is, you know,
absolutely fascinated man time, ever since they started looking at
the stars and figured out every night the stars look about the
same as they did the night before, which is amazing, and they
started to kind of out the period when it looked the same to the
next part when it looked the same. And this dates back a long way.
The ancient Babylonians had their best shot at this using a really
weird base 60 counting system. Why 60? Because they figured out
that it was one of the most divisible numbers of all you know,
1234, they all divided evenly into 60. So guess what, 60 seconds
in a minute? Guess what 60 minutes in an hour? Now it's going to
say, Guess what, 60 hours? Some reason, the day was 24 hours long.
But we have this sort of definition coming even then that the
regularly rotates a period of 86,400 seconds. And if you're
looking up at the stars and you count 86,400 of these, the sky
will look exactly the same as it did last time you're gone around
the world, the Earth once or the Earth has gone once around on
this axle. Cool, hey, but not quite,
George Michaelson 4:03
Ah, there's always a but, isn't there?
Geoff Huston 4:06
Well, what's the definition of a second thing? Well, easy, 6400 of
the time it took to rotate about its own axis. That's second
George Michaelson 4:16
short episode of ping. We're done. See you next time.
Geoff Huston 4:19
So you dial up your ancestor through the time machine and go, How
long is the second dude? Well, he says same thing. 180, 6,400th,
of the time it takes to rotate about its own axis. You go, No,
that's not right, because if I use my seconds and count your time,
that's not actually true. What do you mean?
George Michaelson 4:37
Takes a different number than 864, double zero to do that.
Geoff Huston 4:40
Have you seen those pictures of ice skaters? When they draw their
arms in towards them, they rotate faster, and then they pull them
out again, they rotate slower. So everything happens when a
glacier melts, Ooh, interesting. Or what happens when a large
amount of Earth in continental drift? Shifts around and moves from
the equator towards the poles. Well, all of these things.
George Michaelson 5:05
Well, it's like a very Tubby ballerina Geoff,
Geoff Huston 5:09
it is like a very Tubby ballerina, and there's no friction. So,
oddly enough, the Earth is not a stable, rotating mass. It
changes. It regularly changes. The moon, oddly enough, slows the
earth down slowly, inevitably. And it's around a couple of
milliseconds, 2.3 milliseconds per century is the interest of the
moon. But because stuff moves around on the earth, it kind of
speeds up, slows down volcanic activity. So you get into this
really interesting sort of question, what's a second? Because one
of the ways is just, well, obviously it's 1/86400 of the rotation,
rotational period. So therefore, every time I quote you a time I
have to quote you when that measurement was accurate, well, that's
kind of borked.
George Michaelson 5:58
That's really stupid. And we should say at this point that we've
been able to measure milliseconds with tolerable accuracy,
probably for about 100 years, which means we already know from the
time we started being able to measure milliseconds, we've probably
drifted a bit if our basis of counting is this model of counting
rotations of the Earth. This isn't just incidental. This has
really happened.
Geoff Huston 6:22
Well, we started with quartz oscillators, which gave us
milliseconds. But at the start of the 20th century, we started
looking at a topic - radiation, and there are some behaviors which
are actually astonishingly accurate and regular. Take, for
example, your average cesium, 133 atom. Why would I want to do
that?
George Michaelson 6:43
I don't think I've got any to hand at the moment Geoff, but I'm
going to assume that it exists, and it's not hard to get some.
Geoff Huston 6:49
Well, it's not really hard. They exist. But the beauty of it is
that they emit radiation at a frequency of 9,192,631,770 periods
per second. Absolutely, that's not milliseconds. That's down in
billionth of a second, and it's stable. And so all of a sudden I
have this new measurement of time, which is defined by the atoms.
It's independent of the Earth, it's independent of the solar
system. It's independent of all of that. It is just a period of
time, and it's measured by an atomic action, brilliant.
George Michaelson 7:29
an action that takes place at atomic scale, when materials alter
the amount of energy they contain and drop to a lower or jump to a
higher energy level. The way that system works when you pump
energy into the cesium atom, it regularly flings out things at an
astonishingly regular period
Geoff Huston 7:48
Right. Astonishingly regular so all of us don't count time using
cesium, really, don't we like to get up at the same time in the
morning. We like to go to bed at the same time. We'd like our
clocks to say, well, you know, seven o'clock in the morning is
time to get up, you know, etc. What we don't want is that time to
drift.
George Michaelson 8:09
Yes, your seven o'clock and my seven o'clock, we'd like them to
stay pretty much the same, right?
Geoff Huston 8:14
Right. And we accommodated this thing about seasons and the
rotation around the Sun by every four years, give or take some
exceptions, we added a leap day to basically every four years to
take into account that the rotation around the Sun is 365 and a
quarter days. But what about these weird seconds counters? Well,
for a long time, it didn't matter. But then we started using
quartz oscillators, and then we started to use atomic measures,
and we found that the Earth's rotation is definitely not constant.
George Michaelson 8:48
And when you think about our dependency on services, like global
positioning systems, using spinning objects in space, the accuracy
of systems measuring distance using their orbits would ultimately
relate to the periodicity of the Earth's rotation and our concept
of a second.
Geoff Huston 9:06
We use time in computing much, much more than you'd think, and
highly accurate time I push my files up to the cloud, which is
newer: me or the cloud? Well, time, the timestamp, makes
difference. We're sharing a file. We're sharing updates. Time is
the referee on who was the last to speak.
George Michaelson 9:27
We'd both be doing trades to buy and sell shares, but one of us
has got to go first in order to succeed in buying and selling.
Which one went first? Me, me,
Geoff Huston 9:35
me, me, I've got all the money. Yeah, right. You know. So time
intrudes in all kinds of funny ways. And what we'd like is all of
us to agree to time to arbitrarily high accuracy. Wow. So what we
did is we did this service called U, T, C, Universal Coordinated
Time.
George Michaelson 9:59
U, C, T
Geoff Huston 10:01
Well, French, English, French, universal, coordinated time. UTC,
yes, mais oui. It takes as a correctional piece of data, the
current speed of the Earth's rotation from a weird mob in Paris
called the International Earth Rotation and Reference System,
which actually has the acronym IERS, which actually looks at the
speed of the earth the length of the day, and effectively it tries
to keep this ticking time from UTC into synchronization with the
sun by adding or removing a leap second when it's necessary
George Michaelson 10:47
a leap second. So if a leap year is done religiously, once every
four years, except for some special cases, like every 400 years, I
don't think I'll live long enough to see one. What is a leap
second
Geoff Huston 11:02
Well, because the Earth is nowhere near as regular, and my jumping
up and down has a slight, astronomically small impact on the Earth's
rotational speed. You know, everything affects its speed, and so
it's not regular. And so the Earth has been slowing down for some
time. And every time when it looked like getting outside of point
nine of a second from solar time, because it's slowing down, they
added an extra second, a bonus second,
George Michaelson 11:33
I'm going to live longer. They gave me another second of living
longer!
Geoff Huston 11:37
They did. And it was either the last midnight, the last minute of
June, or bizarrely, the last minute of December, got quick and
longer.
George Michaelson 11:47
But Geoff, you said we're all depending on agreeing what time is,
and we're now highly focused on how accurate time is, and that
means millions and millions of machines sitting out there in the
world knowing what the time is, as some mob in Paris is putting up
a red flag, waving it, saying, This June an extra second.
Geoff Huston 12:12
Yeah, you know, it's kind of okay in June, because most of us
don't have hangovers on the first of July to cope with the mess.
On the other hand, you'd think it's the worst possible time you
can possibly add a leap second is the last second of December when
most of the Earth's population is heavily imbibing and celebrating
and not worrying at all about whether their computers have just
gone into a mess. Because adding a second has, in the past, proved
to be remarkably disruptive for computers, because things that go
sleep one second, that means I'm in a new something or other. And
the answer is, well, sometimes you're in the same minute, and it's
not predictable. It's not every June or every December. It's only
when the Earth has slowed down. So for example, between 2000 and
2006 there were no leap seconds, no extra seconds.
George Michaelson 13:02
Six years without one.
Geoff Huston 13:04
Yeah, it didn't. It wasn't slowing down as much. And bizarrely,
from about the year 2020 until now, the bloody Earth's been
speeding up again, beginning faster. We actually might have to
contemplate shock, horror taking a second away from that final
minute and having a 59 second,
George Michaelson 13:25
I'll live shorter. My life is going to lose a second because the
French are claiming it back. This is not okay.
Geoff Huston 13:31
This is a mess. Now, for a long time, we kept on going, Oh, that
caused mayhem, and it did many systems crash. Oh, we'll do better
next time. When's next time going to happen? No, sometime. And it
was actually a number of folk, including Google, that worked on, I
think, quite an interesting way around this. You get six months
notice from the International Earth's rotation service. You get
six months notice of a leap second. So you've got some time to
prepare. And the idea is you muck around with the slew of time,
and instead of doing a 61 second minute over the preceding few
days, you subtly add a millisecond or two into the time stream,
millisecond here..
George Michaelson 14:17
and hopefully nobody notices.
Geoff Huston 14:19
Well, it's only milliseconds. No one notices, until you get to the
magic time. And guess what? The level is still 60 seconds, because
you've already smeared that second over the previous few hours.
George Michaelson 14:32
But it's a bit happy, isn't it? Geoff. It's not actually a defined
standard mechanism.
Geoff Huston 14:38
Well, negotiate with the Earth, George, negotiate with the earth.
Now, you know, I'm talking about this as if time is some abstract
thing that we can diddle with. It's not, oddly enough, keeping
time is its own art form, and distributing time is even more
amazing because we actually use a protocol. These days, we used to
use radio systems, but these days, it's all on the Internet, and
we all use a protocol called the Network Time Protocol, NTP, RFC,
5905 and it's a remarkably simple but quite deep protocol where
two systems constantly exchange their clock values with each
other, and one of them is a reference service. The other one is in
old speak, we used to call it a slave. I think these days we call
it a dependent clock.
George Michaelson 15:33
much better terminology,
Geoff Huston 15:35
but it's it's trying to synchronize its clock against that
reference signal, and NTP is the way we do this. And so if you
latch on to a highly, highly accurate clock stratum, one, which
traditionally comes from the GPS system, the global positioning
system, you have a remarkably accurate for the time and where the
seconds, leap seconds are inserted or not comes in that stratum,
one version of time as broadcast by those GPS signals.
George Michaelson 16:07
And so if I've got this mechanism right in my head, it consists of
me repeatedly asking you the time, remembering what I thought the
time was asking you, and looking at your answer, and then the time
that's moved forward in me. And by doing this many, many, many,
many times, I built up a picture of the variance in delay between
me and you. And eventually I arrived at a point where I can say, I
know the minimum variance, and I know the worst variance I've
seen. And the next time you tell me the time, I'll take off that
variance, and that is the time I'll put in me. And we kind of come
to an agreement of time. You don't really adjust to me because you
have a highly accurate clock, but I use this mechanism to train
myself to tick at the same time as you even though there's a delay
in between us
Geoff Huston 16:59
exactly, even though that delay might even be slightly variable,
we are synchronizing our clock pulses. Yes, and that's it's
cunning maths. It's a great protocol, and it's really important.
As we've said before, lots and lots and lots of things rely on the
time. From cloud services, our security systems have certificates.
What's a certificate? Well, it has validity. Timers do not believe
this certificate after this particular time, the whole thing about
let's coordinate our actions with the cloud, with anything else.
Time matters. It matters a lot. [George: Yeah], and synchronizing
it is really important.
George Michaelson 17:38
So of course, it's fully official and completely organized,
regulated and managed, and there's a central time poobah. And the
whole thing runs like clockwork, doesn't it?
Geoff Huston 17:48
No, NTP runs in the clear. It does. This is crazy. I I'm a stratum
one server.
George Michaelson 17:55
I'm saying I am.
Geoff Huston 17:56
Well, I'm saying I am. Connect to me. Oh, look, I see you
connecting to a stratum one server. I will answer in its place. I
will start to slew your clock. I will send you into a time warp.
Oh, victim, and it's kind of good you can do that in NTP. Oh, you
can. Why would I do that? Well, why do people do these kinds of
things? Because mucking with the time can be very, very damaging,
and we all know it. I can jam GPS. I need a better way of having
time as an important piece of digital infrastructure. Oh, so what
I really want to do is stop people masquerading as time servers.
George Michaelson 18:37
We want to get rid of this ability to lie in time.
Geoff Huston 18:40
Get rid of the ability to lie in time or misrepresent themselves
as being a better source of time than they really are, or even not
a source of time at all. And so not only is NTP brilliant, but we
need to be able to secure NTP to stop that brilliance being
perverted, and stop people relying on us that is not secured
against the wayward actions of sophisticated attackers.
George Michaelson 19:05
Geoff, I'd like to think that securing something like this would
be a reasonably simple mechanism to take but the problem is,
security is generally adding complexity, and complexity is kind
of like the enemy of timekeeping, because complexity adds delay
and good complexity has to do it in a way that can't use the
variants in that timing to work out how to break the security.
So it begins to feel like adding security to a time service is
actually more complicated than you might think.
Geoff Huston 19:39
Well, think about the problem. It's kind of I need to know whom I
am synchronizing with, and I need to know that the signals I get
from them are from them, but I don't necessarily need to encrypt
the conversation. It's not a secret. It's just the time. And if
you think about what. We do in security, we actually try and solve
a number of problems. Who are you? That's a deep problem. I wish
to know that I'm speaking with the party I intended to speak is
someone else, including into this conversation, representing
themselves as the person I wanted to speak with. That's a piece of
protection. And thirdly, do I need to protect my conversation that
no one can see what we're saying. Well, in the case of time, that
third piece is not relevant. It's not needed, it's overkill.
George Michaelson 20:29
But there's a version of the second piece that perhaps, has
anybody changed. The message I know is coming from you. That's
what someone pretending to be. You could be doing. They could
leave it it came from you, but they could alter what is said,
[Geoff: right] So there's knowing it came from you, and there's
knowing you said it.
Geoff Huston 20:49
So if we add to NTP a key exchange, handshake, Hi, are you the
stratum 1 server I thought I was connecting to, or the stratum 2,
or whatever, and we go, yes, good, fine. Let's work up a session
key, great! not. Every time you send me something, sign it with
your key. I can decode it and know, because it's signed across the
message that that's you saying it, and not someone else faking it.
So now it's very, very hard to intrude in this conversation. I'm
not encrypting the contents. That's a waste of everyone's time,
and it causes extra delay, but I am adding authenticity. And the
reason why I do this is I get back to this area of national
infrastructure and national importance, if we're relying on this
for all kinds of civil things around time, and the more you look,
the more you find time embedded in file management protocols and
security protocols and all kinds of applications I can secure it
then effectively it makes it harder to attack and disrupt me. And
so some countries have taken this on with alacrity. Sweden is
funded a secure NTP system. The Danes have gone that far as well,
de funded nationally this service as a national service. Other
countries have left it to industry itself, and there are secure
time servers out there, all over the net without necessarily
being, being state funded. But it does raise the interesting
question, and I think it's a fascinating question, as one of the
first we're going to ask in this issue about time. Is time public
or private? What do you mean? Is accurate time something you
should pay for, operated as a fee for service basis by commercial
entities, or is it a public service that is accessible for all to
use in a secure manner? Is time so important that it's unwise to
privatize?
George Michaelson 22:53
I'm tending from my own personal philosophy to see this as
something akin to a utility function, you could contest it.
There's no reason why it isn't capable of being contestable. But
I'm not comfortable personally feeling like we're going to say
it's just easier to let commerce sort it out. I kind of feel like,
yes, it's a public utility but that's me,
Geoff Huston 23:21
well, in the US, the GPS system is operated effectively through
public budgets with a fair degree of military participation. Why
are they altruists? Do they appreciate the argument? Oh, I don't
know. A lot of this is to do with hurling bits of metal over large
distances accurately. Time has military application, and so time
is part of that public service, and its rationale has military
interpretations as much as public. So it's kind of, if I left it
entirely to the private sector. Who knows, not a good idea. I'm
more idealistic. I actually think time is a necessary piece of
public infrastructure. And, you know, I feel quite strongly that
its positioning is exactly in the public sector and should remain
so. Time is not a for profit luxury. It's just something we all
need.
George Michaelson 24:11
There was a period where aspects of the national infrastructure,
like the telecommunication system, when it was run by the post
office required a clock, and that clock managed the way signals
were sent through that network. It was essentially a nationally
coordinated tick that allowed aspects of the network to run. There
was a period when radio frequencies used to transmit television
signals were depending on a coordinated clock, and particularly
for sailors at sea, for instance, the institution of sending a
time signal that allows them to reset their clock means they can
navigate safely. So there has been a component that we do things
like this in the wider national interest, and they run as national
services. But then time passes and we stop using radio waves to
transmit TV, and somebody says that isn't vital for us anymore,
because we don't do it that way anymore, and the time signals
disappear. And I find that a little unhappy.
Geoff Huston 25:20
I live in Australia, a country that does have a body, the CSIRO,
that has a time service. Sweden has a time service. Denmark has a
time service. Oh, the poor old UK. I don't know what's public and
what's private these days, anything's possible, but, you know, a
GPS that's a public service in the United States. So that legacy
does persist, and there is still some kind of public aspects of
time, as you point out, that are really important and should not
be a for profit, privatized activity. Time is too important to be
mucked around with like that. I just wanted to leave you with one
other sort of interesting set of thoughts, though. Do you remember
y, 2k?
George Michaelson 25:58
Oh, yes!
Geoff Huston 26:00
So the nature of time. How do we express time? Well, I live in
2026 at the moment. What does that really say? That's 2026 years
past an epoch. So time is actually expressed as an epoch event,
time zero and then a counter that counts the number of things,
years, minute, seconds since that epoch. So [George: yeah], in the
case of y, 2k the epoch was 1900 and the counter was years. And
the interesting thing was two digits of years,
George Michaelson 26:37
but only two digits worth two digits. People were cheap skates.
Geoff Huston 26:43
So at the end of 1999 when it became the year 2000 Guess what? New
epoch? Guess what? That two digit counter, which was your top 2
years, would reset to zero and everyone, oh, my god, Armageddon,
everything will collapse, digital nihilism. Let's all go and camp
in the woods for a month until we get this right? Yeah, yeah,
yeah. Lots of money was spent. Very little happened. I did have to
go into work that night, as did many other IT workers.
George Michaelson 27:11
I paid someone else to go into work because I had young kid. So
for me, it kind of made sense to give them $200 and they kept an
eye on the machines to see that everything was okay. But Geoff,
you're kind of eliding over we actually did have to go and check
what was going to happen. It wasn't just accident that we were all
okay. People invested energy making sure we would be okay.
Geoff Huston 27:36
Well, in some ways, though, it was COBOL, it wasn't every
programming language you use. And I found it kind of laughable
that we were running all our systems on Unix, and the year 2000
was not an event. Why? Oh, that's easy. That's very easy. Time in
Unix has an epoch for UNIX of the first of January, 1970 U, T, C,
and time is the number of seconds standard Atomic Time sections,
you know, that cesium clock those seconds the number of real
seconds that have elapsed since that epoch. So the year 2000 was a
switch from 95,000,200 you know, to the next number. It wasn't
anything special, [George: right] But when does time run out? Oh,
what a tough question. You know, in Unix and in the C language,
there's signed integers and unsigned integers, and variously, time
is the number of seconds since first of January, 1970 when you
express it as a signed counter, who knows why, but you can then
that counter will get to its maximum and then become a negative
number on the 19th of January, 2038, 12 years from now,
George Michaelson 28:56
Not a long time Geoff.
Geoff Huston 28:58
Windows, 32 bit signed value, 19th of January, 2038 oops. Now
maybe some of you got with it read the memo and said, We're going
to make it unsigned. Well, interestingly, you buy yourself a whole
bunch of time right up until around 2106 or so. But it'll still
happen. You'll still clock over. But there comes this next
question, which is sort of relevant and hard to answer. My big,
Grunty, expensive laptop, 64 bits of words. My integer value is 64
bits. What's two to the 64 seconds since the first of January,
1970 let me tell you, because I know you're interested. It's the
fourth of December in the year 292,277,026,596 and don't forget,
the Earth is only 13 though the universe is only 13 billion years
old. But this thing will keep on running for you know, a further
290-odd billion. Is.
George Michaelson 30:00
But do we have to agree, in some mechanistic way that we're going
to uplift time from an epoch into 64 bit numbers? Right?
Geoff Huston 30:10
Oh, no, we've already done. You look at your systems time as a 64
bit number, but I have time in many things, including some flea
bit, you know, eight bit machines or something. What's their time?
There's a whole bunch of embedded stuff that I can't get hold of,
even in my car. There are probably about 300 processors. They
probably all have a value of time somewhere. What's their time are
they using? Well, I don't know what they're using. Let's look at a
few more disasters. What about NTP itself? Oh, hang on, a second.
It carries the time in a quantity that is 32 bits. [George:
What!!!] I'm sorry, and the first 32 bits is the number of seconds
since the first of January, 1900
George Michaelson 30:55
not 1970
Geoff Huston 30:56
when does its time run?
George Michaelson 30:57
It has a different default.
Geoff Huston 30:59
If it's signed, it's it's unsigned, not signed, so you get a fair
deal of time back. But what's 4 billion seconds past the first of
January, 1900 I know you're desperate to know it's the seventh of
February, 2036, so two years before the signed Unix values go
totally ape. NTP goes totally ape. Oh, my God, this is not meant
to happen to us.
George Michaelson 31:23
This is not meant to happen. We're gonna have to fix this.
Geoff Huston 31:27
Oh, it's okay. My file system will fix it, because my file system,
thank you. Macintosh uses HFS plus. HFS plus. Hang on a second,
unsigned 32 bit number. What's the it's epoch. Well, bizarrely,
the first of January, 1904 so on the fifth of February, 2040
there's another crash in all of the HFS file systems. Well, it's
okay. I'm using fat 32/31 of December, 2107 and on and on it goes
even ZFS, which is 64 bits. Thank God that time's out on the
fourth of December in the year 292 billion. Thank God. But you
know, all those 32 bit things are kind of falling due at once.
[George: Yeah] And the real question is, how much of the world is
still in 32 bit, and can we find them?
George Michaelson 32:17
Yeah, because we're talking light bulbs and we're talking
thermostats, and we're talking fridges and we're talking cars with
300 processes. Would you like to guarantee there isn't one of them
that's going to think the time has gone back?
Geoff Huston 32:32
So let me posit the experiment. I take my laptop, I isolate it
from the rest of the world, I put in a date stamp of, I don't
know, a good stamp might be the 20th of January, 2038, or
something, one day after the epoch point. And then I try and
reconnect it to the network with this odd date. What's going to
happen? Well, I know right now my machine is going to go
absolutely flop and die, because nothing works. Nothing works. All
the certificates are invalid. File says nothing is working
because, not because it couldn't cope with the date rollover. But
you can't coexist in two widely disparate times. You can't implant
the time Martian back into the existing time continuum of the
Internet. So oops, I can't test for this. [George: Wow.] I can't.
I have to replicate an entire infrastructure shifted forward in
time to that point of epoch role. So unlike y 2k where at the time
we weren't that interdependent, you could just take a system, roll
its clock forward and see what went, bang, we can't do that this
time, not what we know of. Wow.
George Michaelson 33:38
There's going to have to be some work done on this Geoff.
Geoff Huston 33:41
You think so? I don't think so. I think we spent so much money on
Y2K and ended up going boring, waste of time, not going to do that
again, that we have talked ourselves out of it. And the answer
really is, you know, go and get 64 bit architecture. Do it right.
Just don't go through this problem. Get rid of the cheap stuff.
Well, what about your cars? What about you the oh, I don't know.
Do something. Who's going to do it? I don't know. [George: Wow].
Now the good news is I'll have retired. It may be 12 years away,
but that's 12 too many for my working life. So I watch from the
sidelines if I'm still around, but it is a here and now problem
for the next generation, and it's a tougher problem than Y2K but
it's poisoned by the Y2K debacle.
George Michaelson 34:31
Watching this one emerge is going to be fascinating. Geoff, that's
going to be an interesting one.
Geoff Huston 34:37
The other thing do you know, when we panicked for Y2K 1999 was
coming for years. Oh, he just said, yeah. So sometime, I don't
know, the first of January 2036, when we realize that there'll be
36 days until NTP goes bang in our face, we might panic about the
NTP date, and then someone might politely point out. The unsigned
Unix is going to go bang in about it two years from then. So we
might panic about that, but I think it's going to be a rolling
series of, oh my God. What's happening if we're still, you know,
if we still have computers around at the time, I'm not sure we're
going to do much better than that. Unfortunately.
George Michaelson 34:39
Oh dear,
Geoff Huston 34:39
Are you listening children? You have a problem. Let's see
George Michaelson 35:01
if ping is still running and we can do a follow up.
Geoff Huston 35:12
Not me, not me, find one of the children to talk about it.
George Michaelson 35:28
Thank you, Geoff, that's been fascinating
Geoff Huston 35:31
with that boys and girls see you next time.
George Michaelson 35:36
If you've got a story or research to share here on ping, why not
get in contact by email, to ping@apnic.net or via the APNIC social
media channels, also remember the measurement@apnic.net mailing
list on orbit is there to discuss and share relevant collaborative
opportunities, grants and funding opportunities, jobs and graduate
placings, or to seek feedback from the community on your own
measurement projects, be sure to check out the APNIC website for
All your resource and community needs until next time you
It's kind of okay in June, because most of us don't have hangovers
on the first of July to cope with the mess. On the other hand,
you'd think it's the worst possible time you can possibly add a
leap second is the last second of December when most of the
Earth's population is heavily imbibing and celebrating and not
worrying at all about whether their computers have just gone into
a mess because adding a second has, in the past, proved to be
remarkably disruptive for computers, because things that go sleep
one second, that means I'm in a new something or other. And the
answer is, well, sometimes you're in the same minute, and it's not
predictable. It's not every June or every December. It's only when
the Earth has slowed down. So, for example, between 2000 and 2006
there were no leap seconds, no extra seconds.
George Michaelson 1:00
You're listening to ping, a podcast by APNIC discussing all things
related to measuring the Internet. I'm your host, George
Michaelson, this time, I'm talking to Geoff Huston from APNIC labs
again in his regular monthly spot on ping, Geoff has been looking
at NTP again. NTP is the network time protocol. It's one of the
older systems we depend on, designed and implemented by Dave
Mills, who died in 2024. Dave had been working on time
synchronization from the mid 1970s Geoff has been thinking about
NTP, moves to secure NTP and our increasing dependency on the
underlying concepts of a coordinated sense of time. The dependency
in the modern world on highly synchronized clocks cannot be
overstated. It creeps into every sector of daily life, from
aircraft and space navigation to finance systems and event
scheduling of all kinds, but our model of time is based on the
rotation of the Earth and the length of the second. And
unfortunately, while we now define the length of the second to
astonishingly accurate levels, the rotation of the Earth isn't as
stable as we'd like. Our model of time has to make some
adjustments, and our model of time has been coded over the years
to varying degrees of a start date known as an epoch, and how we
represent time inside the machines. It's all coming a bit unstuck.
Geoff, welcome back to ping. What shall we talk about this time?
Geoff Huston 2:35
Well, I could start with the universe, but I'll scale it back down
a bit. I want to talk about the rotating earth. [George: what!!!]
I really do. It's kind of celestial mechanics, which is, you know,
absolutely fascinated man time, ever since they started looking at
the stars and figured out every night the stars look about the
same as they did the night before, which is amazing, and they
started to kind of out the period when it looked the same to the
next part when it looked the same. And this dates back a long way.
The ancient Babylonians had their best shot at this using a really
weird base 60 counting system. Why 60? Because they figured out
that it was one of the most divisible numbers of all you know,
1234, they all divided evenly into 60. So guess what, 60 seconds
in a minute? Guess what 60 minutes in an hour? Now it's going to
say, Guess what, 60 hours? Some reason, the day was 24 hours long.
But we have this sort of definition coming even then that the
regularly rotates a period of 86,400 seconds. And if you're
looking up at the stars and you count 86,400 of these, the sky
will look exactly the same as it did last time you're gone around
the world, the Earth once or the Earth has gone once around on
this axle. Cool, hey, but not quite,
George Michaelson 4:03
Ah, there's always a but, isn't there?
Geoff Huston 4:06
Well, what's the definition of a second thing? Well, easy, 6400 of
the time it took to rotate about its own axis. That's second
George Michaelson 4:16
short episode of ping. We're done. See you next time.
Geoff Huston 4:19
So you dial up your ancestor through the time machine and go, How
long is the second dude? Well, he says same thing. 180, 6,400th,
of the time it takes to rotate about its own axis. You go, No,
that's not right, because if I use my seconds and count your time,
that's not actually true. What do you mean?
George Michaelson 4:37
Takes a different number than 864, double zero to do that.
Geoff Huston 4:40
Have you seen those pictures of ice skaters? When they draw their
arms in towards them, they rotate faster, and then they pull them
out again, they rotate slower. So everything happens when a
glacier melts, Ooh, interesting. Or what happens when a large
amount of Earth in continental drift? Shifts around and moves from
the equator towards the poles. Well, all of these things.
George Michaelson 5:05
Well, it's like a very Tubby ballerina Geoff,
Geoff Huston 5:09
it is like a very Tubby ballerina, and there's no friction. So,
oddly enough, the Earth is not a stable, rotating mass. It
changes. It regularly changes. The moon, oddly enough, slows the
earth down slowly, inevitably. And it's around a couple of
milliseconds, 2.3 milliseconds per century is the interest of the
moon. But because stuff moves around on the earth, it kind of
speeds up, slows down volcanic activity. So you get into this
really interesting sort of question, what's a second? Because one
of the ways is just, well, obviously it's 1/86400 of the rotation,
rotational period. So therefore, every time I quote you a time I
have to quote you when that measurement was accurate, well, that's
kind of borked.
George Michaelson 5:58
That's really stupid. And we should say at this point that we've
been able to measure milliseconds with tolerable accuracy,
probably for about 100 years, which means we already know from the
time we started being able to measure milliseconds, we've probably
drifted a bit if our basis of counting is this model of counting
rotations of the Earth. This isn't just incidental. This has
really happened.
Geoff Huston 6:22
Well, we started with quartz oscillators, which gave us
milliseconds. But at the start of the 20th century, we started
looking at a topic - radiation, and there are some behaviors which
are actually astonishingly accurate and regular. Take, for
example, your average cesium, 133 atom. Why would I want to do
that?
George Michaelson 6:43
I don't think I've got any to hand at the moment Geoff, but I'm
going to assume that it exists, and it's not hard to get some.
Geoff Huston 6:49
Well, it's not really hard. They exist. But the beauty of it is
that they emit radiation at a frequency of 9,192,631,770 periods
per second. Absolutely, that's not milliseconds. That's down in
billionth of a second, and it's stable. And so all of a sudden I
have this new measurement of time, which is defined by the atoms.
It's independent of the Earth, it's independent of the solar
system. It's independent of all of that. It is just a period of
time, and it's measured by an atomic action, brilliant.
George Michaelson 7:29
an action that takes place at atomic scale, when materials alter
the amount of energy they contain and drop to a lower or jump to a
higher energy level. The way that system works when you pump
energy into the cesium atom, it regularly flings out things at an
astonishingly regular period
Geoff Huston 7:48
Right. Astonishingly regular so all of us don't count time using
cesium, really, don't we like to get up at the same time in the
morning. We like to go to bed at the same time. We'd like our
clocks to say, well, you know, seven o'clock in the morning is
time to get up, you know, etc. What we don't want is that time to
drift.
George Michaelson 8:09
Yes, your seven o'clock and my seven o'clock, we'd like them to
stay pretty much the same, right?
Geoff Huston 8:14
Right. And we accommodated this thing about seasons and the
rotation around the Sun by every four years, give or take some
exceptions, we added a leap day to basically every four years to
take into account that the rotation around the Sun is 365 and a
quarter days. But what about these weird seconds counters? Well,
for a long time, it didn't matter. But then we started using
quartz oscillators, and then we started to use atomic measures,
and we found that the Earth's rotation is definitely not constant.
George Michaelson 8:48
And when you think about our dependency on services, like global
positioning systems, using spinning objects in space, the accuracy
of systems measuring distance using their orbits would ultimately
relate to the periodicity of the Earth's rotation and our concept
of a second.
Geoff Huston 9:06
We use time in computing much, much more than you'd think, and
highly accurate time I push my files up to the cloud, which is
newer: me or the cloud? Well, time, the timestamp, makes
difference. We're sharing a file. We're sharing updates. Time is
the referee on who was the last to speak.
George Michaelson 9:27
We'd both be doing trades to buy and sell shares, but one of us
has got to go first in order to succeed in buying and selling.
Which one went first? Me, me,
Geoff Huston 9:35
me, me, I've got all the money. Yeah, right. You know. So time
intrudes in all kinds of funny ways. And what we'd like is all of
us to agree to time to arbitrarily high accuracy. Wow. So what we
did is we did this service called U, T, C, Universal Coordinated
Time.
George Michaelson 9:59
U, C, T
Geoff Huston 10:01
Well, French, English, French, universal, coordinated time. UTC,
yes, mais oui. It takes as a correctional piece of data, the
current speed of the Earth's rotation from a weird mob in Paris
called the International Earth Rotation and Reference System,
which actually has the acronym IERS, which actually looks at the
speed of the earth the length of the day, and effectively it tries
to keep this ticking time from UTC into synchronization with the
sun by adding or removing a leap second when it's necessary
George Michaelson 10:47
a leap second. So if a leap year is done religiously, once every
four years, except for some special cases, like every 400 years, I
don't think I'll live long enough to see one. What is a leap
second
Geoff Huston 11:02
Well, because the Earth is nowhere near as regular, and my jumping
up and down has a slight, astronomically small impact on the Earth's
rotational speed. You know, everything affects its speed, and so
it's not regular. And so the Earth has been slowing down for some
time. And every time when it looked like getting outside of point
nine of a second from solar time, because it's slowing down, they
added an extra second, a bonus second,
George Michaelson 11:33
I'm going to live longer. They gave me another second of living
longer!
Geoff Huston 11:37
They did. And it was either the last midnight, the last minute of
June, or bizarrely, the last minute of December, got quick and
longer.
George Michaelson 11:47
But Geoff, you said we're all depending on agreeing what time is,
and we're now highly focused on how accurate time is, and that
means millions and millions of machines sitting out there in the
world knowing what the time is, as some mob in Paris is putting up
a red flag, waving it, saying, This June an extra second.
Geoff Huston 12:12
Yeah, you know, it's kind of okay in June, because most of us
don't have hangovers on the first of July to cope with the mess.
On the other hand, you'd think it's the worst possible time you
can possibly add a leap second is the last second of December when
most of the Earth's population is heavily imbibing and celebrating
and not worrying at all about whether their computers have just
gone into a mess. Because adding a second has, in the past, proved
to be remarkably disruptive for computers, because things that go
sleep one second, that means I'm in a new something or other. And
the answer is, well, sometimes you're in the same minute, and it's
not predictable. It's not every June or every December. It's only
when the Earth has slowed down. So for example, between 2000 and
2006 there were no leap seconds, no extra seconds.
George Michaelson 13:02
Six years without one.
Geoff Huston 13:04
Yeah, it didn't. It wasn't slowing down as much. And bizarrely,
from about the year 2020 until now, the bloody Earth's been
speeding up again, beginning faster. We actually might have to
contemplate shock, horror taking a second away from that final
minute and having a 59 second,
George Michaelson 13:25
I'll live shorter. My life is going to lose a second because the
French are claiming it back. This is not okay.
Geoff Huston 13:31
This is a mess. Now, for a long time, we kept on going, Oh, that
caused mayhem, and it did many systems crash. Oh, we'll do better
next time. When's next time going to happen? No, sometime. And it
was actually a number of folk, including Google, that worked on, I
think, quite an interesting way around this. You get six months
notice from the International Earth's rotation service. You get
six months notice of a leap second. So you've got some time to
prepare. And the idea is you muck around with the slew of time,
and instead of doing a 61 second minute over the preceding few
days, you subtly add a millisecond or two into the time stream,
millisecond here..
George Michaelson 14:17
and hopefully nobody notices.
Geoff Huston 14:19
Well, it's only milliseconds. No one notices, until you get to the
magic time. And guess what? The level is still 60 seconds, because
you've already smeared that second over the previous few hours.
George Michaelson 14:32
But it's a bit happy, isn't it? Geoff. It's not actually a defined
standard mechanism.
Geoff Huston 14:38
Well, negotiate with the Earth, George, negotiate with the earth.
Now, you know, I'm talking about this as if time is some abstract
thing that we can diddle with. It's not, oddly enough, keeping
time is its own art form, and distributing time is even more
amazing because we actually use a protocol. These days, we used to
use radio systems, but these days, it's all on the Internet, and
we all use a protocol called the Network Time Protocol, NTP, RFC,
5905 and it's a remarkably simple but quite deep protocol where
two systems constantly exchange their clock values with each
other, and one of them is a reference service. The other one is in
old speak, we used to call it a slave. I think these days we call
it a dependent clock.
George Michaelson 15:33
much better terminology,
Geoff Huston 15:35
but it's it's trying to synchronize its clock against that
reference signal, and NTP is the way we do this. And so if you
latch on to a highly, highly accurate clock stratum, one, which
traditionally comes from the GPS system, the global positioning
system, you have a remarkably accurate for the time and where the
seconds, leap seconds are inserted or not comes in that stratum,
one version of time as broadcast by those GPS signals.
George Michaelson 16:07
And so if I've got this mechanism right in my head, it consists of
me repeatedly asking you the time, remembering what I thought the
time was asking you, and looking at your answer, and then the time
that's moved forward in me. And by doing this many, many, many,
many times, I built up a picture of the variance in delay between
me and you. And eventually I arrived at a point where I can say, I
know the minimum variance, and I know the worst variance I've
seen. And the next time you tell me the time, I'll take off that
variance, and that is the time I'll put in me. And we kind of come
to an agreement of time. You don't really adjust to me because you
have a highly accurate clock, but I use this mechanism to train
myself to tick at the same time as you even though there's a delay
in between us
Geoff Huston 16:59
exactly, even though that delay might even be slightly variable,
we are synchronizing our clock pulses. Yes, and that's it's
cunning maths. It's a great protocol, and it's really important.
As we've said before, lots and lots and lots of things rely on the
time. From cloud services, our security systems have certificates.
What's a certificate? Well, it has validity. Timers do not believe
this certificate after this particular time, the whole thing about
let's coordinate our actions with the cloud, with anything else.
Time matters. It matters a lot. [George: Yeah], and synchronizing
it is really important.
George Michaelson 17:38
So of course, it's fully official and completely organized,
regulated and managed, and there's a central time poobah. And the
whole thing runs like clockwork, doesn't it?
Geoff Huston 17:48
No, NTP runs in the clear. It does. This is crazy. I I'm a stratum
one server.
George Michaelson 17:55
I'm saying I am.
Geoff Huston 17:56
Well, I'm saying I am. Connect to me. Oh, look, I see you
connecting to a stratum one server. I will answer in its place. I
will start to slew your clock. I will send you into a time warp.
Oh, victim, and it's kind of good you can do that in NTP. Oh, you
can. Why would I do that? Well, why do people do these kinds of
things? Because mucking with the time can be very, very damaging,
and we all know it. I can jam GPS. I need a better way of having
time as an important piece of digital infrastructure. Oh, so what
I really want to do is stop people masquerading as time servers.
George Michaelson 18:37
We want to get rid of this ability to lie in time.
Geoff Huston 18:40
Get rid of the ability to lie in time or misrepresent themselves
as being a better source of time than they really are, or even not
a source of time at all. And so not only is NTP brilliant, but we
need to be able to secure NTP to stop that brilliance being
perverted, and stop people relying on us that is not secured
against the wayward actions of sophisticated attackers.
George Michaelson 19:05
Geoff, I'd like to think that securing something like this would
be a reasonably simple mechanism to take but the problem is,
security is generally adding complexity, and complexity is kind
of like the enemy of timekeeping, because complexity adds delay
and good complexity has to do it in a way that can't use the
variants in that timing to work out how to break the security.
So it begins to feel like adding security to a time service is
actually more complicated than you might think.
Geoff Huston 19:39
Well, think about the problem. It's kind of I need to know whom I
am synchronizing with, and I need to know that the signals I get
from them are from them, but I don't necessarily need to encrypt
the conversation. It's not a secret. It's just the time. And if
you think about what. We do in security, we actually try and solve
a number of problems. Who are you? That's a deep problem. I wish
to know that I'm speaking with the party I intended to speak is
someone else, including into this conversation, representing
themselves as the person I wanted to speak with. That's a piece of
protection. And thirdly, do I need to protect my conversation that
no one can see what we're saying. Well, in the case of time, that
third piece is not relevant. It's not needed, it's overkill.
George Michaelson 20:29
But there's a version of the second piece that perhaps, has
anybody changed. The message I know is coming from you. That's
what someone pretending to be. You could be doing. They could
leave it it came from you, but they could alter what is said,
[Geoff: right] So there's knowing it came from you, and there's
knowing you said it.
Geoff Huston 20:49
So if we add to NTP a key exchange, handshake, Hi, are you the
stratum 1 server I thought I was connecting to, or the stratum 2,
or whatever, and we go, yes, good, fine. Let's work up a session
key, great! not. Every time you send me something, sign it with
your key. I can decode it and know, because it's signed across the
message that that's you saying it, and not someone else faking it.
So now it's very, very hard to intrude in this conversation. I'm
not encrypting the contents. That's a waste of everyone's time,
and it causes extra delay, but I am adding authenticity. And the
reason why I do this is I get back to this area of national
infrastructure and national importance, if we're relying on this
for all kinds of civil things around time, and the more you look,
the more you find time embedded in file management protocols and
security protocols and all kinds of applications I can secure it
then effectively it makes it harder to attack and disrupt me. And
so some countries have taken this on with alacrity. Sweden is
funded a secure NTP system. The Danes have gone that far as well,
de funded nationally this service as a national service. Other
countries have left it to industry itself, and there are secure
time servers out there, all over the net without necessarily
being, being state funded. But it does raise the interesting
question, and I think it's a fascinating question, as one of the
first we're going to ask in this issue about time. Is time public
or private? What do you mean? Is accurate time something you
should pay for, operated as a fee for service basis by commercial
entities, or is it a public service that is accessible for all to
use in a secure manner? Is time so important that it's unwise to
privatize?
George Michaelson 22:53
I'm tending from my own personal philosophy to see this as
something akin to a utility function, you could contest it.
There's no reason why it isn't capable of being contestable. But
I'm not comfortable personally feeling like we're going to say
it's just easier to let commerce sort it out. I kind of feel like,
yes, it's a public utility but that's me,
Geoff Huston 23:21
well, in the US, the GPS system is operated effectively through
public budgets with a fair degree of military participation. Why
are they altruists? Do they appreciate the argument? Oh, I don't
know. A lot of this is to do with hurling bits of metal over large
distances accurately. Time has military application, and so time
is part of that public service, and its rationale has military
interpretations as much as public. So it's kind of, if I left it
entirely to the private sector. Who knows, not a good idea. I'm
more idealistic. I actually think time is a necessary piece of
public infrastructure. And, you know, I feel quite strongly that
its positioning is exactly in the public sector and should remain
so. Time is not a for profit luxury. It's just something we all
need.
George Michaelson 24:11
There was a period where aspects of the national infrastructure,
like the telecommunication system, when it was run by the post
office required a clock, and that clock managed the way signals
were sent through that network. It was essentially a nationally
coordinated tick that allowed aspects of the network to run. There
was a period when radio frequencies used to transmit television
signals were depending on a coordinated clock, and particularly
for sailors at sea, for instance, the institution of sending a
time signal that allows them to reset their clock means they can
navigate safely. So there has been a component that we do things
like this in the wider national interest, and they run as national
services. But then time passes and we stop using radio waves to
transmit TV, and somebody says that isn't vital for us anymore,
because we don't do it that way anymore, and the time signals
disappear. And I find that a little unhappy.
Geoff Huston 25:20
I live in Australia, a country that does have a body, the CSIRO,
that has a time service. Sweden has a time service. Denmark has a
time service. Oh, the poor old UK. I don't know what's public and
what's private these days, anything's possible, but, you know, a
GPS that's a public service in the United States. So that legacy
does persist, and there is still some kind of public aspects of
time, as you point out, that are really important and should not
be a for profit, privatized activity. Time is too important to be
mucked around with like that. I just wanted to leave you with one
other sort of interesting set of thoughts, though. Do you remember
y, 2k?
George Michaelson 25:58
Oh, yes!
Geoff Huston 26:00
So the nature of time. How do we express time? Well, I live in
2026 at the moment. What does that really say? That's 2026 years
past an epoch. So time is actually expressed as an epoch event,
time zero and then a counter that counts the number of things,
years, minute, seconds since that epoch. So [George: yeah], in the
case of y, 2k the epoch was 1900 and the counter was years. And
the interesting thing was two digits of years,
George Michaelson 26:37
but only two digits worth two digits. People were cheap skates.
Geoff Huston 26:43
So at the end of 1999 when it became the year 2000 Guess what? New
epoch? Guess what? That two digit counter, which was your top 2
years, would reset to zero and everyone, oh, my god, Armageddon,
everything will collapse, digital nihilism. Let's all go and camp
in the woods for a month until we get this right? Yeah, yeah,
yeah. Lots of money was spent. Very little happened. I did have to
go into work that night, as did many other IT workers.
George Michaelson 27:11
I paid someone else to go into work because I had young kid. So
for me, it kind of made sense to give them $200 and they kept an
eye on the machines to see that everything was okay. But Geoff,
you're kind of eliding over we actually did have to go and check
what was going to happen. It wasn't just accident that we were all
okay. People invested energy making sure we would be okay.
Geoff Huston 27:36
Well, in some ways, though, it was COBOL, it wasn't every
programming language you use. And I found it kind of laughable
that we were running all our systems on Unix, and the year 2000
was not an event. Why? Oh, that's easy. That's very easy. Time in
Unix has an epoch for UNIX of the first of January, 1970 U, T, C,
and time is the number of seconds standard Atomic Time sections,
you know, that cesium clock those seconds the number of real
seconds that have elapsed since that epoch. So the year 2000 was a
switch from 95,000,200 you know, to the next number. It wasn't
anything special, [George: right] But when does time run out? Oh,
what a tough question. You know, in Unix and in the C language,
there's signed integers and unsigned integers, and variously, time
is the number of seconds since first of January, 1970 when you
express it as a signed counter, who knows why, but you can then
that counter will get to its maximum and then become a negative
number on the 19th of January, 2038, 12 years from now,
George Michaelson 28:56
Not a long time Geoff.
Geoff Huston 28:58
Windows, 32 bit signed value, 19th of January, 2038 oops. Now
maybe some of you got with it read the memo and said, We're going
to make it unsigned. Well, interestingly, you buy yourself a whole
bunch of time right up until around 2106 or so. But it'll still
happen. You'll still clock over. But there comes this next
question, which is sort of relevant and hard to answer. My big,
Grunty, expensive laptop, 64 bits of words. My integer value is 64
bits. What's two to the 64 seconds since the first of January,
1970 let me tell you, because I know you're interested. It's the
fourth of December in the year 292,277,026,596 and don't forget,
the Earth is only 13 though the universe is only 13 billion years
old. But this thing will keep on running for you know, a further
290-odd billion. Is.
George Michaelson 30:00
But do we have to agree, in some mechanistic way that we're going
to uplift time from an epoch into 64 bit numbers? Right?
Geoff Huston 30:10
Oh, no, we've already done. You look at your systems time as a 64
bit number, but I have time in many things, including some flea
bit, you know, eight bit machines or something. What's their time?
There's a whole bunch of embedded stuff that I can't get hold of,
even in my car. There are probably about 300 processors. They
probably all have a value of time somewhere. What's their time are
they using? Well, I don't know what they're using. Let's look at a
few more disasters. What about NTP itself? Oh, hang on, a second.
It carries the time in a quantity that is 32 bits. [George:
What!!!] I'm sorry, and the first 32 bits is the number of seconds
since the first of January, 1900
George Michaelson 30:55
not 1970
Geoff Huston 30:56
when does its time run?
George Michaelson 30:57
It has a different default.
Geoff Huston 30:59
If it's signed, it's it's unsigned, not signed, so you get a fair
deal of time back. But what's 4 billion seconds past the first of
January, 1900 I know you're desperate to know it's the seventh of
February, 2036, so two years before the signed Unix values go
totally ape. NTP goes totally ape. Oh, my God, this is not meant
to happen to us.
George Michaelson 31:23
This is not meant to happen. We're gonna have to fix this.
Geoff Huston 31:27
Oh, it's okay. My file system will fix it, because my file system,
thank you. Macintosh uses HFS plus. HFS plus. Hang on a second,
unsigned 32 bit number. What's the it's epoch. Well, bizarrely,
the first of January, 1904 so on the fifth of February, 2040
there's another crash in all of the HFS file systems. Well, it's
okay. I'm using fat 32/31 of December, 2107 and on and on it goes
even ZFS, which is 64 bits. Thank God that time's out on the
fourth of December in the year 292 billion. Thank God. But you
know, all those 32 bit things are kind of falling due at once.
[George: Yeah] And the real question is, how much of the world is
still in 32 bit, and can we find them?
George Michaelson 32:17
Yeah, because we're talking light bulbs and we're talking
thermostats, and we're talking fridges and we're talking cars with
300 processes. Would you like to guarantee there isn't one of them
that's going to think the time has gone back?
Geoff Huston 32:32
So let me posit the experiment. I take my laptop, I isolate it
from the rest of the world, I put in a date stamp of, I don't
know, a good stamp might be the 20th of January, 2038, or
something, one day after the epoch point. And then I try and
reconnect it to the network with this odd date. What's going to
happen? Well, I know right now my machine is going to go
absolutely flop and die, because nothing works. Nothing works. All
the certificates are invalid. File says nothing is working
because, not because it couldn't cope with the date rollover. But
you can't coexist in two widely disparate times. You can't implant
the time Martian back into the existing time continuum of the
Internet. So oops, I can't test for this. [George: Wow.] I can't.
I have to replicate an entire infrastructure shifted forward in
time to that point of epoch role. So unlike y 2k where at the time
we weren't that interdependent, you could just take a system, roll
its clock forward and see what went, bang, we can't do that this
time, not what we know of. Wow.
George Michaelson 33:38
There's going to have to be some work done on this Geoff.
Geoff Huston 33:41
You think so? I don't think so. I think we spent so much money on
Y2K and ended up going boring, waste of time, not going to do that
again, that we have talked ourselves out of it. And the answer
really is, you know, go and get 64 bit architecture. Do it right.
Just don't go through this problem. Get rid of the cheap stuff.
Well, what about your cars? What about you the oh, I don't know.
Do something. Who's going to do it? I don't know. [George: Wow].
Now the good news is I'll have retired. It may be 12 years away,
but that's 12 too many for my working life. So I watch from the
sidelines if I'm still around, but it is a here and now problem
for the next generation, and it's a tougher problem than Y2K but
it's poisoned by the Y2K debacle.
George Michaelson 34:31
Watching this one emerge is going to be fascinating. Geoff, that's
going to be an interesting one.
Geoff Huston 34:37
The other thing do you know, when we panicked for Y2K 1999 was
coming for years. Oh, he just said, yeah. So sometime, I don't
know, the first of January 2036, when we realize that there'll be
36 days until NTP goes bang in our face, we might panic about the
NTP date, and then someone might politely point out. The unsigned
Unix is going to go bang in about it two years from then. So we
might panic about that, but I think it's going to be a rolling
series of, oh my God. What's happening if we're still, you know,
if we still have computers around at the time, I'm not sure we're
going to do much better than that. Unfortunately.
George Michaelson 34:39
Oh dear,
Geoff Huston 34:39
Are you listening children? You have a problem. Let's see
George Michaelson 35:01
if ping is still running and we can do a follow up.
Geoff Huston 35:12
Not me, not me, find one of the children to talk about it.
George Michaelson 35:28
Thank you, Geoff, that's been fascinating
Geoff Huston 35:31
with that boys and girls see you next time.
George Michaelson 35:36
If you've got a story or research to share here on ping, why not
get in contact by email, to ping@apnic.net or via the APNIC social
media channels, also remember the measurement@apnic.net mailing
list on orbit is there to discuss and share relevant collaborative
opportunities, grants and funding opportunities, jobs and graduate
placings, or to seek feedback from the community on your own
measurement projects, be sure to check out the APNIC website for
All your resource and community needs until next time you
