Geoff Huston 0:00
What you've got to worry about is an engineering compromise
between two things. Nothing is permanent. I might want to change
stuff so that if I say to you, "Here's an answer, George, cache it
for a year, then if I want to change it, I've got to be awful
patient. And I'll tell you a tiny story here. When Slack were
signing their Slack.com domain name with DNSSEC. They managed, not
they people, others managed to shoot their foot off a bit and get
it wrong. And what hit the cache was that there is no names in
Slack.com, and that was cached for the TTL that they had suggest
at the time to live the cache cache lifetime, which in Slack's
case happened to be 24 hours. So here's this enormously popular
service, forcibly offline because DNS caching for 24 hours.
George Michaelson 1:02
You're listening to Ping, a podcast by APNIC discussing all things
related to measuring the Internet. I'm your host George
Michaelson. This time, I'm talking again to Geoff Huston from
APNIC Labs in his regular monthly spot on Ping. Geoff has been
looking at what would happen if you had to run a cold start on
your DNS, lots of things have a behavior that as long as the world
is running along fine, you get the benefit of all the things done
before. In the DNS, we call this the cache. Looking at power
supply networks, there's usually already some electricity in the
network from other operators, and if you're a generator, you can
use it to start your motor running before you start generating
electricity to then add to the network. But if you're disconnected
and turned off, and if the network is down and that free
electricity isn't there, cold start for generators is a big
resilience issue. It means planning for simpler engines to run, perhaps a small diesel generator or a nearby hydroelectric dam
that can operate just from gravity and use that to kickstart
things to get a little bit of electricity, so that the electricity
you depend on can start doing its job. Well, the DNS is no
different. In order to cold start your DNS service, there's a
minimum of information you need to find the things that you
actually depend on and will use to find everything else. You have
to populate these into your cache. That critical minimum set of
information and the sequence of queries you do to get things
bootstrapped and running is a fascinating and surprisingly large
amount of activity. Geoff, welcome back to Ping.
Geoff Huston 2:50
Hi, George. You know it's a names day again. [George It is]
Geoff Huston 2:54
I've always said the Internet actually runs on names, and
addresses are merely frippery. They're the adornment of the
Internet, peripheral names are what keeps this entire thing
running.
George Michaelson 3:04
Yeah. So the last time we talked about names, we kind of touched
quite lightly on this aspect of the DNS that makes it work at
scale, caching, which, when it comes down to it, is absolutely a
fundamental concept in computer science and modern engineering.
And I've got the feeling there's a lot more to be said about
cacheing in the DNS. Do you reckon we could have a go on that?
Geoff Huston 3:26
Well, we can, and I'd like to start with what I thought was an
absolutely brilliant presentation by the Internet Software
Consortium, the folk who bring you the Bind name resolution
software. , who works there, he was talking about cold start DNS.
So here's the question: I'm going to resolve a name. What name?
Oh, I'm going to resolve Teams.microsoft.com.
George Michaelson 3:51
Good name.
Geoff Huston 3:52
Good name. A lot of people use it, so we're not mucking around
here. And my resolver just crashed, and I've just started it,
and I'm now looking from the point it started. Says a cold
start, [George: right] How many queries will it take to resolve
teams.Microsoft.Com?
George Michaelson 4:11
In the ordinary course of events, your server hasn't just crashed,
and there's this thing in your server which is the cache of all
things that it's previously ever known and looked up, and you've
just said, "I ain't got that"
Geoff Huston 4:25
Well, I might have done to myself. I might have been the world's
unluckiest user and gone and asked a resolver that has just come
up from the dim, dark depths of breathing its first breath of
electronic oxygen. I might encounter one rarely. [George: yeah]
But the issue is, if I don't have a cache, if I have no jump
start, the DNS does work. You can start resolvers "cold". They do
all the time. And the issue is, what's the burden of giving it a
name to resolve if you don't have a cache to help you?
George Michaelson 4:56
Right. So this was the meat of Ondřej's talk.
Geoff Huston 4:59
Well. This was the surprise thought bubble because he actually
asked the room, and the kind of answers were seven.
Teams.microsoft. com's only got three labels. Okay, 12. The DNS is
a bit extravagant.
George Michaelson 5:11
I can imagine it asking a couple of ancillary questions along the
[Geoff: 15]
George Michaelson 5:16
You take the number of labels, and you think it's got to check for
four and six, and a bit of this, and a bit of that, bit of the
other. Yeah, I could go 15. I'm happy with 15.
Geoff Huston 5:26
If you're running the wrong version of bind, a couple of years
old, the true answer was 247.
George Michaelson 5:33
Oh, that's my bottom hitting the floor with surprise. 200 plus.
240 to [George: resolve one] To resolve the three label name.
George Michaelson 5:42
Oh man, that's broken.
Geoff Huston 5:44
So I went and tried it on my name www.potaroo.net. Three labels.
Yeah,
George Michaelson 5:50
good name,
Geoff Huston 5:51
fine name, fine website, all that kind of stuff. And I rebooted my
resolver and I ran TCP dump. I'm doing packet capture. So what do
I see? Well, the first thing it does is it goes. I've been given a
set of hints for the IP addresses of the root of the DNS, the root
name servers. Don't trust them, so I'll pick one at random, one of
these addresses I've been given, and ask it for all the name
servers of the root. That's query number one. Hi, root root
server. Tell me about the root name service. It's called the
priming query. I'm not sure how old the stuff is that came with
the software release. Let me refresh it with live information.
Query one.
George Michaelson 6:29
I could kind of imagine having picked one at random if it was a
bad hair day and that one at random was offline or unreachable.
There's an obvious outer loop. All right, pick a different one and
try again, and there are 13 of them, and they both have four and
six.
Geoff Huston 6:44
four and six, and there are 26 queries before you give up in disgust. But
by then, you'd have to have snipped the wire on your own computer
not to have got an answer.
George Michaelson 6:52
Yeah, one of these is going to have an anycast instance close to
you, and it's going to say, "Hi, here's the complete list of all
the other root servers addresses and names. Go for your life.
Geoff Huston 7:03
So I pick one. Doesn't matter anyone. And I go dot.dot.potaroo.net. I
wonder who are the name servers of.net. Don't forget, I've got no
cache, no brain, no nothing. I just had a root hints file. I've
now got a true version of the root servers, but now I need to know
the net servers. So I ask a root server at random.
George Michaelson 7:22
Just before we move on, Geoff, was that literally only two queries
for you, or did it actually turn out even that bootstrapping was
more than two queries?
Geoff Huston 7:31
No, no, one, one query,
George Michaelson 7:33
one query.
Geoff Huston 7:34
Ask a root name server that came pre-packaged for the name servers
of the zone dot.
George Michaelson 7:40
One question.
Geoff Huston 7:41
Query type NS. Query name dot. One question, one answer. [George: right]
The next question. Pick any of those IP addresses of the root name
servers. Just one. Pick them. Pick one and say, tell me the name
servers for .NET. Fine. One answer. Here is a list of the names of
the name servers, and because because the name servers are in
bailiwick, the name servers for .NET are in .NET A.GTLD servers
.NET, B. You know, then it actually gratuitously says, "Look, I
know strictly speaking, I should only tell you the names of the
name servers, but look, I'm going to be really friendly, and
you're not going to find out this information any other way
because you don't know the name servers until I tell you. I'm
going to tell you the IP addresses of those name servers.
George Michaelson 8:30
And again, this is extra data given for free, but in one query you
get a bundle of info.
Geoff Huston 8:37
I get this so called glue, and quite frankly, if I didn't get the
glue, I'd be stumped because I don't know who the name servers for
.NET. If you give me said, "Well, here's the name of a name server
in .NET that happens to be in .NET, I'm sitting there going,
"That's circular dude. I can't get out of this hole. You got to
give me some help. And so it does. The DNS gives you help, so
that's my second query, .NET. So I now have some name servers for
.NET and their IP addresses. Thank you, glue. And I go third
query. I'm after a name that's actually potaroo.net. So I ask any
of the name servers for potaroo.net. [George: yeah] And we're
now entering into my territory. I've got two name servers, and
they're both again in bailiwick. [George: yeah] Don't you love
that term, bailiwick?
George Michaelson 9:19
I love that word, bailiwick. It makes me think of a beef eater at
the Tower of London holding a halberd and saying, "You are in my
bailiwick.
Geoff Huston 9:28
It's a wonderful word that reflects the history of England.
bailiwick. Bailey is a French word, and it actually comes from
ribs and law and justice. It talks about the extent of power.
Bailey, French word Norman invasion. Thank you, Normans. The
Normans brought over a whole bunch of words about prisons, police,
and justice. They must have been a fun-loving crowd when they
invaded England.
George Michaelson 9:49
Well, and beef.
Geoff Huston 9:50
Yeah, we're going to lock all you up, you bastards, you Saxons.
We hate you. And Wic, Wic is a wonderful word. It's a Viking
word meaning village.
George Michaelson 9:58
The Internet is a village.
Geoff Huston 10:00
It's a classic term, and it actually got inherited into the DNS
from 18th century Americans, where the Americans wanted to show
how erudite they were, so they started picking up old medieval
English words and using them again. And someone in the DNS, and
I'm going to point the finger of accusation at Paul Vixie and go,
Paul. I blame you. Probably wrong. You revived this and sort of
resurrected the term Baliwick for the DNS. The rest of us haven't
heard this word ever. It's a wonderful word.
George Michaelson 10:29
Well, if it wasn't Paul Vixie, it might have been Paul Mockapetris.
So I think you're safe accusing somebody called Paul.
Geoff Huston 10:36
Oh, accuse Paul. Yeah, Paul. It's all your fault. So anyway, I'm
onto query three. I'm still resolving potaroo.net slowly.
George Michaelson 10:45
There's a moment here. A priory to use a bit of French. You don't
actually know if potaroo is a label directly in .net or if you're
going to be told I don't know. You have to ask somebody else,
Geoff Huston 11:00
George. I'm a very up-to-date, fashionable queryer. I follow
all the latest RFCs. If I was a bearded old timer, crusty from
the 1990s, I would actually ask the root servers, not for the
name servers of .NET. Nothing so silly. I would ask one of these
root name servers, "Tell me the IP address, the A record for
www.potaroo. net, and the root server would come back going in
a gently chiding fashion. No, I don't know that, Geoff. But you
seem to want to know my neighbor. The name servers for .net.
Go ask them. [George: Right] Oh, referral answer. Hi, .net.
Tell me about dub dub dub. potaroo.net. Geoff, calm down. I'm
a .net server. I can't answer that, but I can tell you. And
here's a referral for potaroo.net. Go ask there. But we stopped
doing that because it leaks information like a sieve.
George Michaelson 11:55
Oh well, a whole host of people have just learned that you, Geoff,
are looking for this label. Unassociated parties have learned a
lot about you and your query.
Geoff Huston 12:05
My dot very dot secret dot name dot I don't want other people.to
know I'm dot asking about dot yes that name
George Michaelson 12:12
yeah a lot of people know
Geoff Huston 12:14
the current fashion and most resolvers are now there and it's you
know up around well over half of users query minimization only ask
what's relevant at the point where you're asking. So when I say
you ask the root for the name server is the root, and you ask a
net name server for you know the next one. Now you're deliberately
only minimally exposing information. That's good. So now I've
asked for potaroo.net.
George Michaelson 12:37
So to keep in the background the idea, there's a conversation
about caching here. Having learnt everything about the route and
having learnt the things you need to know in future to ask about
.NET, you have immediately popped them into your local cache.
Geoff Huston 12:54
My resolver will never ask that question again for the time to
live, the cache time of those records. So I'm never going to ask
it again for quite some time, hours, [George: yeah]
sometimes even days, depending on what those GTLs have been set.
George Michaelson 13:08
Great, keep that thought alive. Keep on down the line, Geoff. You
haven't got to the end point yet.
Geoff Huston 13:13
Well, I've got glue records. I've done my third query: is the the
name service for Potaroo that says, "Go ask these people. I go,
aha! Here's the name dub dub dub dot potaroo. Tell me the answer,
oh name server for Potaroo, and it goes. Here's an IP address. You
know, have fun, fill your boots, do what you need. Four queries,
three labels, four queries, standard stuff. Yep.
George Michaelson 13:33
But Geoff teams.microsoft.com.
Geoff Huston 13:36
Ah, let's go to Teams.
George Michaelson 13:38
Three
labels and dub dub dub.potaroo.net. Yeah,
Geoff Huston 13:42
why isn't it four queries? Why is it 247?
George Michaelson 13:45
Yeah, this is not good.
Geoff Huston 13:47
Oh, teams.microsoft.com is not actually teams.microsoft.com. It's
an alias. There is a weird redirection pointer in the DNS called a
Cname, canonical name, and the DNS says, "Actually, you didn't
want that name". What do you mean? I didn't. You didn't. This is
not what you're looking for. With a wave of the hand.
George Michaelson 14:07
This is not the name you're looking for.
Geoff Huston 14:09
This is not the name you're looking for. You're looking for
teams.office. com. Oh, well, okay. Teams.office. com. Then I have
some very sad news for you. This too is not the name you are
really looking for. What do you mean? Oh, you happen to be in this
part of the world looking for this kind of name. Oh, so what kind
of name am I looking for? TMC-g2 dot tm four.office.com. Sounds
silly.
George Michaelson 14:35
I'm guessing that's also a C name, isn't it? Yeah, there's
Geoff Huston 14:38
Another C name coming. Teams-office-com dot s-triple 05 dot dual
s-msh.net. Oh, surely, to God, that's the right name. Surely, no,
there's another Cname.
George Michaelson 14:52
Can we just come back a moment because you started looking for a
thing in.com, which meant from cold start state you had to get the
ns. for Dot com, and you were then looking for Microsoft, which
absolutely exists, and you will have been told go and ask
Microsoft. You hit Microsoft, you're now at three queries, and you
enter this living hell of aliases, and one of those aliases says
no, no, you need to go ask . net, which means you have to go back
to the root and say oh root where's .net?
Geoff Huston 15:23
It's worse than you think. It's way worse than you think. You see,
teams.office.com. That's an interesting name, and that's the
second name in the Cname chain. Don't forget, every time you find
a Cname, you throw everything you've learned into the cache and
start again. Now you've got a cache, but each time the whole query
resolution process starts with the new name. And of course, with
Office. com, you're going to ask the .com servers for the name
servers of Office. com. Tell me all the name servers for Office.
com. Ooh, ns one hyphen o5 dot azure dns.com. Oh, that's in
bailiwick. That's great. Ah, there's more ns two hyphen o5 dot
azure hyphen dns.net. Damn, I've got to go and ask.net.
George Michaelson 16:13
That's not in bailiwick.
Geoff Huston 16:14
It's not ns three hyphen o5 dot azure hyphen dns.org. [George: what?]
Why do you punish me about this? I'm not finished. There's a
fourth name server for office. com ns four hyphen o five.as your
hyphen dns dot info. Jaw drops.
George Michaelson 16:31
That has to be a deliberate decision. Four discretely different
top level domains are the NSes behind this service.
Geoff Huston 16:40
I think I see inside their brains when they were working this out.
I want resilience. What if the .com servers aren't working? And
and normally the answer would be at this point abandoned all hope.
The Internet is dead. Go out and play. No, we will have a backup
in .net. What if they're both down? We will have a backup in .org.
What if all three are down? Well, we will have a backup in .info.
Someone's going to answer out of all of this. Well, possibly.
George Michaelson 17:05
So, their risk management belief was that four discretely
different top level domains got out of the if but maybe, and I'm
looking at that going four discretely different top level domains
costs a cold starter four bootstrapping queries on the route, and
you then intruded on top of that alias transiting.
Geoff Huston 17:29
Let's again look at these name servers for Office. com. By strict
definition, when I ask for the name servers of office.com. I will
get a list of these names, and I will get their IP addresses as
additional glue records,
George Michaelson 17:44
just like you did when you were bootstrapping at the root for
.com.
Geoff Huston 17:48
Should I use them? Ah, well, for the one called
ns105.azuredns.com, I've kind of got no choice. It's in bailiwick.
It's the domain I want to go to. I kind of have to accept the
glue, okay. But what if I get glue for .NET? No, I can't use that.
Why not? It's free, or it might be a lie. [George: right]
I have to resolve it from scratch.
George Michaelson 18:14
This is a moment which has been exploited by hackers in the past.
They're functionally doing a thing that I think is called
poisoning the DNS because
Geoff Huston 18:22
poisoning the cache that I deliberately give you the answer that
you want, but I add these additional IP addresses of name servers
that are mine, and I try and get you, Poor gullible you to believe
them and accept them on face value. And the real thing is, you
should only ever take the in bailiwick. Still love that word. The
in bailiwick addresses because you've got no choice. You're over a
barrel. But all the others don't use them. You could be being
misled. What does that mean?
George Michaelson 18:52
You've got to find the address.
Geoff Huston 18:54
I've got to ask them. I can't accept helpful hints. So I'm now
generating a lot of queries, but let's go one step further because
there's this whole thing about who is authoritative for the name
service of a delegated zone. Ooh, interestingly enough, if you go
through the 300 or so RFCs that define the DNS, you will find that
the parent, the one you've just asked, is not the authoritative
answer point. So when you ask a server in .NET, tell me the name
servers for potaroo.net and it says, "Well, NS1 and NS2. The
answer is, "Well, that's good, but is it right?" And the answer
is, "Well, you're asking the wrong person, dude". Just for a
second, suspend disbelief and ask one of those name servers, i.e.
the child, for the name servers of this name. Hi, ns1.potaroo.
Net. Tell me the name servers of potaroo.net. and you will find
that that answer is authoritative.
George Michaelson 19:52
It might not include the thing you just asked in that answer. It
could give you anything as an answer.
Geoff Huston 19:58
Whenever you get replicated in. In the DNS, you have to live with
the possibility that might not be replicated. It might be
different.
George Michaelson 20:04
Yeah, we touched a little bit on this when we discussed the DELEG
draft on this, and the idea of how to establish authoritatively
what's going on.
Geoff Huston 20:13
And the issue is, if you're appropriately paranoid, you must
always believe the child, not the parent. But that also means
you've got to query the child more queries.
George Michaelson 20:21
Another query. [Geoff: Another]. Is there no end to the number of
queries? I am beginning to get a sense why the number got so
large, Geoff.
Geoff Huston 20:31
We're building up to 247. We're building this up really quickly.
Part of it is: Do you want to be fast, or do you want to be
thorough? Do you want to follow every last buy way, or you just
want to get to the answer, and that's a really interesting
question. Should I resolve the name of every name server of
Office. com, all four of them, but I'm only going to use one of
them? Should I be assiduous and thorough? Should I be the right
version of obsessive compulsive that matches the impulses of
people who code, or should I just go? Damn, milliseconds are on
the line here. I'll have one with the first answer.
George Michaelson 21:05
Kind of want to divert just a moment. I like simple, and there's a
part of me that's sitting here going, the Internet police, the
ones who don't exist, probably should be hitting at Microsoft
saying Cname chains are a really bad idea, and the number of NSes
you put behind a domain-it's not your belief that it needs to be
four independent top-level domains. You've gone too far, and I'm
kind of thinking their rebuttal would be: a) you're not the
Internet police, and b) you're not the boss of me, and c) we
actually are using this to do some things because when you do that
Cname lookup, we're in the position of using a CDN and we can give
you a different answer to the guy over there.
Geoff Huston 21:05
Ah, you see, that's the thing. You see, I actually run a name
server for potaroo.net. and that's me being a crusty old greybeard
because that's not the thing you do. The thing you do is go to
anyone who says, "I'll host your web server for you", and you say,
"Well host it". Now, in the dim dark days, they'd say, "Well,
fine, let me run your entire DNS zone, potaroo.net". And I go,
"No, no, no, no, no, just the name server, dude",
George Michaelson 21:05
mate. That's a lot of asset I've handed over to you.
Geoff Huston 21:05
I'm not going to do that. And so the answer was: How do I lift?
How do I forklift one name out of one zone and drop it into the
control of somebody else? [George: yeah]
And the answer is Cnames, because what it's really saying is that
name dub dub dub.potaroo.net. That's not a real name. That's just
a bit of lipstick on the pig. The pig is over there in a different
sty, and it's got a different name, and it might be an Akamai name
or a Cloudflare name or whatever else, or an Azure name. But the
target of the Cname is where that forklift is happening. Okay.
Yeah. So let's go one step further because I can. All Akamai names
tend to be 2 Cnames deep. Why two? One to lift it out of its home.
Hi, that wasn't the name you thought. It's actually under control
of Academy as a host name. Two. Oh, where are you, George? You're
in Australia. I have a server in Brisbane. I'm going to Cname the
generic name to a server really close to you. [George: yeah]
So that when you go to this resource, the speed's going to rock.
George Michaelson 21:05
Right.
Geoff Huston 21:05
So I do the two to one to isolate, two to steer. Now Microsoft are
doing five because
George Michaelson 21:05
because the Internet police haven't knocked on their door,
Geoff Huston 21:05
and you kind of go, "couldn't you do better". And I think the
answer was, "Well, we could have, but we didn't [George: so]
yeah,
George Michaelson 21:16
you have given me a nice feeling about cold start involving a lot
more query than I possibly could have imagined, and I feel like
there are two stories in this that we probably could have some fun
talking about. And the first one is this word "cache, and so I get
a thing from you, and I decide to hang on to it for the length of
time you tell me you think you're going to keep it before you
might change it.
Geoff Huston 24:09
Ah, it's not a rule; it's only a guideline. Only
George Michaelson 24:12
guideline, and I'm in this beautiful position that I've built up a
model of all the things I've ever been told and the age they're
in. And the thing is, that model means I might not be asking all
these questions. I had to under cold start, but the next time I go
and look at guardian.net, I don't have to do this dance over.net.
Yeah, exactly. The second time you go to teams.microsoft.com or
one of your app starts, it comes back in an instant because it's
gone through that rigmarole and loaded the cache with basically
the answer. It's gone through all the intermediate stages of
discovering that answer. And interestingly, when you ask a second
time, it doesn't try and rediscover. It says to the recursive
resolver, "Hey, have you got teams. Microsoft. Com's IP address?"
Yeah, here it is. Do." Done. I'd look up.
If we think about the investment that people like Cloudflare and
Google have in public DNS, at the cost of them knowing everything
I do, they're probably not in a cold start world very often. I
mean, thermonuclear devices have not gone off in Western America,
so the service that operates out of Mountain View that feeds their
DNS had a cold start when they turned it on, but they've been
running continuously now for a decade, and it's not cold start.
They have a lot of data.
Geoff Huston 25:33
What you've got to worry about is an engineering compromise
between two things. Nothing is permanent. I might want to change
stuff so that if I say to you, "Here's an answer, George, cache it
for a year", then if I want to change it, I've got to be awful
patient. And I'll tell you a tiny story here. When Slack were
signing their Slack.com domain name with DNSSEC, they managed not
they people others managed to shoot their foot off a bit and get
it wrong. And what hit the cache was that there is no names in
Slack. com.
George Michaelson 26:10
Oh wow!
Geoff Huston 26:11
And that was cached for the TTL that they had suggested the time
to live, the cache cache lifetime, which in Slack's case happened
to be 24 hours. So here's this enormously popular service,
forcibly offline because DNS caching for 24 hours.
George Michaelson 26:27
But you did just say it's only a suggestion; it's not an
instruction.
Geoff Huston 26:31
Yes, and some folk make it longer, and some folk stupidly make it
shorter. But it is only a suggestion, and certainly when I say to
you, "Here's an answer. Remember it for three seconds." Most
recursive resolves would say, "Geoff, Geoff, get serious. If I'm
gonna going to process this answer, three's too short. Minimum of
pick a number: 15, 20 60, seconds.
George Michaelson 26:55
Yeah, but if you have a giant foot gun and you and the person
publishing decide, yeah, it's a day. Then your harm of
mispublishing has 24 hours of duration in the distributed world.
Geoff Huston 27:07
Right, but if I'm looking at teams.microsoft. com and I'm looking
at Cold Start, if I put a cache lifetime of how long, one second,
then once it expires, I'm back into the hell of rediscovery.
George Michaelson 27:20
Right, because cache ejection doesn't make you go and re query it.
You wait till somebody needs it, at which point you don't have the
cache state.
Geoff Huston 27:28
Well, people keep on tinkering, and some implementations will
query for you just in case to be sure, to be sure. But most of the
time, no. The next person who comes along will find basically an
empty cache and have to do it all over again. Now it won't be a
full top-down discovery, but certainly bits of this will have
disappeared. You've got to discover a lot more than you thought.
So there is this issue about what are you optimizing for:
performance or resilience? [George: yeah]
and that's the real trade-off when you're designing a DNS system.
I can put in 13 name servers. Look at the root, each with a v4 and
a V6 address. That's 26 points. Is anyone really going to hang
around while your recursive resolver unsuccessfully makes 26
queries, only to learn you pulled the Ethernet wire out of the
back of your laptop? No. So you know you don't need to go full
overboard. You know all guns blazing on resilience. You don't need
to over equip, and most folks think four is a good number for name
servers. Oddly enough, odd numbers are evil. Six is a good number.
Five is an evil number. Don't know why, but they never go the full
13 or something outrageous like that. That just really doesn't
work.
George Michaelson 28:40
Is there a strong sense whether it should all be in bailiwick or
there should be some out of bailiwick, or is there no guidance in
this?
Geoff Huston 28:47
There's a whole bunch of guidance saying do not accept gratuitous
offers of glue authority records for names that are outside the
bailiwick of the name server. Do not accept them because people
will lie and will poison. So, if you're after performance in
bailiwick, if you're after resilience, keep a few to one side. The
don't use all out of bailiwick because that means every time you
got a problem. Probably unwise to use all in bailiwick for the
same kind of issue around poison. But in bailiwick, certainly
better than out of bailiwick on the whole.
George Michaelson 29:20
Yeah, and we have no better mechanism than 2 Cnames to provide
hand off to a third party and localization to a specific location.
Geoff Huston 29:30
Well, 2 3 4, depends on what you're trying to achieve
administratively, but it is how do you lift names out of one space
and get them to another? There was one other technique that I
thought was truly, truly, truly awesomely weird, which was weird
games in the DNS. We called it Napter, N-A-P-T-R, and it actually
said something entirely different. A C name says, "Don't look at
me, look at this name, go and resolve that. A NAPTR is. Brain
explodes. A regular expression,
George Michaelson 30:03
A Turing complete program.
Geoff Huston 30:05
You shouldn't have been drinking while while I said that. It is
honestly apply this transform for the name you're looking at to
get a new name and go and look for that instead. So if I say that
star.com becomes star.foo.baz.com, then no matter what I ask for
in.com, the regular expression transform will match it to a new
name generically by using regular expressions.
George Michaelson 30:30
Wow, that's powerful. That's scary. That's complicated.
Geoff Huston 30:31
Well, it never took off. It never took off. It was very scary, and it was so
scary that they invented a new one called Snapter. What was
Snapter? That was the simple Napter because the power of regular
expressions mean. You know that game, The Towers of Hanoi.
George Michaelson 30:50
Oh yeah,
Geoff Huston 30:51
you could play that in the DNS with Napter Records. [George: yeah]
Geoff Huston 30:53
because the awesome power of regular expression. So, so there's a
number of techniques of how do you move something. You see, the
problem is that URLs have this sort of duality about what it is
and where it is. [George: Yeah]. And the conventional URL, as
distinct from a URI, a URL is actually a locator, and it's not an
identity. But we all use it as identities. So when I go and change
the organization of my web page and put things in different places
and so on? How do I keep all the old URLs, or do I not change the
organization of my web page because dot dot dot?
George Michaelson 31:33
Right, I'm a strong believer in the persistence of URLs because
the behavior, and if we invoke the caching word, there is a
massive amount of offline cached state held in everybody's browser
tabs and browser history, and if you randomly decide you don't
want to support the name, but I've gone into history search and
find the critical document under the old name, I'm in a world of
hurt.
Geoff Huston 31:56
You are, but the problem is that when people you know move to new
servers, go to a different commercial provider, etc. etc. They
actually want to reorganize their data. They actually want to
reorganize things, but they've got to do so with the same URLs,
which you actually can't use as a true locator. So you need
translation systems in the DNS.
George Michaelson 32:18
Yeah, if we walked into URIs as the default instance of how we do
this? We'd have the necessary level of indirection that Cnames is
providing, and we might be able to stave URIs as persistent
things.
Geoff Huston 32:32
Oh, everyone knows that academics live in ivory towers. And
George Michaelson 32:35
thank you for calling my apartment an ivory tower. Does that mean
it's worth more?
Geoff Huston 32:39
And no one listens to them now. It made sense, but it gelled
with the people at the time. Locators sort of, "Well, right,
done there. On to the next problem. [George: Yeah], you know,
I've spent two minutes thinking about this. On to the next. So
yeah, you're right.
George Michaelson 32:52
Last time we spoke, there was a component of this that was the
kick off was the tragedy of the commons, and I think there's a
flavor of risk of tragedy in the commons in the bootstrap
situation you've just described. Here we have something that is
truly critical for the global community that use Microsoft
product. And if you walk into a world where there was a local
power outage, you have a 250 query burden in the global DNS
framework for you to learn how to get to Microsoft. That doesn't
sound to me like a good situation.
Geoff Huston 33:25
Oh, optimizing for disasters is never fun, and I don't think
that's worthwhile. But what is more realistic is that bits of the
Internet may not be there at the exact time you need it. So when
we're talking about resilience, you actually do want to spread
your assets, but for performance you want to think hard. So in
domain name server names, out of domain, prefer in domain. Spread
them across many top levels versus don't don't spread them.
Whatever happened with Office. com and its use of com Info. Net
and org, that's a bad idea. It doesn't help resilience. A Cname
chain that's five long is three too long. It really is. I can
understand one. That's brilliant. I can understand the
administrative convenience of two: one to lift and a variable one
to translate. [George: Yeah]. I can't really understand three.
Don't use them. They're expensive. And as for the number of name
servers of a domain, two, if they're any cast, in other words,
they're implemented everywhere, but it's two distinct names with
each of them have a v4 and a V6 address. That's four addresses.
It's good. Three is kind of
George Michaelson 34:31
odd numbers are evil.
Geoff Huston 34:33
Odd numbers are evil, and what disaster are you thinking about?
And even when you get to four, it's kind of if you needed four,
each with two addresses, something's happening so badly that it's
not worth optimizing for. Don't bother. Two is a good answer. So
resilience and diversity, you've got to think about how and what
level you want to do it. [George: yeah]
Geoff Huston 34:51
trying to do it all in the DNS is pushing the DNS well beyond it.
So let's have a look at a couple more examples, and this one I
kind of like. Google. It has four name servers, which I would
argue is two too many, and they're all anycast. They're
everywhere. Each of those four have one IPv4 address and one IPv6
address.
George Michaelson 35:10
So eight different address instances can supply Google,
Geoff Huston 35:14
and they're all in Google.com.
George Michaelson 35:16
Is that resilient?
Geoff Huston 35:17
So if I ask for the name servers of Google.com, I get back an
answer that goes: Here are four names, and here are eight IP
addresses. Go for it. Nothing more to ask.
George Michaelson 35:28
But they all reside within Google's own assets of BGP routing,
global connectivity, and naming.
Geoff Huston 35:35
No, no, hang on, hang on. They might all reside within Google's
assets, but they're in different routed prefixes, but in both four
and six [George: right] They don't fate share. We're learning,
George. We are getting better at this, [George: right]
And so that's not a bad idea. Let's go over to Cloudflare. net.
George Michaelson 35:52
There's kind of an ancillary behavior that means Google's
operational practices now has to have a black book requirement:
you never ever play with the routing plane in all of these IP
spaces at the same time. If you have to do something,
Geoff Huston 36:07
that'd be shooting your own foot off. That would be a very bad
idea. Yeah,
George Michaelson 36:09
they're obviously moving risk into different aspects of reliability
engineering for provision of what it is to be Google.
Geoff Huston 36:16
Well, and it's a minimal piece of engineering, but I think it
gives resilience and performance. Let's go to Cloudflare.
Cloudflare. net. Five name servers. That's an evil number.
George Michaelson 36:27
Odd number.
Geoff Huston 36:27
But okay, five. The odd number. I'm suspicious too. They're
inventively numbered NS1, NS2, all up to NS5. Are they in
bailiwick? Yes, they're all in Cloudflare. net. So far, so good.
Each name has three IPV4 and three IPV6 addresses. Yeah, of course
you can do that. It's legal because it's not the names that
matter. It is the addresses. But now those five, [George: yeah]
right, have become 15 times two, which is 30.
George Michaelson 36:57
That's a lot of
Geoff Huston 36:58
IP addresses.
George Michaelson 36:59
I've tried A. I've tried B, I've tried C, I tried D.
Geoff Huston 37:02
You're going to give up. Resolvers aren't that persistent, and you
kind of go. I'm not sure what you were thinking, Cloudflare.
You've kind of gone overboard, and there's a whole bunch of stuff
that no one is ever going to do. It's good that you're in
bailiwick. Full points, putting three V4 and three V6 against
every name server. [George: yeah] I think you had an enthusiastic
afternoon. I think that was actually a silly thought. You should
have gone to sleep instead.
George Michaelson 37:26
I would have said do two and have three of each if you like, but
do two and have one of each probably would have been as good.
Geoff Huston 37:33
Direct correlation between names and addresses is for the DNS
simple. Stacking IP addresses up against a single name is I don't
know. I think the best adjective I can use is I think it's
distasteful. It's not illegal. It's not wrong. It just it offends
me. And you kind of go why instead of oh you're hanging too much
off a single name. Don't do that. [George: yeah] keep it simple,
one to one.
George Michaelson 37:57
So there are different strategies out there.
Geoff Huston 37:59
Oh, there are very different strategies because what you're trying
to do, and this gets back to the whole thing about what Microsoft
was trying to achieve and why, and even the whole thing about
caching, is that you're trying to optimize both performance and
resilience. Now, caching will give you most of the performance you
need, except when the cache expires, and if you're using short
time to live because I want to change things on the fly. So if you
have very long time to live in the cache, everything is difficult
to change. So if you want stuff that's malleable and efficient,
you've got to think about what information you're loading into the
DNS and where and how, and try and optimize the DNS performance.
Maybe there's an AI tool that will help you. Maybe there isn't. I
suspect you will find hallucinations more easily than you're going
to find truth. But there's an awful good stuff to read about this
topic, and some good presentations to watch about engineering both
resilience and performance in the DNS.
George Michaelson 38:56
So Ondřej's presentation covered off on a lot of this.
Geoff Huston 38:59
No, it just was a mere starting point as to there are hidden
gotchas in the DNS. His story was actually, you know, bind and 247
queries. [George: Yeah], we started trimming out the fat. We
started saying, "I'm not going to ask the child. I'm just going to
rely on the parent. I'm not going to resolve all the name seven
names. I'm only going to use one. I'm going to take the short path
to the answer and only go down the byways and you know diversions
if the shortest path didn't work. I'm going to code for optimal
performance, [George: yeah], and only exercise resilience if the
shortest path didn't work. So his story was about how engines
negotiate, but the DNS is two parts. It's the engines that go
through the minefield that the folk who will actually configure
their names set for those engines. And the answer is, don't get
too clever when you're setting up this sort of field of
coordination and configuration for your names. Keep it simple.
George Michaelson 39:56
But you've written up the story that you found as a trigger point
in this, the aspect of behavior here that is what we've just been
looking at.
Geoff Huston 40:04
I've written up a story about what I think it leads to. This is
why there are a couple of really good, you know, suggestions about
configuring your names. Use in bailiwick names. Avoid spreading
out to multiple top levels. Keep your Cnames short, and quite
frankly, don't have a whole heap of name server names or
addresses. You're just wasting everyone's time, including your
own. Four guidelines. If you do that, you won't get into too much
trouble. There are so many ways you can handle yourself. Try not
to use this set.
George Michaelson 40:33
Going to emerge as a formalism in IETF, or you think this one will
just stay in soft operational practice?
Geoff Huston 40:39
Oh, a lot of the DNS is folklore, and I think that's the way it
should be. I think the mania for trying to document everything and
having it change in two weeks has led to 2000. How much we up to
now? RFC 10,000. Coming soon. No one's read them all. We got to
the point where they've actually ceased to be a useful reference.
And in some ways, the folklore, the common knowledge in the DNS,
is more topical, more up to date, and more accurate. Oh,
George Michaelson 41:03
fightin talk.
Geoff Huston 41:05
Whoa. Well, you know, maybe that, or I'm just sick and tired of
writing RFCs, so I've stopped one or the other.
George Michaelson 41:10
That's been fascinating, Geoff. Thank you.
Geoff Huston 41:13
It's been fun, and I hope, dear listener, you're kept along for
the ride. Thanks indeed.
George Michaelson 41:18
If you've got a story or research to share here on Ping. Why not
get in contact by email to ping@apnic.net or via the APNIC social
media channels? Also, remember the measurement at APNIC. net
mailing list on Orbit is there to discuss and share relevant
collaborative opportunities, grants and funding opportunities,
jobs and graduate placings, or to seek feedback from the community
on your own measurement projects, be sure to check out the APNIC
website for all your resource and community needs. Until next
time.
What you've got to worry about is an engineering compromise
between two things. Nothing is permanent. I might want to change
stuff so that if I say to you, "Here's an answer, George, cache it
for a year, then if I want to change it, I've got to be awful
patient. And I'll tell you a tiny story here. When Slack were
signing their Slack.com domain name with DNSSEC. They managed, not
they people, others managed to shoot their foot off a bit and get
it wrong. And what hit the cache was that there is no names in
Slack.com, and that was cached for the TTL that they had suggest
at the time to live the cache cache lifetime, which in Slack's
case happened to be 24 hours. So here's this enormously popular
service, forcibly offline because DNS caching for 24 hours.
George Michaelson 1:02
You're listening to Ping, a podcast by APNIC discussing all things
related to measuring the Internet. I'm your host George
Michaelson. This time, I'm talking again to Geoff Huston from
APNIC Labs in his regular monthly spot on Ping. Geoff has been
looking at what would happen if you had to run a cold start on
your DNS, lots of things have a behavior that as long as the world
is running along fine, you get the benefit of all the things done
before. In the DNS, we call this the cache. Looking at power
supply networks, there's usually already some electricity in the
network from other operators, and if you're a generator, you can
use it to start your motor running before you start generating
electricity to then add to the network. But if you're disconnected
and turned off, and if the network is down and that free
electricity isn't there, cold start for generators is a big
resilience issue. It means planning for simpler engines to run, perhaps a small diesel generator or a nearby hydroelectric dam
that can operate just from gravity and use that to kickstart
things to get a little bit of electricity, so that the electricity
you depend on can start doing its job. Well, the DNS is no
different. In order to cold start your DNS service, there's a
minimum of information you need to find the things that you
actually depend on and will use to find everything else. You have
to populate these into your cache. That critical minimum set of
information and the sequence of queries you do to get things
bootstrapped and running is a fascinating and surprisingly large
amount of activity. Geoff, welcome back to Ping.
Geoff Huston 2:50
Hi, George. You know it's a names day again. [George It is]
Geoff Huston 2:54
I've always said the Internet actually runs on names, and
addresses are merely frippery. They're the adornment of the
Internet, peripheral names are what keeps this entire thing
running.
George Michaelson 3:04
Yeah. So the last time we talked about names, we kind of touched
quite lightly on this aspect of the DNS that makes it work at
scale, caching, which, when it comes down to it, is absolutely a
fundamental concept in computer science and modern engineering.
And I've got the feeling there's a lot more to be said about
cacheing in the DNS. Do you reckon we could have a go on that?
Geoff Huston 3:26
Well, we can, and I'd like to start with what I thought was an
absolutely brilliant presentation by the Internet Software
Consortium, the folk who bring you the Bind name resolution
software. , who works there, he was talking about cold start DNS.
So here's the question: I'm going to resolve a name. What name?
Oh, I'm going to resolve Teams.microsoft.com.
George Michaelson 3:51
Good name.
Geoff Huston 3:52
Good name. A lot of people use it, so we're not mucking around
here. And my resolver just crashed, and I've just started it,
and I'm now looking from the point it started. Says a cold
start, [George: right] How many queries will it take to resolve
teams.Microsoft.Com?
George Michaelson 4:11
In the ordinary course of events, your server hasn't just crashed,
and there's this thing in your server which is the cache of all
things that it's previously ever known and looked up, and you've
just said, "I ain't got that"
Geoff Huston 4:25
Well, I might have done to myself. I might have been the world's
unluckiest user and gone and asked a resolver that has just come
up from the dim, dark depths of breathing its first breath of
electronic oxygen. I might encounter one rarely. [George: yeah]
But the issue is, if I don't have a cache, if I have no jump
start, the DNS does work. You can start resolvers "cold". They do
all the time. And the issue is, what's the burden of giving it a
name to resolve if you don't have a cache to help you?
George Michaelson 4:56
Right. So this was the meat of Ondřej's talk.
Geoff Huston 4:59
Well. This was the surprise thought bubble because he actually
asked the room, and the kind of answers were seven.
Teams.microsoft. com's only got three labels. Okay, 12. The DNS is
a bit extravagant.
George Michaelson 5:11
I can imagine it asking a couple of ancillary questions along the
[Geoff: 15]
George Michaelson 5:16
You take the number of labels, and you think it's got to check for
four and six, and a bit of this, and a bit of that, bit of the
other. Yeah, I could go 15. I'm happy with 15.
Geoff Huston 5:26
If you're running the wrong version of bind, a couple of years
old, the true answer was 247.
George Michaelson 5:33
Oh, that's my bottom hitting the floor with surprise. 200 plus.
240 to [George: resolve one] To resolve the three label name.
George Michaelson 5:42
Oh man, that's broken.
Geoff Huston 5:44
So I went and tried it on my name www.potaroo.net. Three labels.
Yeah,
George Michaelson 5:50
good name,
Geoff Huston 5:51
fine name, fine website, all that kind of stuff. And I rebooted my
resolver and I ran TCP dump. I'm doing packet capture. So what do
I see? Well, the first thing it does is it goes. I've been given a
set of hints for the IP addresses of the root of the DNS, the root
name servers. Don't trust them, so I'll pick one at random, one of
these addresses I've been given, and ask it for all the name
servers of the root. That's query number one. Hi, root root
server. Tell me about the root name service. It's called the
priming query. I'm not sure how old the stuff is that came with
the software release. Let me refresh it with live information.
Query one.
George Michaelson 6:29
I could kind of imagine having picked one at random if it was a
bad hair day and that one at random was offline or unreachable.
There's an obvious outer loop. All right, pick a different one and
try again, and there are 13 of them, and they both have four and
six.
Geoff Huston 6:44
four and six, and there are 26 queries before you give up in disgust. But
by then, you'd have to have snipped the wire on your own computer
not to have got an answer.
George Michaelson 6:52
Yeah, one of these is going to have an anycast instance close to
you, and it's going to say, "Hi, here's the complete list of all
the other root servers addresses and names. Go for your life.
Geoff Huston 7:03
So I pick one. Doesn't matter anyone. And I go dot.dot.potaroo.net. I
wonder who are the name servers of.net. Don't forget, I've got no
cache, no brain, no nothing. I just had a root hints file. I've
now got a true version of the root servers, but now I need to know
the net servers. So I ask a root server at random.
George Michaelson 7:22
Just before we move on, Geoff, was that literally only two queries
for you, or did it actually turn out even that bootstrapping was
more than two queries?
Geoff Huston 7:31
No, no, one, one query,
George Michaelson 7:33
one query.
Geoff Huston 7:34
Ask a root name server that came pre-packaged for the name servers
of the zone dot.
George Michaelson 7:40
One question.
Geoff Huston 7:41
Query type NS. Query name dot. One question, one answer. [George: right]
The next question. Pick any of those IP addresses of the root name
servers. Just one. Pick them. Pick one and say, tell me the name
servers for .NET. Fine. One answer. Here is a list of the names of
the name servers, and because because the name servers are in
bailiwick, the name servers for .NET are in .NET A.GTLD servers
.NET, B. You know, then it actually gratuitously says, "Look, I
know strictly speaking, I should only tell you the names of the
name servers, but look, I'm going to be really friendly, and
you're not going to find out this information any other way
because you don't know the name servers until I tell you. I'm
going to tell you the IP addresses of those name servers.
George Michaelson 8:30
And again, this is extra data given for free, but in one query you
get a bundle of info.
Geoff Huston 8:37
I get this so called glue, and quite frankly, if I didn't get the
glue, I'd be stumped because I don't know who the name servers for
.NET. If you give me said, "Well, here's the name of a name server
in .NET that happens to be in .NET, I'm sitting there going,
"That's circular dude. I can't get out of this hole. You got to
give me some help. And so it does. The DNS gives you help, so
that's my second query, .NET. So I now have some name servers for
.NET and their IP addresses. Thank you, glue. And I go third
query. I'm after a name that's actually potaroo.net. So I ask any
of the name servers for potaroo.net. [George: yeah] And we're
now entering into my territory. I've got two name servers, and
they're both again in bailiwick. [George: yeah] Don't you love
that term, bailiwick?
George Michaelson 9:19
I love that word, bailiwick. It makes me think of a beef eater at
the Tower of London holding a halberd and saying, "You are in my
bailiwick.
Geoff Huston 9:28
It's a wonderful word that reflects the history of England.
bailiwick. Bailey is a French word, and it actually comes from
ribs and law and justice. It talks about the extent of power.
Bailey, French word Norman invasion. Thank you, Normans. The
Normans brought over a whole bunch of words about prisons, police,
and justice. They must have been a fun-loving crowd when they
invaded England.
George Michaelson 9:49
Well, and beef.
Geoff Huston 9:50
Yeah, we're going to lock all you up, you bastards, you Saxons.
We hate you. And Wic, Wic is a wonderful word. It's a Viking
word meaning village.
George Michaelson 9:58
The Internet is a village.
Geoff Huston 10:00
It's a classic term, and it actually got inherited into the DNS
from 18th century Americans, where the Americans wanted to show
how erudite they were, so they started picking up old medieval
English words and using them again. And someone in the DNS, and
I'm going to point the finger of accusation at Paul Vixie and go,
Paul. I blame you. Probably wrong. You revived this and sort of
resurrected the term Baliwick for the DNS. The rest of us haven't
heard this word ever. It's a wonderful word.
George Michaelson 10:29
Well, if it wasn't Paul Vixie, it might have been Paul Mockapetris.
So I think you're safe accusing somebody called Paul.
Geoff Huston 10:36
Oh, accuse Paul. Yeah, Paul. It's all your fault. So anyway, I'm
onto query three. I'm still resolving potaroo.net slowly.
George Michaelson 10:45
There's a moment here. A priory to use a bit of French. You don't
actually know if potaroo is a label directly in .net or if you're
going to be told I don't know. You have to ask somebody else,
Geoff Huston 11:00
George. I'm a very up-to-date, fashionable queryer. I follow
all the latest RFCs. If I was a bearded old timer, crusty from
the 1990s, I would actually ask the root servers, not for the
name servers of .NET. Nothing so silly. I would ask one of these
root name servers, "Tell me the IP address, the A record for
www.potaroo. net, and the root server would come back going in
a gently chiding fashion. No, I don't know that, Geoff. But you
seem to want to know my neighbor. The name servers for .net.
Go ask them. [George: Right] Oh, referral answer. Hi, .net.
Tell me about dub dub dub. potaroo.net. Geoff, calm down. I'm
a .net server. I can't answer that, but I can tell you. And
here's a referral for potaroo.net. Go ask there. But we stopped
doing that because it leaks information like a sieve.
George Michaelson 11:55
Oh well, a whole host of people have just learned that you, Geoff,
are looking for this label. Unassociated parties have learned a
lot about you and your query.
Geoff Huston 12:05
My dot very dot secret dot name dot I don't want other people.to
know I'm dot asking about dot yes that name
George Michaelson 12:12
yeah a lot of people know
Geoff Huston 12:14
the current fashion and most resolvers are now there and it's you
know up around well over half of users query minimization only ask
what's relevant at the point where you're asking. So when I say
you ask the root for the name server is the root, and you ask a
net name server for you know the next one. Now you're deliberately
only minimally exposing information. That's good. So now I've
asked for potaroo.net.
George Michaelson 12:37
So to keep in the background the idea, there's a conversation
about caching here. Having learnt everything about the route and
having learnt the things you need to know in future to ask about
.NET, you have immediately popped them into your local cache.
Geoff Huston 12:54
My resolver will never ask that question again for the time to
live, the cache time of those records. So I'm never going to ask
it again for quite some time, hours, [George: yeah]
sometimes even days, depending on what those GTLs have been set.
George Michaelson 13:08
Great, keep that thought alive. Keep on down the line, Geoff. You
haven't got to the end point yet.
Geoff Huston 13:13
Well, I've got glue records. I've done my third query: is the the
name service for Potaroo that says, "Go ask these people. I go,
aha! Here's the name dub dub dub dot potaroo. Tell me the answer,
oh name server for Potaroo, and it goes. Here's an IP address. You
know, have fun, fill your boots, do what you need. Four queries,
three labels, four queries, standard stuff. Yep.
George Michaelson 13:33
But Geoff teams.microsoft.com.
Geoff Huston 13:36
Ah, let's go to Teams.
George Michaelson 13:38
Three
labels and dub dub dub.potaroo.net. Yeah,
Geoff Huston 13:42
why isn't it four queries? Why is it 247?
George Michaelson 13:45
Yeah, this is not good.
Geoff Huston 13:47
Oh, teams.microsoft.com is not actually teams.microsoft.com. It's
an alias. There is a weird redirection pointer in the DNS called a
Cname, canonical name, and the DNS says, "Actually, you didn't
want that name". What do you mean? I didn't. You didn't. This is
not what you're looking for. With a wave of the hand.
George Michaelson 14:07
This is not the name you're looking for.
Geoff Huston 14:09
This is not the name you're looking for. You're looking for
teams.office. com. Oh, well, okay. Teams.office. com. Then I have
some very sad news for you. This too is not the name you are
really looking for. What do you mean? Oh, you happen to be in this
part of the world looking for this kind of name. Oh, so what kind
of name am I looking for? TMC-g2 dot tm four.office.com. Sounds
silly.
George Michaelson 14:35
I'm guessing that's also a C name, isn't it? Yeah, there's
Geoff Huston 14:38
Another C name coming. Teams-office-com dot s-triple 05 dot dual
s-msh.net. Oh, surely, to God, that's the right name. Surely, no,
there's another Cname.
George Michaelson 14:52
Can we just come back a moment because you started looking for a
thing in.com, which meant from cold start state you had to get the
ns. for Dot com, and you were then looking for Microsoft, which
absolutely exists, and you will have been told go and ask
Microsoft. You hit Microsoft, you're now at three queries, and you
enter this living hell of aliases, and one of those aliases says
no, no, you need to go ask . net, which means you have to go back
to the root and say oh root where's .net?
Geoff Huston 15:23
It's worse than you think. It's way worse than you think. You see,
teams.office.com. That's an interesting name, and that's the
second name in the Cname chain. Don't forget, every time you find
a Cname, you throw everything you've learned into the cache and
start again. Now you've got a cache, but each time the whole query
resolution process starts with the new name. And of course, with
Office. com, you're going to ask the .com servers for the name
servers of Office. com. Tell me all the name servers for Office.
com. Ooh, ns one hyphen o5 dot azure dns.com. Oh, that's in
bailiwick. That's great. Ah, there's more ns two hyphen o5 dot
azure hyphen dns.net. Damn, I've got to go and ask.net.
George Michaelson 16:13
That's not in bailiwick.
Geoff Huston 16:14
It's not ns three hyphen o5 dot azure hyphen dns.org. [George: what?]
Why do you punish me about this? I'm not finished. There's a
fourth name server for office. com ns four hyphen o five.as your
hyphen dns dot info. Jaw drops.
George Michaelson 16:31
That has to be a deliberate decision. Four discretely different
top level domains are the NSes behind this service.
Geoff Huston 16:40
I think I see inside their brains when they were working this out.
I want resilience. What if the .com servers aren't working? And
and normally the answer would be at this point abandoned all hope.
The Internet is dead. Go out and play. No, we will have a backup
in .net. What if they're both down? We will have a backup in .org.
What if all three are down? Well, we will have a backup in .info.
Someone's going to answer out of all of this. Well, possibly.
George Michaelson 17:05
So, their risk management belief was that four discretely
different top level domains got out of the if but maybe, and I'm
looking at that going four discretely different top level domains
costs a cold starter four bootstrapping queries on the route, and
you then intruded on top of that alias transiting.
Geoff Huston 17:29
Let's again look at these name servers for Office. com. By strict
definition, when I ask for the name servers of office.com. I will
get a list of these names, and I will get their IP addresses as
additional glue records,
George Michaelson 17:44
just like you did when you were bootstrapping at the root for
.com.
Geoff Huston 17:48
Should I use them? Ah, well, for the one called
ns105.azuredns.com, I've kind of got no choice. It's in bailiwick.
It's the domain I want to go to. I kind of have to accept the
glue, okay. But what if I get glue for .NET? No, I can't use that.
Why not? It's free, or it might be a lie. [George: right]
I have to resolve it from scratch.
George Michaelson 18:14
This is a moment which has been exploited by hackers in the past.
They're functionally doing a thing that I think is called
poisoning the DNS because
Geoff Huston 18:22
poisoning the cache that I deliberately give you the answer that
you want, but I add these additional IP addresses of name servers
that are mine, and I try and get you, Poor gullible you to believe
them and accept them on face value. And the real thing is, you
should only ever take the in bailiwick. Still love that word. The
in bailiwick addresses because you've got no choice. You're over a
barrel. But all the others don't use them. You could be being
misled. What does that mean?
George Michaelson 18:52
You've got to find the address.
Geoff Huston 18:54
I've got to ask them. I can't accept helpful hints. So I'm now
generating a lot of queries, but let's go one step further because
there's this whole thing about who is authoritative for the name
service of a delegated zone. Ooh, interestingly enough, if you go
through the 300 or so RFCs that define the DNS, you will find that
the parent, the one you've just asked, is not the authoritative
answer point. So when you ask a server in .NET, tell me the name
servers for potaroo.net and it says, "Well, NS1 and NS2. The
answer is, "Well, that's good, but is it right?" And the answer
is, "Well, you're asking the wrong person, dude". Just for a
second, suspend disbelief and ask one of those name servers, i.e.
the child, for the name servers of this name. Hi, ns1.potaroo.
Net. Tell me the name servers of potaroo.net. and you will find
that that answer is authoritative.
George Michaelson 19:52
It might not include the thing you just asked in that answer. It
could give you anything as an answer.
Geoff Huston 19:58
Whenever you get replicated in. In the DNS, you have to live with
the possibility that might not be replicated. It might be
different.
George Michaelson 20:04
Yeah, we touched a little bit on this when we discussed the DELEG
draft on this, and the idea of how to establish authoritatively
what's going on.
Geoff Huston 20:13
And the issue is, if you're appropriately paranoid, you must
always believe the child, not the parent. But that also means
you've got to query the child more queries.
George Michaelson 20:21
Another query. [Geoff: Another]. Is there no end to the number of
queries? I am beginning to get a sense why the number got so
large, Geoff.
Geoff Huston 20:31
We're building up to 247. We're building this up really quickly.
Part of it is: Do you want to be fast, or do you want to be
thorough? Do you want to follow every last buy way, or you just
want to get to the answer, and that's a really interesting
question. Should I resolve the name of every name server of
Office. com, all four of them, but I'm only going to use one of
them? Should I be assiduous and thorough? Should I be the right
version of obsessive compulsive that matches the impulses of
people who code, or should I just go? Damn, milliseconds are on
the line here. I'll have one with the first answer.
George Michaelson 21:05
Kind of want to divert just a moment. I like simple, and there's a
part of me that's sitting here going, the Internet police, the
ones who don't exist, probably should be hitting at Microsoft
saying Cname chains are a really bad idea, and the number of NSes
you put behind a domain-it's not your belief that it needs to be
four independent top-level domains. You've gone too far, and I'm
kind of thinking their rebuttal would be: a) you're not the
Internet police, and b) you're not the boss of me, and c) we
actually are using this to do some things because when you do that
Cname lookup, we're in the position of using a CDN and we can give
you a different answer to the guy over there.
Geoff Huston 21:05
Ah, you see, that's the thing. You see, I actually run a name
server for potaroo.net. and that's me being a crusty old greybeard
because that's not the thing you do. The thing you do is go to
anyone who says, "I'll host your web server for you", and you say,
"Well host it". Now, in the dim dark days, they'd say, "Well,
fine, let me run your entire DNS zone, potaroo.net". And I go,
"No, no, no, no, no, just the name server, dude",
George Michaelson 21:05
mate. That's a lot of asset I've handed over to you.
Geoff Huston 21:05
I'm not going to do that. And so the answer was: How do I lift?
How do I forklift one name out of one zone and drop it into the
control of somebody else? [George: yeah]
And the answer is Cnames, because what it's really saying is that
name dub dub dub.potaroo.net. That's not a real name. That's just
a bit of lipstick on the pig. The pig is over there in a different
sty, and it's got a different name, and it might be an Akamai name
or a Cloudflare name or whatever else, or an Azure name. But the
target of the Cname is where that forklift is happening. Okay.
Yeah. So let's go one step further because I can. All Akamai names
tend to be 2 Cnames deep. Why two? One to lift it out of its home.
Hi, that wasn't the name you thought. It's actually under control
of Academy as a host name. Two. Oh, where are you, George? You're
in Australia. I have a server in Brisbane. I'm going to Cname the
generic name to a server really close to you. [George: yeah]
So that when you go to this resource, the speed's going to rock.
George Michaelson 21:05
Right.
Geoff Huston 21:05
So I do the two to one to isolate, two to steer. Now Microsoft are
doing five because
George Michaelson 21:05
because the Internet police haven't knocked on their door,
Geoff Huston 21:05
and you kind of go, "couldn't you do better". And I think the
answer was, "Well, we could have, but we didn't [George: so]
yeah,
George Michaelson 21:16
you have given me a nice feeling about cold start involving a lot
more query than I possibly could have imagined, and I feel like
there are two stories in this that we probably could have some fun
talking about. And the first one is this word "cache, and so I get
a thing from you, and I decide to hang on to it for the length of
time you tell me you think you're going to keep it before you
might change it.
Geoff Huston 24:09
Ah, it's not a rule; it's only a guideline. Only
George Michaelson 24:12
guideline, and I'm in this beautiful position that I've built up a
model of all the things I've ever been told and the age they're
in. And the thing is, that model means I might not be asking all
these questions. I had to under cold start, but the next time I go
and look at guardian.net, I don't have to do this dance over.net.
Yeah, exactly. The second time you go to teams.microsoft.com or
one of your app starts, it comes back in an instant because it's
gone through that rigmarole and loaded the cache with basically
the answer. It's gone through all the intermediate stages of
discovering that answer. And interestingly, when you ask a second
time, it doesn't try and rediscover. It says to the recursive
resolver, "Hey, have you got teams. Microsoft. Com's IP address?"
Yeah, here it is. Do." Done. I'd look up.
If we think about the investment that people like Cloudflare and
Google have in public DNS, at the cost of them knowing everything
I do, they're probably not in a cold start world very often. I
mean, thermonuclear devices have not gone off in Western America,
so the service that operates out of Mountain View that feeds their
DNS had a cold start when they turned it on, but they've been
running continuously now for a decade, and it's not cold start.
They have a lot of data.
Geoff Huston 25:33
What you've got to worry about is an engineering compromise
between two things. Nothing is permanent. I might want to change
stuff so that if I say to you, "Here's an answer, George, cache it
for a year", then if I want to change it, I've got to be awful
patient. And I'll tell you a tiny story here. When Slack were
signing their Slack.com domain name with DNSSEC, they managed not
they people others managed to shoot their foot off a bit and get
it wrong. And what hit the cache was that there is no names in
Slack. com.
George Michaelson 26:10
Oh wow!
Geoff Huston 26:11
And that was cached for the TTL that they had suggested the time
to live, the cache cache lifetime, which in Slack's case happened
to be 24 hours. So here's this enormously popular service,
forcibly offline because DNS caching for 24 hours.
George Michaelson 26:27
But you did just say it's only a suggestion; it's not an
instruction.
Geoff Huston 26:31
Yes, and some folk make it longer, and some folk stupidly make it
shorter. But it is only a suggestion, and certainly when I say to
you, "Here's an answer. Remember it for three seconds." Most
recursive resolves would say, "Geoff, Geoff, get serious. If I'm
gonna going to process this answer, three's too short. Minimum of
pick a number: 15, 20 60, seconds.
George Michaelson 26:55
Yeah, but if you have a giant foot gun and you and the person
publishing decide, yeah, it's a day. Then your harm of
mispublishing has 24 hours of duration in the distributed world.
Geoff Huston 27:07
Right, but if I'm looking at teams.microsoft. com and I'm looking
at Cold Start, if I put a cache lifetime of how long, one second,
then once it expires, I'm back into the hell of rediscovery.
George Michaelson 27:20
Right, because cache ejection doesn't make you go and re query it.
You wait till somebody needs it, at which point you don't have the
cache state.
Geoff Huston 27:28
Well, people keep on tinkering, and some implementations will
query for you just in case to be sure, to be sure. But most of the
time, no. The next person who comes along will find basically an
empty cache and have to do it all over again. Now it won't be a
full top-down discovery, but certainly bits of this will have
disappeared. You've got to discover a lot more than you thought.
So there is this issue about what are you optimizing for:
performance or resilience? [George: yeah]
and that's the real trade-off when you're designing a DNS system.
I can put in 13 name servers. Look at the root, each with a v4 and
a V6 address. That's 26 points. Is anyone really going to hang
around while your recursive resolver unsuccessfully makes 26
queries, only to learn you pulled the Ethernet wire out of the
back of your laptop? No. So you know you don't need to go full
overboard. You know all guns blazing on resilience. You don't need
to over equip, and most folks think four is a good number for name
servers. Oddly enough, odd numbers are evil. Six is a good number.
Five is an evil number. Don't know why, but they never go the full
13 or something outrageous like that. That just really doesn't
work.
George Michaelson 28:40
Is there a strong sense whether it should all be in bailiwick or
there should be some out of bailiwick, or is there no guidance in
this?
Geoff Huston 28:47
There's a whole bunch of guidance saying do not accept gratuitous
offers of glue authority records for names that are outside the
bailiwick of the name server. Do not accept them because people
will lie and will poison. So, if you're after performance in
bailiwick, if you're after resilience, keep a few to one side. The
don't use all out of bailiwick because that means every time you
got a problem. Probably unwise to use all in bailiwick for the
same kind of issue around poison. But in bailiwick, certainly
better than out of bailiwick on the whole.
George Michaelson 29:20
Yeah, and we have no better mechanism than 2 Cnames to provide
hand off to a third party and localization to a specific location.
Geoff Huston 29:30
Well, 2 3 4, depends on what you're trying to achieve
administratively, but it is how do you lift names out of one space
and get them to another? There was one other technique that I
thought was truly, truly, truly awesomely weird, which was weird
games in the DNS. We called it Napter, N-A-P-T-R, and it actually
said something entirely different. A C name says, "Don't look at
me, look at this name, go and resolve that. A NAPTR is. Brain
explodes. A regular expression,
George Michaelson 30:03
A Turing complete program.
Geoff Huston 30:05
You shouldn't have been drinking while while I said that. It is
honestly apply this transform for the name you're looking at to
get a new name and go and look for that instead. So if I say that
star.com becomes star.foo.baz.com, then no matter what I ask for
in.com, the regular expression transform will match it to a new
name generically by using regular expressions.
George Michaelson 30:30
Wow, that's powerful. That's scary. That's complicated.
Geoff Huston 30:31
Well, it never took off. It never took off. It was very scary, and it was so
scary that they invented a new one called Snapter. What was
Snapter? That was the simple Napter because the power of regular
expressions mean. You know that game, The Towers of Hanoi.
George Michaelson 30:50
Oh yeah,
Geoff Huston 30:51
you could play that in the DNS with Napter Records. [George: yeah]
Geoff Huston 30:53
because the awesome power of regular expression. So, so there's a
number of techniques of how do you move something. You see, the
problem is that URLs have this sort of duality about what it is
and where it is. [George: Yeah]. And the conventional URL, as
distinct from a URI, a URL is actually a locator, and it's not an
identity. But we all use it as identities. So when I go and change
the organization of my web page and put things in different places
and so on? How do I keep all the old URLs, or do I not change the
organization of my web page because dot dot dot?
George Michaelson 31:33
Right, I'm a strong believer in the persistence of URLs because
the behavior, and if we invoke the caching word, there is a
massive amount of offline cached state held in everybody's browser
tabs and browser history, and if you randomly decide you don't
want to support the name, but I've gone into history search and
find the critical document under the old name, I'm in a world of
hurt.
Geoff Huston 31:56
You are, but the problem is that when people you know move to new
servers, go to a different commercial provider, etc. etc. They
actually want to reorganize their data. They actually want to
reorganize things, but they've got to do so with the same URLs,
which you actually can't use as a true locator. So you need
translation systems in the DNS.
George Michaelson 32:18
Yeah, if we walked into URIs as the default instance of how we do
this? We'd have the necessary level of indirection that Cnames is
providing, and we might be able to stave URIs as persistent
things.
Geoff Huston 32:32
Oh, everyone knows that academics live in ivory towers. And
George Michaelson 32:35
thank you for calling my apartment an ivory tower. Does that mean
it's worth more?
Geoff Huston 32:39
And no one listens to them now. It made sense, but it gelled
with the people at the time. Locators sort of, "Well, right,
done there. On to the next problem. [George: Yeah], you know,
I've spent two minutes thinking about this. On to the next. So
yeah, you're right.
George Michaelson 32:52
Last time we spoke, there was a component of this that was the
kick off was the tragedy of the commons, and I think there's a
flavor of risk of tragedy in the commons in the bootstrap
situation you've just described. Here we have something that is
truly critical for the global community that use Microsoft
product. And if you walk into a world where there was a local
power outage, you have a 250 query burden in the global DNS
framework for you to learn how to get to Microsoft. That doesn't
sound to me like a good situation.
Geoff Huston 33:25
Oh, optimizing for disasters is never fun, and I don't think
that's worthwhile. But what is more realistic is that bits of the
Internet may not be there at the exact time you need it. So when
we're talking about resilience, you actually do want to spread
your assets, but for performance you want to think hard. So in
domain name server names, out of domain, prefer in domain. Spread
them across many top levels versus don't don't spread them.
Whatever happened with Office. com and its use of com Info. Net
and org, that's a bad idea. It doesn't help resilience. A Cname
chain that's five long is three too long. It really is. I can
understand one. That's brilliant. I can understand the
administrative convenience of two: one to lift and a variable one
to translate. [George: Yeah]. I can't really understand three.
Don't use them. They're expensive. And as for the number of name
servers of a domain, two, if they're any cast, in other words,
they're implemented everywhere, but it's two distinct names with
each of them have a v4 and a V6 address. That's four addresses.
It's good. Three is kind of
George Michaelson 34:31
odd numbers are evil.
Geoff Huston 34:33
Odd numbers are evil, and what disaster are you thinking about?
And even when you get to four, it's kind of if you needed four,
each with two addresses, something's happening so badly that it's
not worth optimizing for. Don't bother. Two is a good answer. So
resilience and diversity, you've got to think about how and what
level you want to do it. [George: yeah]
Geoff Huston 34:51
trying to do it all in the DNS is pushing the DNS well beyond it.
So let's have a look at a couple more examples, and this one I
kind of like. Google. It has four name servers, which I would
argue is two too many, and they're all anycast. They're
everywhere. Each of those four have one IPv4 address and one IPv6
address.
George Michaelson 35:10
So eight different address instances can supply Google,
Geoff Huston 35:14
and they're all in Google.com.
George Michaelson 35:16
Is that resilient?
Geoff Huston 35:17
So if I ask for the name servers of Google.com, I get back an
answer that goes: Here are four names, and here are eight IP
addresses. Go for it. Nothing more to ask.
George Michaelson 35:28
But they all reside within Google's own assets of BGP routing,
global connectivity, and naming.
Geoff Huston 35:35
No, no, hang on, hang on. They might all reside within Google's
assets, but they're in different routed prefixes, but in both four
and six [George: right] They don't fate share. We're learning,
George. We are getting better at this, [George: right]
And so that's not a bad idea. Let's go over to Cloudflare. net.
George Michaelson 35:52
There's kind of an ancillary behavior that means Google's
operational practices now has to have a black book requirement:
you never ever play with the routing plane in all of these IP
spaces at the same time. If you have to do something,
Geoff Huston 36:07
that'd be shooting your own foot off. That would be a very bad
idea. Yeah,
George Michaelson 36:09
they're obviously moving risk into different aspects of reliability
engineering for provision of what it is to be Google.
Geoff Huston 36:16
Well, and it's a minimal piece of engineering, but I think it
gives resilience and performance. Let's go to Cloudflare.
Cloudflare. net. Five name servers. That's an evil number.
George Michaelson 36:27
Odd number.
Geoff Huston 36:27
But okay, five. The odd number. I'm suspicious too. They're
inventively numbered NS1, NS2, all up to NS5. Are they in
bailiwick? Yes, they're all in Cloudflare. net. So far, so good.
Each name has three IPV4 and three IPV6 addresses. Yeah, of course
you can do that. It's legal because it's not the names that
matter. It is the addresses. But now those five, [George: yeah]
right, have become 15 times two, which is 30.
George Michaelson 36:57
That's a lot of
Geoff Huston 36:58
IP addresses.
George Michaelson 36:59
I've tried A. I've tried B, I've tried C, I tried D.
Geoff Huston 37:02
You're going to give up. Resolvers aren't that persistent, and you
kind of go. I'm not sure what you were thinking, Cloudflare.
You've kind of gone overboard, and there's a whole bunch of stuff
that no one is ever going to do. It's good that you're in
bailiwick. Full points, putting three V4 and three V6 against
every name server. [George: yeah] I think you had an enthusiastic
afternoon. I think that was actually a silly thought. You should
have gone to sleep instead.
George Michaelson 37:26
I would have said do two and have three of each if you like, but
do two and have one of each probably would have been as good.
Geoff Huston 37:33
Direct correlation between names and addresses is for the DNS
simple. Stacking IP addresses up against a single name is I don't
know. I think the best adjective I can use is I think it's
distasteful. It's not illegal. It's not wrong. It just it offends
me. And you kind of go why instead of oh you're hanging too much
off a single name. Don't do that. [George: yeah] keep it simple,
one to one.
George Michaelson 37:57
So there are different strategies out there.
Geoff Huston 37:59
Oh, there are very different strategies because what you're trying
to do, and this gets back to the whole thing about what Microsoft
was trying to achieve and why, and even the whole thing about
caching, is that you're trying to optimize both performance and
resilience. Now, caching will give you most of the performance you
need, except when the cache expires, and if you're using short
time to live because I want to change things on the fly. So if you
have very long time to live in the cache, everything is difficult
to change. So if you want stuff that's malleable and efficient,
you've got to think about what information you're loading into the
DNS and where and how, and try and optimize the DNS performance.
Maybe there's an AI tool that will help you. Maybe there isn't. I
suspect you will find hallucinations more easily than you're going
to find truth. But there's an awful good stuff to read about this
topic, and some good presentations to watch about engineering both
resilience and performance in the DNS.
George Michaelson 38:56
So Ondřej's presentation covered off on a lot of this.
Geoff Huston 38:59
No, it just was a mere starting point as to there are hidden
gotchas in the DNS. His story was actually, you know, bind and 247
queries. [George: Yeah], we started trimming out the fat. We
started saying, "I'm not going to ask the child. I'm just going to
rely on the parent. I'm not going to resolve all the name seven
names. I'm only going to use one. I'm going to take the short path
to the answer and only go down the byways and you know diversions
if the shortest path didn't work. I'm going to code for optimal
performance, [George: yeah], and only exercise resilience if the
shortest path didn't work. So his story was about how engines
negotiate, but the DNS is two parts. It's the engines that go
through the minefield that the folk who will actually configure
their names set for those engines. And the answer is, don't get
too clever when you're setting up this sort of field of
coordination and configuration for your names. Keep it simple.
George Michaelson 39:56
But you've written up the story that you found as a trigger point
in this, the aspect of behavior here that is what we've just been
looking at.
Geoff Huston 40:04
I've written up a story about what I think it leads to. This is
why there are a couple of really good, you know, suggestions about
configuring your names. Use in bailiwick names. Avoid spreading
out to multiple top levels. Keep your Cnames short, and quite
frankly, don't have a whole heap of name server names or
addresses. You're just wasting everyone's time, including your
own. Four guidelines. If you do that, you won't get into too much
trouble. There are so many ways you can handle yourself. Try not
to use this set.
George Michaelson 40:33
Going to emerge as a formalism in IETF, or you think this one will
just stay in soft operational practice?
Geoff Huston 40:39
Oh, a lot of the DNS is folklore, and I think that's the way it
should be. I think the mania for trying to document everything and
having it change in two weeks has led to 2000. How much we up to
now? RFC 10,000. Coming soon. No one's read them all. We got to
the point where they've actually ceased to be a useful reference.
And in some ways, the folklore, the common knowledge in the DNS,
is more topical, more up to date, and more accurate. Oh,
George Michaelson 41:03
fightin talk.
Geoff Huston 41:05
Whoa. Well, you know, maybe that, or I'm just sick and tired of
writing RFCs, so I've stopped one or the other.
George Michaelson 41:10
That's been fascinating, Geoff. Thank you.
Geoff Huston 41:13
It's been fun, and I hope, dear listener, you're kept along for
the ride. Thanks indeed.
George Michaelson 41:18
If you've got a story or research to share here on Ping. Why not
get in contact by email to ping@apnic.net or via the APNIC social
media channels? Also, remember the measurement at APNIC. net
mailing list on Orbit is there to discuss and share relevant
collaborative opportunities, grants and funding opportunities,
jobs and graduate placings, or to seek feedback from the community
on your own measurement projects, be sure to check out the APNIC
website for all your resource and community needs. Until next
time.
