Full Transcript for DNS Query Duplication

use it without charge. It wasn't anybody's land. It was everyone's land. It was the commons, and and the problem was it only worked if everyone was restrained in the number of beasts they sent out on the commons going eat grass. Because as soon as someone got greedy and sent out, you know, more, the place just went to rack and ruin because all the grass got eaten. It turned to mud. You know, bad things happen, and it actually led to, according to the economists who studied this, a subsequent move of the fencing of the commons to try and sort of, if you privatized it, people would look after it better. But there's a generic kind of principle about unconstrained human behavior [George: yeah], free resources tend to get exploited to the point of overconsumption and degradation. George Michaelson 3:49 It feels a little bit like a version of the prisoner's dilemma that the only good answer in the prisoner's dilemma is for them both to do the thing that's mutually beneficial, and if either one of them tries to get their own safety secured. It stuffs up the whole thing. The Commons. Everybody's got to be a good player. And the minute there's one bad apple that takes more than their free rights, the whole thing goes to pot. Geoff Huston 4:13 So, free resources get abused. They just do. Free money. Free whatever. And so there's this aspect of this applied to the Internet's domain name system. George Michaelson 4:25 Well, I'm not paying for it, so there's a quality that makes it feel to me very free. Geoff Huston 4:31 Well, if I run an authoritative server where I answer queries that other users have made, so if you want to know about potaroo.net. It's my server that's on the hook. Does anyone pay me? George Michaelson 4:44 So, answer here means specifically be the authoritative answer because there might be many other DNS players along the path who feel like they are air quotes providing the answer, but ultimately somebody has to have asked you, Geoff, in order. For that to happen, Geoff Huston 5:01 right? Someone needs to have sort of seeded the DNS with the original sets of you know data, and that's actually an act that doesn't get compensated by by the people who actually are making the queries. Queries are free, and if you observe from that, well, do people over query because they're free? That's a really interesting question. George Michaelson 5:25 Yeah, Geoff Huston 5:25 you see, the DNS works and it's really efficient because it uses the UDP protocol, the datagram protocol, the protocol that says every packet is an adventure. If I send you a UDP data packet. You may or may not get it. UDP doesn't care. You see, this actually, oddly enough, and let me digress a tiny little bit, was actually one of the issues with the early forays into packet networking in the 1960s. We had been used to a form of networking which, when we moved from analog to digital signals. We regarded the signal we were switching, even digital, as a consistent stream of packets, and we ended up switching the most incredibly expensive thing we could possibly switch: time. So that if you had a common resource switch between, say, eight people for the first eighth of a second, it was user A, and then the next eighth, user B, etc. In other words, you tried to sort of fair share amongst competing people by actually switching on on a common view of the time, which is interesting. But what it meant was that the network was carrying the integrity of the conversations. It didn't matter how clever or dumb you were when you attached your speaker and microphone to this network. The network itself would preserve that digital data as it switched it through the system. Circuit switch systems. When we went to packetization, there was a debate, and it was a furious debate. The debate was in a packet switch network. The packets don't go all the way from A to B. It's actually a series of of hops. There are intermediate switches, and so you eat the first switch that then passes it to the second switch, then the third, then the fourth. Now it doesn't really matter how you define the sequence of switches, whether it's predetermined, whether it's you know like the Internet dynamically discovered, doesn't really matter. But the issue is when one intermediate system hands a packet to the next intermediate system, does it care that the next one received it? Ooh, big question. Because if you care X 25 then you actually get a whole sequence of A sends it to B, B acts it back to A, B sends it to C, C acts it back to A. What does this mean? Well, if someone pops a packet in to this system, it will either come out really reliably, or the interior of the network will go, oh my God! All hope is lost. Let me destroy the circuit. Let me give you an indication of failure. George Michaelson 8:05 Because every hop along the path, people are saying, "Did you get it? And the person on the other end is saying, "No. In which case, you have the burden to retransmit it. So bounded in time, it's going to get there, or the whole thing's going to say, "We've got a problem", Geoff Huston 8:21 right? So X 25 based networks, which were really popular for a small window of time in the g 1980s worked on this principle of reliable hop by hop transmission. One of the networks that I did work on, DEC net, and a venerable protocol called DDCMP, and it was actually the same kind of thing. All the intermediate systems effectively explicitly acknowledge successful receipt of a packet, and so the thing injecting packets didn't have to care. Once you passed the packet to the network, you could assume that it was going

to come out the other end. What you couldn't assume was speed, because you know, if someone had stuffed it up, they'd sit there and try and try and try for quite some time before they gave up on the entire conversation. George Michaelson 9:08 So, if we bring this back to an Internet context and the idea of Internet protocol, the actual packets that are the Internet protocol packets, whether they be four or six, they're kind of stuck in the weeds on this, they're not saying absolutely must be on the network like DDCMP or X 25 They're actually saying don't care what I run on piece of string, giant fiber optic 100 gigabit channel, totally up to you. I'm just a packet, mate. Geoff Huston 9:35 Well, that was the genius, and I would say of the Internet that looking at everyone else who was busy doing reliable, reliable switching and effectively synthesizing a circuit out of these hop by hop packet elements, the Internet said, "Nah, every packet is an adventure. It's not the network's problem to identify and fix it. George Michaelson 9:57 I think every packet is an adventure. Needs to be on a T-shirt. I think that's brilliant. Geoff Huston 10:02 That's right. We're off on a big adventure. Some of us might not make it. Whoops. And that was kind of it. The argument was, and it got encapsulated in what what we call the end to end principle, that it was left to the two machines at either end who were communicating to figure out if any packets that hadn't made it, and repair the damage. George Michaelson 10:23 Right, Geoff Huston 10:24 and you kind of go, well, that was just IP, and you go, well, no, that was Ethernet. George Michaelson 10:28 No, Geoff Huston 10:28 it was Ethernet. It was a whole generation of successor data protocol, you know, designs that effectively said, well, this is a packet system. I don't actually care about the reliability of the packets. That's up to the communicating systems at either end. All I care about is just pushing it through, and if I fail, neh; let's just move on with our lives, right? George Michaelson 10:49 So we don't actually routinely construct applications and services into IP packets. We've got a layering model, and there's a choice moment coming, which I think brings us back to DNS and UDP. How do you make something on top of this have the behaviors you need to know is this unreliable or reliable? Geoff Huston 11:09 Yes, a lot of stuff use TCP. This reliable transport. Now TCP has overheads because there's no network state, but there really is state at either end. Instead of I to send you a packet, I have to go through a dance. Hi, George. I've got a number. Here's the number I'm thinking. The packet is going to be called a synchronized packet, so it's a SYN. George Michaelson 11:30 Hi, Geoff. Here's an ACK for that. And by the way, here's a number I made up that is my number that you should remember when I tell you things. Geoff Huston 11:39 Ah, George, I see you've received my packet and sended my number back number back to me, and I've got your number. Let me acknowledge your number. This is the third part of the three-way handshake, and now we both have sequencing numbers that the others using. George Michaelson 11:55 Terribly polite protocol, isn't it, Geoff? So polite. Oh, thank you for sending me that packet. Geoff Huston 12:01 Oh, thank you. Here's my first data packet. Now I know I said magic number 10. Well, it's got 10 bytes in it, so the new magic number will be 20. George Michaelson 12:11 Thank you, Geoff. I got that magic number 20. Here's my answer with magic number two. Geoff Huston 12:16 Ah, I see that you've received magic number 20. I will delete my copy because you have it. I don't need to resend it. Thank you, George. Oh, and by the way, let me acknowledge 2. You see how it kind of moves. That instead of doing hop by hop, did you get it? Did you get it? Did you get it? The two end systems play the did you get it dance by advancing their respective sequence numbers. The cost. Well, it's a three-way hands shape to start with. George Michaelson 12:44 So there's a bit of delay coming in. Geoff Huston 12:46 This whole thing about let's exchange sequence numbers works perfectly fine if we've got stuff to say to each other. But George only wanted to say hi. George Michaelson 12:56 Short conversation. Geoff Huston 12:58 That's two characters. That's that's two bytes, so it's kind of SYN SYN ACK ACK " hi" ACK FIN FIN ACK FIN. You know, that's a lot of packets to go " hi" George Michaelson 13:10 "hi" yeah. Geoff Huston 13:12 And when the approach came to think about implementing the domain name system and its resolution as an Internet protocol, it was very quickly realized that there were many, many points of conversation, many points of authority potentially, but each conversation would be tiny. Hi, do you know about dub dub dub.apnic.net George Michaelson 13:34 No, Geoff Huston 13:34 done. You know that's it. I don't need handshake handshake. You know, blah blah blah. It's just you know, fine. Tell me yes or no. I don't care. George Michaelson 13:44 I think tied up in this is sort of a subsidiary story that if we were talking about a medium that was so atrociously noisy, one packet in two never got through. You might be very uncomfortable with that class of behavior. But we are living in a world where the behavior assumption was most packets get through, and you'll need to say, "I didn't get that", or you'll need as a sender to say, "Looks like you didn't get that. I think I'll do it again". Is kind of down in the threshold where you're not really

paranoid about checking every single time. It mostly works. Geoff Huston 14:19 You know, history makes liars of us all, and you're right. That was the thinking when networks were unreliable. TCP kind of added strength to it, but if networks were basically reliable, things were fine. And then we put the Internet on mobile phones, and the loss rate, at least initially, was awesome. It was one of the few systems that had a loss rate of one in every 10 packets never made it. George Michaelson 14:42 Wow! Geoff Huston 14:42 It was pretty bad. And you know what collapsed TCP. George Michaelson 14:46 Seriously, because Geoff Huston 14:47 if the loss rate is high enough, it never gets. I suppose what I'd call a full head of steam. It always just sputters. Hi, I's one packet. Because the way TCP works is that if you send me an ACK for a packet. I'm going. Well, that's fine. Here's two packets, and you send me the ACKs for those. And I go, well, George, this is cooking. Here's 4. George Michaelson 15:08 I'll ACK that. Geoff Huston 15:08 8, 16, George Michaelson 15:10 I'll ACK that. Geoff Huston 15:11 You know, TCP has this system, which ironically is called slow start, which is anything but. [George: Yeah] it's actually exponential start, and it just zooms forward. If I've got a lot to say, TCP works brilliantly as long as as long as the loss rate isn't bad. If the loss rate's around two 3% max, TCP is okay George Michaelson 15:32 because you get the second part of the TCP behavior, which is under loss. The back off is extreme. Geoff Huston 15:38 Oh, it's very extreme. Oh, a packet loss! Panic! Ring the fire alarms! You know, ring the bell. Let's let's back right off. And so, oddly enough, you might have thought the TCP was sort of perfect for crappy networks. It's actually very imperfect. And what actually became really quite neat was UDP, because applications expected damage. It's kind of yeah, right? You know, lost a packet. It's not a big drama, and it was UDP was always intended for short transactions. Although some people abused it and started doing weird and wacko things using gigabytes of data over UDP, but they were loonies. George Michaelson 16:14 Well, they were loonies then, Geoff. But if we look at modern parlance and QUIC using UDP for megabytes of data has come back into fashion, but we've drifted slightly here. The point is, UDP was the way to say I'm not going to invest in a layer to do all of that reliability checking for me. I'm going to fling packets out with the least overhead, particularly when I'm asking questions like DNS questions. What's the address of this name? Geoff Huston 16:42 Single packet question, single packet answer. And if packets get damaged, it's left to the DNS application to figure it out and ask again somewhere else.

George Michaelson 16:52 Ah, now you just said ask again. Geoff Huston 16:55 Well, yes, because if you don't get an answer, what are you meant to do? Go and sulk? I'm sorry to get answer. Nobody likes me. It doesn't work like that. George Michaelson 17:05 It's not the " Eeyore" network, is it? Geoff Huston 17:08 Oh, absolutely not. I need an answer. I'm going to get one. I'm going to keep on asking til I get one. And so it's persisted under loss, right? Because that's the way we designed it. The application has to sort of make up for a network that isn't perfect. Okay, George Michaelson 17:25 right Geoff Huston 17:25 now I said before queries are free. So when do you re query? George Michaelson 17:31 Now I'm a good person, Geoff. I want to go to Internet heaven, so I'm going to say the right thing to do is to re query when you're really sure you have to, and not re query any other time. There you go. I'm a good player in the Commons. Geoff Huston 17:48 Well, you've also read RFC 1536 that said you'll wait for about five seconds. George Michaelson 17:53 That's a long time. Geoff Huston 17:54 Oh, continents have drifted in five seconds. You know, attention span. Entire generations of attention spans have come and gone. [George: Squirrel] exactly. But that was the advice, you know. Be patient, wait. Don't splutter packets into the network. And of course, since then, the networks have got bigger. But at the same time, we have got way more impatient. We are extremely impatient about this. And so, what's the best way to get an answer quickly, George Michaelson 18:24 Vincent? Twice, Vincent twice. Geoff Huston 18:27 Well, ask, ask, ask, ask, ask. You know, George Michaelson 18:29 are you there yet? Are we there yet? Are we there yet? Are we there yet? Geoff Huston 18:33 Yeah, I want to drive from Canberra to Sydney. Okay, how do I get there fast? George Michaelson 18:36 We there yet? Geoff Huston 18:37 Sent five cars, but I'm only in one of them. Sometimes it doesn't work. George Michaelson 18:43 That's the million man month statement about you can get a baby in under nine months if you get nine women to carry a baby, isn't it? Geoff Huston 18:51 That's the same kind of fallacy, you know. But oddly enough, in the DNS, it's not quite a fallacy. You see, George Michaelson 18:57 what? Geoff Huston 18:58 Well, the thing is, if this stuff is unreliable, and let's let's go to our magic 10% Then 10% of the time, I'm not going to get an answer, but I'm going to wait for five seconds. So 10% of folk will have a five second delay, and they'll ask again. 10% of them will not get an answer for another five seconds. So 1% of folk will be waiting for at least 10 seconds for an answer, and 10% of them won't get it, etc. etc. George Michaelson 19:24

Well, in a network of 2 billion users, 1% is an astronomically large number. Geoff Huston 19:30 Well, it is, and the whole issue comes that maybe that's the wrong strategy. George Michaelson 19:35 Yeah. Geoff Huston 19:36 So one thing you can do is not wait five seconds. I will work out reasonable approximation is for when you should respond. A packet can circle round the globe in a second. In fact, you can do it for half a second if you're lucky, and on you know the right pieces of cable. So I'll wait for a second because if you haven't answered it, George Michaelson 19:54 yeah. Geoff Huston 19:54 In fact, damn it, 300 milliseconds. George Michaelson 19:57 You know what? I would feel like this. Was still in the class of good behavior in the commons. If you're genuinely likely to get an answer sooner than that, waiting 300 feels like a pretty good fit for human centric delay, the experience of waiting for a thing to happen, and the reality sometimes you have to do it again. I could be happy in that world, and I'd love to believe it's another short episode, and you're telling me you went out and looked, and that's exactly what you saw because we're good citizens in the commons of the DNS. That's true, isn't it, Geoff? Geoff Huston 20:29 I'm sorry. Yeah, right. So we went and looked. We went and looked for a day using this ad based experiment, and we sent out a large number of names, oddly enough, because we don't want caching to happen in the DNS, which glosses over everything. All these names are unique, but that's easy. You just make a long name and you know add some variable fields every time. So on this day, we did 150 million unique names that got asked for by users, and they're unique, and they're unique in a couple of ways. One, the name only exists just once in time. That's it. Secondly, the only the only systems that can answer that DNS query are ones we run. So we see the query because we're the only people that are authoritative, and no one can cache it because it's unique. George Michaelson 21:18 You kind of brought the cache word to the table, and I think we're probably going to want to come back and talk about that in another episode because there is so much in cache. Cache is fascinating. Geoff Huston 21:30 If it wasn't for the cache, there wouldn't be the DNS. But that's a talk for a different day. George Michaelson 21:34 But the thing is, you've constructed a world with lots of unique labels where caching simply can't apply because they've never existed in the system before. Geoff Huston 21:43 Yep, 150 million of them in 24 hours. We were very busy, George Michaelson 21:46 good-sized experiment. Geoff Huston 21:48 These all over the planet, but we received 250 million. In fact, 270 million actual queries. So 150 million becomes 270 million in the DNS. George Michaelson 22:00 Well, it's not quite double, but it's a hell of a lot closer to double than a small percentage are repeated. That's most queries are repeated, Geoff. Geoff Huston 22:12 On average, most queries are repeated. On average, the DNS is going " to be sure, to be sure", and on average, it's it's sending out the query a second time. Now, one theory is the Internet's far worse than anyone ever suspected. They've all wasted their obligatory 300 milliseconds or whatever their time out might be. They haven't got an answer in time, and they're requery. George Michaelson 22:33 Well, I just want to put my hand up here and say we're recording this using a system that is transmitting live audio and video, and effectively, is like a two to 300 kilobit stream continuously from me to you and you to me. And if the level of behavior in that system demanded every single packet that we are doing was sent twice, I'm fairly comfortable we'd be seeing artifacts like delay or break up on this video, Geoff. That's what we used to see in old noisy systems when things could not reliably get data through. So if you and me can reliably send hundreds of kilobits of data to do this video and audio recording, I'm not buying that the DNS is suffering a noisy network and HAS to retransmit. You've just said it IS retransmitting, but you haven't said it HAD to do it. Geoff Huston 23:23 Well, we go a bit further because we're seeing all these queries, and so what we actually do is then go. Well, a duplicate query has the same query name a dot b dot c and the same query type a record quad a whatever because we've got a recording of every query we see, why don't we time the interval between the first query and the duplicate query? Why don't we actually see how long people are waiting? How patient? George Michaelson 23:50 That's quite interesting. So you don't actually need their clock. You just have to have your own clock recording the interval between seeing it once and seeing it again. Geoff Huston 24:00 Well, I just record the time when I see a query. Go back over the logs and go. Well, that was the duplicate of the first query. What's the time difference between the first and the second? George Michaelson 24:10 And if you're using a system like DNS cap or TCP dump pcaps, you actually automatically get a packet timer, don't you? An arrival time, which means you can do the packet interval, either from a log or from a data stream. Geoff Huston 24:23 Pcaps, pcap's is an extraordinary amount of data that wade through. I'm using logs. It's good enough for for this jazz. So yeah, the log says, you know, here's a time interval between queries. And we've said before, this is not time out. This is not a crappy network. So what sort of time interval do people like to do

because as I said, half of all the traffic we see, almost half, are duplicates. So of the duplicates, just looking at the ones that weren't the first, but it's a query that happened before, right? George Michaelson 24:55 Yeah, Geoff Huston 24:55 45% of all those duplicates come to us within 10. Milliseconds of the original. Oh my God! George Michaelson 25:03 Just before we drill down there, this is quite a skewed distribution, isn't it? I mean, a small number of these duplicates probably arrive a significant time after, like more than 300 milliseconds. Yeah. Geoff Huston 25:14 Well, a small number might well be the first time didn't make it. I'm twiddling my thumbs and I'm doing a DNS timeout, and I'm trying again. George Michaelson 25:22 Real loss. Geoff Huston 25:23 Real loss will cause retransmission. George Michaelson 25:25 Now to come back to the astonishment moment, almost half of the retransmitted packets are 10 milliseconds or less. [Geoff: Less] Holy cow, Geoff. Holy cow. Geoff Huston 25:39 Well, let's up the clock a little bit and go 20 milliseconds, 20 thousandths of a second, two hundredths of a second. That's not a timeout. No, 60-5% of all queries occur almost like we'd say tap tap, and it's the same query. George Michaelson 25:56 I am astounded, Geoff. I mean, to me, that shriek's deliberately coded. Geoff Huston 26:01 Okay, so let's dig a little deeper because it's always fun to dig a little deeper. Do you know about happy eyeballs, dear listeners? And George, maybe folk do. George Michaelson 26:11 Got a kind of vague feel. It's an odd thing, isn't it? It's this idea that we need to construct a way to handle how do we make six happen if four and six both exist, it's in that space, isn't it? Geoff Huston 26:23 It's in that space, and it used to be when I'm opening up a TCP session that a Linux operating system would wait for 108 seconds before saying that's not going to work. So if you didn't know if you could open it using V4 or V6, and you tried in say v4 the TCP part of the operating system would say, "Hey, I'm working on this three way handshake. Leave it to me. Oh, that SYN didn't make it. Let me time out and try again. Oh, that SYN didn't make it, and it'd take 108 seconds to go. I don't think it's working. Let's try V6 George Michaelson 26:59 I have a story for you from the deep dark days of the early Internet, when people still used the file called host. txt, and it took so long to serially check line by line that file to find where I was coming from that when I used the protocol called FTP to try and download code from London, getting it from a machine at the University of Maryland, the 30-second timer on the login would time out and disconnect me before it had worked out who I was. Geoff Huston 27:30 Steam-powered computers, George. They were wonderful. Turn off the steam. Go to electronics. Oh, you British! My apologies to all the British listeners. I didn't mean it. This whole idea, though, of duplicating became a technique where instead of waiting for conventional timers, you do try and speed up the process. So what we called happy eyeballs was you actually try a connection in v4 but impatiently. In fact, you do it in six first, and if that doesn't work, you do a rapid switch over to b4 George Michaelson 28:05 You kind of set two. Geoff Huston 28:07 It's a two horse race. George Michaelson 28:08 Two horse race. It's not wait for one then the other. You fire them off with a just little bit of an advantage. Geoff Huston 28:15 You handicap the one you prefer, and if you kind of go, oh well, if you've implemented both protocols over the same paths and it's roughly the same time. Whoever you handicap will win the race. So I handicap six. I send out the two horses at the same time. Six has a slight favor. You'll make a V6 connection if you can, but you'll rapidly fall back to v4 [George: r ight] and so you go. That's good. George Michaelson 28:36 Putting this into the context of the DNS, I'd love to say, Geoff, it must be that you're measuring happy eyeballs, then. That's where you're going, isn't it? Geoff Huston 28:43 No, not really. I'm trying to find an explanation as to why almost two thirds of all the duplicates occur within 20 milliseconds. Someone is doing this, bang bang, and happy eyeballs is part of it because certainly in the DNS, if I kind of want to make sure, then testing V4 and V6 is not a bad strategy. But there's more to it than that. In theory, I'm meant to have names with multiple authoritative name servers, and if I want to increase the resilience, a very clever recursive resolver, one that's making the queries, would go, "Ah, Geoff, I see you've got four name servers out there that can answer my question. Let me send the same question to all four at once". Queries are free. I'll just take the first one to answer. George Michaelson 29:28 Right, but Geoff, that wouldn't look like if you air quote it duplicates unless you measured across the whole system. Whereas you're at a single point from what you've been. Geoff Huston 29:37 So , that's where the argument breaks down. So let's see what other clever things people do. One of them, which we have noticed in a number of ISPs, is they're not really sure about their own DNS infrastructure. So, well, it's been in a dusty cupboard for 10 years. There's a bit of mold and bit rot growing on the shazzy. Might not be working. George Michaelson 30:01 The perversity of the Internet, where you trust the remote party more than your own equipment that you bought and set up to run service. Geoff Huston 30:10

Google's public DNS has an awesome reputation. Cloudflare's all one has an awesome reputation. So tell you what, let me send the query to one of these open resolvers, and at the same time, work diligently hard to resolve the name myself. If I get an answer from the open resolver, I'm done. Otherwise, I'll keep on working. Yeah, not a bad strategy. George Michaelson 30:30 I could believe in this strategy, Geoff. But again, Geoff Huston 30:33 you could believe it. That would occur. George Michaelson 30:35 I bring back to the table that would not, in strict sense, normally look like a duplicate to you would it because you would see different originators asking you the same question, Geoff Huston 30:46 same name. A duplicate is the same name, not the same asker. George Michaelson 30:50 Ah, ah ha. Geoff Huston 30:52 So v4 and V6 if they're asking for the same name, are duplicates. Cloudflare or Google and My Resolver are duplicates if they're asking for the same query type, the same query number. George Michaelson 31:04 Right. So this begins to sound like a possible collection of reasons you would see twice. Geoff Huston 31:10 An optimization. Yeah. You know, send off four race horses down different tracks. Whichever one comes back, I'll rub with it. You know, the v4 racehorse, the V6 the one to Google, the one to Cloudflare, the one to open DNS, whatever queries are free. Just send them all out. Take the first answer you get. You're done. Sounds good. George Michaelson 31:26 Mostly got a few if but maybes. But yeah, I could be there. Geoff Huston 31:30 So 1/5 of all the duplicates come from the same IP address. Jaw dropping moment. George Michaelson 31:36 Wait, I'm sorry. 45% of all the duplicates are rapid fire. Geoff Huston 31:40 Yes, George Michaelson 31:40 and 1/5 of the duplicates Geoff Huston 31:43 of the rapid fire duplicates of the ones that are tap tap come from the same IP address. George Michaelson 31:49 I'm not using two people to ask this question for me. I'm asking it twice. Wow, Geoff Huston 31:54 what's your problem, George? Sorry, what? George Michaelson 31:57 Didn't hear you. What did you say? Geoff Huston 31:58 Yeah, what? Sorry, what? Sorry, what? I'm in a group. Sorry, what? George Michaelson 32:02 You know, entering the room as a tragedy of the commons. I'm sitting here, frankly, aghast, Geoff, because this is taking a service that is notionally free, but actually has a publicly distributed cost. It is not free to provide the DNS either as an authority or as a public resolver. It's a cost, and it's a packet cost in terms of congestion on the wire. And you've just said to me there could be two or three or four or 10 duplications to get an answer from a system that we already agree is overwhelmingly reliable, even if you're

using the UDP shout into the void protocol, Geoff Huston 32:41 60% of the duplicates are only single duplicates. If I say one or two duplicates, so in other words, a total of three packets, I get up to 80% So most of the time, it really is tap tap, and it's so bizarre. Not tap tap tap. George Michaelson 32:55 But isn't that an abuse of the system? Geoff Huston 32:57 Well, from my selfish perspective, as the resolver, I just want the answer, dude, and I'm incredibly impatient. So someone else has the cost to bear. Tough on them. George Michaelson 33:06 Ah, Geoff Huston 33:07 it's the selfish. It's the tragedy of the commons all over again. And I'm sitting there going, "Wow, where is this?" Well, we know where it is because we know who's asking. George Michaelson 33:18 Oh dear, name and shame time. That comes up so often in these measurements, doesn't it? Geoff Huston 33:23 It does, but I'll be vague. I'll be general. The hot spots appear to be where there is, if you will, network infrastructure that may not be so crash hot. So oddly enough, the Indian subcontinent, including Bangladesh, we see a lot of duplicates. Double the rate coming out of the U.S. and Europe. That doesn't mean the U.S. and Europe don't have it. They do, but the rate is much higher in our servers that serve the Indian subcontinent, and oddly enough, in China and Hong Kong in that area, where again the duplicate rate is pretty much double what we see in other parts of the world. George Michaelson 33:57 So China, we know, has a national strategic view of DNS in the context of Internet, and it is extremely common for people to use intermediary devices because they can satisfy national delivery goals by buying legitimated systems that do the things the Chinese state considers necessary for safe Internet. So, for China, my spidey sense is saying this sounds like intermediaries. Geoff Huston 34:25 It's hard to say, George, because in our experiment we don't get to see what happened to start this off. We know an ad got placed in a browser. We know that, but we can't trace the DNS from that machine all the way through to the one machine we can see the machine that is asking our server the question. So between the user and that final sort of hopping off point, it's all black to us. We can't tell. But it does strike me as amazing that almost one half of the DNS are ghost cars. They're nothing. They're not real. They're just duplicates, and most of those duplicates occur almost back to back with the original query. And so the answer set, the answer set, is going out to all of them, duplicate and original. [George: Yeah], it's not as if I don't answer the duplicate. I've already answered one. George Michaelson 35:17 No, Geoff Huston 35:17 we answer everything. George Michaelson 35:18 I'm kind

of wishing there was the facility in the packet of DNS under this condition of behavior that allowed a flag to be put in the question, saying this is the second time I've asked. Because the thing you don't know is the origination order of these two packets. You know the arrival order. You can't actually know which of the two was. Geoff Huston 35:38 You can't tell, and I can't tell. But if it's the same IP address and all that's varied is the source port, does it matter which was the original? George Michaelson 35:46 No, not really. I mean, it's kind of irrelevant. Geoff Huston 35:49 Exactly. So, in some ways, the most bizarre behavior, where it's almost as if the switch is duplicating packets. Now, that used to happen, and I remember seeing it in about 1990, that certain switches from a certain really cheap vendor decided that when life was tough, instead of really switching one packet, they'd turn one packet into two for free. George Michaelson 36:11 Because the world always gets better when you put more stuff on the network, right? Geoff Huston 36:15 More packets equals more joy, you know. But those days, thankfully, George Michaelson 36:21 every packet's an adventure, Geoff. Geoff Huston 36:23 Even when it gets cloned, but you know they were cloned packets. These are point of difference. It's it's a DNS source port that's actually changed. George Michaelson 36:32 Wow, Geoff, this is really fascinating. I think you might need to move to other forms of experimentation, like doing packet trace on the client side resolver to see if you can actually see two packet queries being initiated out of some system. Geoff Huston 36:47 Oh, at some point we resort back to understanding which resolvers are doing this and understanding who runs them, and actually dropping them a note saying, "Hi, we've noticed something rather bizarre about the DNS. Yeah, you are running. Who's vendor's code? Are you using? What version is it? How have you customized it?" George Michaelson 37:06 So, dear listener, if you happen to know the reason that this behavior is now ubiquitous in the global Internet, Geoff would love to know. Geoff Huston 37:15 Oh, I would love to know. So much of the Internet happens without us looking, and even if you were running a system that did it, do you know? Nobody knows. George Michaelson 37:24 Oh God, I wouldn't have a clue. Geoff Huston 37:26 No, no one looks down at this detail. Quite frankly, even if you run an authoritative name server trying to find these rapid duplicates, you've actually really got to look hard. And so you kind of wonder, George, what else is out there of the Internet? Complete swap. The only difference is no one's bothered looking for it. [George: Yeah] you know it's out there, just waiting to be uncovered. George Michaelson 37:49 Folks, there's a world of measurement out there. We should all be looking at this stuff. That's really fascinating, Geoff. You've written this one up on your blog. Geoff Huston 37:56 It's coming up in the next couple of days. I'm just in the final process of crossing the eyes, dotting the Ts, or whatever you do with I's and T s. George Michaelson 38:04 By the time this one goes to air, I'm sure you'll have it online, so we'll include it in the web page with it. That was really great, Geoff. Thanks. Geoff Huston 38:11 Thanks, George. Cheers. Till next time. George Michaelson 38:14 If you've got a story or research to share here on Ping, why not get in contact by email to ping at APNIC. net or via the APNIC social media channels. Also, remember the measurement at APNIC. net mailing list on Orbit is there to discuss and share relevant collaborative opportunities, grants and funding opportunities, jobs or graduate placings, or to seek feedback from the community on your own measurement projects, be sure to check out the APNIC website for all your resource and community needs. Until next time.