From 619bc4955ac05fa6702de82633eb82401bd4afed Mon Sep 17 00:00:00 2001 From: Pavel 'LEdoian' Turinsky Date: Mon, 30 Oct 2023 00:55:07 +0100 Subject: [PATCH] New article about DNS and forgoten AAAA for NS --- content/forgetting-dns6.rst | 312 +++++ content/images/forgetting-dns6/image1.svg | 1468 +++++++++++++++++++++ content/images/forgetting-dns6/image2.svg | 1468 +++++++++++++++++++++ content/images/forgetting-dns6/image3.svg | 1468 +++++++++++++++++++++ content/images/forgetting-dns6/image4.svg | 1468 +++++++++++++++++++++ content/images/forgetting-dns6/image5.svg | 1468 +++++++++++++++++++++ content/images/forgetting-dns6/layers.map | 5 + content/images/forgetting-dns6/make.sh | 12 + content/images/forgetting-dns6/source.svg | 1462 ++++++++++++++++++++ 9 files changed, 9131 insertions(+) create mode 100644 content/forgetting-dns6.rst create mode 100644 content/images/forgetting-dns6/image1.svg create mode 100644 content/images/forgetting-dns6/image2.svg create mode 100644 content/images/forgetting-dns6/image3.svg create mode 100644 content/images/forgetting-dns6/image4.svg create mode 100644 content/images/forgetting-dns6/image5.svg create mode 100644 content/images/forgetting-dns6/layers.map create mode 100755 content/images/forgetting-dns6/make.sh create mode 100644 content/images/forgetting-dns6/source.svg diff --git a/content/forgetting-dns6.rst b/content/forgetting-dns6.rst new file mode 100644 index 0000000..e7eca02 --- /dev/null +++ b/content/forgetting-dns6.rst @@ -0,0 +1,312 @@ +Do not forget about IPv6 DNS +@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + +:slug: forgetting-dns6 +:date: 2023-10-28 23:31 +:tags: ipv6-only, dns +:category: networking +:keywords: dns, ipv6, deployment, bug +:lang: en +:translation: false +:status: draft + +Do you think IPv6-only internet works OK? I am going to tell you that it does +not, but it is not immediately visible. TL;DR: The internet can be broken also +by forgetting to add AAAA records of the *nameservers*. This creates IPv4 +requirement for the resolving even when the target is reachable using IPv6. + +Quick recap +=========== + +Connecting to a website is easy, right? You type in the name, you get the front page. + +.. figure:: {static}/images/forgetting-dns6/image1.svg + :width: 50% + + This is a very naïve idea of connecting to a server. + +OK, it is a bit harder: the computer needs an IP address, so we need to use +this magic box called DNS. The flow looks something like this: + +.. figure:: {static}/images/forgetting-dns6/image2.svg + :width: 50% + + Slightly better, now we at least know the machine-readable address. + +And for IPv6-only, everything on the picture has to have IPv6 connectivity and AAAA DNS records. + +Reaching IPv4 land from IPv6-only +--------------------------------- + +There are :s:few many sites that still only support IPv4. To reach them, we +need someone, who can reach both the IPv4- and IPv6-land, to go there on our +behalf – a proxy. This proxy can be ad-hoc (I often use ``ssh -D``), or there +are well-known protocols like NAT64 with DNS64 to do that in a standard and +lightweight manner.[^This is very much the same as when you try to reach the +IPv4-public-land from IPv4-private-land, that is, from a private range of IP +addresses. This is called either just NAT, or NAT44, meaning IPv4-to-IPv4 NAT.] +In that case, the connection looks like this: + +.. figure:: {static}/images/forgetting-dns6/image3.svg + :width: 100% + + And now we can reach the whole internet. + +You might already know that you need some workaround like this to reach GitHub. +What I think you didn't know, you need similar workaround to reach the Wikipedia. + +Disclaimer: While I am sad that GitHub lives in the past and it is stupid that +they do not have IPv6, I do not want to shame Wikipedia in particular. +It is just an example I found out recently. I am aware of several other +sites suffering from the same problem, including at least one IPv6 test.[^There +are several more tests that do not even have the AAAA record, lol.] (It would +be nice if they added the missing piece in the puzzle, though.) + +Enter DNS +========= + +Our picture has one unexplored magic box: the DNS. As per the definition (which +I just made up and was not bothered to even fully formulate): + +> yada yada distributed database of records attached to the strings – domain +names. The records hold various information about the domain, depending on the type. + +There are three interesting types of records: A records give IPv4 addresses, +AAAA give IPv6 addresses, and NS give names of servers who know about the +particular subtree of the database. And to actually resolve the final AAAA +record, the (recursive) resolver starts at the *root zone* and tries to find +the answer.[^In my example, there is a recursive DNS resolver external to my +machine, in order not to complicate it too much. Yes, the real deployment is +often trickier.] The resolution algorithm can be visualised like this: + +.. figure:: {static}/images/forgetting-dns6/image4.svg + :width: 100% + + Yeah, it's a mess. + +There is one extra tricky bit: the NS records contain *names*, not addresses, +so when resolving, we need *two* queries for each layer (very simplified): +first we ask for the final domain (``blog.ledoian.cz``) and get a NS record +(when the server does not have the answer) and then we need to ask for the A or +AAAA record of the name from that record, so that we can connect to the server +mentioned in the NS record. + +You might start to see the issue. When the DNS was just a black box, we could +paint the whole picture green and call it a day. And from the regular user's +point of view, that is the case, just use some public DNS like 1.1.1.1, 8.8.8.8 +or 9.9.9.9. Oh, right, I meant these easy-to-remember addresses: +2606:4700:4700::1111, 2001:4860:4860::8888 and 2620:fe::fe, respectively. The +point is, they will give you the answer, because they are dual-stack, not IPv6-only. + +In a way, those servers (or other dual-stack resolvers) act like another proxy, +similar to the SSH, NAT64 and NAT44 ones mentioned earlier. This may not be +much of a problem for many people. But if you have any reason to use your own +recursive DNS server (privacy reasons, DNSSEC validation, ISP provides bad +service, you are the ISP, …) *inside* an IPv6-only network, you *will* have +issues.[^I have not yet tried to run a recursive DNS in a network with DNS64 +and NAT64. Could be fun :-D My wild guess is that I would need CLAT (i.e. the +full 464XLAT deployment) to make that work, since the resolver is connecting +directly to IPv4 addresses and would need to learn to use NAT64 to resolve +them. (The CLAT could be built right into the resolver, though).] + +Example: Wikipedia +================== + +Let's now see this in action. You know Wikipedia, right? And you can reach +Wikipedia on IPv6, right? It has an AAAA record (don't mind the CNAME, that +means that the server is really called some other way):: + + $ dig en.wikipedia.org AAAA + […] + en.wikipedia.org. 18737 IN CNAME dyna.wikimedia.org. + dyna.wikimedia.org. 323 IN AAAA 2a02:ec80:600:ed1a::1 + +And this record does work:: + + $ ncat --ssl 2a02:ec80:600:ed1a::1 443 < + […] + +But we can dig deeper: let's see what servers we are really asking:: + + $ dig en.wikipedia.org AAAA +trace + + ; <<>> DiG … <<>> en.wikipedia.org AAAA +trace + ;; global options: +cmd + . 78918 IN NS e.root-servers.net. + . 78918 IN NS f.root-servers.net. + . 78918 IN NS g.root-servers.net. + . 78918 IN NS h.root-servers.net. + . 78918 IN NS i.root-servers.net. + . 78918 IN NS j.root-servers.net. + . 78918 IN NS k.root-servers.net. + . 78918 IN NS l.root-servers.net. + . 78918 IN NS m.root-servers.net. + . 78918 IN NS a.root-servers.net. + . 78918 IN NS b.root-servers.net. + . 78918 IN NS c.root-servers.net. + . 78918 IN NS d.root-servers.net. + ;; Received 525 bytes from … in 0 ms + + org. 172800 IN NS c0.org.afilias-nst.info. + org. 172800 IN NS a2.org.afilias-nst.info. + org. 172800 IN NS a0.org.afilias-nst.info. + org. 172800 IN NS b0.org.afilias-nst.org. + org. 172800 IN NS b2.org.afilias-nst.org. + org. 172800 IN NS d0.org.afilias-nst.org. + ;; Received 788 bytes from 202.12.27.33#53(m.root-servers.net) in 24 ms + + wikipedia.org. 3600 IN NS ns0.wikimedia.org. + wikipedia.org. 3600 IN NS ns1.wikimedia.org. + wikipedia.org. 3600 IN NS ns2.wikimedia.org. + ;; Received 658 bytes from 2001:500:48::1#53(b2.org.afilias-nst.org) in 20 ms + + en.wikipedia.org. 86400 IN CNAME dyna.wikimedia.org. + ;; Received 94 bytes from 208.80.153.231#53(ns1.wikimedia.org) in 132 ms + +Hey, there are IPv4 addresses in there! I know, this is cheating, the output is +run from a dual-stack machine. But we can still simulate IPv6-only resolution +by adding ``-6`` flag:: + + $ dig en.wikipedia.org AAAA +trace -6 + + ; <<>> DiG … <<>> en.wikipedia.org AAAA +trace -6 + ;; global options: +cmd + . 78915 IN NS d.root-servers.net. + . 78915 IN NS e.root-servers.net. + . 78915 IN NS f.root-servers.net. + . 78915 IN NS g.root-servers.net. + . 78915 IN NS h.root-servers.net. + . 78915 IN NS i.root-servers.net. + . 78915 IN NS j.root-servers.net. + . 78915 IN NS k.root-servers.net. + . 78915 IN NS l.root-servers.net. + . 78915 IN NS m.root-servers.net. + . 78915 IN NS a.root-servers.net. + . 78915 IN NS b.root-servers.net. + . 78915 IN NS c.root-servers.net. + ;; Received 525 bytes from … in 0 ms + + org. 172800 IN NS d0.org.afilias-nst.org. + org. 172800 IN NS c0.org.afilias-nst.info. + org. 172800 IN NS b2.org.afilias-nst.org. + org. 172800 IN NS a0.org.afilias-nst.info. + org. 172800 IN NS b0.org.afilias-nst.org. + org. 172800 IN NS a2.org.afilias-nst.info. + ;; Received 816 bytes from 2001:500:2::c#53(c.root-servers.net) in 8 ms + + wikipedia.org. 3600 IN NS ns0.wikimedia.org. + wikipedia.org. 3600 IN NS ns1.wikimedia.org. + wikipedia.org. 3600 IN NS ns2.wikimedia.org. + couldn't get address for 'ns0.wikimedia.org': not found + couldn't get address for 'ns1.wikimedia.org': not found + couldn't get address for 'ns2.wikimedia.org': not found + dig: couldn't get address for 'ns0.wikimedia.org': no more + + +Some of those IPv4 addresses were benign – the respective servers are reachable +both using IPv4 and IPv6 address, or there is an alternative server that is +reachable using IPv6. That is the case for the root nameserver – in the second +case, we used C, which has IPv6 address (2001:500:2::c). In fact, the M server +also has IPv6 address, but dig chose the IPv4 one (it should not matter):: + + $ dig m.root-servers.net AAAA + […] + m.root-servers.net. 77991 IN AAAA 2001:dc3::35 + +But the latter case is the bigger issue. For the domain ``wikipedia.org`` there +are three nameservers:: + + $ dig wikipedia.org NS -6 + […] + wikipedia.org. 86400 IN NS ns0.wikimedia.org. + wikipedia.org. 86400 IN NS ns1.wikimedia.org. + wikipedia.org. 86400 IN NS ns2.wikimedia.org. + +This resolution is the last one that worked in IPv6-only mode, because none of +these three servers has AAAA record (some of them may have IPv6, which we do not learn about):: + + $ dig ns0.wikimedia.org AAAA + […] + ;; Got answer: + ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59468 + ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1 + +The NOERROR status says the domain name exists, but we got zero answers for +AAAA records. This is the case for all three nameservers. And here is the +ultimate picture of what is happening and what goes wrong. + +.. figure:: {static}/images/forgetting-dns6/image5.svg + :width: 100% + + The breakage in action + +Also note that the connection from the laptop to the DNS resolver may in fact +consist of a chain of several (caching, non-recursive) DNS resolvers, so that +the final DNS resolver can have dual-stack connectivity. + +The problems with this state +============================ + +So, what is the deal. We *just* need to have a dual-stack DNS resolver +somewhere, and that's it, no? Well, yes but actually no. + +There are two problems with this: First, this means that any new ISP needs to +have *at least some* IPv4 address, even if they intend to just use IPv6 +services. IPv4 addresses are scarce, `expensive +`__ and small +blocks `don't route well +`__, +which is not great both from the +new ISP's and from overal routing's point of view. It also hinders IPv6 +deployment and postpones IPv4 abandonment, needlessly. + +The second issue is that this is not very visible. We are building IPv6 world, +but deep inside, it still relies on IPv4, which might lead to great surprise +when we start cutting off IPv4 internet. And it might lead to false sense of +having IPv6 deployed, which is not true to the whole extent. + +Insert "It was DNS" meme here. + +Solution +======== + +The solution of this state is simple: get IPv6 connectivity to your +authoritative DNS server (or use another) and do not forget to add an AAAA +record for it in DNS. If the DNS server already has IPv6, it is probably just +adding a single line to the zone file (and a second one for the DNSSEC +signature), which should not be a big deal. + +Unfortunately, this needs to be done for the whole DNS chain. +Especially domain names at universities are infamous for very nested domains. +A domain name may looks like +``machine.department.location.faculty.university.some-common.suffix``. That +tree is deep, and so is the resolution of this problem. + +Amusing bug of almost good deployment +===================================== + +We have seen there may be multiple NS records for a domain, and thus +multiple nameservers. This is good for redundancy. But this does not mean that +the servers will have the same records – they are only supposed to give +equivalent answers. + +I have come across a silly misconfiguration: a domain which has several +nameservers, which serve a *slightly* different set of NS records for its +subdomain. Specifically, the servers which were only reachable using IPv4 were +*exactly* the servers that knew about one additional nameserver for the +subdomain, which, incidentally, was the *only* one that was IPv6-capable. + +So, while all the correct records were present in DNS (somewhat/somewhere), this still +meant that IPv6-only resolution was doomed to fail, because the IPv6 nameserver +chain was broken. diff --git a/content/images/forgetting-dns6/image1.svg b/content/images/forgetting-dns6/image1.svg new file mode 100644 index 0000000..bb5229e --- /dev/null +++ b/content/images/forgetting-dns6/image1.svg @@ -0,0 +1,1468 @@ + + + +blog.ledoian.cz diff --git a/content/images/forgetting-dns6/image2.svg b/content/images/forgetting-dns6/image2.svg new file mode 100644 index 0000000..37f4455 --- /dev/null +++ b/content/images/forgetting-dns6/image2.svg @@ -0,0 +1,1468 @@ + + + +DNSblog.ledoian.cz diff --git a/content/images/forgetting-dns6/image3.svg b/content/images/forgetting-dns6/image3.svg new file mode 100644 index 0000000..469483a --- /dev/null +++ b/content/images/forgetting-dns6/image3.svg @@ -0,0 +1,1468 @@ + + + +IPv6-landblog.ledoian.czProxyDNS diff --git a/content/images/forgetting-dns6/image4.svg b/content/images/forgetting-dns6/image4.svg new file mode 100644 index 0000000..539883a --- /dev/null +++ b/content/images/forgetting-dns6/image4.svg @@ -0,0 +1,1468 @@ + + + +IPv6-landblog.ledoian.czProxyDNSa.ns.nic.cz knowsabout *.czWhere is blog.ledoian.cz?ns.web4u.cz knowsabout *.ledoian.czWhere is blog.ledoian.cz?ns.web4u.cza.ns.nic.czk.root-servers.net2a01:4f8:c0c:36b8::7365:6c66Where is blog.ledoian.cz?blog.ledoian.cz is at2a01:4f8:c0c:36b8::7365:6c66 diff --git a/content/images/forgetting-dns6/image5.svg b/content/images/forgetting-dns6/image5.svg new file mode 100644 index 0000000..204abeb --- /dev/null +++ b/content/images/forgetting-dns6/image5.svg @@ -0,0 +1,1468 @@ + + + +IPv6-landen.wikipedia.orgDNSIt is possible to use IPv4only if both ends support it.Without this answer, we don't knowWikipedia's IPv6 address!Dual-stackresolvere.g. Google diff --git a/content/images/forgetting-dns6/layers.map b/content/images/forgetting-dns6/layers.map new file mode 100644 index 0000000..e4cd194 --- /dev/null +++ b/content/images/forgetting-dns6/layers.map @@ -0,0 +1,5 @@ +image1.svg:img1:-2 -2 54.997375 16.451033 +image2.svg:img2:-2 -2 54.997375 33.725821 +image3.svg:img3:-2 -2 99.143173 42.9856 +image4.svg:img4 D3|img3:-2 -2 115.76167 66.238632 +image5.svg:img5:-2 -2 92.378784 61.578362 diff --git a/content/images/forgetting-dns6/make.sh b/content/images/forgetting-dns6/make.sh new file mode 100755 index 0000000..3bb9a58 --- /dev/null +++ b/content/images/forgetting-dns6/make.sh @@ -0,0 +1,12 @@ +#!/bin/sh + +set -eux + +while IFS=: read fn layers viewbox; do + # Pass 1: pick correct layers + sed -re "/$layers/"' { N; s/none/inline/; }' source.svg > "$fn" + # Pass 2: crop images + inkscape "$fn" -D --export-overwrite + # Pass 3: add margins + sed -ri 's/^ viewBox="[^"]*"/ viewBox="'"$viewbox"'"/' "$fn" +done < layers.map diff --git a/content/images/forgetting-dns6/source.svg b/content/images/forgetting-dns6/source.svg new file mode 100644 index 0000000..349478c --- /dev/null +++ b/content/images/forgetting-dns6/source.svg @@ -0,0 +1,1462 @@ + + + +