blog/content/forgetting-dns6.rst

Do not forget about IPv6 DNS
@@@@@@@@@@@@@@@@@@@@@@@@@@@@

:slug: forgetting-dns6
:date: 2024-01-24 03:20
:tags: ipv6-only, dns
:category: networking
:keywords: dns, ipv6, deployment, bug
:lang: en
:translation: false
:status: published

Do you think IPv6-only internet works OK? I am going to tell you that it does
not, but it is not immediately visible. TL;DR: The internet can be broken also
by forgetting to add AAAA records of the *nameservers*. This creates IPv4
requirement for the resolving even when the target is reachable using IPv6.

Quick recap
===========

Connecting to a website is easy, right? You type in the name, you get the front page.

.. figure:: {static}/images/forgetting-dns6/image1.svg
   :width: 50%
   
   This is a very naïve idea of connecting to a server.

OK, it is a bit harder: the computer needs an IP address, so we need to use
this magic box called DNS. The flow looks something like this:

.. figure:: {static}/images/forgetting-dns6/image2.svg
   :width: 50%

   Slightly better, now we at least know the machine-readable address.

And for IPv6-only everything on the picture has to have IPv6 connectivity and AAAA DNS records.

Reaching IPv4 land from IPv6-only
---------------------------------

There are few^H^H^Hmany sites that still only support IPv4. To reach them, we
need someone who can reach both the IPv4- and IPv6-land to go there on our
behalf – a proxy. This proxy can be ad-hoc (I often use ``ssh -D``) or there
are well-known protocols like NAT64 with DNS64 to do that in a standard and
lightweight manner. [#nat44]_
In that case, the connection looks like this:

.. figure:: {static}/images/forgetting-dns6/image3.svg
   :width: 100%

   And now we can reach the whole internet.

You might already know that you need some workaround like this to reach GitHub.
What I think you might not know, you need similar workaround to reach the Wikipedia.

Disclaimer: I like Wikipedia and this is not meant to shame them, just use as
an example. I am aware of several other sites suffering from the same problem,
including at least one IPv6 test. [#test-aaaa]_ (It would be nice if they added
the missing piece in the puzzle, though.)

Enter DNS
=========

Our picture has one unexplored magic box: the DNS. As per the definition (which
I just made up and was not bothered to even fully formulate):

    yada yada distributed database of records attached to the strings – domain
    names. The records hold various information about the domain depending on the type.

There are three interesting types of records: A records give IPv4 addresses,
AAAA give IPv6 addresses and NS give names of servers who know about the
particular subtree of the database. And to actually resolve the final AAAA
record the (recursive) resolver starts at the *root zone* and tries to find
the answer. [#dns-simplification]_ The resolution algorithm can be visualised like this:

.. figure:: {static}/images/forgetting-dns6/image4.svg
   :width: 100%

   Yeah, it's a mess.

There is one extra tricky bit: the NS records contain *names*, not addresses,
so when resolving we need *two* queries for each layer (very simplified):
first we ask for the final domain (``blog.ledoian.cz``) and get a NS record
(when the server does not have the answer) and then we need to ask for the A or
AAAA record of the name from that record, so that we can connect to the server
mentioned in the NS record. (This allows a nameserver to be made redundant
and/or reside on other types of network.)

You might start to see the issue. When the DNS was just a black box, we could
paint the whole picture green and call it a day. And from the regular user's
point of view, that is the case, just use some public DNS like 1.1.1.1, 8.8.8.8
or 9.9.9.9. Oh, right, I meant these easy-to-remember addresses:
2606:4700:4700::1111, 2001:4860:4860::8888 and 2620:fe::fe, respectively. The
point is, they will give you the answer because they are dual-stack, not IPv6-only.

In a way, those servers (or other dual-stack resolvers) act like another proxy,
similar to the SSH, NAT64 and NAT44 ones mentioned earlier. This may not be
much of a problem for many people. But if you have any reason to use your own
recursive DNS server (privacy reasons, DNSSEC validation, ISP provides bad
service, you are the ISP, …) *inside* an IPv6-only network, you *will* have
issues. [#dns-behind-nat64]_

Example: Wikipedia
==================

Let's now see this in action. You know Wikipedia, right? And you can reach
Wikipedia on IPv6, right? It has an AAAA record (don't mind the CNAME, that
means that the server is really called some other way)::

    $ dig en.wikipedia.org AAAA
    […]
    en.wikipedia.org.	18737	IN	CNAME	dyna.wikimedia.org.
    dyna.wikimedia.org.	323	IN	AAAA	2a02:ec80:600:ed1a::1

And this record does work::

    $ ncat --ssl 2a02:ec80:600:ed1a::1 443 <<GO
    GET /wiki/Main_Page HTTP/1.1
    Host: en.wikipedia.org
    
    GO
    HTTP/1.1 200 OK
    […]
    content-language: en
    content-type: text/html; charset=UTF-8
    content-length: 98078

    <!DOCTYPE html>
    […]

But we can dig deeper: let's see what servers we are really asking::

    $ dig en.wikipedia.org AAAA +trace

    ; <<>> DiG … <<>> en.wikipedia.org AAAA +trace
    ;; global options: +cmd
    .			78918	IN	NS	e.root-servers.net.
    .			78918	IN	NS	f.root-servers.net.
    .			78918	IN	NS	g.root-servers.net.
    .			78918	IN	NS	h.root-servers.net.
    .			78918	IN	NS	i.root-servers.net.
    .			78918	IN	NS	j.root-servers.net.
    .			78918	IN	NS	k.root-servers.net.
    .			78918	IN	NS	l.root-servers.net.
    .			78918	IN	NS	m.root-servers.net.
    .			78918	IN	NS	a.root-servers.net.
    .			78918	IN	NS	b.root-servers.net.
    .			78918	IN	NS	c.root-servers.net.
    .			78918	IN	NS	d.root-servers.net.
    ;; Received 525 bytes from … in 0 ms

    org.			172800	IN	NS	c0.org.afilias-nst.info.
    org.			172800	IN	NS	a2.org.afilias-nst.info.
    org.			172800	IN	NS	a0.org.afilias-nst.info.
    org.			172800	IN	NS	b0.org.afilias-nst.org.
    org.			172800	IN	NS	b2.org.afilias-nst.org.
    org.			172800	IN	NS	d0.org.afilias-nst.org.
    ;; Received 788 bytes from 202.12.27.33#53(m.root-servers.net) in 24 ms

    wikipedia.org.		3600	IN	NS	ns0.wikimedia.org.
    wikipedia.org.		3600	IN	NS	ns1.wikimedia.org.
    wikipedia.org.		3600	IN	NS	ns2.wikimedia.org.
    ;; Received 658 bytes from 2001:500:48::1#53(b2.org.afilias-nst.org) in 20 ms

    en.wikipedia.org.	86400	IN	CNAME	dyna.wikimedia.org.
    ;; Received 94 bytes from 208.80.153.231#53(ns1.wikimedia.org) in 132 ms

Hey, there are IPv4 addresses in there! I know, this is cheating, the output is
from a dual-stack machine. But we can still simulate IPv6-only resolution
by adding ``-6`` flag::

    $ dig en.wikipedia.org AAAA +trace -6

    ; <<>> DiG … <<>> en.wikipedia.org AAAA +trace -6
    ;; global options: +cmd
    .			78915	IN	NS	d.root-servers.net.
    .			78915	IN	NS	e.root-servers.net.
    .			78915	IN	NS	f.root-servers.net.
    .			78915	IN	NS	g.root-servers.net.
    .			78915	IN	NS	h.root-servers.net.
    .			78915	IN	NS	i.root-servers.net.
    .			78915	IN	NS	j.root-servers.net.
    .			78915	IN	NS	k.root-servers.net.
    .			78915	IN	NS	l.root-servers.net.
    .			78915	IN	NS	m.root-servers.net.
    .			78915	IN	NS	a.root-servers.net.
    .			78915	IN	NS	b.root-servers.net.
    .			78915	IN	NS	c.root-servers.net.
    ;; Received 525 bytes from … in 0 ms

    org.			172800	IN	NS	d0.org.afilias-nst.org.
    org.			172800	IN	NS	c0.org.afilias-nst.info.
    org.			172800	IN	NS	b2.org.afilias-nst.org.
    org.			172800	IN	NS	a0.org.afilias-nst.info.
    org.			172800	IN	NS	b0.org.afilias-nst.org.
    org.			172800	IN	NS	a2.org.afilias-nst.info.
    ;; Received 816 bytes from 2001:500:2::c#53(c.root-servers.net) in 8 ms

    wikipedia.org.		3600	IN	NS	ns0.wikimedia.org.
    wikipedia.org.		3600	IN	NS	ns1.wikimedia.org.
    wikipedia.org.		3600	IN	NS	ns2.wikimedia.org.
    couldn't get address for 'ns0.wikimedia.org': not found
    couldn't get address for 'ns1.wikimedia.org': not found
    couldn't get address for 'ns2.wikimedia.org': not found
    dig: couldn't get address for 'ns0.wikimedia.org': no more


Some of those IPv4 addresses were benign – the respective servers are reachable
both using IPv4 and IPv6 address or there is an alternative server that is
reachable using IPv6. That is the case for the root nameserver – in the second
case, we used C, which has IPv6 address (2001:500:2::c). In fact, the M server
also has IPv6 address, but dig chose the IPv4 one (it should not matter)::

    $ dig m.root-servers.net AAAA
    […]
    m.root-servers.net.	77991	IN	AAAA	2001:dc3::35

But the latter case is the bigger issue. For the domain ``wikipedia.org`` there
are three nameservers::

    $ dig wikipedia.org NS -6
    […]
    wikipedia.org.		86400	IN	NS	ns0.wikimedia.org.
    wikipedia.org.		86400	IN	NS	ns1.wikimedia.org.
    wikipedia.org.		86400	IN	NS	ns2.wikimedia.org.

This is the last answer that we could get on an IPv6-only network, because none of
these three servers has AAAA record (some of them may have IPv6 address unknown to us)::

    $ dig ns0.wikimedia.org AAAA
    […]
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59468
    ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

The NOERROR status says the domain name exists, but we got zero answers for
AAAA records. This is the case for all three nameservers. And here is the
ultimate picture of what is happening and what goes wrong.

.. figure:: {static}/images/forgetting-dns6/image5.svg
   :width: 100%

   The breakage in action

Also note that the connection from the laptop to the DNS resolver may in fact
consist of a chain of several (caching, non-recursive) DNS resolvers, so that
the final DNS resolver can have dual-stack connectivity.

The problems with this state
============================

So, what is the deal. We *just* need to have a dual-stack DNS resolver
somewhere, and that's it, no? Well, yes but actually yes.

There are two problems with this: First, this means that any new ISP needs to
have *at least some* IPv4 address, even if they intend to just use IPv6
services. IPv4 addresses are scarce, `expensive
<https://blog.apnic.net/2021/12/16/opinion-ipv4-address-markets/>`__ and small
blocks `don't route well
<https://labs.ripe.net/author/stephen_strowes/visibility-of-ipv4-and-ipv6-prefix-lengths-in-2019/>`__,
which is not great both from the
new ISP's and from overal routing's point of view. It also hinders IPv6
deployment and postpones IPv4 abandonment, needlessly.

The second issue is that this is not very visible. We are building IPv6 world,
but deep inside it still relies on IPv4, which might lead to great surprise
when we start cutting off IPv4 internet. And it might lead to false sense of
having IPv6 deployed, which is not true to the whole extent.

Insert "It was DNS" meme here.

Solution
========

The solution of this state is simple: get IPv6 connectivity to your
authoritative DNS server (or use another) and do not forget to add an AAAA
record for it in DNS. If the DNS server already has IPv6 it is probably just
a matter of adding a single line to the zone file (and a second one for the DNSSEC
signature), which should not be a big deal.

Unfortunately, this needs to be done for the whole DNS chain.
Especially domain names at universities are infamous for very nested domains.
A domain name may look like
``machine.department.location.faculty.university.some-common.suffix``. That
tree is deep and so is the resolution of this problem.

Amusing bug of almost good deployment
=====================================

We have seen there may be multiple NS records for a domain and thus
multiple nameservers. This is good for redundancy. But this does not mean that
the servers will have the same records – they are only supposed to give
equivalent answers (as far as I know).

I have come across a silly misconfiguration: a domain which has several
nameservers, which serve a *slightly* different set of NS records for its
subdomain. Specifically, the servers which were only reachable using IPv4 were
*exactly* the servers that knew about one additional nameserver for the
subdomain, which, incidentally, was the *only* one that was IPv6-capable.

So, while all the correct records were present in DNS (somewhat/somewhere), this still
meant that IPv6-only resolution was doomed to fail because the IPv6 nameserver
chain was broken. (This has already been fixed.)

-----

.. [#nat44] This is very much the same as when you try to reach the
   IPv4-public-land from IPv4-private-land, that is, from a private range of IP
   addresses. This is called either just NAT, or NAT44 to denote IPv4-to-IPv4 NAT.

.. [#test-aaaa] There are several more tests that do not even have the AAAA
   record, lol.

.. [#dns-simplification] In my example, there is a single recursive DNS
   resolver external to my machine in order not to complicate it too much.
   The real deployment is often trickier.

.. [#dns-behind-nat64] I have not yet tried to run a recursive DNS in a network
   with DNS64 and NAT64. Could be fun :-D My wild guess is that I would need
   CLAT (i.e. the full 464XLAT deployment) to make that work, since the
   resolver is connecting directly to IPv4 addresses and would need to learn to
   use NAT64 to resolve them. (The CLAT could be built right into the resolver,
   though).