So, I get asked every once and awhile why I think it might not be a great idea to add AAAA records to popular websites, to spread IPv6 adoption.  The reason I always give: there's still lots of proxy and DNS servers out there that go belly-up when confronted with AAAA records.

Here's another reason why:

So, fed up with the myriad of bugs on the Nokia E71, I decided to upgrade to the latest latest version of the firmware for my phone: 200.21.118.  Unfortunately, it didn't fix the Bluetooth and disconnect issues, and only addressed the Flickr photo-sharing application.  It also (or so I thought) introduced a DNS resolver issue with AAAA and A records, but this turned out to be coincidental, and due to AT&T adding broken IPv6 support to their DNS servers!  Read on for my explanation…

I first realized this issue when I started having some strange DNS resolution issues when connecting to mail, web, SSH, etc. on my servers.  This only happened over the WAN interface (it's AT&T's UMTS network), not over Wi-Fi.  Resolution only succeeded 10% of the time, which was fairly painful.  At first, I thought it was a fluke, but now I've come to realize it was only having issues on domains that have NS records with IPv6 glue.  Connecting to stuff at the office worked just fine, and browsing the web seemed ok for most sites.

I used IfInfo to determine what DNS servers AT&T was providing, and it turned out to be 172.18.145.103 (which translates to 209.183.35.23, alpinetdns.mycingular.net, apparently).  Now, here's what confused me at first, still thinking that the new Nokia E71 version had tweaked their resolver to support AAAA records: I ran tcpdumps, both IPv4 and IPv6, on my DNS servers when trying to load a prolixium.com URL, and saw /nothing/ when the queries failed.  When they succeeded, I only saw an A record lookup.  I went into the advanced access point settings and pointed the AT&T connection to 4.2.2.1 and 4.2.2.2, Level3's anycasted DNS servers, and then tried again:

16:39:02.155276 IP ics2.Atlanta1.Level3.net.39513 > ns3.antiderivative.net.domain: 57546 [1au] A? start.prolixium.com. (48)

Works perfectly!  But.. only an A record?

I switched back to automatic DNS (the 172 stuff) and tried to pull up a page created by one of my friends: http://mathemagi.ca/math/.  Now, mathemagi.ca only has an A record, but my DNS servers that have IPv6 glue are authoritative for it.  Bingo - page didn't load, and the DNS lookup timed-out.  Switch back to Level3's servers, and it works.

Here's what I figure happened: AT&T is deploying IPv6 on their wireless network, but are breaking things along the way.  I suspect that the 172.18.145.103 DNS server is sitting on a network segment where stateless autoconfiguration has been enabled by accident, before there is actually IPv6 transit available.  The DNS server probably had an IPv6-enabled networking stack with autoconfiguration enabled.  It autoconfigured an IPv6 address that doesn't work, and is trying to send ns2.antiderivative.net and ns3.antiderivative.net (my DNS servers) queries via IPv6, that are being blackholed somewhere else in the network.  Eventually this should time out and fall back to IPv4, but I suspect the DNS timeout on the phone is lower than the timeout on the DNS server - so stuff just breaks.

The root servers and GTLD servers are both online with IPv6, so this could potentially make that DNS server fail for every query.  My guess is that the DNS server either has them all cached, or is a slave of some sorts, so it doesn't have to time out sending DNS queries via IPv6 to the root and GTLDs.

I'm going to stick with Level3's DNS servers, for now, until IPv6 gets correctly deployed on AT&T's wireless network, or they remove stateless autoconfiguration from the DNS server (or segment).

Bah, way to spread IPv6 adoption :-(

Update 2009/03/22:

Thanks to some private replies I received from this news posting, AT&T has since resolved the DNS issue.  It seems that 172.18.145.103 is a virtual IP on some load balancer, distributing load between 209.183.35.23 and 209.183.33.23.  The latter server seems to have been the one with issues, and has since been taken out of rotation.

I still believe it has something to do with an IPv6 messup as described above, as every site I tried that failed has AAAA glue.  Perhaps only 209.183.33.23 has autoconfiguration enabled.  However, some other sites were affected two, so this is inconclusive.

New comments are currently disabled for this entry.