Help debugging issue with new nameservers

twheel · June 28, 2021, 3:21pm

I recently set up new nameservers (ns1.wheelockweb.com, ns2.wheelockweb.com, ns3.wheelockweb.com), but now certificate renewals for my clients are failing with

ValueError: Challenge did not pass for dev.grovespringfarm.com: {'identifier': {'type': 'dns', 'value': 'dev.grovespringfarm.com'}, 'status': 'invalid', 'expires': '2021-07-05T15:17:41Z', 'challenges': [{'type': 'http-01', 'status': 'invalid', 'error': {'type': 'urn:ietf:params:acme:error:dns', 'detail': 'DNS problem: query timed out looking up A for dev.grovespringfarm.com', 'status': 400}, 'url': 'https://acme-v02.api.letsencrypt.org/acme/chall-v3/14365511253/TniMkA', 'token': 'Tt2i39sASEtEFGnKEcreX7lDJnn-VOIbP-mQDi96J3A', 'validated': '2021-06-28T15:17:42Z'}]}

Testing at unboundtest.com, it seems like the issue is with my new nameservers, but I can’t figure out what the results are telling me (where the problem is). Can anybody help?

https://unboundtest.com/m/A/dev.grovespringfarm.com/2SB26K33

Query results for A dev.grovespringfarm.com
----- Unbound logs -----
Jun 28 13:47:41 unbound[1780797:0] notice: init module 0: validator
Jun 28 13:47:41 unbound[1780797:0] notice: init module 1: iterator
Jun 28 13:47:41 unbound[1780797:0] info: start of service (unbound 1.12.0).
Jun 28 13:47:42 unbound[1780797:0] info: 127.0.0.1 dev.grovespringfarm.com. A IN
Jun 28 13:47:42 unbound[1780797:0] info: resolving dev.grovespringfarm.com. A IN
Jun 28 13:47:42 unbound[1780797:0] info: priming . IN NS
Jun 28 13:47:42 unbound[1780797:0] info: response for . NS IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <.> 2001:500:200::b#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was ANSWER
Jun 28 13:47:42 unbound[1780797:0] info: priming successful for . NS IN
Jun 28 13:47:42 unbound[1780797:0] info: response for dev.grovespringfarm.com. A IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <.> 2001:503:ba3e::2:30#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was REFERRAL
Jun 28 13:47:42 unbound[1780797:0] info: response for dev.grovespringfarm.com. A IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <com.> 192.54.112.30#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was REFERRAL
Jun 28 13:47:42 unbound[1780797:0] info: resolving ns2.wheelockweb.com. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: resolving ns3.wheelockweb.com. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: resolving ns1.wheelockweb.com. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: response for ns3.wheelockweb.com. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <com.> 2001:502:8cc::30#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was REFERRAL
Jun 28 13:47:42 unbound[1780797:0] info: resolving ns1.wheelockweb.net. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: resolving ns3.wheelockweb.net. A IN
Jun 28 13:47:42 unbound[1780797:0] info: resolving ns3.wheelockweb.net. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: resolving ns1.wheelockweb.net. A IN
Jun 28 13:47:42 unbound[1780797:0] info: response for ns1.wheelockweb.com. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <com.> 2001:501:b1f9::30#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was REFERRAL
Jun 28 13:47:42 unbound[1780797:0] info: resolving ns2.wheelockweb.net. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: resolving ns2.wheelockweb.net. A IN
Jun 28 13:47:42 unbound[1780797:0] info: response for ns1.wheelockweb.net. A IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <.> 192.36.148.17#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was REFERRAL
Jun 28 13:47:42 unbound[1780797:0] info: response for ns2.wheelockweb.net. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <.> 192.203.230.10#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was REFERRAL
Jun 28 13:47:42 unbound[1780797:0] info: response for ns2.wheelockweb.net. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <net.> 192.35.51.30#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was REFERRAL
Jun 28 13:47:42 unbound[1780797:0] info: resolving dns1.registrar-servers.com. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: resolving dns1.registrar-servers.com. A IN
Jun 28 13:47:42 unbound[1780797:0] info: response for ns1.wheelockweb.net. A IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <net.> 2001:502:7094::30#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was REFERRAL
Jun 28 13:47:42 unbound[1780797:0] info: resolving dns2.registrar-servers.com. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: resolving dns2.registrar-servers.com. A IN
Jun 28 13:47:42 unbound[1780797:0] info: response for dns1.registrar-servers.com. A IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <com.> 2001:503:d2d::30#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was REFERRAL
Jun 28 13:47:42 unbound[1780797:0] info: response for dns2.registrar-servers.com. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <com.> 192.54.112.30#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was REFERRAL
Jun 28 13:47:42 unbound[1780797:0] info: response for ns1.wheelockweb.net. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <.> 198.97.190.53#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was REFERRAL
Jun 28 13:47:42 unbound[1780797:0] info: response for ns1.wheelockweb.net. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <net.> 192.43.172.30#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was REFERRAL
Jun 28 13:47:42 unbound[1780797:0] info: response for ns2.wheelockweb.com. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <com.> 192.26.92.30#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was REFERRAL
Jun 28 13:47:42 unbound[1780797:0] info: response for dns1.registrar-servers.com. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <com.> 2001:502:1ca1::30#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was REFERRAL
Jun 28 13:47:42 unbound[1780797:0] info: response for dns1.registrar-servers.com. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <registrar-servers.com.> 2001:502:f3ff::204#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was ANSWER
Jun 28 13:47:42 unbound[1780797:0] info: response for dns2.registrar-servers.com. A IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <com.> 192.26.92.30#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was REFERRAL
Jun 28 13:47:42 unbound[1780797:0] info: response for dns1.registrar-servers.com. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <registrar-servers.com.> 204.74.66.4#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was ANSWER
Jun 28 13:47:42 unbound[1780797:0] info: response for ns2.wheelockweb.net. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <wheelockweb.net.> 2610:a1:1024::200#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was ANSWER
Jun 28 13:47:42 unbound[1780797:0] info: response for ns2.wheelockweb.net. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <wheelockweb.net.> 2610:a1:1024::200#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was nodata ANSWER
Jun 28 13:47:42 unbound[1780797:0] info: response for ns2.wheelockweb.net. A IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <.> 2001:500:12::d0d#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was REFERRAL
Jun 28 13:47:42 unbound[1780797:0] info: response for dns2.registrar-servers.com. A IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <registrar-servers.com.> 51.89.217.44#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was ANSWER
Jun 28 13:47:42 unbound[1780797:0] info: response for dns2.registrar-servers.com. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <registrar-servers.com.> 51.89.217.44#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was ANSWER
Jun 28 13:47:42 unbound[1780797:0] info: response for ns3.wheelockweb.net. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <.> 199.7.83.42#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was REFERRAL
Jun 28 13:47:42 unbound[1780797:0] info: response for dns1.registrar-servers.com. A IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <registrar-servers.com.> 51.89.217.44#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was ANSWER
Jun 28 13:47:42 unbound[1780797:0] info: response for ns1.wheelockweb.net. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <wheelockweb.net.> 156.154.133.200#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was ANSWER
Jun 28 13:47:42 unbound[1780797:0] info: resolving dns1.registrar-servers.com. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: resolving dns1.registrar-servers.com. A IN
Jun 28 13:47:42 unbound[1780797:0] info: response for ns1.wheelockweb.net. A IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <wheelockweb.net.> 156.154.133.200#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was ANSWER
Jun 28 13:47:42 unbound[1780797:0] info: response for ns1.wheelockweb.net. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <wheelockweb.net.> 156.154.133.200#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was nodata ANSWER
Jun 28 13:47:42 unbound[1780797:0] info: response for ns3.wheelockweb.net. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <net.> 2001:503:231d::2:30#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was REFERRAL
Jun 28 13:47:42 unbound[1780797:0] info: response for ns3.wheelockweb.net. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <wheelockweb.net.> 2610:a1:1024::200#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was ANSWER
Jun 28 13:47:42 unbound[1780797:0] info: response for ns3.wheelockweb.net. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <wheelockweb.net.> 156.154.133.200#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was nodata ANSWER
Jun 28 13:47:42 unbound[1780797:0] info: response for ns2.wheelockweb.net. A IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <net.> 2001:502:1ca1::30#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was REFERRAL
Jun 28 13:47:42 unbound[1780797:0] info: response for ns2.wheelockweb.net. A IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <wheelockweb.net.> 2610:a1:1024::200#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was ANSWER
Jun 28 13:47:42 unbound[1780797:0] info: response for ns3.wheelockweb.net. A IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <.> 199.9.14.201#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was REFERRAL
Jun 28 13:47:42 unbound[1780797:0] info: response for ns3.wheelockweb.net. A IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <net.> 192.41.162.30#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was REFERRAL
Jun 28 13:47:42 unbound[1780797:0] info: response for ns3.wheelockweb.net. A IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <wheelockweb.net.> 156.154.132.200#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was ANSWER
Jun 28 13:47:42 unbound[1780797:0] info: response for ns2.wheelockweb.com. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <wheelockweb.com.> 192.227.155.222#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was ANSWER
Jun 28 13:47:42 unbound[1780797:0] info: resolving ns3.wheelockweb.net. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: resolving ns3.wheelockweb.net. A IN
Jun 28 13:47:42 unbound[1780797:0] info: response for ns3.wheelockweb.com. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <wheelockweb.com.> 192.227.155.222#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was ANSWER
Jun 28 13:47:42 unbound[1780797:0] info: resolving ns2.wheelockweb.net. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: resolving ns2.wheelockweb.net. A IN
Jun 28 13:47:42 unbound[1780797:0] info: response for ns1.wheelockweb.com. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <wheelockweb.com.> 192.227.155.222#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was ANSWER
Jun 28 13:47:42 unbound[1780797:0] info: response for ns3.wheelockweb.net. A IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <wheelockweb.net.> 156.154.133.200#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was ANSWER
Jun 28 13:47:42 unbound[1780797:0] info: response for ns3.wheelockweb.net. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <wheelockweb.net.> 156.154.133.200#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was ANSWER
Jun 28 13:47:42 unbound[1780797:0] info: response for ns2.wheelockweb.net. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <wheelockweb.net.> 156.154.133.200#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was ANSWER
Jun 28 13:47:42 unbound[1780797:0] info: response for ns2.wheelockweb.net. A IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <wheelockweb.net.> 156.154.132.200#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was ANSWER
Jun 28 13:47:42 unbound[1780797:0] info: response for ns2.wheelockweb.net. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <wheelockweb.net.> 2610:a1:1024::200#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was nodata ANSWER
Jun 28 13:47:42 unbound[1780797:0] info: response for ns3.wheelockweb.net. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <wheelockweb.net.> 156.154.133.200#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was nodata ANSWER
Jun 28 13:47:42 unbound[1780797:0] info: response for dns1.registrar-servers.com. A IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <registrar-servers.com.> 51.222.46.83#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was ANSWER
Jun 28 13:47:42 unbound[1780797:0] info: response for ns3.wheelockweb.com. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <wheelockweb.com.> 205.185.120.156#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was nodata ANSWER
Jun 28 13:47:42 unbound[1780797:0] info: response for ns2.wheelockweb.com. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <wheelockweb.com.> 192.227.155.222#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was nodata ANSWER
Jun 28 13:47:42 unbound[1780797:0] info: response for ns1.wheelockweb.com. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <wheelockweb.com.> 192.210.243.231#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was nodata ANSWER
Jun 28 13:47:42 unbound[1780797:0] info: response for dns2.registrar-servers.com. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <registrar-servers.com.> 51.89.217.44#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was ANSWER
Jun 28 13:47:42 unbound[1780797:0] info: response for dns1.registrar-servers.com. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <registrar-servers.com.> 51.89.217.44#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was ANSWER
Jun 28 13:47:42 unbound[1780797:0] info: response for dns1.registrar-servers.com. AAAA IN
Jun 28 13:47:42 unbound[1780797:0] info: reply from <registrar-servers.com.> 51.89.217.44#53
Jun 28 13:47:42 unbound[1780797:0] info: query response was ANSWER
Jun 28 13:47:46 unbound[1780797:0] info: Capsforid: timeouts, starting fallback


Error running query: read udp 127.0.0.1:55721->127.0.0.1:1053: i/o timeout

rmbolger · June 28, 2021, 3:50pm

So here's what is weird. When you ask the TLD servers what the NS records are for grovespringfarm.com, you do get the ns1-3.wheelockweb.com in the response, but the response comes with glue record IP addresses that don't match the IPs those names actually resolve to.

>dig ns grovespringfarm.com @m.gtld-servers.net

; <<>> DiG 9.16.1 <<>> ns grovespringfarm.com @m.gtld-servers.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57453
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 3, ADDITIONAL: 4
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;grovespringfarm.com.           IN      NS

;; AUTHORITY SECTION:
grovespringfarm.com.    172800  IN      NS      ns1.wheelockweb.com.
grovespringfarm.com.    172800  IN      NS      ns2.wheelockweb.com.
grovespringfarm.com.    172800  IN      NS      ns3.wheelockweb.com.

;; ADDITIONAL SECTION:
ns1.wheelockweb.com.    172800  IN      A       168.235.83.159
ns2.wheelockweb.com.    172800  IN      A       23.226.235.113
ns3.wheelockweb.com.    172800  IN      A       199.101.97.208

;; Query time: 22 msec
;; SERVER: 192.55.83.30#53(192.55.83.30)
;; WHEN: Mon Jun 28 08:43:39 Pacific Daylight Time 2021
;; MSG SIZE  rcvd: 162

Notice the list of IPs in the ADDITIONAL section. But then if you query the names directly, you get a different set of IPs.

>dig +noall +answer ns1.wheelockweb.com
ns1.wheelockweb.com.    14049   IN      A       192.227.155.222

>dig +noall +answer ns2.wheelockweb.com
ns2.wheelockweb.com.    14065   IN      A       192.210.243.231

>dig +noall +answer ns3.wheelockweb.com
ns3.wheelockweb.com.    14066   IN      A       205.185.120.156

I get query timeouts when trying to query the IPs from the ADDITIONAL section directly. But things appear to work against the IPs from the direct resolution of the names.

>dig +noall +answer soa grovespringfarm.com @168.235.83.159
;; connection timed out; no servers could be reached

>dig +noall +answer soa grovespringfarm.com @ns1.wheelockweb.com
grovespringfarm.com.    14400   IN      SOA     ns1.wheelockweb.com. registrar.wheelockweb.com. 2021050402 14400 3600 1209600 7200

twheel · June 28, 2021, 3:55pm

Thanks. I believe that may be an issue that my registrar (namecheap) needs to fix. They have a function in their DNS management section to register nameservers, but I couldn’t get it to work when I tried to register the new nameservers. (The glue records are the old nameservers and need to change to the IPs that you get when you query directly.)

I have a trouble ticket at namecheap and will update this thread when that issue is resolved.

rg305 · June 28, 2021, 6:11pm

If you don't have too many domains to deal with, you could just create three new name servers (not reusing the same exact names) and then switch over to them (once they all work as expected).

twheel · June 28, 2021, 7:47pm

Thanks for the help, rmbolger and rg305. Namecheap fixed the bug that was preventing me from updating the glue records and all is working as expected now.

system · July 28, 2021, 7:48pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.