CAA servfail with register.com

TeQuilYa · December 20, 2022, 8:35pm

My domain is:

I ran this command:
sudo certbot renew --cert-name saintjoisd.socs.net

It produced this output:
Renewing an existing certificate for www.saintjoisd.net
Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
Domain: www.saintjoisd.net
Type: dns
Detail: DNS problem: SERVFAIL looking up CAA for www.saintjoisd.net - the domain's nameservers may be malfunctioning

My web server is (include version):
Apache 2.4.37

The operating system my web server runs on is (include version):
Rocky Linux release 8.7 (Green Obsidian)

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know):
yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot 1.32.2

Additional Details:
I've checked over various tools to try and remedy this myself but haven't had any luck other than letsdebug confirming where the problem lies:

It all seems to stem from a SERVFAIL on the CAA record instead of a NOERROR return.

I don't have direct access to my client's DNS to verify configuration or make changes. I'm doing my best to track down what needs to happen so I can summarize it and pass it on to their tech folks. I think adding a CAA record to specifically allow letsencrypt certificates might work, but I've read in other posts that register.com does not support CAA records so that may not be an option?

certbot has created and renewed certificates for this site in the past, I'm not sure what my client could have changed in the last 2-3 months to break that.

As a troubleshooting measure I was able to create --dry-run certificates on their top-level domain, so the problem seems related to the www.saintjoisd.net subdomain specifically.

Thanks for any assistance.

Osiris · December 20, 2022, 8:52pm

For the apex domain, resolving the CAA RR seems to work indeed, see https://unboundtest.com/m/CAA/saintjoisd.net/GIJYHVCG

However, there seems to be a DNSSEC issue with the www subdomain: https://unboundtest.com/m/CAA/www.saintjoisd.net/5XB7RNVE

Usually, DNSViz provides a nice overview regarding DNSSEC issues (www.saintjoisd.net | DNSViz), but it seems their code has stopped validating a quite common DNSSEC algorithm. It even stopped validating the root zone . and .com TLD. So that doesn't make much sense to me, I think it's a (fairly recent) bug introduced in their systems. So no help there.

petercooperjr · December 20, 2022, 9:46pm

What needs to happen is for the DNS server to properly respond to requests.

That has been a workaround that some people have used when they couldn't get their DNS provider to behave sometimes. Some people have found that turning off DNSSEC and re-enabling it fixed whatever broken configuration their DNS server was using. But really the problem is the DNS server not working.

There's some more info on CAA in the documentation, if you haven't seen it yet. Particularly, look at the "CAA Errors" section.

TeQuilYa · December 20, 2022, 9:57pm

Thanks for the quick reply. I had looked over some of those tools output previously, but I'm not sure what normal datavis output is. The help topic sticky points to this too that seemed to indicate DNSSEC was ok:

I will continue trying to sort out how DNSSEC relates to the SERVFAIL, what tools are correct about whether DNSSEC is correct or broken and what might help resolve it.

I did find some historical reference that in the past this client's www record may have been a CNAME, so I may suggest they try putting that back. Although I fear that the reason they mucked around with changing it from a CNAME to an A record was driven by something else (perhaps the DNSSEC stuff on the apex record is behind that as well).

Thanks for providing some new things to consider though.

petercooperjr · December 21, 2022, 1:55am

You're overcomplicating things. You can see the failure with any DNS resolver trying to get the the CAA record for www.saintjoisd.net.

$ dig CAA www.saintjoisd.net

; <<>> DiG 9.16.27-RH <<>> CAA www.saintjoisd.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 40391
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

MikeMcQ · December 21, 2022, 3:25am

Hmmm. Maybe not any resolver From an AWS EC2 US East Coast instance it resolved fine (repeatedly). But, unboundtest still shows SERVFAIL. DNSViz no longer has the spurious errors Osiris pointed out and now just warnings about a faulty DS record.

dig CAA www.saintjoisd.net

; <<>> DiG 9.18.1-1ubuntu1.2-Ubuntu <<>> CAA www.saintjoisd.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42978
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

rg305 · December 21, 2022, 3:42am

What happens when you follow the authoritative DNS path?
[from root (".") to www.saintjoisd.net.]

MikeMcQ · December 21, 2022, 4:01am

Me? Like this?

dig CAA www.saintjoisd.net @a.root-servers.net

; <<>> DiG 9.18.1-1ubuntu1.2-Ubuntu <<>> CAA www.saintjoisd.net @a.root-servers.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53913
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 13, ADDITIONAL: 27
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;www.saintjoisd.net.            IN      CAA

;; AUTHORITY SECTION:
net.                    172800  IN      NS      e.gtld-servers.net.
net.                    172800  IN      NS      f.gtld-servers.net.
net.                    172800  IN      NS      m.gtld-servers.net.
net.                    172800  IN      NS      i.gtld-servers.net.
net.                    172800  IN      NS      j.gtld-servers.net.
net.                    172800  IN      NS      b.gtld-servers.net.
net.                    172800  IN      NS      a.gtld-servers.net.
net.                    172800  IN      NS      c.gtld-servers.net.
net.                    172800  IN      NS      k.gtld-servers.net.
net.                    172800  IN      NS      h.gtld-servers.net.
net.                    172800  IN      NS      l.gtld-servers.net.
net.                    172800  IN      NS      g.gtld-servers.net.
net.                    172800  IN      NS      d.gtld-servers.net.

Then

dig CAA www.saintjoisd.net @a.gtld-servers.net

; <<>> DiG 9.18.1-1ubuntu1.2-Ubuntu <<>> CAA www.saintjoisd.net @a.gtld-servers.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43506
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 2, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;www.saintjoisd.net.            IN      CAA

;; AUTHORITY SECTION:
saintjoisd.net.         172800  IN      NS      dns101.register.com.
saintjoisd.net.         172800  IN      NS      dns102.register.com.

Lastly this and same for @dns102.register.com

dig CAA www.saintjoisd.net @dns101.register.com

; <<>> DiG 9.18.1-1ubuntu1.2-Ubuntu <<>> CAA www.saintjoisd.net @dns101.register.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28772
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;www.saintjoisd.net.            IN      CAA

;; AUTHORITY SECTION:
saintjoisd.net.         3600    IN      SOA     DNS101.REGISTER.COM. root.REGISTER.COM. 122111212 10800 3600 604800 3600

;; Query time: 16 msec
;; SERVER: 162.159.24.117#53(dns101.register.com) (UDP)
;; WHEN: Wed Dec 21 03:49:54 UTC 2022
;; MSG SIZE  rcvd: 107

rg305 · December 21, 2022, 4:04am

Yeah, that is what I meant.
[but just in parallel - to all authoritative DNS servers]

I do see that the authoritative DNS servers are hosted by CloudFlare.
Perhaps that has something to do with this.

rg305 · December 21, 2022, 4:05am

Where you got the error...
Can you run?:

dig CAA www.saintjoisd.net @dns101.register.com
dig CAA www.saintjoisd.net @dns102.register.com

And

nslookup dns101.register.com
nslookup dns102.register.com

MikeMcQ · December 21, 2022, 4:18am

Is there a short-hand command and result that would show you what you are looking for? That's a lot of data to post if you want to see all of it above for each one.

You mean for @petercooperjr right? Because he got SERVFAIL but I got NOERROR

For me though the IP's for those two name servers (no AAAA)

dig +noall +answer dns101.register.com
dns101.register.com.    244     IN      A       162.159.24.117
dig +noall +answer dns102.register.com
dns102.register.com.    269     IN      A       162.159.25.158

TeQuilYa · December 21, 2022, 5:10am

Thanks, I dig get that dig info as well. But I don't know how to correct it. It may not be complicated but I don't know what the fix to pass on to my client is yet.

If register.com can't do CAA as some have said then the client may be SOL. If they have messed up their DNSSEC somehow I'm not skilled enough yet to help them fix it. I don't know if they need really the DNSSEC or if they can just remove it as some other register.com users have done when they ran into this problem (but I suspect they will report that the need it for some other project on their apex or subdomains).

Thanks for spending time trying to educate me.

TeQuilYa · December 21, 2022, 5:19am

Here are the short answers first:

$ nslookup dns101.register.com
Server:         10.5.1.134
Address:        10.5.1.134#53

Non-authoritative answer:
Name:   dns101.register.com
Address: 162.159.24.117

$ nslookup dns102.register.com
Server:         10.5.1.134
Address:        10.5.1.134#53

Non-authoritative answer:
Name:   dns102.register.com
Address: 162.159.25.158

And the longer ones:

$ dig CAA www.saintjoisd.net @dns101.register.com

; <<>> DiG 9.11.36-RedHat-9.11.36-5.el8_7.2 <<>> CAA www.saintjoisd.net @dns101.register.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59000
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;www.saintjoisd.net.            IN      CAA

;; AUTHORITY SECTION:
saintjoisd.net.         3600    IN      SOA     DNS101.REGISTER.COM. root.REGISTER.COM. 122111212 10800 3600 604800 3600

;; Query time: 34 msec
;; SERVER: 162.159.24.117#53(162.159.24.117)
;; WHEN: Tue Dec 20 23:16:29 CST 2022
;; MSG SIZE  rcvd: 107

$ dig CAA www.saintjoisd.net @dns102.register.com

; <<>> DiG 9.11.36-RedHat-9.11.36-5.el8_7.2 <<>> CAA www.saintjoisd.net @dns102.register.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9975
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;www.saintjoisd.net.            IN      CAA

;; AUTHORITY SECTION:
saintjoisd.net.         3600    IN      SOA     DNS101.REGISTER.COM. root.REGISTER.COM. 122111212 10800 3600 604800 3600

;; Query time: 55 msec
;; SERVER: 162.159.25.158#53(162.159.25.158)
;; WHEN: Tue Dec 20 23:16:43 CST 2022
;; MSG SIZE  rcvd: 107

Interesting that dns101/102 return NOERROR, but my default servers and google's public return SERVFAIL:

$ dig CAA www.saintjoisd.net @8.8.8.8

; <<>> DiG 9.11.36-RedHat-9.11.36-5.el8_7.2 <<>> CAA www.saintjoisd.net @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 43465
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;www.saintjoisd.net.            IN      CAA

;; Query time: 76 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Tue Dec 20 23:18:22 CST 2022
;; MSG SIZE  rcvd: 47

Thanks very much from each of you spending time trying to help me make some progress.

rg305 · December 21, 2022, 5:32am

Well...
My best guess is that they are behind some unhinged DDoS protected network device.

TeQuilYa · December 21, 2022, 5:39am

Thanks for adding the detail that the dnsviz output has changed throughout the day. Yesterday when looking I only saw the one warning about SHA1 being ignored when better algorithms were available. Then when late afternoon today it was showing a large number of different errors mentioned by osiris, I thought I had just screwed up my checks yesterday. But it turns out the same checks now are back to only showing the SHA1 being ignored warning.

Between dataviz only showing the SHA1 warning and dnssec-analyzer showing all green I thought the problem lay somewhere other than DNSSEC. That was the point at which I posted my questions here.

TeQuilYa · December 21, 2022, 5:44am

So many factors at play. We have multiple clients using cloudflare, so that alone probably isn't it. I was pleasantly surprised when the first cloudflare clients made it through letsencrypt ACME renewals cleanly when they started showing up.

I suspect it is the combination of register.com, cloudflare and DNSSEC that has proven too much for poor certbot...

petercooperjr · December 21, 2022, 12:03pm

For some reason AWS doesn't turn on DNSSEC checking by default, and you have to specifically turn it on. By "any DNS resolver" I meant "any DNS resolver which is at least kinda trying to comply with current standard practice", but I suppose I should have been more clear.

The fix is to have the DNS provider fix their broken DNSSEC. Neither you nor your client can really do anything, you need to have the DNS provider fix it, or change the domain to use a DNS provider that isn't so broken.

You're probably running into that DnsViz only shows the result for CAA if specifically asked for in the advanced settings.

First on the Analyze tab:

Then on the DNSSEC tab:

(I don't know if you need it in both tabs, but I figure it can't hurt.) And then you can see the DNSSEC errors:

NSEC proving non-existence of www.saintjoisd.net/CAA: No NSEC RR matches the SNAME (www.saintjoisd.net).

NSEC proving non-existence of www.saintjoisd.net/CAA: No NSEC RR matches the SNAME (www.saintjoisd.net).

NSEC proving non-existence of www.saintjoisd.net/CAA: The following queries resulted in an answer response, even though the NSEC records indicate that the queried names don't exist: www.saintjoisd.net/A

NSEC proving non-existence of www.saintjoisd.net/CAA: The following queries resulted in an answer response, even though the NSEC records indicate that the queried names don't exist: www.saintjoisd.net/A

Osiris · December 21, 2022, 12:40pm

@petercooperjr Weird, I'm still getting 15 warnings with (or some variants with different algo's such as SHA-386):

RRSIG ./DNSKEY alg 8, id 20326: Validation of DNSSEC algorithm 8 (RSASHA256) is not supported by this code, so the cryptographic status of this RRSIG is unknown.

.. instead of the errors you're showing. Not sure why? I've tried Chrome and Edge, just in case DNSViz makes the browser to all the calculations..

petercooperjr · December 21, 2022, 1:05pm

I was seeing warnings along those lines too, but I thought the errors were a separate thing showing that the DNSSEC responses were wrong. You're just seeing the warnings, and not the errors too, when picking to look at the CAA record?

Nummer378 · December 21, 2022, 2:06pm

@petercooperjr is correct. The non-existence records on subdomains of saintjoisd.net are all broken. I had a look at this yesterday, but a short skim didn't see a huge issue. While I have no idea what's (sometimes) broken with DnsViz , the error messages are clear + reproduceable for me.

With DNSSEC, nameservers have to proof that a given record does not exist. Otherwise anyone could just truncate or convert responses into NXDOMAIN, which is a problem (it could be used to disable DNSSEC entirely). DNSSEC uses either NSEC or NSEC3 for this. Both have a slight disadvantage, as they allow zone enumeration (getting all records a given zone has), but its a bit more difficult to do so with NSEC3.

The domain in question here uses NSEC (which is actually easier to debug in cases like this), but the signing on subdomains is broken. While queries to saintjoisd.net all result in correct answers, any query to a subdomain with a type that does not exist results in a broken NSEC record.

As an example (this applies to any subdomain due to wildcards, see below):

For a given query dig CAA www.saintjoisd.net, the nameserver claims (via NSEC) that the entire subdomain does not exist (i.e. there is no www.saintjoisd.net, only saintjoisd.net), yet the response is a NOERROR one (with empty answer section, i.e. no matching records). If the entire subdomain was non-existing the response should have been NXDOMAIN instead. Actually, if we do a query dig A www.saintjoisd.net we can actually get a non-empty A response (with correct RRSIG), so the subdomain does actually exist. This is not stated by the nameserver though, resulting in a broken NSEC record and hence failing DNSSEC validation.

This issue happens with all record types that don't exist (i.e. AAAA, SSHFP...) all result in this exact same error. It is not limited to CAA records, it applies to all non-existence proofs.

A few things I discovered while testing:

The zone appears to be live signed, i.e. DNSSEC records are generated on the fly. This is a bit unusual, but not a problem in itself.
The zone uses a wildcard, i.e. dig A anysubodmain.saintjoisd.net resolves to the same IP address (www." also appears to be a part of the wildcard). The existence RRsigs are correctly signed here, unlike the NSEC for the same subdomain. This wildcard is a probable cause for the nameserver trouble, as the nameserver does not properly disclose it. With NSEC, wildcards have to be disclosed in the NSEC response*, otherwise the resolver can't possibly validate it. The nameserver here does not disclose the wildcard at all, instead pretending that no subdomains exist.

My suggestion to @TeQuilYa would be to speak with their DNS provider and see if they can fix the DNSSEC issue. If they are unable to do so, you have some more options:

Disable DNSSEC entirely (weakens security, but resolves the issue)
Try to check if the issue also appears without the wildcard. The NSEC issue might disappear, but its not guaranteed - there might be more dragons in the shadows.
Adding a CAA record will result in a non-empty response with RRSIG, also working around the broken NSEC response. This does not fix your other record types (AAAA for example) though, so I wouldn't recommend it.

Some more trivia: Cloudflare, which supports Extended DNSSEC errors, also complains about the NSEC records on subdomains:

# dig AAAA www.saintjoisd.net +dnssec @1.1.1.1

; <<>> DiG 9.16.33-Debian <<>> AAAA www.saintjoisd.net +dnssec @1.1.1.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 6991
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1232
; EDE: 6 (DNSSEC Bogus): (proof of non-existence of www.saintjoisd.net. AAAA)
;; QUESTION SECTION:
;www.saintjoisd.net.            IN      AAAA

*As the zone uses live signing, it could also theoretically get away with not disclosing the wildcard (instead generating NSEC records on the fly), but at the very least it has to properly generate the NSEC records.