DNS problem: SERVFAIL looking up CAA

rvanoosterhout · April 6, 2021, 6:56pm

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:
lbc-tel.werkonderweg.nl

I ran this command:
certbot certonly --standalone --non-interactive -d lbc-tel.werkonderweg.nl --email xxxx@xxxx.nl --agree-tos

It produced this output:
DNS problem: SERVFAIL looking up CAA for lbc-tel.werkonderweg.nl

My web server is (include version):
nginx,1.10.3

The operating system my web server runs on is (include version):
Debian Stretch 9.13

My hosting provider, if applicable, is:
Selfhosted

I can login to a root shell on my machine (yes or no, or I don't know):
yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
0.28.0

Extra information:

CAA records are present:

$ dig @ns01.is.nl caa werkonderweg.nl +short
0 issue "[letsencrypt.org](http://letsencrypt.org/)"
0 issuewild "[comodoca.com](http://comodoca.com/)"
0 issue "[symantec.com](http://symantec.com/)"
0 iodef "mailto:caa@accent.nl"
0 issue “[comodoca.com](http://comodoca.com/)”

Previous servers which are running on a letsencrypt certificate are also not working anymore when we remove the certificate and reapply for a certificate.

Using --dry-run gives the same result.

jvanasco · April 6, 2021, 7:07pm

You can use https://unboundtest.com/ to debug your DNS, that closely replicates how LetsEncrypt checks your DNS

rvanoosterhout · April 6, 2021, 7:23pm

Unboundtest on my main domain gives the CAA records present:

https://unboundtest.com/m/CAA/werkonderweg.nl/HK4F76I4

petercooperjr · April 6, 2021, 7:39pm

Yeah, but doing a CAA request on the full domain (which Let's Encrypt is obligated to do first) gives a DNSSEC validation failure.

https://unboundtest.com/m/CAA/lbc-tel.werkonderweg.nl/7OJFJQKS

Apr 06 19:39:02 unbound[1218286:0] info: validate(nodata): sec_status_bogus

https://dns.google/query?name=lbc-tel.werkonderweg.nl&rr_type=CAA&ecs=

DNSSEC validation failure.

You need to fix the DNSSEC for the non-existence of the CAA record of the full domain name.

Osiris · April 6, 2021, 7:54pm

Hm, strangest thing. When I ask my ISPs resolver to resolve it, I too get a SERVFAIL. But when I test with dnsviz, it says everything is good?

https://dnsviz.net/d/lbc-tel.werkonderweg.nl/dnssec/

I specifically asked for a CAA record too when "Analyzing" it, I saw it did a CAA record lookup in the output.. Why doesn't it show this error?

petercooperjr · April 6, 2021, 7:59pm

Definitely something odd. I can reproduce with dig on any public recursive DNS server I've tried (like dig @1.1.1.1 caa lbc-tel.werkonderweg.nl +dnssec gives a status: SERVFAIL), but when going at the authoritative nameservers directly I'm not seeing problems (though unboundtest did so I think I'm just not knowing how to use dig right for this).

I mean, as a workaround I kind of expect that adding a CAA record for the full name would work, but there's something weird going on with DNSSEC.

Osiris · April 6, 2021, 8:01pm

I think it's also something lacking from dnsviz.. When I request specifically a request from the authorative DNS servers directly, I'm getting a NSEC3 containing reply:

;; AUTHORITY SECTION:
werkonderweg.nl.	14400	IN	SOA	ns01.is.nl. domain-admin.is.nl. 2021040100 10800 3600 604800 14400
werkonderweg.nl.	14400	IN	RRSIG	SOA 8 2 86400 20210415000000 20210325000000 16786 werkonderweg.nl. ws5e4LC5FEd1oeYErVXrxeLhszTVWAxbLIHXR6Uu3v1HK6LA5gXOcVyn gZDJqYoXWV5k8kO+oSHg7trozHALahsm2d5MgWERgYcrlB1V3/jQerpo DYnIfCEskfbfs42lh9uNvas7AuxNLs4AamIewIgsUvMqxWKqh3s1mNM1 dEg=
804t4dv6v1dqacgifch3ugdkde8l1u02.werkonderweg.nl. 14400	IN NSEC3 1 0 1 AB 83C5EDNKDGAQG8B4C0OSQ2MM859QKAVG A RRSIG
804t4dv6v1dqacgifch3ugdkde8l1u02.werkonderweg.nl. 14400	IN RRSIG NSEC3 8 3 14400 20210415000000 20210325000000 16786 werkonderweg.nl. vWC5cwZS7mehAXxaRKV7qnffhoP0a6af3MX3Z/Z/M9wuNgPjZNlzmccB CZ5TA/18/oc4FUGTJKMV08l2oPXdrj4MaK9ZFuPquuYKT9yXKHQQhvzY V65ztBVc5ocijCvFpmPd9XsZcrWicLMbF4g9fyORK+mUGqD0kVtHHN+P BNI=

If I tell dnsviz it should request a CAA record, I would THINK it should show me these NSEC3 replies, and validate them of course.. However, it only shows the succesful A resource record, although I didn't ask for it.....

Also, the Verisign debugger from the Google Public DNS query isn't helpfull at all too: it doesn't have the ability to specify CAA records...

Could it perhaps be because werkonderweg.nl. has two zone-DNSKEY RRs set up (in total three, but the other one is a zone+sep DNSKEY)? One with ID=16786 and one with ID=50520? Perhaps this confuses resolvers.. And I'm not sure if this is correct according to the RFCs.

JuergenAuer · April 6, 2021, 8:26pm

Hi @rvanoosterhout

your configuration is buggy - see your check created this morning - https://check-your-website.server-daten.de/?q=lbc-tel.werkonderweg.nl - but my output is buggy too.

CAA-Query sends a valid NSEC3 RR as result with the hashed query name "819dl4eo9f4b0pu4q7ld4ar26asqam7o" between the hashed NSEC3-owner "804t4dv6v1dqacgifch3ugdkde8l1u02" and the hashed NextOwner "83c5ednkdgaqg8b4c0osq2mm859qkavg". So the zone confirmes the not-existence of that CAA RR.

804t4dv6v1dqacgifch3ugdkde8l1u02 < 819dl4eo9f4b0pu4q7ld4ar26asqam7o < 83c5ednkdgaqg8b4c0osq2mm859qkavg

So that's a NXDOMAIN proof, not a NoData proof. NoData would require, that the hashed NSEC3-owner = the hashed query name , not <.

PS: Short:
NSEC3-Owner < QueryName < NextDomainName --> NXDOMAIN
NSEC3-Owner = QueryName < NextDomainName --> NoData, QueryName exists, but the specific RR doesn't exist.

But your subdomain has an A record, that's not a wildcard. So your subdomain exists as name in your zone (not only a wildcard expansion).

So the result is bogus -> Servfail.

My local Unbound instance confirms that:

CAA check is bogus:

[1617740658] libunbound[21376:0] info: validator operate: query lbc-tel.werkonderweg.nl. CAA IN
[1617740658] libunbound[21376:0] info: validator operate: chased to . TYPE0 CLASS0
[1617740658] libunbound[21376:0] debug: NODATA response failed to prove NODATA status with NSEC/NSEC3
[1617740658] libunbound[21376:0] info: validate(nodata): sec_status_bogus
lbc-tel.werkonderweg.nl. has no CAA record (BOGUS (security failure))
validation failure <lbc-tel.werkonderweg.nl. CAA IN>: nodata proof failed from 2001:9a0:2002:1::53:2 and 217.149.76.228

A check is secure:

[1617740722] libunbound[23064:0] info: validator operate: query werkonderweg.nl. DNSKEY IN
[1617740722] libunbound[23064:0] info: validated DNSKEY werkonderweg.nl. DNSKEY IN
[1617740722] libunbound[23064:0] debug: validator[module 0] operate: extstate:module_wait_subquery event:module_event_pass
[1617740722] libunbound[23064:0] info: validator operate: query lbc-tel.werkonderweg.nl. A IN
[1617740722] libunbound[23064:0] info: validate(positive): sec_status_secure
[1617740722] libunbound[23064:0] info: validation success lbc-tel.werkonderweg.nl. A IN
lbc-tel.werkonderweg.nl. has address 185.103.17.93 (secure)

PS:

Now the bug is fixed. New output, the third row is yellow:

CAA-Query sends a valid NSEC3 RR as result with the hashed query name "819dl4eo9f4b0pu4q7ld4ar26asqam7o" between the hashed NSEC3-owner "804t4dv6v1dqacgifch3ugdkde8l1u02" and the hashed NextOwner "83c5ednkdgaqg8b4c0osq2mm859qkavg". So the zone confirmes the not-existence of that CAA RR.

Bitmap: A, RRSIG Validated: RRSIG-Owner 804t4dv6v1dqacgifch3ugdkde8l1u02.werkonderweg.nl., Algorithm: 8, 3 Labels, original TTL: 14400 sec, Signature-expiration: 15.04.2021, 00:00:00 +, Signature-Inception: 25.03.2021, 00:00:00 +, KeyTag 16786, Signer-Name: werkonderweg.nl

Status: Fatal / bogus. NoError+NoDataResult sent, the answer says, the query name exists, the NSEC3 covers the Query Name, but there are not enough informations about wildcards (-1): NoError - there must be a confirmed wildcard expansion to create the query name. Recalculate the zone or update the name server software. Or there is a Man in the middle, who has removed one of the required NSEC3-Records, so DNSSEC works.

JuergenAuer · April 6, 2021, 8:59pm

No, the answer doesn't show an A RR validation.

That's

Type - two values - number of iterations (1) - Salt to add - Hash of the NextOwnerName (83C5EDNKDGAQG8B4C0OSQ2MM859QKAVG) - then follows the BitMap. That's a list of existing RR with that domain name (A and RRSIG exists).

More then one DNSKEY are good, allowed, sometimes required.

Most zones have two DNSKEY (see my own zone).

One is the Secure Entry Point, that's the DNSKEY that is used in the parent DS and that DNSKEY is used to sign the set of DNSKEY.

The other DNSKEYs are used to sign the data of the zone.

So it's possible to change one of these DNSKEY without changing the parent DS.

And if a DNSKEY is changed, such a zone has temporary 3 DNSKEY.

Osiris · April 6, 2021, 9:17pm

I was talking about the dnsviz tool. It shows an A RR when asked for a CAA. That's just how the tool operates: it always also resolves A and AAAA RRs, even if you didn't ask for them. And for some reason, it leaves the RR I did request out.. And also doesn't report any DNSSEC errors unfortunately.

Ah, OK, good to know 3 is also allowed.

JuergenAuer · April 6, 2021, 9:23pm

I don't understand. Hope, the link works:

https://dnsviz.net/d/lbc-tel.werkonderweg.nl/dnssec/?rr=257&a=all&ds=all&ta=.&tk=

Use " DNSSEC options ( show )" and select CAA.

Screenshots:

Osiris · April 6, 2021, 9:35pm

Hmkay.. I've never used this option before. And IMO I shouldn't have to use it. At the "Analyse" options page, I specifically asked for CAA RR as extra RR type. I would have expected the software to be at least so smart to report this extra selected RR type without me having to search for it.....

Ah well, now I know where to find it.

rvanoosterhout · April 7, 2021, 1:26pm

The problem is solved by disabling the DNSSEC by my DNS provider

petercooperjr · April 7, 2021, 1:31pm

Well, ideally your DNS provider would fix their DNSSEC implementation instead. I'd keep bugging them about it if I were you. But glad to hear that you have a workaround for now.

JuergenAuer · April 7, 2021, 2:12pm

Normally, that's bad.

If your provider has a DNSSEC implementation, he should update that configuration.

You may fix it partial by adding a CAA to your subdomain.

Existing RR are valid, the bug happens, if there is a NSEC proof of the not-existence of the CAA required.

--

DNSSEC is an amazing feature.

rvanoosterhout · April 8, 2021, 1:34pm

The problem has been solved by turning DNSSEC on again. According to my DNS provider, there was a mismatch in the keys.

system · May 8, 2021, 1:35pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
SERVFAIL Looking up CAA Help	2	456	November 26, 2020
DNS problem: SERVFAIL looking up CAA for www.188589.com Help	4	1345	October 19, 2018
SERVFAIL looking up CAA .co domain Help	5	1420	December 7, 2017
CAA record prevents issuing the certificate: SERVFAIL Help	19	3863	March 6, 2021
SERVFAIL looking up A for Help	5	859	January 9, 2022

DNS problem: SERVFAIL looking up CAA

Related topics