Failing to renew certificate: "DNS problem: server failure at resolver looking up CAA" (multiple domains, wildcard DNS)

Help
1

My domain is: nsrc.vtp.kenet.or.ke and various subdomains. There are actually two A record entries in DNS, one of which is a wildcard:

nsrc.vtp.kenet.or.ke.      A 197.136.7.252
*.p.nsrc.vtp.kenet.or.ke.  A 197.136.7.252

I am using dehydrated -c. The domains.txt file lists 71 individual domains. For avoidance of doubt: that's 71 different domains using HTTP-01 challenge, and I'm not attempting to get a wildcard certificate.

It produced this output:

...
 + Forcing renew.
 + Checking expire date of existing cert...
 + Valid till Jul 26 19:26:33 2024 GMT (Longer than 30 days). Ignoring because renew was forced!
 + Signing domains...
 + Generating private key...
 + Generating signing request...
 + Requesting new certificate order from CA...
 + Received 71 authorizations URLs from the CA
 + Handling authorization for elk.p.nsrc.vtp.kenet.or.ke
 + Found valid authorization for elk.p.nsrc.vtp.kenet.or.ke
... (lots more of these) ...
 + 0 pending challenge(s)
 + Requesting certificate...
  + ERROR: An error occurred while sending post-request to https://acme-v02.api.letsencrypt.org/acme/finalize/1091617847/266269961837 (Status 403)

Details:
HTTP/2 403
server: nginx
date: Sun, 05 May 2024 13:41:33 GMT
content-type: application/problem+json
boulder-requester: 1091617847
cache-control: public, max-age=0, no-cache
link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
replay-nonce: KBbvukTe41tMXs77iVYIWZMIBOxl6UEUWS8VAJEqTvm3FYdm3Hg

{
  "type": "urn:ietf:params:acme:error:caa",
  "detail": "Error finalizing order :: Rechecking CAA for \"librenms.campus3.p.nsrc.vtp.kenet.or.ke\" and 69 more identifiers failed. Refer to sub-problems for more information",
  "status": 403,
  "subproblems": [
...
...
    {
      "type": "urn:ietf:params:acme:error:caa",
      "detail": "Error finalizing order :: While processing CAA for oob.host2.campus5.p.nsrc.vtp.kenet.or.ke: DNS problem: server failure at resolver looking up CAA for oob.host2.campus5.p.nsrc.vtp.kenet.or.ke",
      "status": 403,
      "identifier": {
        "type": "dns",
        "value": "oob.host2.campus5.p.nsrc.vtp.kenet.or.ke"
      }
    }
  ]
}

(Here just showing the last of the failed domains, but they all have the same error).

You'll note that there are 70 failures, versus 71 domains. The one domain which doesn't give an error is the top-level domain itself, nsrc.vtp.kenet.or.ke. All the failing ones are <something>.p.nsrc.vtp.kenet.or.ke

# cat le.json | jq -r '.subproblems[].identifier.value' | wc -l
70
# cat le.json | jq -r '.subproblems[].identifier.value' | grep '\.p\.nsrc\.vtp\.kenet\.or\.ke' | wc -l
70

The previous certificate was fine (it was issued on Apr 27). I recently added two new hostnames to it, which forces a regeneration of the whole certificate. I did this a couple of days ago and it was failing then, and I've tried again daily and it's still failing in the same way today. Hence I don't think it's a transient problem.

The error suggests a problem with DNS. However I don't have any problem resolving from outside:

% dig @8.8.8.8 oob.host2.campus5.p.nsrc.vtp.kenet.or.ke. caa

; <<>> DiG 9.10.6 <<>> @8.8.8.8 oob.host2.campus5.p.nsrc.vtp.kenet.or.ke. caa
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58979
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;oob.host2.campus5.p.nsrc.vtp.kenet.or.ke. IN CAA

;; AUTHORITY SECTION:
kenet.or.ke.		300	IN	SOA	ns1.kenet.or.ke. hostmaster.kenet.or.ke. 2024050202 14400 7200 604800 300

i.e. there's no CAA record, which is fine. And ditto if I go direct to the authoritative servers (ns1/ns2/ns3.kenet.or.ke), e.g.

% dig +norec @ns1.kenet.or.ke. oob.host2.campus5.p.nsrc.vtp.kenet.or.ke. caa

; <<>> DiG 9.10.6 <<>> +norec @ns1.kenet.or.ke. oob.host2.campus5.p.nsrc.vtp.kenet.or.ke. caa
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58992
;; flags: qr aa; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;oob.host2.campus5.p.nsrc.vtp.kenet.or.ke. IN CAA

;; AUTHORITY SECTION:
kenet.or.ke.		300	IN	SOA	ns1.kenet.or.ke. hostmaster.kenet.or.ke. 2024050202 14400 7200 604800 300

It works with +tcp too.

Is there possibly some connectivity issue between Letsencrypt servers and this network? Are there logs that can show this? All three nameservers are in AS36914 (which obviously isn't good, but I don't have control over that)

Alternatively, has something changed at Letsencrypt which may be causing these CAA lookups to fail?

Thanks,

Brian.

EDIT: I've now put it back to the original list of 69 domains and attempted a force renew, and I get the same errors. Therefore, adding the extra domains wasn't the underlying cause; something else has changed since April 27th which is preventing LE from checking for the non-existent CAA records.

2 Likes
2

looks like tree climbing, not really cached nature and a lot of domains will hit wildcard dns servers a lot: (like unboundtest hits secX.rcode0.net 28 times. are you sure it didn't ratelimited by dns server?

3 Likes
3

Have you tried adding a CAA record at that level? Or even at a deeper level?

All the domain name validations succeeded but Let's Encrypt must re-check CAA records to ensure it is still allowed to issue a cert. Adding a CAA eliminates some queries because otherwise it must check every level. There is an overall timeout for walking the tree and usually the error says something like timeout. But, when no other options come to mind it might be worth trying.

There is a cache where CAA re-checks are not needed but it is fairly short and I don't recall the exact duration. Just mentioning it as a confounding factor when debugging such difficult problems.

More details about CAA checking here:

4 Likes
4

It turns out ratelimiting has been configured at the authoritative servers. I'll have to work with them to see if it's possible to tune this.

3 Likes
5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.