DNS problem: SERVFAIL looking up A/AAAA on my domain

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:
services.mambasoft.it

I ran this command:
certbot renew

It produced this output:
Renewing an existing certificate for services.mambasoft.it

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
Domain: services.mambasoft.it
Type: dns
Detail: DNS problem: SERVFAIL looking up A for services.mambasoft.it - the domain's nameservers may be malfunctioning; DNS problem: SERVFAIL looking up AAAA for services.mambasoft.it - the domain's nameservers may be malfunctioning

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

Failed to renew certificate services.mambasoft.it with error: Some challenges have failed.

My web server is (include version):
Apache 2.4.54

The operating system my web server runs on is (include version):
openmamba GNU/Linux (2024.4 rolling)

My hosting provider, if applicable, is:
Hetzner dedicated server

I can login to a root shell on my machine (yes or no, or I don't know):
Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
2.9.0

I don't think this is your only problem, but one of your authoritative DNS servers (2a01:4f8:151:7444::1 ns3.mambasoft.net) isn't responding over TCP:

ISC EDNS compliance tester result for services.mambasoft.it

Unboundtest, which is a DNS resolver configured very similarly to how Let's Encrypt's servers are configured, with very verbose logging, tries to describe why it's failing to resolve the name, though I haven't quite figured out what's it's trying to say yet.

https://unboundtest.com/m/A/services.mambasoft.it/SFC63S5W

I think it might be related to a broken DNSSEC configuration, not for the mambasoft.it domain, but for the mambasoft.net name that its authoritative resolvers use. Not entirely sure, but maybe it points you in the right direction:

https://dnsviz.net/d/ns3.mambasoft.net/dnssec/

5 Likes

It's also, uh, unusual that there are six NS records listed for mambasoft.net:

ns1.mambasoft.it
ns2.mambasoft.it
ns3.mambasoft.it
ns1.mambasoft.net
ns2.mambasoft.net
ns3.mambasoft.net

With the corresponding .it and .net names having the same IPs. I don't know as it's wrong in any way, but I also don't know if it's what's intended.

3 Likes

Thanks for pointing out some problems on which I'm investigating.
So far I've resolved ns3.mambasoft.net not responding over TCP.

The configuration became this complex, including the six NS for mambasoft.net, because I had to replace glue records for mambasoft.it with mambasoft.net. BTW the six ns entries match the whois record for mambasoft.net.

I'm checking for problems related to DNSSEC on mambasoft.net, it seems ok but DS is missing on parent domain (.net), could it be this the cause? I'm trying to resolve this with my provider or will try disabling DNSSEC.

2 Likes

Yeah, it looks like mambasoft.net is signing but without anyone caring about the signature (because there's no DS record on .net as you say). That shouldn't be causing a problem, though.

Like I said, Unboundtest is trying to say what it doesn't like, but there's just so much logging output that it's hard for mortals to actually understand.

4 Likes

It's not clear for me that Unboundtest says that it doesn't like something, I mean if I ask it to check other well-known domains it doesn't say they are ok or not clearly. Maybe you refer to search, e.g., for "error" matches in lines, in this case I get only this match but I don't know if it may be relevant:

May 24 15:49:04 unbound1.19[2242100:0] debug: outnettcp got tcp error -1

Yeah, the output is hard to parse. So the key result is at the top, saying "status: SERVFAIL". That's the result that Let's Encrypt's servers are returning when you try to get a certificate, as Unbound is reporting that there's an error looking up the domain.

Near the end, it says error: SERVFAIL <services.mambasoft.it. AAAA IN>: exceeded the maximum nameserver nxdomains

But I can't figure out exactly which query it's seeing nxdomains for.

4 Likes

I am now getting correct response for A record using unboundtest

But, AAAA and CAA queries still get SERVFAIL. These records are not required but they must give a proper "not found" and not SERVFAIL.

https://unboundtest.com/m/A/services.mambasoft.it/27AUSDNO

3 Likes

I've removed DNSSEC management a few minutes ago, it does not seem to solve but should simplify the diagnosis.
CAA, A, AAAA for services.mambasoft.it (or what else in the domain) do exist so they can't return "not found", I suppose SERVFAIL is an error definition from Let's Encrypt side. After some or many tries (within days or weeks) I use to have success in renewal, for random reasons, it is frustrating that I can't understand and fix this.

2 Likes

I've found some inconsistency, where sometimes unboundtest worked and sometimes it gave SERVFAIL. Though maybe it's related to changes to the domain happening.

4 Likes

In the last hours I succeeded renewing mambasoft.it, www.mambasoft.it and another subdomain (as said after randomly retrying), services.mambasoft.it is still failing. BTW the problem would come back on next renewal...

2 Likes

I've always had suspects of random network reachability problems from Let's Encrypt servers, even though 3 nameservers in different locations are there for the purpose of ensuring that queries are resolved.

By the way now also services.mambasoft.it succeeded:

Congratulations, all renewals succeeded:
/etc/letsencrypt/live/services.mambasoft.it/fullchain.pem (success)

I've fixed two things: ns3 reachabiity on TCP/IPV6 and disabled DNSSEC but, due to maybe propagation delays, to be sure that the problem is fixed I need to wait and see if I get errors on future renewals.

Thank you all for your help!

3 Likes

Consider combining your certs to handle all your (sub)domains. DNS validation can give you wildcard certificates to ease your workload. Just a thought.

4 Likes

I had a DNS wildcard certificate in the past but it required me to do some implementations to update DNS entries automatically and share certificates between servers, so I opted for the simplest working solution. Maybe I will retry in the future, thanks for the hint.

3 Likes