Certbot renew failing to renew my certificates - SERVFAIL looking up A for my site

angelo_v · October 1, 2020, 1:29am

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: howdenaces.com

I ran this command: sudo certbot renew --dry-run

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/howdenaces.com.conf

Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for www.howdenaces.com
http-01 challenge for howdenaces.com
Waiting for verification...
Challenge failed for domain www.howdenaces.com
http-01 challenge for www.howdenaces.com
Cleaning up challenges
Attempting to renew cert (howdenaces.com) from /etc/letsencrypt/renewal/howdenaces.com.conf produced an unexpected error: Some challenges have failed.. Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/howdenaces.com/fullchain.pem (failure)

** DRY RUN: simulating 'certbot renew' close to cert expiry
** (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/howdenaces.com/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
** (The test certificates above have not been saved.)

1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:

The following errors were reported by the server:

Domain: www.howdenaces.com
Type: dns
Detail: DNS problem: SERVFAIL looking up A for www.howdenaces.com -
the domain's nameservers may be malfunctioning

My web server is (include version): Apache

The operating system my web server runs on is (include version): Ubuntu 20.04

My hosting provider, if applicable, is: DigitalOcean

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): I use the terminal and fileZilla to manage the server, as well DigitalOceans' control panel for the records and firewall settings.

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 0.40.0

Not really sure what I'm missing here, I haven't changed any of the records since I had this setup over a month ago, and when I checked recently, I suddenly got the privacy error. I could still enter the site through ssh via the terminal, and when I tried to run certbot renew, that's when I saw the error.

_az · October 1, 2020, 2:08am

There is a referral loop in your DNS setup. I don't know how it would have worked the first time, if indeed you haven't changed anything since.

Go to your DigitalOcean DNS control panel.

You should find something like:

www.howdenaces.com.     86396   IN      NS      ns1.digitalocean.com.
www.howdenaces.com.     86396   IN      NS      ns2.digitalocean.com.

you need to get rid of these NS records, because they result in an infinite loop.

If you can't find them, could you please post a screenshot of the DNS interface?

angelo_v · October 1, 2020, 2:33am

@_az
Should I remove the ones in the black rectangle or the blue one?

rg305 · October 1, 2020, 2:36am

the black www entries are definitely wrong
www don't use NS entries (normally)

_az · October 1, 2020, 2:37am

Indeed, I think once you delete the black ones, everything should be fixed.

angelo_v · October 1, 2020, 2:37am

@rg305 @_az I deleted the NS records in the black rectangle, do I have to wait for a propagation time before trying to run the sudo certbot renew --dry-run again? Or do I have to reset the server or something?

_az · October 1, 2020, 2:38am

Wait a couple of minutes. Some of the DigitalOcean nameservers are still showing the old records.

angelo_v · October 1, 2020, 2:39am

Gotcha, will wait for a couple minutes before running the sudo certbot renew --dry-run script again.

rg305 · October 1, 2020, 2:39am

--dry-run won't count against you, so you could try it now.

angelo_v · October 1, 2020, 2:42am

I got this error instead:

1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: www.howdenaces.com
   Type:   dns
   Detail: No valid IP addresses found for www.howdenaces.com

And after running it again, it went back to this error:

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: www.howdenaces.com
   Type:   dns
   Detail: DNS problem: SERVFAIL looking up A for www.howdenaces.com -
   the domain's nameservers may be malfunctioning

rg305 · October 1, 2020, 2:44am

OK now you need an A record for www = 128.199.142.171
OR
a CNAME for www to howdenaces.com

Strike all that - you have a www record.
hmm...

_az · October 1, 2020, 2:45am

A record for www is already in there.

I think you just need to keep waiting, because I still see the bad referrals happening. (But less and less often).

DigitalOcean has some wacky anycast stuff powered by Cloudflare, right? it's probably taking a a while because of that.

rg305 · October 1, 2020, 2:46am

the infinite loop begins to unwind...

angelo_v · October 1, 2020, 2:57am

@_az @rg305

I'll let this sit for a couple more minutes then try from time to time. I just tried again right now but it's still giving the SERVFAIL error.

rg305 · October 1, 2020, 2:58am

Try deleting the www entry
wait one minute
then add the www entry back in (128.199.142.171)

_az · October 1, 2020, 2:59am

You might be in for an extended wait, because DO's servers are still randomly serving the bad referral:

howdenaces.com.         172800  IN      NS      ns1.digitalocean.com.
howdenaces.com.         172800  IN      NS      ns2.digitalocean.com.
howdenaces.com.         172800  IN      NS      ns3.digitalocean.com.
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN NSEC3 1 1 0 - CK0Q1GIN43N1ARRC9OSM6QPQR81H5M9A NS SOA RRSIG DNSKEY NSEC3PARAM
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN RRSIG NSEC3 8 2 86400 20201004044123 20200927033123 24966 com. mfMj0mWhB7y6rOePA3ZB43jj8Qi/8U8+mHOaQtTMs/Dko4vx3RgNQpc1 qLCNOdgVS2iz/M+OXWwUPxl2pZIDjWM0V+12tVFHJRNvlZEihsTD6J+m IwRaFqx7t98vQHls5R02Rl+7zq2RenAEdFiFno249WDp6LBb51i+tC50 dymKNGwudjNZ8k/o/1c7nnVBjMMY9MbIxO48LI57CpmJuw==
S5QTBE54H6UTQRGODJ33A90CV2P829PB.com. 86400 IN NSEC3 1 1 0 - S5QTO72LA1GRP3500UMC5CQGVEV1H125 NS DS RRSIG
S5QTBE54H6UTQRGODJ33A90CV2P829PB.com. 86400 IN RRSIG NSEC3 8 2 86400 20201007042802 20200930031802 24966 com. Tfc6FCN6ZcFv8wU2h9BUqKKr7PMgSIUC820cFIrUSmPA7+oXpK4Is5s7 kUZla0fR7nFWB/+Osqscj4yvyqioj6PJsCJkr9Zkc+SowjV3sEW1MmOt UfZpx7YGXDLyCUiqzWpEo0l7F4UOfRr+XUfk2R/rKUIFU+85xrVhivfL ifBavOWIP/DDMEEa11Y6vepmD/myilotqimqTXD5MCvzSQ==
;; Received 795 bytes from 192.35.51.30#53(f.gtld-servers.net) in 143 ms

www.howdenaces.com.     83229   IN      NS      ns1.digitalocean.com.
www.howdenaces.com.     83229   IN      NS      ns2.digitalocean.com.
;; Received 96 bytes from 2400:cb00:2049:1::c629:dead#53(ns3.digitalocean.com) in 183 ms

www.howdenaces.com.     83229   IN      NS      ns1.digitalocean.com.
www.howdenaces.com.     83229   IN      NS      ns2.digitalocean.com.
;; BAD (HORIZONTAL) REFERRAL
;; Received 96 bytes from 173.245.58.51#53(ns1.digitalocean.com) in 195 ms

www.howdenaces.com.     83229   IN      NS      ns1.digitalocean.com.
www.howdenaces.com.     83229   IN      NS      ns2.digitalocean.com.
;; BAD (HORIZONTAL) REFERRAL
;; Received 96 bytes from 173.245.58.51#53(ns1.digitalocean.com) in 195 ms

www.howdenaces.com.     2952    IN      A       128.199.142.171
;; Received 63 bytes from 173.245.59.41#53(ns2.digitalocean.com) in 23 ms

Maybe give it an hour or whatever. I don't think there's any way to force it to purge.

rg305 · October 1, 2020, 3:03am

That is what I tried to do...
[delete/wait/add it back in]

Maybe switching to a new type like CNAME would hurry things along...
God only knows.. well, DO and CF should know too.

angelo_v · October 1, 2020, 3:12am

@rg305 I tried the delete/wait/add for the www record, but sadly, it didn't work. I then removed it and made it a CNAME record instead of an A record, but that didn't work as well. I think I'll just have to wait this one out.

rg305 · October 1, 2020, 3:17am

Try adding a few random entries
xxx = 1.2.3.4
qqq = 5.6.7.8
zzz = 9.0.0.0
see if that awakens the sync

angelo_v · October 1, 2020, 3:24am

Tried adding the random entries as such but to no avail.