Certbot renew failing to renew my certificates - SERVFAIL looking up A for my site

let is simmer a bit...
a watched (DNS) pot will never boil - lol

I did see the SOA record change on all three NS servers...

2 Likes

@_az, is that error always from the same NS1 server?

1 Like

No I still see it on NS2 and NS3 as well.

Precisely why I am muting this thread for at least the next 6 hours :laughing:.

3 Likes

@_az @rg305
I tried it again and now I'm getting a CAA record error instead. How do I resolve this? Or is this still a DNS problem that I'll have to wait out?

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: www.howdenaces.com
   Type:   dns
   Detail: DNS problem: SERVFAIL looking up CAA for www.howdenaces.com
   - the domain's nameservers may be malfunctioning

I made a CAA record to allow letsencrypt.org to issue a certificate, then I reset my server.

When I got back in, the sudo certbot renew --dry-run command wasn't giving errors anymore, however, when I tried to run sudo certbot --apache, I got the following error:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

Plugins selected: Authenticator apache, Installer apache

Which names would you like to activate HTTPS for?

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

1: howdenaces.com

2: www.howdenaces.com

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Select the appropriate numbers separated by commas and/or spaces, or leave input

blank to select all options shown (Enter 'c' to cancel): 

Cert is due for renewal, auto-renewing...

Renewing an existing certificate

An unexpected error occurred:

There were too many requests of a given type :: Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/

Please see the logfiles in /var/log/letsencrypt for more details.

When I tried to run sudo certbot renew, I got the following instead:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/howdenaces.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate
Attempting to renew cert (howdenaces.com) from /etc/letsencrypt/renewal/howdenaces.com.conf produced an unexpected error: urn:ietf:params:acme:error:rateLimited :: There were too many requests of a given type :: Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/. Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/howdenaces.com/fullchain.pem (failure)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/howdenaces.com/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)

Looks like I hit a wall here. When will the renewal attempt refresh? If it would take long, is it advisable for me to just nuke everything and start a fresh server install then get a new certificate? Or would that just be futile?

1 Like

Hello Angelo :slightly_smiling_face:

Let's bring this thing home!

I just ran quite a few tests on your DNS zone looking up the CAA for www.howdenaces.com via the CNAME to howdenaces.com. In all but one of my trials a correct CAA record was returned. I also got passing results from letsdebug.net for www.howdenaces.com and howdenaces.com. You have clearly hit the Failed Validation limit on the Let's Encrypt production server. This rate-limit clears very quickly so try the following when you can:

sudo certbot renew

There is a Failed Validation limit of 5 failures per account, per hostname, per hour. This limit is higher on our staging environment, so you can use that environment to debug connectivity problems. Exceeding the Failed Validations limit is reported with the error message too many failed authorizations recently.

1 Like

@griffin Thank you! I'll wait for another hour just to make sure that the rate-limit has cleared. I'll update this thread after trying.

2 Likes

Considering this is still an issue, I would suggest contacting Digital Ocean's support.

You can provide this command to reproduce the issue:

dig +trace www.howdenaces.com

it's still doing this weird stuff:

www.howdenaces.com.     60536   IN      NS      ns1.digitalocean.com.
www.howdenaces.com.     60536   IN      NS      ns2.digitalocean.com.
;; Received 96 bytes from 198.41.222.173#53(ns3.digitalocean.com) in 108 ms

www.howdenaces.com.     60536   IN      NS      ns1.digitalocean.com.
www.howdenaces.com.     60536   IN      NS      ns2.digitalocean.com.
;; BAD (HORIZONTAL) REFERRAL
;; Received 96 bytes from 173.245.58.51#53(ns1.digitalocean.com) in 148 ms

www.howdenaces.com.     60536   IN      NS      ns1.digitalocean.com.
www.howdenaces.com.     60536   IN      NS      ns2.digitalocean.com.
;; BAD (HORIZONTAL) REFERRAL
;; Received 96 bytes from 173.245.58.51#53(ns1.digitalocean.com) in 196 ms

www.howdenaces.com.     60535   IN      NS      ns1.digitalocean.com.
www.howdenaces.com.     60535   IN      NS      ns2.digitalocean.com.
;; BAD (HORIZONTAL) REFERRAL
;; Received 96 bytes from 2400:cb00:2049:1::adf5:3b29#53(ns2.digitalocean.com) in 196 ms

howdenaces.com.         3600    IN      A       128.199.142.171
www.howdenaces.com.     43200   IN      CNAME   howdenaces.com.
;; Received 77 bytes from 2400:cb00:2049:1::adf5:3b29#53(ns2.digitalocean.com) in 196 ms
3 Likes

Thank you. I take it that there are still NS records that are causing problems?

2 Likes

If DigitalOcean doesn't resolve this issue soon enough, I'm thinking of just deleting my server and starting a new one. With that in mind, I can't just go wipe the current server and reinstall everything, correct? Since it's still gonna have the same IP Address and same Records and things won't get resolved. If I'm gonna start, I should start with a brand new server with a different IP. Would that be the best course of action?

1 Like

The problem is in DNS, that may not fix anything.

2 Likes

Gotcha. This is such a drag, I'm still getting the CAA records error. Hopefully the DNS refreshes soon.

2 Likes

I'm wondering what I'm supposed to be seeing wrong here:

The DNS for the site uses ns[1-3].DitigalOcean.com [witch uses CloudFlare]
[see: https://dnsspy.io/scan/howdenaces.com]
So, it depends on which CF node/site you hit.
Lucky you, you are hitting a node that isn't showing a problem.

1 Like

Thanks for that. :slightly_smiling_face:

I thought I was losing my mind... again. :woozy_face:

On another note, in the output of the link you gave me, I know it's just an address, but this is just not encouraging:
2400:cb00:2049:1::c629:dead

0xdeadbeef anyone? :cow: :hamburger:

I can't confirm that you are NOT - lol

Someone's not a dead head !

1 Like

Are you familiar with 0xdeadbeef?

Not in any urban dictionary way.
But I am familliar with dead head stickers on Cadillacs.

1 Like

But to put your mind at ease:

dig www.howdenaces.com @2400:cb00:2049:1::c629:dead

; <<>> DiG 9.11.3-1ubuntu1.13-Ubuntu <<>> www.howdenaces.com @2400:cb00:2049:1::c629:dead
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39694
;; flags: qr aa rd; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;www.howdenaces.com.            IN      A

;; ANSWER SECTION:
www.howdenaces.com.     43200   IN      CNAME   howdenaces.com.
howdenaces.com.         3600    IN      A       128.199.142.171

;; Query time: 50 msec
;; SERVER: 2400:cb00:2049:1::c629:dead#53(2400:cb00:2049:1::c629:dead)
;; WHEN: Thu Oct 01 17:28:15 UTC 2020
;; MSG SIZE  rcvd: 77
1 Like

0xdeadbeef is a magic debug value.

Magic debug values are specific values written to memory during allocation or deallocation, so that it will later be possible to tell whether or not they have become corrupted, and to make it obvious when values taken from uninitialized memory are being used. Memory is usually viewed in hexadecimal, so memorable repeating or hexspeak values are common. Numerically odd values may be preferred so that processors without byte addressing will fault when attempting to use them as pointers (which must fall at even addresses). Values should be chosen that are away from likely addresses (the program code, static data, heap data, or the stack). Similarly, they may be chosen so that they are not valid codes in the instruction set for the given architecture.

Since it is very unlikely, although possible, that a 32-bit integer would take this specific value, the appearance of such a number in a debugger or memory dump most likely indicates an error such as a buffer overflow or an uninitialized variable.

"Dead beef", Famously used on IBM systems such as the RS/6000, also used in the classic Mac OS operating systems, OPENSTEP Enterprise, and the Commodore Amiga. On Sun Microsystems' Solaris, marks freed kernel memory (KMEM_FREE_PATTERN)

https://en.m.wikipedia.org/wiki/Magic_number_(programming)#Magic_debug_values

1 Like

I still don't see the issue.