Unable to renew my wildcard certificate

My domain is: kiyo.ooo

Web server: nginx/1.14.2
OS: Ubuntu 18.04.2 LTS x86_64
Machine type: VPS relying on LXC
I can login to a root shell on my machine: Yes
Control panel: No
My client is: certbot 0.31.0

I ran this command: /usr/bin/certbot renew --post-hook "nginx -s reload" --test-cert --break-my-certs

It produced this output:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/kiyo.ooo.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator dns-cloudflare, Installer nginx
Renewing an existing certificate
Performing the following challenges:
dns-01 challenge for kiyo.ooo
dns-01 challenge for kiyo.ooo
Waiting 10 seconds for DNS changes to propagate
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (kiyo.ooo) from /etc/letsencrypt/renewal/kiyo.ooo.conf produced an unexpected error: Failed authorization procedure. kiyo.ooo (dns-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Incorrect TXT record "wy5t6N3WCmTO0w16hqJZKlp6B6x1qumzAJzpiriDlNY" (and 1 more) found at _acme-challenge.kiyo.ooo. Skipping.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/kiyo.ooo/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Running post-hook command: nginx -s reload
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: kiyo.ooo
   Type:   unauthorized
   Detail: Incorrect TXT record
   "wy5t6N3WCmTO0w16hqJZKlp6B6x1qumzAJzpiriDlNY" (and 1 more) found at
   _acme-challenge.kiyo.ooo

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

I checked the logs with verbose enabled, and there is no auth issue with Cloudflare, Certbot can perfectly change my DNS records.
I ran this command with the staging server for a wildcard certificate, because the renewal of my working certificate is unfortunately rate-limited.
What could i do?
Thanks.

Hi @Conobi

checking your domain there is no TXT entry visible. But your SOA record says ( https://check-your-website.server-daten.de/?q=kiyo.ooo ):

|Domain:|kiyo.ooo|
|---|---|
|Primary:|lou.ns.cloudflare.com|
|Mail:|dns.cloudflare.com|
|Serial:|2030650622|
|Refresh:|10000|
|Retry:|2400|
|Expire:|604800|
|TTL:|3600|
|num Entries:|4|

You have a standard TTL of 3600 seconds = one hour. So that

looks too short.

Perhaps add --debug-challenges, then Certbot waits, recheck your domain if the TXT entries are visible. Then confirm the next step.

There are lots of reasons that it could be important to increase this delay, but the TTL isn't a reliable indicator here, because unlike most clients, Let's Encrypt always directly checks the authoritative nameserver. If you have a TTL of 86400 seconds, it doesn't take 86400 seconds until the authoritative nameserver starts to serve the new records; in fact, the amount of time it takes should be independent of the TTL.

The delay before DNS zone changes are served from an authoritative server depends on the DNS provider's infrastructure, and can vary widely, but TTL isn't a good clue to the size of this delay, and it's not directly connected to "propagation delay" as that concept is used in most parts of the DNS world (which does relate to TTL and cached copies of old DNS records).

Cloudflare is supposed to deploy DNS record changes in "<5 seconds". I think it's usually a lot faster than that, too.

Certbot waits 10 seconds by default. (Which is what I choose, too.)

@Conobi

Maybe Cloudflare was just running a little slowly?

I'd suggest trying again one or two times.

If it still doesn't work, try adding "--dns-cloudflare-propagation-seconds 15".

For what it's worth, you can use --dry-run instead of those two options. It will also issue a certificate with the staging environment, but Certbot will just discard it without permanently modifying /etc/letsencrypt/live/.

Edit: FYI, I just successfully issued a few certificates using a 5 second delay and the staging environment.

Thanks for your answers. Howewer, it still doesn’t work, even with --dns-cloudflare-propagation-seconds 30.
I now have more details. The detailed issue is:

Attempting to renew cert (kiyo.ooo) from /etc/letsencrypt/renewal/kiyo.ooo.conf produced an unexpected error: Failed authorization procedure. kiyo.ooo (dns-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Incorrect TXT record "NIzc9k9CXJjxkzsOx9LkGTe2mOKwQelxJ-gap-_3xqQ" (and 1 more) found at _acme-challenge.kiyo.ooo, kiyo.ooo (dns-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Incorrect TXT record "NIzc9k9CXJjxkzsOx9LkGTe2mOKwQelxJ-gap-_3xqQ" (and 1 more) found at _acme-challenge.kiyo.ooo. Skipping.

When I try with --dry-run, I don’t have any errors, everything works perfectly, which is weird?

Also, sometimes Certbot simply doesn’t works before another weird error:

Writing new private key to /etc/letsencrypt/archive/kiyo.ooo/privkey3.pem.
Attempting to renew cert (kiyo.ooo) from /etc/letsencrypt/renewal/kiyo.ooo.conf produced an unexpected error: [Errno 2] No such file or directory: '/etc/letsencrypt/archive/kiyo.ooo/privkey2.pem'. Skipping.
Traceback was:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/certbot/renewal.py", line 452, in handle_renewal_request
    main.renew_cert(lineage_config, plugins, renewal_candidate)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 1193, in renew_cert
    renewed_lineage = _get_and_save_cert(le_client, config, lineage=lineage)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 116, in _get_and_save_cert
    renewal.renew_cert(config, domains, le_client, lineage)
  File "/usr/lib/python3/dist-packages/certbot/renewal.py", line 317, in renew_cert
    lineage.save_successor(prior_version, new_cert, new_key.pem, new_chain, config)
  File "/usr/lib/python3/dist-packages/certbot/storage.py", line 1108, in save_successor
    old_mode = stat.S_IMODE(os.stat(old_privkey).st_mode) & \
FileNotFoundError: [Errno 2] No such file or directory: '/etc/letsencrypt/archive/kiyo.ooo/privkey2.pem'

That's a curious error. Looks like multiple instances of certbot are running.

Or you have some overlapping certificate definitions.

What says

certbot certificates

Oh, what's that?

Rechecked your domain via https://check-your-website.server-daten.de/?q=kiyo.ooo

There are new certificates:

CRT-Id Issuer not before not after Domain names LE-Duplicate next LE
1370430423 CN=Let's Encrypt Authority X3, O=Let's Encrypt, C=US 2019-04-09 15:11:31 2019-07-08 15:11:31 *.kiyo.ooo, kiyo.ooo duplicate nr. 5 next Letsencrypt certificate: 2019-04-10 16:18:34
1368225157 CN=Let's Encrypt Authority X3, O=Let's Encrypt, C=US 2019-04-09 15:10:25 2019-07-08 15:10:25 *.kiyo.ooo, kiyo.ooo duplicate nr. 4
1368073188 CN=Let's Encrypt Authority X3, O=Let's Encrypt, C=US 2019-04-09 01:29:42 2019-07-08 01:29:42 *.kiyo.ooo, kiyo.ooo duplicate nr. 3
1364506503 CN=Let's Encrypt Authority X3, O=Let's Encrypt, C=US 2019-04-08 17:19:17 2019-07-07 17:19:17 *.kiyo.ooo, kiyo.ooo duplicate nr. 2
1360005621 CN=Let's Encrypt Authority X3, O=Let's Encrypt, C=US 2019-04-03 16:18:34 2019-07-02 16:18:34 *.kiyo.ooo, kiyo.ooo duplicate nr. 1
1357640422 CN=Let's Encrypt Authority X3, O=Let's Encrypt, C=US 2019-04-02 13:34:52 2019-07-01 13:34:52 *.kiyo.ooo, kiyo.ooo
1356475481 CN=Let's Encrypt Authority X3, O=Let's Encrypt, C=US 2019-04-01 20:18:00 2019-06-30 20:18:00 *.kiyo.ooo, kiyo.ooo
1356004171 CN=Let's Encrypt Authority X3, O=Let's Encrypt, C=US 2019-04-01 11:04:08 2019-06-30 11:04:08 *.kiyo.ooo, kiyo.ooo
1355637038 CN=Let's Encrypt Authority X3, O=Let's Encrypt, C=US 2019-04-01 05:17:34 2019-06-30 05:17:34 *.kiyo.ooo, kiyo.ooo

5 wildcard certificates created in the last 7 days.

Looks like you have multipe cron jobs. Older weeks - the same picture.

5 certificates between 2019-03-24 and 2019-03-27, same with 2019-03-17 and 2019-03-20 and earlier.

You have 34 active Letsencrypt certificates.

Yep, one of my cron tab has gone wrong; it is why I only use the staging server for now.

  Certificate Name: kiyo.ooo
    Domains: *.kiyo.ooo kiyo.ooo
    Expiry Date: 2019-03-25 18:57:50+00:00 (INVALID: EXPIRED)
    Certificate Path: /etc/letsencrypt/live/kiyo.ooo/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/kiyo.ooo/privkey.pem

The weird thing is the fact that even if I’ve successfully got the certificates, they are not saved on my server.

Can you post “sudo ls -alR /etc/letsencrypt/{archive,live,renewal}”?

Do you have an orphaned process which blocks files?

Perhaps reboot your server.

Here is the results of sudo ls -alR /etc/letsencrypt/{archive,live,renewal} :

/etc/letsencrypt/archive:
total 40
drwx------ 10 root root 4096 avril 10 09:10 .
drwxr-xr-x  9 root root 4096 avril 10 12:10 ..
drwxr-xr-x  2 root root 4096 avril 10 09:23 kiyo.ooo
drwxr-xr-x  2 root root 4096 déc.  25 20:57 kiyo.ooo-0001
drwxr-xr-x  2 root root 4096 avril  9 20:59 kiyo.ooo-0002

/etc/letsencrypt/archive/kiyo.ooo:
total 16
drwxr-xr-x  2 root root 4096 avril 10 09:23 .
drwx------ 10 root root 4096 avril 10 09:10 ..
-rw-------  1 root root 1704 avril 10 09:23 privkey3.pem
-rw-------  1 root root 1704 avril 10 09:11 privkey.pem

/etc/letsencrypt/archive/kiyo.ooo-0001:
total 40
drwxr-xr-x  2 root root 4096 déc.  25 20:57 .
drwx------ 10 root root 4096 avril 10 09:10 ..
-rw-r--r--  1 root root 1907 déc.  25 20:40 cert1.pem
-rw-r--r--  1 root root 1911 déc.  25 20:57 cert2.pem
-rw-r--r--  1 root root 1647 déc.  25 20:40 chain1.pem
-rw-r--r--  1 root root 1647 déc.  25 20:57 chain2.pem
-rw-r--r--  1 root root 3554 déc.  25 20:40 fullchain1.pem
-rw-r--r--  1 root root 3558 déc.  25 20:57 fullchain2.pem
-rw-r--r--  1 root root 1708 déc.  25 20:40 privkey1.pem
-rw-r--r--  1 root root 1704 déc.  25 20:57 privkey2.pem

/etc/letsencrypt/archive/kiyo.ooo-0002:
total 24
drwxr-xr-x  2 root root 4096 avril  9 20:59 .
drwx------ 10 root root 4096 avril 10 09:10 ..
-rw-r--r--  1 root root 1854 avril  9 20:59 cert1.pem
-rw-r--r--  1 root root 1679 avril  9 20:59 chain1.pem
-rw-r--r--  1 root root 3533 avril  9 20:59 fullchain1.pem
-rw-------  1 root root 1708 avril  9 20:59 privkey1.pem

/etc/letsencrypt/live:
total 28
drwx------ 6 root root 4096 avril  9 21:05 .
drwxr-xr-x 9 root root 4096 avril 10 12:10 ..
drwxr-xr-x 2 root root 4096 mars  13 05:00 kiyo.ooo
-rw-r--r-- 1 root root  740 déc.  20 19:01 README

/etc/letsencrypt/live/kiyo.ooo:
total 12
drwxr-xr-x 2 root root 4096 mars  13 05:00 .
drwx------ 6 root root 4096 avril  9 21:05 ..
lrwxrwxrwx 1 root root   37 mars  13 05:00 cert.pem -> ../../archive/kiyo.ooo-0001/cert2.pem
lrwxrwxrwx 1 root root   38 mars  13 05:00 chain.pem -> ../../archive/kiyo.ooo-0001/chain2.pem
lrwxrwxrwx 1 root root   42 mars  13 05:00 fullchain.pem -> ../../archive/kiyo.ooo-0001/fullchain2.pem
lrwxrwxrwx 1 root root   40 mars  13 05:00 privkey.pem -> ../../archive/kiyo.ooo-0001/privkey2.pem
-rw-r--r-- 1 root root  692 déc.  25 20:40 README

/etc/letsencrypt/renewal:
total 24
drwxr-xr-x 2 root root 4096 avril 10 09:26 .
drwxr-xr-x 9 root root 4096 avril 10 12:10 ..
-rw-r--r-- 1 root root  595 mars  13 05:00 kiyo.ooo.conf

I tried to restart the server, it didn’t change anything. It doesn’t seems like a process block the writing of files.

/etc/letsencrypt/archive/kiyo.ooo/ is almost empty.

/etc/letsencrypt/live/kiyo.ooo/ has symlinks to ../../kiyo.ooo-0001/ instead of ../../kiyo.ooo/.

/etc/letsencrypt/live/kiyo.ooo-0001/ and /etc/letsencrypt/live/kiyo.ooo-0002/ are missing.

/etc/letsencrypt/renewal/kiyo.ooo-0001.conf and /etc/letsencrypt/renewal/kiyo.ooo-0002.conf are missing.

Certbot’s not going to be able to save new certificates for kiyo.ooo when its files are that damaged.

Can you also post the contents of /etc/letsencrypt/renewal/kiyo.ooo.conf?

1 Like

It was this. I cleaned up all my files, and remade my symlinks. There is no more issues, thanks everybody for your help!

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.