Numbered suffixes in the /live/ directory

[Luc]

Problem: certbot always grabs a new certificate instead of keeping one that is not yet up for renewal (–keep is used). It saves the output to /etc/letsencrypt/live/$domain-0001/ instead of just …/live/$domain/. The next run, it’ll be -0002, etc.

Template:

My domain is: lucgommans.nl

I ran this command: certbot certonly --agree-tos -n --webroot -w /var/www/html/ --keep --cert-name lgms --email user@example.com -d lgms.nl,lucgommans.nl,www.lgms.nl,www.lucgommans.nl

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for lgms.nl
http-01 challenge for lucgommans.nl
http-01 challenge for www.lgms.nl
http-01 challenge for www.lucgommans.nl
Using the webroot path /var/www/html for all unmatched domains.
Waiting for verification…
Cleaning up challenges
live directory exists for lgms

On the second run, the last line of output is omitted; instead, it reports:

IMPORTANT NOTES:

• Congratulations! Your certificate and chain have been saved at:
  /etc/letsencrypt/live/lgms-0001/fullchain.pem
  Your key file has been saved at:
  /etc/letsencrypt/live/lgms-0001/privkey.pem
  Your cert will expire on 2019-06-30. To obtain a new or tweaked
  version of this certificate in the future, simply run certbot
  again. To non-interactively renew all of your certificates, run
  “certbot renew”

(Emphasis mine)

My web server is (include version): nginx

The operating system my web server runs on is (include version): Debian stretch

My hosting provider, if applicable, is: myself

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): certbot 0.28.0

I used a very similar command on my previous server (it used ‘letsencrypt’ instead of ‘certbot’, and the --agree-tos, -n, and --cert-name flags are new, the rest is the same) and it did not have this issue. Running letsencrypt --version gives certbot 0.31.0. It has had this behaviour for years, though, so it’s not as if it could have been broken in 0.28. And it’s a different host so it’s not a downgrade that might mess up data files.

I found a handful of other pages about this, among which this other forum thread, but they seem to be about different things (in the case of the forum thread, I am not requesting partially overlapping names but 100% identical names).

Thanks!

PS. I could not really tell from the forum descriptions whether to post in Server or in Help: the former mentions certbot, but the latter also totally applies, so I chose server as the one with a more specific name.

[jmorahan]

@Luc

live directory exists for lgms

Hmm.

Did you perhaps rename or delete some of the files or directories under /etc/letsencrypt/ at some point? It sounds like maybe the directory layout isn't quite as certbot expects it to be.

Could you share the output of this command?

ls -alR /etc/letsencrypt

[Luc]

etc/letsencrypt:
total 40
drwxr-xr-x  9 root root 4096 Apr  1 23:49 ./
drwxr-xr-x 53 root root 4096 Mar 31 19:35 ../
drwx------  3 root root 4096 Apr  1 22:37 accounts/
drwx------ 12 root root 4096 Apr  1 23:49 archive/
-rw-r--r--  1 root root  121 May 26  2018 cli.ini
drwxr-xr-x  2 root root 4096 Apr  1 23:49 csr/
drwx------  2 root root 4096 Apr  1 23:49 keys/
drwxr-xr-x 13 root root 4096 Apr  1 23:49 live/
drwxr-xr-x  2 root root 4096 Apr  1 23:49 renewal/
drwxr-xr-x  5 root root 4096 Mar 24 19:34 renewal-hooks/

etc/letsencrypt/accounts:
total 12
drwx------ 3 root root 4096 Apr  1 22:37 ./
drwxr-xr-x 9 root root 4096 Apr  1 23:49 ../
drwx------ 3 root root 4096 Apr  1 22:37 acme-v02.api.letsencrypt.org/

etc/letsencrypt/accounts/acme-v02.api.letsencrypt.org:
total 12
drwx------ 3 root root 4096 Apr  1 22:37 ./
drwx------ 3 root root 4096 Apr  1 22:37 ../
drwx------ 3 root root 4096 Apr  1 22:38 directory/

etc/letsencrypt/accounts/acme-v02.api.letsencrypt.org/directory:
total 12
drwx------ 3 root root 4096 Apr  1 22:38 ./
drwx------ 3 root root 4096 Apr  1 22:37 ../
drwx------ 2 root root 4096 Apr  1 22:38 f29e0c3b7039a28444fd8de2d19ec4fd/

etc/letsencrypt/accounts/acme-v02.api.letsencrypt.org/directory/f29e0c3b7039a28444fd8de2d19ec4fd:
total 20
drwx------ 2 root root 4096 Apr  1 22:38 ./
drwx------ 3 root root 4096 Apr  1 22:38 ../
-rw-r--r-- 1 root root   72 Apr  1 22:38 meta.json
-r-------- 1 root root 1632 Apr  1 22:38 private_key.json
-rw-r--r-- 1 root root   78 Apr  1 22:38 regr.json

etc/letsencrypt/archive:
total 48
drwx------ 12 root root 4096 Apr  1 23:49 ./
drwxr-xr-x  9 root root 4096 Apr  1 23:49 ../
drwxr-xr-x  2 root root 4096 Apr  1 22:43 lgms-0001/

etc/letsencrypt/archive/lgms-0001:
total 24
drwxr-xr-x  2 root root 4096 Apr  1 22:43 ./
drwx------ 12 root root 4096 Apr  1 23:49 ../
-rw-r--r--  1 root root 1956 Apr  1 22:43 cert1.pem
-rw-r--r--  1 root root 1647 Apr  1 22:43 chain1.pem
-rw-r--r--  1 root root 3603 Apr  1 22:43 fullchain1.pem
-rw-r--r--  1 root root 1708 Apr  1 22:43 privkey1.pem

etc/letsencrypt/csr:
total 96
drwxr-xr-x 2 root root 4096 Apr  1 23:49 ./
drwxr-xr-x 9 root root 4096 Apr  1 23:49 ../
-rw-r--r-- 1 root root 1115 Apr  1 22:38 0000_csr-certbot.pem
-rw-r--r-- 1 root root  976 Apr  1 22:38 0001_csr-certbot.pem
-rw-r--r-- 1 root root 1175 Apr  1 22:38 0002_csr-certbot.pem
-rw-r--r-- 1 root root  960 Apr  1 22:38 0003_csr-certbot.pem
-rw-r--r-- 1 root root  944 Apr  1 22:38 0004_csr-certbot.pem
-rw-r--r-- 1 root root 1017 Apr  1 22:38 0005_csr-certbot.pem
-rw-r--r-- 1 root root 1289 Apr  1 22:39 0006_csr-certbot.pem
-rw-r--r-- 1 root root 1115 Apr  1 22:43 0007_csr-certbot.pem
-rw-r--r-- 1 root root  976 Apr  1 22:43 0008_csr-certbot.pem
-rw-r--r-- 1 root root  960 Apr  1 22:43 0009_csr-certbot.pem
-rw-r--r-- 1 root root  944 Apr  1 22:44 0010_csr-certbot.pem
-rw-r--r-- 1 root root 1017 Apr  1 22:44 0011_csr-certbot.pem
-rw-r--r-- 1 root root 1208 Apr  1 22:44 0012_csr-certbot.pem
-rw-r--r-- 1 root root 1115 Apr  1 22:47 0013_csr-certbot.pem
-rw-r--r-- 1 root root  976 Apr  1 22:47 0014_csr-certbot.pem
-rw-r--r-- 1 root root 1115 Apr  1 22:49 0015_csr-certbot.pem
-rw-r--r-- 1 root root  976 Apr  1 22:50 0016_csr-certbot.pem
-rw-r--r-- 1 root root 1163 Apr  1 22:51 0017_csr-certbot.pem
-rw-r--r-- 1 root root 1163 Apr  1 22:51 0018_csr-certbot.pem
-rw-r--r-- 1 root root 1013 Apr  1 22:54 0019_csr-certbot.pem
-rw-r--r-- 1 root root 1013 Apr  1 22:54 0020_csr-certbot.pem
-rw-r--r-- 1 root root 1115 Apr  1 23:49 0021_csr-certbot.pem

etc/letsencrypt/keys:
total 96
drwx------ 2 root root 4096 Apr  1 23:49 ./
drwxr-xr-x 9 root root 4096 Apr  1 23:49 ../
-rw------- 1 root root 1704 Apr  1 22:38 0000_key-certbot.pem
-rw------- 1 root root 1704 Apr  1 22:38 0001_key-certbot.pem
-rw------- 1 root root 1704 Apr  1 22:38 0002_key-certbot.pem
-rw------- 1 root root 1704 Apr  1 22:38 0003_key-certbot.pem
-rw------- 1 root root 1708 Apr  1 22:38 0004_key-certbot.pem
-rw------- 1 root root 1704 Apr  1 22:38 0005_key-certbot.pem
-rw------- 1 root root 1708 Apr  1 22:39 0006_key-certbot.pem
-rw------- 1 root root 1704 Apr  1 22:43 0007_key-certbot.pem
-rw------- 1 root root 1708 Apr  1 22:43 0008_key-certbot.pem
-rw------- 1 root root 1704 Apr  1 22:43 0009_key-certbot.pem
-rw------- 1 root root 1704 Apr  1 22:44 0010_key-certbot.pem
-rw------- 1 root root 1704 Apr  1 22:44 0011_key-certbot.pem
-rw------- 1 root root 1704 Apr  1 22:44 0012_key-certbot.pem
-rw------- 1 root root 1704 Apr  1 22:47 0013_key-certbot.pem
-rw------- 1 root root 1704 Apr  1 22:47 0014_key-certbot.pem
-rw------- 1 root root 1704 Apr  1 22:49 0015_key-certbot.pem
-rw------- 1 root root 1704 Apr  1 22:50 0016_key-certbot.pem
-rw------- 1 root root 1704 Apr  1 22:51 0017_key-certbot.pem
-rw------- 1 root root 1708 Apr  1 22:51 0018_key-certbot.pem
-rw------- 1 root root 1704 Apr  1 22:54 0019_key-certbot.pem
-rw------- 1 root root 1704 Apr  1 22:54 0020_key-certbot.pem
-rw------- 1 root root 1704 Apr  1 23:49 0021_key-certbot.pem

etc/letsencrypt/live:
total 56
drwxr-xr-x 13 root root 4096 Apr  1 23:49 ./
drwxr-xr-x  9 root root 4096 Apr  1 23:49 ../
-rw-r--r--  1 root root  740 Apr  1 22:38 README
drwxr-xr-x  2 root root 4096 Apr  1 22:43 lgms/

etc/letsencrypt/live/asdf/lgms:
total 16
drwxr-xr-x 2 root root 4096 Mar 22 18:52 ./
drwxr-xr-x 9 root root 4096 Apr  1 22:54 ../
-rw-r--r-- 1 root root 1216 Mar 22 18:52 fullchain.pem
-rw------- 1 root root 1704 Mar 22 18:52 privkey.pem

etc/letsencrypt/live/lgms:
total 12
drwxr-xr-x  2 root root 4096 Apr  1 22:43 ./
drwxr-xr-x 13 root root 4096 Apr  1 23:49 ../
-rw-r--r--  1 root root  692 Apr  1 22:43 README
lrwxrwxrwx  1 root root   33 Apr  1 22:43 cert.pem -> ../../archive/lgms-0001/cert1.pem
lrwxrwxrwx  1 root root   34 Apr  1 22:43 chain.pem -> ../../archive/lgms-0001/chain1.pem
lrwxrwxrwx  1 root root   38 Apr  1 22:43 fullchain.pem -> ../../archive/lgms-0001/fullchain1.pem
lrwxrwxrwx  1 root root   36 Apr  1 22:43 privkey.pem -> ../../archive/lgms-0001/privkey1.pem

etc/letsencrypt/renewal:
total 48
drwxr-xr-x 2 root root 4096 Apr  1 23:49 ./
drwxr-xr-x 9 root root 4096 Apr  1 23:49 ../
-rw-r--r-- 1 root root    0 Apr  1 22:38 lgms.conf
-rw-r--r-- 1 root root  648 Apr  1 22:43 lgms-0001.conf

etc/letsencrypt/renewal-hooks:
total 20
drwxr-xr-x 5 root root 4096 Mar 24 19:34 ./
drwxr-xr-x 9 root root 4096 Apr  1 23:49 ../
drwxr-xr-x 2 root root 4096 Mar 24 19:34 deploy/
drwxr-xr-x 2 root root 4096 Mar 24 19:34 post/
drwxr-xr-x 2 root root 4096 Mar 24 19:34 pre/

etc/letsencrypt/renewal-hooks/deploy:
total 8
drwxr-xr-x 2 root root 4096 Mar 24 19:34 ./
drwxr-xr-x 5 root root 4096 Mar 24 19:34 ../

etc/letsencrypt/renewal-hooks/post:
total 8
drwxr-xr-x 2 root root 4096 Mar 24 19:34 ./
drwxr-xr-x 5 root root 4096 Mar 24 19:34 ../

etc/letsencrypt/renewal-hooks/pre:
total 8
drwxr-xr-x 2 root root 4096 Mar 24 19:34 ./
drwxr-xr-x 5 root root 4096 Mar 24 19:34 ../

[jmorahan]

Right. So in /etc/letsencrypt/archive you have an lgms-0001 subdirectory, but no lgms. In /etc/letsencrypt/live you have an lgms subdirectory, but no lgms-0001. The symbolic links in /etc/letsencrypt/live/lgms are pointing to the actual files in /etc/letsencrypt/archive/lgms-0001.

This will confuse certbot. It expects that the subdirectories under /etc/letsencrypt/archive and /etc/letsencrypt/live for a given certificate will have the same names.

Since you’re using certonly and --webroot, this should be easy to fix - as long as you’re not already at the rate limit! (note to future readers: this method will probably not work if you’re obtaining certificates a different way):

• First back up the entire /etc/letsencrypt directory and all its contents just in case;
• Delete both /etc/letsencrypt/live/lgms and /etc/letsencrypt/archive/lgms-0001, as well as the two files in /etc/letsencrypt/renewal
• Run your certbot certonly command again to get a new certificate, hopefully with the correct name;
• Then double check your nginx configuration to make sure it’s pointed at the correct location for the certificate files.
• Finally reload nginx to pick up the renewed certificate and verify that everything still works.

If you hit the rate limit, restore your backup and try again later

It’s also possible to fix this without requesting a new certificate, but it’s a bit more complex.

[Luc]

Thanks a lot for the elaborate response!

Looking at the docs, I probably hit the rate limit precisely (“Duplicate Certificate limit of 5 certificates per week”) for one of my less important domains that I tested with, but I could circumvent it by adding a bogus subdomain so it counts as a new certificate (weird that one can circumvent a work limit by adding more work).

It works! It now places the certificates in /live/$certname/, and when running the script again, it reports that the cert is not due for renewal. The final test will be when they are up for renewal, but so far so good

The cause must have been that I added the directories and certificates manually to /etc/letsencrypt/live so that nginx would work with self-signed certificates, before taking the server into production use (putting it on the IP address that the domain names refer to) and requesting real certificates. This issue would probably have been prevented by emptying the /live/ directory before requesting the certificates. I tried that later, but by then there was stuff in /archive/ and other directories, which probably confused certbot (I didn’t know that).

Thanks again for your help!