How to prevent creation of '/etc/letsencrypt/live/domain.tld-0001' when removing domains from a `domain.tld` multidomain certificate?


#6

I totally agree.
The approach I described above worked out as long as no subdomains were removed.
As soon as the old subdomain was gone out of the nameservers the approach of keeping the old subdomain names stopped working.

So I had to modify my scripting to delete any previous certs/config/etc. of that domain.
When I just wanted to update my previous answer here, I realized that you had found exactly the same “solution”. :slightly_smiling:


The /etc/letsencrypt/live folder
#7

Thanks for filing issue #2071; this is a real ongoing limitation of our client.


#8

any updates on this ?


#9

The corresponding GitHub issue #2071 is currently assigned to the 0.6.0 milestone. The project at this moment seems to be working towards 0.5.0 minor release. So not the next, but the “after the next” minor release should make us happy :slight_smile: (unless the milestone is reassigned to a later release).


#10

For all interested, 0.5.0 got released a few hours ago, which means that GitHub issue #2071, which is targeted at 0.6.0 and should solve this problem, is targeted at the next minor release.


#11

Hello!
I know I’m replying years later, but I recently had this issue, and this solution worked for me in the three servers I am running CertBot:

Has far as I understand, sometimes for some reason when you add a domain in your CertBot command when you already have a Certificate, a duplicate Certificate is created, leaving the old one appended with 0001, and creating a new one with the new domain added, without 0001.

The name of a certificate is the same as the first domain after -d in the command.
So for example, if you create a certificate (for example with webroot plugin):

./certbot-auto --webroot -w /var/www/html -n -d example.com -d existing.com

The name of it (you can consult your certificates with ./certbot-auto certificates) would be “example.com

Now if you add a new domain to that certificate, using --expand flag, is when a duplicate certificate would be generated:

./certbot-auto --webroot -w /var/www/html -n --expand -d example.com -d existing.com -d newdomain.com

And you might have a new certifcate called “example.com-0001”. The certificates would be now:

The same when removing a domain from the certificate.

So the solution I found for this is stop using --expand flag in the first place, as the official doc says:

Consider using --cert-name instead of --expand , as it gives more control over which certificate is modified and it lets you remove domains as well as adding them.
(https://certbot.eff.org/docs/using.html#re-creating-and-updating-existing-certificates)

First, you can delete the old certificates and then check that they are not being used in fact:
./certbot-auto delete --cert-name example.com-0001

And now, using --cert-name flag, add the new domain. This flag would add to the Certificate Name specified the domains wanted:

./certbot-auto --webroot -w /var/www/html -n --cert-name example.com -d example.com -d existing.com -d newdomain.com

Remember to add all the domains, otherwise it would “smash” the domains you had certificated. And also remember of the rate limit of 100 domains per certificate.

Hope it helps.

Thanks.


#13

@carito, your advice is good—thanks for sharing your experience.

The reason that the --expand flag might confuse people is that it was originally meant as an alternative to asking users “Do you want to expand the existing cert?” by automatically answering “yes” to this question. However, the trouble is that if the requested certificate is missing any of the old names, that question would never have been presented and --expand is silently ignored (it doesn’t look around for a certificate that the user might have wanted to expand). Perhaps we should change this behavior in some way and present a warning in this case, since it seems like a common source of confusion.

Using --cert-name whenever you want to change the coverage of an existing certificate is definitely the intended approach today.


#14

I’ve read through this thread twice and I’m afraid I’m still not clear on how to proceed with my situation.

I had 2 certs installed on my server (Amazon Linux 2 running Apache): dev.mydomain.com and mydomain.com. The only domain currently active on this server is mydomain.com.

The cert expires on 10/13, so tonight I ran: sudo certbot renew

It couldn’t renew dev.mydomain.com because that domain name now points to another server that’s behind a firewall. No problem, I don’t need a cert for that domain name on this server anymore.

Then the renewal failed for mydomain.com. Here’s what Certbot said:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/mydomain.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/certbot/renewal.py", line 64, in _reconstitute
    renewal_candidate = storage.RenewableCert(full_path, config)
  File "/usr/lib/python2.7/site-packages/certbot/storage.py", line 439, in __init__
    self._check_symlinks()
  File "/usr/lib/python2.7/site-packages/certbot/storage.py", line 498, in _chec k_symlinks
    "expected {0} to be a symlink".format(link))
CertStorageError: expected /etc/letsencrypt/live/mydomain.com/cert.pem to be a symlink
Renewal configuration file /etc/letsencrypt/renewal/mydomain.com.conf is broken. Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/dev.mydomain.com/fullchain.pem (failure)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/dev.mydomain.com/fullchain.pem (failure)

Additionally, the following renewal configurations were invalid:
  /etc/letsencrypt/renewal/mydomain.com.conf (parsefail)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 1 parse failure(s)

Then I ran sudo certbot

It seemed to work fine–it only asked if I wanted a cert for mydomain.com (since dev.mydomain.com is no longer listed in httpd.conf).

But when I checked my cert in the browser the expiration date is still 10/13. I took a peek in /etc/letsencrypt/live and found a new directory named mydomain.com-0001. Is that where my new cert files are?

How do I clean up this mess properly? Thanks!


#15

certbot delete --cert-name mydomain.com-0001

certbot certonly --cert-name mydomain.com -d mydomain.com

You need to specify ---cert-name to reduce the coverage of an existing certificate rather than creating a duplicative certificate (historically, we assumed that people wouldn’t usually want to do this).


#16

Thank you.

I did the delete part and it was fine.

On the second part, I did a --dry-run and got a prompt I haven’t encountered before:

How would you like to authenticate with the ACME CA?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: Apache Web Server plugin - Beta (apache)
2: Spin up a temporary webserver (standalone)
3: Place files in webroot directory (webroot)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Not sure how to proceed here–this is a production server so I don’t really want to just guess and risk downtime on the SSL. Do I choose 1? I’ve run certbot a dozen times or more and never got this prompt, but then again I haven’t used the certonly parameter yet.

Thanks for your help.

Just out of curiosity I ran sudo certbot renew --debug --dry-run

Got this:
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/mydomain.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/certbot/renewal.py", line 64, in _reconstitute
    renewal_candidate = storage.RenewableCert(full_path, config)
  File "/usr/lib/python2.7/site-packages/certbot/storage.py", line 439, in __init__
    self._check_symlinks()
  File "/usr/lib/python2.7/site-packages/certbot/storage.py", line 498, in _check_symlinks
    "expected {0} to be a symlink".format(link))
CertStorageError: expected /etc/letsencrypt/live/mydomain.com/cert.pem to be a symlink
Renewal configuration file /etc/letsencrypt/renewal/mydomain.com.conf is broken. Skipping.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

No renewals were attempted.

Additionally, the following renewal configurations were invalid:
  /etc/letsencrypt/renewal/mydomain.com.conf (parsefail)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Exiting abnormally:
Traceback (most recent call last):
  File "/bin/certbot", line 9, in <module>
    load_entry_point('certbot==0.27.1', 'console_scripts', 'certbot')()
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 1364, in main
    return config.func(config, plugins)
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 1276, in renew
    renewal.handle_renewal_request(config)
  File "/usr/lib/python2.7/site-packages/certbot/renewal.py", line 455, in handle_renewal_request
    len(renew_failures), len(parse_failures)))
Error: 0 renew failure(s), 1 parse failure(s)

Is it saying that /etc/letsencrypt/live/mydomain.com/cert.pem is not a symlink? So it’s an actual file instead?

And /etc/letsencrypt/renewal/mydomain.com.conf is frigged up too?

Would I be better off just deleting the whole /etc/letsencrypt/ directory and starting from scratch? Or can I salvage this somehow?

It is a production server, so be best not to have SSL broken for more than a couple minutes, maybe in the predawn hours. For now SSL seems to be functioning fine. Please advise how to proceed–much appreciated.


#17

Did you previously manually delete or rename any files in /etc/letsencrypt?


#18

I manually deleted /etc/letsencrypt/live/dev.mydomain.com/ at one point.

Should I delete the whole /etc/letsencrypt directory and start over?


#19

That might be an easy way to eliminate the errors from Certbot, since it would eliminate inconsistencies that are confusing Certbot, but it’s not going to be a good way to meet your goal of avoiding server downtime because the copies of your certificates and keys are all located inside that directory. If you delete them, your web server won’t be able to start at all, and won’t be able to serve your site in HTTPS.

So I would suggest either planning for some downtime and proceeding with that plan (in the awareness that HTTPS won’t be available on your site in the meantime and that you may also need to manually edit your web server configuration to disable HTTPS), or else posting the output of

ls -lR /etc/letsencrypt

so we can consider how to undo the effects of deleting /etc/letsencrypt/live/dev.mydomain.com so that Certbot will no longer be confused by that.


#20

Thanks for your help on this.

Here’s the output fromsudo ls -lR /etc/letsencrypt:

/etc/letsencrypt:
total 4
drwx------ 5 root root  122 Aug 17 02:43 accounts
drwx------ 3 root root   36 Oct  5 13:45 archive
drwxr-xr-x 2 root root  118 Oct  4 04:33 csr
drwx------ 2 root root  118 Oct  4 04:33 keys
drwx------ 3 root root   36 Oct  5 13:45 live
-rw-r--r-- 1 root root 1591 Aug 17 02:43 options-ssl-apache.conf
drwxr-xr-x 2 root root   41 Oct  5 13:45 renewal
drwxr-xr-x 5 root root   43 Jul 31 19:29 renewal-hooks

/etc/letsencrypt/accounts:
total 0
drwxr-xr-x 3 root root 23 Aug 17 02:43 acme-staging-v02.api.letsencrypt.org
drwxr-xr-x 3 root root 23 Aug 17 02:43 acme-v01.api.letsencrypt.org
drwx------ 3 root root 23 Jul 31 19:29 acme-v02.api.letsencrypt.org

/etc/letsencrypt/accounts/acme-staging-v02.api.letsencrypt.org:
total 0
drwxr-xr-x 3 root root 46 Aug 17 02:43 directory

/etc/letsencrypt/accounts/acme-staging-v02.api.letsencrypt.org/directory:
total 0
drwxr-xr-x 2 root root 64 Aug 17 02:43 f1ebf87b111af639896242fd4c5b4df9

/etc/letsencrypt/accounts/acme-staging-v02.api.letsencrypt.org/directory/f1ebf87b111af639896242fd4c                                                            5b4df9:
total 12
-rw-r--r-- 1 root root  103 Aug 17 02:43 meta.json
-rw-r--r-- 1 root root 1632 Aug 17 02:43 private_key.json
-rw-r--r-- 1 root root   85 Aug 17 02:43 regr.json

/etc/letsencrypt/accounts/acme-v01.api.letsencrypt.org:
total 0
drwxr-xr-x 3 root root 46 Aug 17 02:43 directory

/etc/letsencrypt/accounts/acme-v01.api.letsencrypt.org/directory:
total 0
drwxr-xr-x 2 root root 64 Aug 17 02:43 7de914e3b34b63cc3cc0b3e4390d5ae3

/etc/letsencrypt/accounts/acme-v01.api.letsencrypt.org/directory/7de914e3b34b63cc3cc0b3e4390d5ae3:
total 12
-rw-r--r-- 1 root root  102 Aug 17 02:43 meta.json
-rw-r--r-- 1 root root 1632 Aug 17 02:43 private_key.json
-rw-r--r-- 1 root root  770 Aug 17 02:43 regr.json

/etc/letsencrypt/accounts/acme-v02.api.letsencrypt.org:
total 0
drwx------ 3 root root 46 Jul 31 19:29 directory

/etc/letsencrypt/accounts/acme-v02.api.letsencrypt.org/directory:
total 0
drwx------ 2 root root 64 Jul 31 19:29 ee5cfe4a5103a47d1d5c10cdd2ecfd06

/etc/letsencrypt/accounts/acme-v02.api.letsencrypt.org/directory/ee5cfe4a5103a47d1d5c10cdd2ecfd06:
total 12
-rw-r--r-- 1 root root  102 Jul 31 19:29 meta.json
-r-------- 1 root root 1632 Jul 31 19:29 private_key.json
-rw-r--r-- 1 root root   78 Jul 31 19:29 regr.json

/etc/letsencrypt/archive:
total 0
drwxr-xr-x 2 root root 160 Aug 17 02:43 mydomain.com

/etc/letsencrypt/archive/mydomain.com:
total 32
-rw-r--r-- 1 root root 2175 Aug 17 02:43 cert1.pem
-rw-r--r-- 1 root root 2171 Aug 17 02:43 cert2.pem
-rw-r--r-- 1 root root 1647 Aug 17 02:43 chain1.pem
-rw-r--r-- 1 root root 1647 Aug 17 02:43 chain2.pem
-rw-r--r-- 1 root root 3822 Aug 17 02:43 fullchain1.pem
-rw-r--r-- 1 root root 3818 Aug 17 02:43 fullchain2.pem
-rw-r--r-- 1 root root 1704 Aug 17 02:43 privkey1.pem
-rw-r--r-- 1 root root 1704 Aug 17 02:43 privkey2.pem

/etc/letsencrypt/csr:
total 16
-rw-r--r-- 1 root root 936 Aug 17 02:43 0000_csr-certbot.pem
-rw-r--r-- 1 root root 936 Aug 17 02:43 0001_csr-certbot.pem
-rw-r--r-- 1 root root 940 Oct  4 04:31 0002_csr-certbot.pem
-rw-r--r-- 1 root root 936 Oct  4 04:33 0003_csr-certbot.pem

/etc/letsencrypt/keys:
total 16
-rw------- 1 root root 1704 Aug 17 02:43 0000_key-certbot.pem
-rw------- 1 root root 1704 Aug 17 02:43 0001_key-certbot.pem
-rw------- 1 root root 1704 Oct  4 04:31 0002_key-certbot.pem
-rw------- 1 root root 1704 Oct  4 04:33 0003_key-certbot.pem

/etc/letsencrypt/live:
total 0
drwxr-xr-x 2 root root 93 Aug 17 02:43 mydomain.com

/etc/letsencrypt/live/mydomain.com:
total 20
-rw-r--r-- 1 root root 2171 Aug 17 02:43 cert.pem
-rw-r--r-- 1 root root 1647 Aug 17 02:43 chain.pem
-rw-r--r-- 1 root root 3818 Aug 17 02:43 fullchain.pem
-rw-r--r-- 1 root root 1704 Aug 17 02:43 privkey.pem
-rw-r--r-- 1 root root  543 Aug 17 02:43 README

/etc/letsencrypt/renewal:
total 4
-rw-r--r-- 1 root root 569 Aug 17 02:43 mydomain.com.conf

/etc/letsencrypt/renewal-hooks:
total 0
drwxr-xr-x 2 root root 6 Jul 31 19:29 deploy
drwxr-xr-x 2 root root 6 Jul 31 19:29 post
drwxr-xr-x 2 root root 6 Jul 31 19:29 pre

/etc/letsencrypt/renewal-hooks/deploy:
total 0

/etc/letsencrypt/renewal-hooks/post:
total 0

/etc/letsencrypt/renewal-hooks/pre:
total 0

#21

The directory structure looks okay… in most ways.

Those files (except README) are supposed to be symlinks to the files in archive. For example, cert.pem is supposed to be a symlink to ../../archive/mydomain.com/cert2.pem at the moment.

Somehow the symlinks got transformed into copies of the files they originally pointed to (or so it appears).

Do you know how that happened? Maybe copying the directory, or restoring a backup? Maybe on August 17 at 02:43?

In any case, Certbot should be happy again once they’re fixed.


#22

I migrated the site to a new server in August and it must’ve happened then. When I copied over the /live/ directory I must not have properly preserved the symlinks. I think there’s a flag you use to do that when decompressing the directory, and I bet what happened is I didn’t use that flag. My best guess anyway.

Is the solution to create new symlinks to the files in archive? What’s the proper way to do that? Would it be:

ln -s /etc/letsencrypt/archive/mydomain.com/cert1.pem /etc/letsencrypt/live/mydomain.com/cert.pem

Then once that’s done, I can run cerbot renew (I’ll try a dry run first)?

Thanks for identifying the issue.


#23

Hi @EdGeis,

The right way should be this:

Before doing any change, please, backup (as user root):

tar zcvf /root/backup_etc_letsencrypt_2018-Oct-06.tar.gz /etc/letsencrypt/

And then, recreate the symbolic links pointing to the last files inside archive/mydomain.com

cd /etc/letsencrypt/live/mydomain.com/
ln -sf ../../archive/mydomain.com/cert2.pem cert.pem
ln -sf ../../archive/mydomain.com/chain2.pem chain.pem
ln -sf ../../archive/mydomain.com/fullchain2.pem fullchain.pem
ln -sf ../../archive/mydomain.com/privkey2.pem privkey.pem

Good luck,
sahsanu


#24

Got it…thank you. Will give this a try (I have a backup server I can test this on by modifying my .hosts file).

So next time it renews, does it add cert3.pem etc. to /etc/letscenrcrypt/archive/mydomain.com? Just wondering about the sequential numerals added to the file names there…


#25

Yes, and it will also automatically update the symlinks to point to cert3.pem and the other files.


#26

Got it, thanks. I just created the symlinks on a backup server and tried a renewal dry-run. Other than the DNS not pointing to the right IP because it’s a different server, everything else checked out OK. In the wee hours tonight I’ll do this on the production server and try a renewal, then report back. Should work…I guess the nuclear option would be to just vaporize /etc/letsencrypt/ and run certbot from the top again.