How to prevent creation of '/etc/letsencrypt/live/domain.tld-0001' when removing domains from a `domain.tld` multidomain certificate?

Help
1

Hi,

I am analyzing how we would use letsencrypt in our company. I already successfully issued and deployed 2 certs, but now want to think how to make the process scalable and work when using it for all of our websites with as much automation as possible.

I stumbled upon an issue: let’s say you issue a multidomain certificate for domains domain.tld,s1.domain.tld,s2.domain.tld. It is available at /etc/letsencrypt/live/domain.tld, to which I point my web server.

Now I setup a cron script to renew that every once in a while and use it that way successfully for some time. Everybody is happy.

Then I stop using the domain s2.domain.tld for good (or I want to add it to another multidomain certificate) and to keep things clean, I want that domain to no longer be present in the certificate. So I remove the domain from the list - I start renewing only with domain list domain.tld,s1.domain.tld. But at that moment, letsencrypt creates a new certificate location, /etc/letsencrypt/live/domain.tld-0001. Aesthetic issues aside (my eyes really want to explode when looking at the “-0001” stuff), this messes things up as the web server still uses certificates from /etc/letsencrypt/live/domain.tld and not the new ones, so it would require me to reconfigure the webserver to use the new ones. And if I wanted things clean, to also remove the /etc/letsencrypt/live/domain.tld, or better, somehow “move” /etc/letsencrypt/live/domain.tld-0001 to /etc/letsencrypt/live/domain.tld, but I cannot see how I could do that without some downtime (and more importantly, it is a lot of hassle I don’t want to have).

So is there any way to “explain” to letsencrypt that I really want to stop using the s2.domain.tld in that multidomain certificate, and that is should not create a new “branch” of certificates (i.e. domain.tld-0001) and instead really remove the domain s2.domain.tld from the existing (original) certificate “branch” in /etc/letsencrypt/live/domain.tld?

UPDATE: I tried editing /etc/letsencrypt/renewal/domain.tld and remove the s2.domain.tld from the domains key and the corresponding entry in [[webroot_map]] section, but letsencrypt still creates the new domain.tld-0001 “branch” in /etc/letsencrypt (in live, archive, renewal). The /etc/letsencrypt/renewal/domain.tld.conf and /etc/letsencrypt/renewal/domain.tld-0001.conf differ only in this way:

--- /etc/letsencrypt/renewal/domain.tld.conf  2016-01-03 00:05:29.000000000 +0100
+++ /etc/letsencrypt/renewal/domain.tld-0001.conf     2016-01-03 00:05:48.00000000
0 +0100                                                                                            
@@ -1,7 +1,7 @@
-cert = /etc/letsencrypt/live/domain.tld/cert.pem
-privkey = /etc/letsencrypt/live/domain.tld/privkey.pem
-chain = /etc/letsencrypt/live/domain.tld/chain.pem
-fullchain = /etc/letsencrypt/live/domain.tld/fullchain.pem
+cert = /etc/letsencrypt/live/domain.tld-0001/cert.pem
+privkey = /etc/letsencrypt/live/domain.tld-0001/privkey.pem
+chain = /etc/letsencrypt/live/domain.tld-0001/chain.pem
+fullchain = /etc/letsencrypt/live/domain.tld-0001/fullchain.pem
 
 # Options and defaults used in the renewal process
 [renewalparams]
@@ -14,7 +14,7 @@
 installer = none
 config_dir = /etc/letsencrypt-testonly
 text_mode = True
-func = <function obtain_cert at 0x3665500>
+func = <function obtain_cert at 0x2c4c500>
 staging = True
 prepare = False
 work_dir = /var/lib/letsencrypt

None of these two differences seem to really shed any light: the first one is obvious (if newly named “branch” is created, update the paths accordingly), and second one too (one function will have different addresses in two different letsencrypt runs).

END UPDATE

Thanks,
Boris

3 Likes
2

Hello @dusek,

This post won’t be too useful but…

1) Remove a domain to renew existing certificate.

I’ve been testing it with several domains and several options and no matter what I do, I’m always get a new -t0001 domain with new certs. So for this you should be waiting till someone could explain how we could do it (without removing all related data to this existing certificate).

Note: I tested it several times using --duplicate, --expand and --reinstall switches but nothing works, all of them create a new -t000x suffix.

2) Add a domain to renew existing certificate.

I’ve tested it creating a cert for 2 domains and when renewed, I’ve added a new domain and it worked fine using --expand switch, it creates a new cert inside the same dir (without adding -t0001 suffix to domain dir).

Help for expand switch:

  --expand              If an existing cert covers some subset of the
                        requested names, always expand and replace it with the
                        additional names. (default: False)

The config file I’ve used in /etc/letsencrypt/test.domain.tld.ini

staging
text
renew-by-default
agree-tos
register-unsafely-without-email
rsa-key-size = 4096
authenticator = webroot
webroot-path = /path/to/domain.tld
domains = t1.domain.tld,t2.domain.tld

The command used to created the first certificate for 2 domains:

./letsencrypt-auto certonly --config /etc/letsencrypt/test.domain.tld.ini

Once created, I modified the ini file /etc/letsencrypt/test.domain.tld.ini to add a new domain (t3.domain.tld):

staging
text
renew-by-default
agree-tos
register-unsafely-without-email
rsa-key-size = 4096
authenticator = webroot
webroot-path = /path/to/domain.tld
domains = t1.domain.tld,t2.domain.tld,t3.domain.tld

And this time I executed the letsencrypt-auto command but adding --expand switch.

./letsencrypt-auto certonly --config /etc/letsencrypt/test.domain.tld.ini --expand

And it created a new cert for 3 domains in the same place, replacing previous cert.

At least you know that adding new domain won’t break your config :wink:

Cheers,
sahsanu

3

Hello @sahsanu,

Re 1. (removing domain): It’s always nice to hear that someone else has the same issue too, even though it does not help a bit :slightly_smiling: I looked at the letsencrypt code and it seems that letsencrypt gets the list of domains not from the .conf file in /etc/letsencrypt/renewal, but from the certificate itself. So as it stands, until there is an explicit option to tell letsencrypt to remove a SAN domain from a multidomain certificate, what I want to do will not be possible.

Re 2. (adding domain): Yes, I noticed that and it worked nicely for me for quite some time too. Note that according to letsencrypt-auto --help all, --expand is automatically assumed when using the --renew-by-default option (but it’s perhaps better to be explicit just in case).

Btw., I became enough confident about the problem to create a github issue for my problem: https://github.com/letsencrypt/letsencrypt/issues/2071

1 Like
4

Hello,

i have the same issue.
As a workaround my system will remember old subdomains and keeps requesting them along with the current subdomains, although they might exist no longer.
So i will not receive -0001 certs at the price of the cert having (a little) larger size.

Br,
Thomas

5

I used this technique that might be risky.

Scan /etc/letsencrypt for all folders containing domain.tld and delete them.

Run letsecrypt-auto with your new parameters.

I don’t know whether this is a good idea or not. This time it worked for me.

The risk could be that the service could refuse to grant a renewed cert if you can’t prove that you have access to the old one. Though that is only my speculation.

My suggestion would be to add an --alias argument to let the service know what domain to overwrite.

6

I totally agree.
The approach I described above worked out as long as no subdomains were removed.
As soon as the old subdomain was gone out of the nameservers the approach of keeping the old subdomain names stopped working.

So I had to modify my scripting to delete any previous certs/config/etc. of that domain.
When I just wanted to update my previous answer here, I realized that you had found exactly the same “solution”. :slightly_smiling:

7

Thanks for filing issue #2071; this is a real ongoing limitation of our client.

1 Like
8

any updates on this ?

9

The corresponding GitHub issue #2071 is currently assigned to the 0.6.0 milestone. The project at this moment seems to be working towards 0.5.0 minor release. So not the next, but the “after the next” minor release should make us happy :slight_smile: (unless the milestone is reassigned to a later release).

1 Like
10

For all interested, 0.5.0 got released a few hours ago, which means that GitHub issue #2071, which is targeted at 0.6.0 and should solve this problem, is targeted at the next minor release.

2 Likes
11

Hello!
I know I'm replying years later, but I recently had this issue, and this solution worked for me in the three servers I am running CertBot:

Has far as I understand, sometimes for some reason when you add a domain in your CertBot command when you already have a Certificate, a duplicate Certificate is created, leaving the old one appended with 0001, and creating a new one with the new domain added, without 0001.

The name of a certificate is the same as the first domain after -d in the command.
So for example, if you create a certificate (for example with webroot plugin):

./certbot-auto --webroot -w /var/www/html -n -d example.com -d existing.com

The name of it (you can consult your certificates with ./certbot-auto certificates) would be "example.com"

Now if you add a new domain to that certificate, using --expand flag, is when a duplicate certificate would be generated:

./certbot-auto --webroot -w /var/www/html -n --expand -d example.com -d existing.com -d newdomain.com

And you might have a new certifcate called "example.com-0001". The certificates would be now:

The same when removing a domain from the certificate.

So the solution I found for this is stop using --expand flag in the first place, as the official doc says:

Consider using --cert-name instead of --expand , as it gives more control over which certificate is modified and it lets you remove domains as well as adding them.
(User Guide — Certbot 2.7.0.dev0 documentation)

First, you can delete the old certificates and then check that they are not being used in fact:
./certbot-auto delete --cert-name example.com-0001

And now, using --cert-name flag, add the new domain. This flag would add to the Certificate Name specified the domains wanted:

./certbot-auto --webroot -w /var/www/html -n --cert-name example.com -d example.com -d existing.com -d newdomain.com

Remember to add all the domains, otherwise it would "smash" the domains you had certificated. And also remember of the rate limit of 100 domains per certificate.

Hope it helps.

Thanks.

1 Like
13

@carito, your advice is good—thanks for sharing your experience.

The reason that the --expand flag might confuse people is that it was originally meant as an alternative to asking users “Do you want to expand the existing cert?” by automatically answering “yes” to this question. However, the trouble is that if the requested certificate is missing any of the old names, that question would never have been presented and --expand is silently ignored (it doesn’t look around for a certificate that the user might have wanted to expand). Perhaps we should change this behavior in some way and present a warning in this case, since it seems like a common source of confusion.

Using --cert-name whenever you want to change the coverage of an existing certificate is definitely the intended approach today.

1 Like
14

I’ve read through this thread twice and I’m afraid I’m still not clear on how to proceed with my situation.

I had 2 certs installed on my server (Amazon Linux 2 running Apache): dev.mydomain.com and mydomain.com. The only domain currently active on this server is mydomain.com.

The cert expires on 10/13, so tonight I ran: sudo certbot renew

It couldn’t renew dev.mydomain.com because that domain name now points to another server that’s behind a firewall. No problem, I don’t need a cert for that domain name on this server anymore.

Then the renewal failed for mydomain.com. Here’s what Certbot said:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/mydomain.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/certbot/renewal.py", line 64, in _reconstitute
    renewal_candidate = storage.RenewableCert(full_path, config)
  File "/usr/lib/python2.7/site-packages/certbot/storage.py", line 439, in __init__
    self._check_symlinks()
  File "/usr/lib/python2.7/site-packages/certbot/storage.py", line 498, in _chec k_symlinks
    "expected {0} to be a symlink".format(link))
CertStorageError: expected /etc/letsencrypt/live/mydomain.com/cert.pem to be a symlink
Renewal configuration file /etc/letsencrypt/renewal/mydomain.com.conf is broken. Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/dev.mydomain.com/fullchain.pem (failure)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/dev.mydomain.com/fullchain.pem (failure)

Additionally, the following renewal configurations were invalid:
  /etc/letsencrypt/renewal/mydomain.com.conf (parsefail)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 1 parse failure(s)

Then I ran sudo certbot

It seemed to work fine–it only asked if I wanted a cert for mydomain.com (since dev.mydomain.com is no longer listed in httpd.conf).

But when I checked my cert in the browser the expiration date is still 10/13. I took a peek in /etc/letsencrypt/live and found a new directory named mydomain.com-0001. Is that where my new cert files are?

How do I clean up this mess properly? Thanks!

15

certbot delete --cert-name mydomain.com-0001

certbot certonly --cert-name mydomain.com -d mydomain.com

You need to specify ---cert-name to reduce the coverage of an existing certificate rather than creating a duplicative certificate (historically, we assumed that people wouldn't usually want to do this).

16

Thank you.

I did the delete part and it was fine.

On the second part, I did a --dry-run and got a prompt I haven’t encountered before:

How would you like to authenticate with the ACME CA?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: Apache Web Server plugin - Beta (apache)
2: Spin up a temporary webserver (standalone)
3: Place files in webroot directory (webroot)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Not sure how to proceed here–this is a production server so I don’t really want to just guess and risk downtime on the SSL. Do I choose 1? I’ve run certbot a dozen times or more and never got this prompt, but then again I haven’t used the certonly parameter yet.

Thanks for your help.

Just out of curiosity I ran sudo certbot renew --debug --dry-run

Got this:
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/mydomain.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/certbot/renewal.py", line 64, in _reconstitute
    renewal_candidate = storage.RenewableCert(full_path, config)
  File "/usr/lib/python2.7/site-packages/certbot/storage.py", line 439, in __init__
    self._check_symlinks()
  File "/usr/lib/python2.7/site-packages/certbot/storage.py", line 498, in _check_symlinks
    "expected {0} to be a symlink".format(link))
CertStorageError: expected /etc/letsencrypt/live/mydomain.com/cert.pem to be a symlink
Renewal configuration file /etc/letsencrypt/renewal/mydomain.com.conf is broken. Skipping.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

No renewals were attempted.

Additionally, the following renewal configurations were invalid:
  /etc/letsencrypt/renewal/mydomain.com.conf (parsefail)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Exiting abnormally:
Traceback (most recent call last):
  File "/bin/certbot", line 9, in <module>
    load_entry_point('certbot==0.27.1', 'console_scripts', 'certbot')()
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 1364, in main
    return config.func(config, plugins)
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 1276, in renew
    renewal.handle_renewal_request(config)
  File "/usr/lib/python2.7/site-packages/certbot/renewal.py", line 455, in handle_renewal_request
    len(renew_failures), len(parse_failures)))
Error: 0 renew failure(s), 1 parse failure(s)

Is it saying that /etc/letsencrypt/live/mydomain.com/cert.pem is not a symlink? So it’s an actual file instead?

And /etc/letsencrypt/renewal/mydomain.com.conf is frigged up too?

Would I be better off just deleting the whole /etc/letsencrypt/ directory and starting from scratch? Or can I salvage this somehow?

It is a production server, so be best not to have SSL broken for more than a couple minutes, maybe in the predawn hours. For now SSL seems to be functioning fine. Please advise how to proceed–much appreciated.

17

Did you previously manually delete or rename any files in /etc/letsencrypt?

18

I manually deleted /etc/letsencrypt/live/dev.mydomain.com/ at one point.

Should I delete the whole /etc/letsencrypt directory and start over?

19

That might be an easy way to eliminate the errors from Certbot, since it would eliminate inconsistencies that are confusing Certbot, but it's not going to be a good way to meet your goal of avoiding server downtime because the copies of your certificates and keys are all located inside that directory. If you delete them, your web server won't be able to start at all, and won't be able to serve your site in HTTPS.

So I would suggest either planning for some downtime and proceeding with that plan (in the awareness that HTTPS won't be available on your site in the meantime and that you may also need to manually edit your web server configuration to disable HTTPS), or else posting the output of

ls -lR /etc/letsencrypt

so we can consider how to undo the effects of deleting /etc/letsencrypt/live/dev.mydomain.com so that Certbot will no longer be confused by that.

20

Thanks for your help on this.

Here’s the output fromsudo ls -lR /etc/letsencrypt:

/etc/letsencrypt:
total 4
drwx------ 5 root root  122 Aug 17 02:43 accounts
drwx------ 3 root root   36 Oct  5 13:45 archive
drwxr-xr-x 2 root root  118 Oct  4 04:33 csr
drwx------ 2 root root  118 Oct  4 04:33 keys
drwx------ 3 root root   36 Oct  5 13:45 live
-rw-r--r-- 1 root root 1591 Aug 17 02:43 options-ssl-apache.conf
drwxr-xr-x 2 root root   41 Oct  5 13:45 renewal
drwxr-xr-x 5 root root   43 Jul 31 19:29 renewal-hooks

/etc/letsencrypt/accounts:
total 0
drwxr-xr-x 3 root root 23 Aug 17 02:43 acme-staging-v02.api.letsencrypt.org
drwxr-xr-x 3 root root 23 Aug 17 02:43 acme-v01.api.letsencrypt.org
drwx------ 3 root root 23 Jul 31 19:29 acme-v02.api.letsencrypt.org

/etc/letsencrypt/accounts/acme-staging-v02.api.letsencrypt.org:
total 0
drwxr-xr-x 3 root root 46 Aug 17 02:43 directory

/etc/letsencrypt/accounts/acme-staging-v02.api.letsencrypt.org/directory:
total 0
drwxr-xr-x 2 root root 64 Aug 17 02:43 f1ebf87b111af639896242fd4c5b4df9

/etc/letsencrypt/accounts/acme-staging-v02.api.letsencrypt.org/directory/f1ebf87b111af639896242fd4c                                                            5b4df9:
total 12
-rw-r--r-- 1 root root  103 Aug 17 02:43 meta.json
-rw-r--r-- 1 root root 1632 Aug 17 02:43 private_key.json
-rw-r--r-- 1 root root   85 Aug 17 02:43 regr.json

/etc/letsencrypt/accounts/acme-v01.api.letsencrypt.org:
total 0
drwxr-xr-x 3 root root 46 Aug 17 02:43 directory

/etc/letsencrypt/accounts/acme-v01.api.letsencrypt.org/directory:
total 0
drwxr-xr-x 2 root root 64 Aug 17 02:43 7de914e3b34b63cc3cc0b3e4390d5ae3

/etc/letsencrypt/accounts/acme-v01.api.letsencrypt.org/directory/7de914e3b34b63cc3cc0b3e4390d5ae3:
total 12
-rw-r--r-- 1 root root  102 Aug 17 02:43 meta.json
-rw-r--r-- 1 root root 1632 Aug 17 02:43 private_key.json
-rw-r--r-- 1 root root  770 Aug 17 02:43 regr.json

/etc/letsencrypt/accounts/acme-v02.api.letsencrypt.org:
total 0
drwx------ 3 root root 46 Jul 31 19:29 directory

/etc/letsencrypt/accounts/acme-v02.api.letsencrypt.org/directory:
total 0
drwx------ 2 root root 64 Jul 31 19:29 ee5cfe4a5103a47d1d5c10cdd2ecfd06

/etc/letsencrypt/accounts/acme-v02.api.letsencrypt.org/directory/ee5cfe4a5103a47d1d5c10cdd2ecfd06:
total 12
-rw-r--r-- 1 root root  102 Jul 31 19:29 meta.json
-r-------- 1 root root 1632 Jul 31 19:29 private_key.json
-rw-r--r-- 1 root root   78 Jul 31 19:29 regr.json

/etc/letsencrypt/archive:
total 0
drwxr-xr-x 2 root root 160 Aug 17 02:43 mydomain.com

/etc/letsencrypt/archive/mydomain.com:
total 32
-rw-r--r-- 1 root root 2175 Aug 17 02:43 cert1.pem
-rw-r--r-- 1 root root 2171 Aug 17 02:43 cert2.pem
-rw-r--r-- 1 root root 1647 Aug 17 02:43 chain1.pem
-rw-r--r-- 1 root root 1647 Aug 17 02:43 chain2.pem
-rw-r--r-- 1 root root 3822 Aug 17 02:43 fullchain1.pem
-rw-r--r-- 1 root root 3818 Aug 17 02:43 fullchain2.pem
-rw-r--r-- 1 root root 1704 Aug 17 02:43 privkey1.pem
-rw-r--r-- 1 root root 1704 Aug 17 02:43 privkey2.pem

/etc/letsencrypt/csr:
total 16
-rw-r--r-- 1 root root 936 Aug 17 02:43 0000_csr-certbot.pem
-rw-r--r-- 1 root root 936 Aug 17 02:43 0001_csr-certbot.pem
-rw-r--r-- 1 root root 940 Oct  4 04:31 0002_csr-certbot.pem
-rw-r--r-- 1 root root 936 Oct  4 04:33 0003_csr-certbot.pem

/etc/letsencrypt/keys:
total 16
-rw------- 1 root root 1704 Aug 17 02:43 0000_key-certbot.pem
-rw------- 1 root root 1704 Aug 17 02:43 0001_key-certbot.pem
-rw------- 1 root root 1704 Oct  4 04:31 0002_key-certbot.pem
-rw------- 1 root root 1704 Oct  4 04:33 0003_key-certbot.pem

/etc/letsencrypt/live:
total 0
drwxr-xr-x 2 root root 93 Aug 17 02:43 mydomain.com

/etc/letsencrypt/live/mydomain.com:
total 20
-rw-r--r-- 1 root root 2171 Aug 17 02:43 cert.pem
-rw-r--r-- 1 root root 1647 Aug 17 02:43 chain.pem
-rw-r--r-- 1 root root 3818 Aug 17 02:43 fullchain.pem
-rw-r--r-- 1 root root 1704 Aug 17 02:43 privkey.pem
-rw-r--r-- 1 root root  543 Aug 17 02:43 README

/etc/letsencrypt/renewal:
total 4
-rw-r--r-- 1 root root 569 Aug 17 02:43 mydomain.com.conf

/etc/letsencrypt/renewal-hooks:
total 0
drwxr-xr-x 2 root root 6 Jul 31 19:29 deploy
drwxr-xr-x 2 root root 6 Jul 31 19:29 post
drwxr-xr-x 2 root root 6 Jul 31 19:29 pre

/etc/letsencrypt/renewal-hooks/deploy:
total 0

/etc/letsencrypt/renewal-hooks/post:
total 0

/etc/letsencrypt/renewal-hooks/pre:
total 0
21

The directory structure looks okay... in most ways.

Those files (except README) are supposed to be symlinks to the files in archive. For example, cert.pem is supposed to be a symlink to ../../archive/mydomain.com/cert2.pem at the moment.

Somehow the symlinks got transformed into copies of the files they originally pointed to (or so it appears).

Do you know how that happened? Maybe copying the directory, or restoring a backup? Maybe on August 17 at 02:43?

In any case, Certbot should be happy again once they're fixed.