Symlinks not correct in /live folder

Help
1

Hi,

We have run into a quite urgent problem. The symlinks inside the live folder used to link to

May 13 09:39 cert.pem -> ../../archive/janmaes.be-0003/cert1.pem
May 13 09:39 chain.pem -> ../../archive/janmaes.be-0003/chain1.pem
May 13 09:39 fullchain.pem -> ../../archive/janmaes.be-0003/fullchain1.pem
May 13 09:39 privkey.pem -> ../../archive/janmaes.be-0003/privkey1.pem

However the certificates are now being generated inside the original /archive/janmaes.be folder with an extra increment in ID; Example:

Dec 7 16:39 cert1.pem
May 13 09:39 cert2.pem
Dec 7 16:39 chain1.pem
May 13 09:39 chain2.pem
Dec 7 16:39 fullchain1.pem
May 13 09:39 fullchain2.pem
Dec 7 16:39 privkey1.pem
May 13 09:39 privkey2.pem

The symlinks however are not being updated to this new location.
Does anybody have any idea how to solve this? We have about 50 clients using letsencrypt, I fear the crons have been renewing every time (luckely we run them weekly) and I'm afraid to run into quota problems.

I have solved the problem below by adding an extra domain to the request, the main problem above remains

Additional urgent problem: certbot keeps renewing these certificates, the console shows it generated a new certificated but with a renewel date in the past. But they count against quota. During debugging I locked out maesontime.be,www.maesontime.be:

There were too many requests of a given type :: Error creating new cert :: too many certificates already issued for exact set of domains: maesontime.be,www.maesontime.be

Could someone increase the quota on these so we can restore ssl to the site. I deleted all certs for these domains to fix the problem forgetting about the quota.

2

Hi @maarten,

Could you please show the exact command used in cron?.

I think you are messing the letsencrypt conf. Could you please also show the contents of the files found by this command?:

ls -la /etc/letsencrypt/renewal/*janmaes.be*

Also, please, show the output of these commands:

ls -la /etc/letsencrypt/live/*janmaes.be*
ls -la /etc/letsencrypt/archive/*janmaes.be*

Cheers,
sahsanu

1 Like
3

Hi,

The system has been running for over a year without any problems. A little background info: the letsencrypt service runs inside a docker container inside a jenkins CI system. Reverse proxy rules for each client redirect the signing challenge to this system.
Here are the outputs:

ls -la /etc/letsencrypt/renewal/janmaes.be

-rw-r--r-- 1 root root 590 Feb 11 09:39 /etc/letsencrypt/renewal/janmaes.be-0001.conf
-rw-r--r-- 1 root root 590 Mar 7 19:19 /etc/letsencrypt/renewal/janmaes.be-0002.conf
-rw-r--r-- 1 root root 590 Mar 7 19:24 /etc/letsencrypt/renewal/janmaes.be-0003.conf
-rw-r--r-- 1 root root 565 May 13 09:39 /etc/letsencrypt/renewal/janmaes.be.conf

ls -la /etc/letsencrypt/live/janmaes.be

lrwxrwxrwx 1 root root 39 May 13 09:39 cert.pem -> ../../archive/janmaes.be-0003/cert1.pem
lrwxrwxrwx 1 root root 40 May 13 09:39 chain.pem -> ../../archive/janmaes.be-0003/chain1.pem
lrwxrwxrwx 1 root root 44 May 13 09:39 fullchain.pem -> ../../archive/janmaes.be-0003/fullchain1.pem
lrwxrwxrwx 1 root root 42 May 13 09:39 privkey.pem -> ../../archive/janmaes.be-0003/privkey1.pem
-rw-r--r-- 1 root root 543 Mar 7 19:24 README

ls -la /etc/letsencrypt/archive/janmaes.be

/etc/letsencrypt/archive/janmaes.be:
total 40
drwxr-xr-x 2 root root 4096 Dec 7 16:46 .
drwx------ 102 root root 4096 May 16 19:00 ..
-rw-r--r-- 1 root root 1814 Dec 7 16:39 cert1.pem
-rw-r--r-- 1 root root 1805 May 13 09:39 cert2.pem
-rw-r--r-- 1 root root 1647 Dec 7 16:39 chain1.pem
-rw-r--r-- 1 root root 1647 May 13 09:39 chain2.pem
-rw-r--r-- 1 root root 3461 Dec 7 16:39 fullchain1.pem
-rw-r--r-- 1 root root 3452 May 13 09:39 fullchain2.pem
-rw-r--r-- 1 root root 1708 Dec 7 16:39 privkey1.pem
-rw-r--r-- 1 root root 1704 May 13 09:39 privkey2.pem

/etc/letsencrypt/archive/janmaes.be-0001:
total 40
drwxr-xr-x 2 root root 4096 Feb 11 09:39 .
drwx------ 102 root root 4096 May 16 19:00 ..
-rw-r--r-- 1 root root 1805 Dec 10 09:39 cert1.pem
-rw-r--r-- 1 root root 1805 Feb 11 09:39 cert2.pem
-rw-r--r-- 1 root root 1647 Dec 10 09:39 chain1.pem
-rw-r--r-- 1 root root 1647 Feb 11 09:39 chain2.pem
-rw-r--r-- 1 root root 3452 Dec 10 09:39 fullchain1.pem
-rw-r--r-- 1 root root 3452 Feb 11 09:39 fullchain2.pem
-rw-r--r-- 1 root root 1704 Dec 10 09:39 privkey1.pem
-rw-r--r-- 1 root root 1704 Feb 11 09:39 privkey2.pem

/etc/letsencrypt/archive/janmaes.be-0002:
total 24
drwxr-xr-x 2 root root 4096 Mar 7 19:19 .
drwx------ 102 root root 4096 May 16 19:00 ..
-rw-r--r-- 1 root root 1805 Mar 7 19:19 cert1.pem
-rw-r--r-- 1 root root 1647 Mar 7 19:19 chain1.pem
-rw-r--r-- 1 root root 3452 Mar 7 19:19 fullchain1.pem
-rw-r--r-- 1 root root 1704 Mar 7 19:19 privkey1.pem

/etc/letsencrypt/archive/janmaes.be-0003:
total 24
drwxr-xr-x 2 root root 4096 Mar 7 19:24 .
drwx------ 102 root root 4096 May 16 19:00 ..
-rw-r--r-- 1 root root 1805 Mar 7 19:24 cert1.pem
-rw-r--r-- 1 root root 1647 Mar 7 19:24 chain1.pem
-rw-r--r-- 1 root root 3452 Mar 7 19:24 fullchain1.pem
-rw-r--r-- 1 root root 1704 Mar 7 19:24 privkey1.pem

4

I requested the contents of the above files ;).

You are not showing the complete output, I can't see nor the . .. dir references nor the real dir that should be shown in the first line of the output.

Also, please, show the exact line you use in your cron to renew the certs.

Cheers,
sahsanu

5

This problem is symptomatic of making slight changes to which domain names are included in a certificate request, e.g. adding or removing a single name (like the www prefix). Certbot handles these requests in a way that can be confusing.

What you have here are four different certificate lineages which all relate to the same domain name. certbot renew will try to renew all of them 60 days before their respective expiration dates, while certbot certonly will try to renew a specific one, which may not always be the one that you expect.

I suggest running certbot certificates to get an overview of the contents and renewal status of each one and then thinking about which one matches what you really want. It would be possible to delete the other ones, or alternatively to update one with a name and location that you want to match the contents that you want.

There is no way to change the rate limits in this situation. You can run lectl

to calculate when you will be able to issue certificates again.

If you know the specific commands that you’ve been running that perform the certificate requests and renewals, I can try to tell you more about how this happened and how it might be avoided in the future. I’m sorry that Certbot’s behavior in this case was unexpected.

6

Thanks for the explanation. The command we run at the moment is

./certbot-auto certonly --expand --non-interactive --agree-tos --email *** --webroot -w /tmp/letsencrypt/ -d janmaes.be,www.janmaes.be

Trough the course of the project url's get added and removed like you described so this will probably be the problem. Do you have any suggestions how to avoid this in the future? I guess we will have to make our renewal script a bit smarter and also build in some cleanup work for the lineages we don't want. What is the safest way to proceed now and clean up the old lineages once we find the correct ones? Just delete the conf file?

@sahsanu
I don't know if it's still relevant but here is the complete command

root@jenkins:/home/calibrate# ls -la /etc/letsencrypt/live/janmaes.be
total 12
drwxr-xr-x 2 root root 4096 May 13 09:39 .
drwx------ 94 root root 4096 May 16 19:00 ..
lrwxrwxrwx 1 root root 39 May 13 09:39 cert.pem -> ../../archive/janmaes.be-0003/cert1.pem
lrwxrwxrwx 1 root root 40 May 13 09:39 chain.pem -> ../../archive/janmaes.be-0003/chain1.pem
lrwxrwxrwx 1 root root 44 May 13 09:39 fullchain.pem -> ../../archive/janmaes.be-0003/fullchain1.pem
lrwxrwxrwx 1 root root 42 May 13 09:39 privkey.pem -> ../../archive/janmaes.be-0003/privkey1.pem
-rw-r--r-- 1 root root 543 Mar 7 19:24 README

cat /etc/letsencrypt/renewal/janmaes.be-0001.conf

# renew_before_expiry = 30 days
version = 0.11.1
cert = /etc/letsencrypt/live/janmaes.be-0001/cert.pem
privkey = /etc/letsencrypt/live/janmaes.be-0001/privkey.pem
chain = /etc/letsencrypt/live/janmaes.be-0001/chain.pem
fullchain = /etc/letsencrypt/live/janmaes.be-0001/fullchain.pem
archive_dir = /etc/letsencrypt/archive/janmaes.be-0001

# Options used in the renewal process
[renewalparams]
authenticator = webroot
installer = None
account = 6a7f9bd03c7fdcc03d9d6b5f9f02a1ab
webroot_path = /tmp/letsencrypt,
[[webroot_map]]
janmaes.be = /tmp/letsencrypt
www.janmaes.be = /tmp/letsencrypt

cat /etc/letsencrypt/renewal/janmaes.be-0002.conf

# renew_before_expiry = 30 days
version = 0.12.0
archive_dir = /etc/letsencrypt/archive/janmaes.be-0002
cert = /etc/letsencrypt/live/janmaes.be-0002/cert.pem
privkey = /etc/letsencrypt/live/janmaes.be-0002/privkey.pem
chain = /etc/letsencrypt/live/janmaes.be-0002/chain.pem
fullchain = /etc/letsencrypt/live/janmaes.be-0002/fullchain.pem

# Options used in the renewal process
[renewalparams]
authenticator = webroot
installer = None
account = 6a7f9bd03c7fdcc03d9d6b5f9f02a1ab
webroot_path = /tmp/letsencrypt,
[[webroot_map]]
janmaes.be = /tmp/letsencrypt
www.janmaes.be = /tmp/letsencrypt

cat /etc/letsencrypt/renewal/janmaes.be-0003.conf

# renew_before_expiry = 30 days
version = 0.12.0
archive_dir = /etc/letsencrypt/archive/janmaes.be-0003
cert = /etc/letsencrypt/live/janmaes.be-0003/cert.pem
privkey = /etc/letsencrypt/live/janmaes.be-0003/privkey.pem
chain = /etc/letsencrypt/live/janmaes.be-0003/chain.pem
fullchain = /etc/letsencrypt/live/janmaes.be-0003/fullchain.pem

# Options used in the renewal process
[renewalparams]
authenticator = webroot
installer = None
account = 6a7f9bd03c7fdcc03d9d6b5f9f02a1ab
webroot_path = /tmp/letsencrypt,
[[webroot_map]]
janmaes.be = /tmp/letsencrypt
www.janmaes.be = /tmp/letsencrypt

cat /etc/letsencrypt/renewal/janmaes.be.conf

# renew_before_expiry = 30 days
version = 0.14.0
cert = /etc/letsencrypt/live/janmaes.be/cert.pem
privkey = /etc/letsencrypt/live/janmaes.be/privkey.pem
chain = /etc/letsencrypt/live/janmaes.be/chain.pem
fullchain = /etc/letsencrypt/live/janmaes.be/fullchain.pem
archive_dir = /etc/letsencrypt/archive/janmaes.be

# Options used in the renewal process
[renewalparams]
authenticator = webroot
installer = None
account = 6a7f9bd03c7fdcc03d9d6b5f9f02a1ab
webroot_path = /tmp/letsencrypt,
[[webroot_map]]
janmaes.be = /tmp/letsencrypt
www.janmaes.be = /tmp/letsencrypt
7

You can use the name which certbot certificates shows for the lineage, and specify that with --cert-name. This is exactly the same name as the part before .conf in /etc/letsencrypt/renewal. You can do this for your renewals with certonly (in order to be clear that that is the exact certificate you want to act upon). It is then an alternative to -d. If you want to add or remove domain names, you can specify the complete (new) list of domain names that the certificate should cover with -d, while also specifying which certificate it is with --cert-name.

You can also use the name from certbot certificates to delete a lineage with certbot delete. Alternatively, you can delete all of the files related to the undesired certificates in /etc/letsencrypt/renewal, /etc/letsencrypt/live, and /etc/letsencrypt/archive. certbot delete can do this for you, and if there are other things that ought to be deleted as of a future version of Certbot, it will also handle deleting those.

1 Like
8

ok, thx for the support. We will look into expanding our script with the commands you mentioned.
I’ll mark this one as solved.

9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.