Can't renew cert cause of rateLimited

sahsanu · October 9, 2017, 10:00am

That is your problem, you are reaching the limit of 5 certificates per 7 days for the same subset of domains.

Take a look to rate-limits page.

We also have a Duplicate Certificate limit of 5 certificates per week. A certificate is considered a duplicate of an earlier certificate if they contain the exact same set of hostnames, ignoring capitalization and ordering of hostnames. For instance, if you requested a certificate for the names [www.example.com, example.com], you could request four more certificates for [www.example.com, example.com] during the week. If you changed the set of names by adding [blog.example.com], you would be able to request additional certificates.

Seems you have a cron job (or similar) scheduled to run twice a day between 9:00-10:00 UTC and 21:00-22:00 UTC.

CRT ID     DOMAIN (CN)          VALID FROM             VALID TO               EXPIRES IN  SANs
226861944  mail.cloud-vngl.de   2017-Oct-08 21:59 UTC  2018-Jan-06 21:59 UTC  89 days     cloud-vngl.de
                                                                                          mail.cloud-vngl.de
                                                                                          smtp.cloud-vngl.de
                                                                                          www.cloud-vngl.de

226485135  mail.cloud-vngl.de   2017-Oct-08 09:25 UTC  2018-Jan-06 09:25 UTC  89 days     cloud-vngl.de
                                                                                          mail.cloud-vngl.de
                                                                                          smtp.cloud-vngl.de
                                                                                          www.cloud-vngl.de

223028722  mail.cloud-vngl.de   2017-Oct-03 09:23 UTC  2018-Jan-01 09:23 UTC  84 days     cloud-vngl.de
                                                                                          mail.cloud-vngl.de
                                                                                          smtp.cloud-vngl.de
                                                                                          www.cloud-vngl.de

222745518  mail.cloud-vngl.de   2017-Oct-02 21:41 UTC  2017-Dec-31 21:41 UTC  83 days     cloud-vngl.de
                                                                                          mail.cloud-vngl.de
                                                                                          smtp.cloud-vngl.de
                                                                                          www.cloud-vngl.de

222288599  mail.cloud-vngl.de   2017-Oct-02 09:56 UTC  2017-Dec-31 09:56 UTC  83 days     cloud-vngl.de
                                                                                          mail.cloud-vngl.de
                                                                                          smtp.cloud-vngl.de
                                                                                          www.cloud-vngl.de

221897043  mail.cloud-vngl.de   2017-Oct-01 21:26 UTC  2017-Dec-30 21:26 UTC  82 days     cloud-vngl.de
                                                                                          mail.cloud-vngl.de
                                                                                          smtp.cloud-vngl.de
                                                                                          www.cloud-vngl.de

221544637  mail.cloud-vngl.de   2017-Oct-01 09:59 UTC  2017-Dec-30 09:59 UTC  82 days     cloud-vngl.de
                                                                                          mail.cloud-vngl.de
                                                                                          smtp.cloud-vngl.de
                                                                                          www.cloud-vngl.de
[...]

The problem is that your scheduled task, a cronjob, a systemd timer or similar is trying to force a renew every time it runs and it shouldn't, do you have in your /etc/letsencrypt/cli.ini file or /etc/letsencrypt/renewal/mail.cloud-vngl.de.conf a directive like force-renewal ?. What is the command used in your cronjob, systemd timer or whatever you are using to renew your certs?.

Cheers,
sahsanu

teneco · October 9, 2017, 12:26pm

Hi sahsanu,

i found that strange cron job; it triggers indead twice a day with the following command:

root test -x /usr/bin/certbot -a ! -d /run/systemd/system && perl -e ‘sleep int(rand(3600))’ && certbot -q renew

it is a file “certbot” in /etc/cron.d.

I will remove it; cause there is another cron job that is triggered twice a week just “certbot renew”.

cheers, Stefan

danb35 · October 9, 2017, 1:31pm

That isn't a problem; in fact, it's the recommended way to do the renewal. But normally, "certbot renew" checks your existing certificates, and only renews certs with less than 30 days' validity remaining. For you, it appears to be forcing a renewal every time, which is why you're running into the rate limits. As @sahsanu says, take a look in /etc/letsencrypt/cli.ini or /etc/letsencrypt/renewal/mail.cloud-vngl.de.conf to see if there's anything unusual that might be causing this (you may want to just post the contents of both files; I don't believe they have sensitive information in them).

teneco · October 9, 2017, 2:12pm

The first file doesn't exist. The content of the second one:

# renew_before_expiry = 30 days
version = 0.17.0
cert = /etc/letsencrypt/live/mail.cloud-vngl.de/cert.pem
privkey = /etc/letsencrypt/live/mail.cloud-vngl.de/privkey.pem
chain = /etc/letsencrypt/live/mail.cloud-vngl.de/chain.pem
fullchain = /etc/letsencrypt/live/mail.cloud-vngl.de/fullchain.pem
archive_dir = /etc/letsencrypt/archive/mail.cloud-vngl.de

# Options used in the renewal process
[renewalparams]
authenticator = apache
installer = apache
account = 123456789012345678

mnordhoff · October 9, 2017, 2:24pm

How about “ls -l /etc/letsencrypt/live/mail.cloud-vngl.de/”?

sahsanu · October 9, 2017, 2:32pm

Hi @teneco,

I suppose you installed certbot using Ubuntu package manager, the above certbot cron job means:

If /usr/bin/certbot exists, is executable and the dir /run/systemd/system doesn't exist, execute a sleep command with random time between 0 and 3600 seconds and after that, execute certbot renew command.

So, if your OS is using systemd it will have this dir /run/systemd/system and above certbot command won't run... never because the package of certbot installs a systemd timer, similar to cron.

You can check the timers used in your system:

systemctl list-timers --all

If you want to stop and disable the certbot.timer

systemctl stop certbot.timer
systemctl disable certbot.timer

And to be sure that in case a new certbot update doesn't activate certbot.timer again you could mask it.

systemctl mask certbot.timer

But I think your problem is not the cronjob, nor timer, nor renewal conf. I think you have messed the dir structure used by certbot .

Please, show the output of following commands:

ls -lRa /etc/letsencrypt/live/mail.cloud-vngl.*

ls -lRa /etc/letsencrypt/archive/mail.cloud-vngl.*

Cheers,
sahsanu

bytecamp · October 9, 2017, 3:00pm

Although off-topic, I think this is a poor design decision. There is no benefit in supporting systemd.timer (at least I don’t see one) in favor of and next to crond. Cron is there - use it.

teneco · October 9, 2017, 3:56pm

root@mail: ls -lRa /etc/letsencrypt/live/mail.cloud-vngl.*
/etc/letsencrypt/live/mail.cloud-vngl.de:
insgesamt 12
drwxr-xr-x 2 root root 4096 Okt 9 12:56 .
drwx------ 6 root root 4096 Aug 2 09:55 …
lrwxrwxrwx 1 root root 47 Okt 9 12:56 cert.pem -> …/…/archive/mail.cloud-vngl.de-0001/cert1.pem
lrwxrwxrwx 1 root root 48 Okt 9 12:56 chain.pem -> …/…/archive/mail.cloud-vngl.de-0001/chain1.pem
lrwxrwxrwx 1 root root 52 Okt 9 12:56 fullchain.pem -> …/…/archive/mail.cloud-vngl.de-0001/fullchain1.pem
lrwxrwxrwx 1 root root 50 Okt 9 12:56 privkey.pem -> …/…/archive/mail.cloud-vngl.de-0001/privkey1.pem
-rw-r–r-- 1 root root 543 Jul 27 15:50 README

/etc/letsencrypt/live/mail.cloud-vngl.de-bak:
insgesamt 8
drwxr-xr-x 2 root root 4096 Jan 18 2017 .
drwx------ 6 root root 4096 Aug 2 09:55 …
lrwxrwxrwx 1 root root 42 Jan 18 2017 cert.pem -> …/…/archive/mail.cloud-vngl.de/cert3.pem
lrwxrwxrwx 1 root root 43 Jan 18 2017 chain.pem -> …/…/archive/mail.cloud-vngl.de/chain3.pem
lrwxrwxrwx 1 root root 47 Jan 18 2017 fullchain.pem -> …/…/archive/mail.cloud-vngl.de/fullchain3.pem
lrwxrwxrwx 1 root root 45 Jan 18 2017 privkey.pem -> …/…/archive/mail.cloud-vngl.de/privkey3.pem
root@mail:

root@mail:/# ls -lRa /etc/letsencrypt/archive/mail.cloud-vngl.*
/etc/letsencrypt/archive/mail.cloud-vngl.de:
insgesamt 56
drwxr-xr-x 2 root root 4096 Jan 18 2017 .
drwx------ 6 root root 4096 Jul 27 15:50 …
-rw-r–r-- 1 root root 2232 Nov 24 2016 cert1.pem
-rw-r–r-- 1 root root 1879 Okt 9 12:56 cert2.pem
-rw-r–r-- 1 root root 1960 Jan 18 2017 cert3.pem
-rw-r–r-- 1 root root 1647 Nov 24 2016 chain1.pem
-rw-r–r-- 1 root root 1647 Okt 9 12:56 chain2.pem
-rw-r–r-- 1 root root 1647 Jan 18 2017 chain3.pem
-rw-r–r-- 1 root root 3879 Nov 24 2016 fullchain1.pem
-rw-r–r-- 1 root root 3526 Okt 9 12:56 fullchain2.pem
-rw-r–r-- 1 root root 3607 Jan 18 2017 fullchain3.pem
-rw-r–r-- 1 root root 3272 Nov 24 2016 privkey1.pem
-rw-r–r-- 1 root root 1704 Okt 9 12:56 privkey2.pem
-rw-r–r-- 1 root root 1704 Jan 18 2017 privkey3.pem

/etc/letsencrypt/archive/mail.cloud-vngl.de-0001:
insgesamt 24
drwxr-xr-x 2 root root 4096 Jul 27 15:50 .
drwx------ 6 root root 4096 Jul 27 15:50 …
-rw-r–r-- 1 root root 2224 Jul 27 15:50 cert1.pem
-rw-r–r-- 1 root root 1647 Jul 27 15:50 chain1.pem
-rw-r–r-- 1 root root 3871 Jul 27 15:50 fullchain1.pem
-rw-r–r-- 1 root root 3272 Jul 27 15:50 privkey1.pem
root@mail:/#

teneco · October 9, 2017, 3:59pm

root@mail:/# ls -l /etc/letsencrypt/live/mail.cloud-vngl.de
insgesamt 4
lrwxrwxrwx 1 root root 47 Okt 9 12:56 cert.pem -> ../../archive/mail.cloud-vngl.de-0001/cert1.pem
lrwxrwxrwx 1 root root 48 Okt 9 12:56 chain.pem -> ../../archive/mail.cloud-vngl.de-0001/chain1.pem
lrwxrwxrwx 1 root root 52 Okt 9 12:56 fullchain.pem -> ../../archive/mail.cloud-vngl.de-0001/fullchain1.pem
lrwxrwxrwx 1 root root 50 Okt 9 12:56 privkey.pem -> ../../archive/mail.cloud-vngl.de-0001/privkey1.pem
-rw-r--r-- 1 root root 543 Jul 27 15:50 README
root@mail:/#

sahsanu · October 9, 2017, 4:50pm

@teneco, Uff, you have messed it a bit, it is not a good idea to rename the created dirs and you did it

Let’s see if this certificate is covering your domains.

openssl x509 -in /etc/letsencrypt/archive/mail.cloud-vngl.de/cert2.pem -noout -text | grep -E '(DNS:|Not After)'

And show also the output of these commands:

ls -la /etc/letsencrypt/renewal/

certbot certificates

With the result of above commans lets see if we can fix that mess ;).

Cheers,
sahsanu

teneco · October 9, 2017, 5:55pm

Not After : Jan 7 09:56:29 2018 GMT
DNS:cloud-vngl.de, DNS:mail.cloud-vngl.de, DNS:smtp.cloud-vngl.de, DNS:www.cloud-vngl.de

root@mail:/# ls -la /etc/letsencrypt/renewal/
insgesamt 12
drwxr-xr-x 2 root root 4096 Okt 9 16:14 .
drwxr-xr-x 8 root root 4096 Okt 9 15:34 …
-rw-r–r-- 1 root root 0 Jul 27 15:37 mail.cloud-vngl.de.bak2
-rw-r–r-- 1 root root 493 Okt 9 12:56 mail.cloud-vngl.de.conf
root@mail:
root@mail:/# certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log

Found the following certs:
  Certificate Name: mail.cloud-vngl.de
    Domains: mail.cloud-vngl.de,cloud-vngl.de,smtp.cloud-vngl.de,www.cloud-vngl.de
    Expiry Date: 2017-10-25 12:51:00+00:00 (VALID: 15 days)
    Certificate Path: /etc/letsencrypt/live/mail.cloud-vngl.de/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/mail.cloud-vngl.de/privkey.pem
-------------------------------------------------------------------------------
root@mail:/

sahsanu · October 9, 2017, 6:09pm

I’m leaving now so I can’t answer but I’ll do tomorrow morning.

teneco · October 9, 2017, 6:17pm

thank you a lot so far.
Stefan

sahsanu · October 10, 2017, 8:00am

Hi @teneco,

Ok, lets see if we can fix it.

Login as root and execute below commands:

First, backup… just in case

cd
tar zcvf backup_etc_letsencrypt-2017-Oct-10.tar.gz /etc/letsencrypt/

Now, we will reorganize your certs.

rm -f /etc/letsencrypt/archive/mail.cloud-vngl.de/*
mv /etc/letsencrypt/archive/mail.cloud-vngl.de-0001/* /etc/letsencrypt/archive/mail.cloud-vngl.de/
rm -rf  /etc/letsencrypt/archive/mail.cloud-vngl.de-0001/
cd /etc/letsencrypt/live/mail.cloud-vngl.de/
rm *.pem
ln -s ../../archive/mail.cloud-vngl.de/cert1.pem cert.pem
ln -s ../../archive/mail.cloud-vngl.de/chain1.pem chain.pem
ln -s ../../archive/mail.cloud-vngl.de/fullchain1.pem fullchain.pem
ln -s ../../archive/mail.cloud-vngl.de/privkey1.pem privkey.pem

And now try to renew the cert:

certbot renew

If the cert has been renewed, reload Apache:

service apache2 reload

I hope this helps.

Edit: Added rm *.pem to command list to avoid errors when issuing ln commands.

Cheers,
sahsanu

teneco · October 10, 2017, 9:14am

I got an error here: "The destination ‘/etc/letsencrypt/archive/mail.c’ is not a folder.

sahsanu · October 10, 2017, 9:15am

You need to copy the entire line:

mv /etc/letsencrypt/archive/mail.cloud-vngl.de-0001/* /etc/letsencrypt/archive/mail.cloud-vngl.de/

teneco · October 10, 2017, 9:37am

OK, sorry.
Next problem here:

’ could not establish link ‘cert.pem’, file already exists’
or original:
ln: Die symbolische Verknüpfung ‘cert.pem’ konnte nicht angelegt werden: Die Datei existiert bereits
We are in the folder ‘/etc/letsencrypt/live/mail.cloud-vngl.de/’ and there the files already exist.

sahsanu · October 10, 2017, 9:46am

My fault.

cd /etc/letsencrypt/live/mail.cloud-vngl.de/
rm *.pem

And after that you can continue with ln commands.

teneco · October 10, 2017, 10:01am

Yes it did:
Congratulations, all renewals succeeded. The following certs have been renewed:
/etc/letsencrypt/live/mail.cloud-vngl.de/fullchain.pem (success)

Thank you very much.
In the past I tried to make a backup of one the folders (/etc/letsencrypt/live/mail.cloud-vngl.de-bak), that was not a good idea.

sahsanu · October 10, 2017, 10:09am

Glad you get it working. I can confirm your apache server is showing the new cert

Next time don’t rename/move the dirs

Have a nice day.
sahsanu