Certificate renewal impossible, after file system manipulation

Help
1

Hello,

recently i had to update my (allready expired) SSL certificate for my domain. I ran the command certbot renew, which placed a valid certificate in my file system but kept the expired one at the same time. The old one was placed into /etc/letsencrypt/live/watchtrainer.de and the new one in _/etc/letsencrypt/live/watchtrainer.de-0001._Unfortunately the webserver still used the expired one in the first mentioned, old directory.

These two directories have one thing in common. They both contain the key files_fullchain.pem_ and privkey.pem. Obviously, my nginx server still used the old ones to encrypt sessions. So I found myself removing both key files in the old directory (…/watchtrainer.de) to eventually replace them by symbolic links to the new keys in the new folder (…/watchtrainer.de-0001).

The operation was succesfull. The server uses the valid certificate but the command certbot renew displays an error related to my manipulations. Can I somehow completely remove the certs and start from scratch or can i just ignore the errors ?

My domain is:
watchtrainer.de

I ran this command:
certbot renew

It produced this output:

**Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/watchtrainer.de.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator nginx, Installer nginx
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
Renewing an existing certificate
Performing the following challenges:
tls-sni-01 challenge for api.watchtrainer.de
tls-sni-01 challenge for watchtrainer.de
tls-sni-01 challenge for www.watchtrainer.de
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (watchtrainer.de) from /etc/letsencrypt/renewal/watchtrainer.de.conf produced an unexpected error: urn:ietf:params:acme:error:rateLimited :: There were too many requests of a given type :: Error finalizing order :: too many certificates already issued for exact set of domains: api.watchtrainer.de,watchtrainer.de,www.watchtrainer.de: see https://letsencrypt.org/docs/rate-limits/. Skipping.

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/watchtrainer.de-0001.conf
-------------------------------------------------------------------------------
Cert not yet due for renewal
Plugins selected: Authenticator nginx, Installer None
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/watchtrainer.de/fullchain.pem (failure)

-------------------------------------------------------------------------------

The following certs are not due for renewal yet:
  /etc/letsencrypt/live/watchtrainer.de-0001/fullchain.pem expires on 2019-01-30 (skipped)
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/watchtrainer.de/fullchain.pem (failure)
-------------------------------------------------------------------------------
1 renew failure(s), 0 parse failure(s)**

My web server is (include version):
nginx/1.12.2

The operating system my web server runs on is (include version):
CentOS Linux release 7.5.1804 (Core)

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):

2

It looks like you are hitting the "duplicate certificate" rate limit, from issuing the same certificate over and over again. You should check your crontab to see if you have something forcing renewals very rapidly. If not, this was probably a byproduct of trying various commands, and will not be an issue by the time you actually need to renew.

3

Hi @PhilippW,

As @jsha said you have hit the duplicated certificates rate limit (5 certs in 7 days).

CRT ID     CERT TYPE   DOMAIN (CN)      VALID FROM             VALID TO               EXPIRES IN  SANs
909996751  Final cert  watchtrainer.de  2018-Nov-01 14:33 UTC  2019-Jan-30 14:33 UTC  89 days     watchtrainer.de
                                                                
909207271  Final cert  watchtrainer.de  2018-Nov-01 07:41 UTC  2019-Jan-30 07:41 UTC  89 days     api.watchtrainer.de
                                                                                                  watchtrainer.de
                                                                                                  www.watchtrainer.de
                                                                
907682026  Final cert  watchtrainer.de  2018-Oct-31 19:41 UTC  2019-Jan-29 19:41 UTC  89 days     api.watchtrainer.de
                                                                                                  watchtrainer.de
                                                                                                  www.watchtrainer.de
                                                                
906315221  Final cert  watchtrainer.de  2018-Oct-31 07:41 UTC  2019-Jan-29 07:41 UTC  88 days     api.watchtrainer.de
                                                                                                  watchtrainer.de
                                                                                                  www.watchtrainer.de
                                                                
904770107  Final cert  watchtrainer.de  2018-Oct-30 19:41 UTC  2019-Jan-28 19:41 UTC  88 days     api.watchtrainer.de
                                                                                                  watchtrainer.de
                                                                                                  www.watchtrainer.de
                                                                
904255341  Final cert  watchtrainer.de  2018-Oct-30 15:11 UTC  2019-Jan-28 15:11 UTC  87 days     api.watchtrainer.de
                                                                                                  watchtrainer.de
                                                                                                  www.watchtrainer.de
                                                                
904212046  Final cert  watchtrainer.de  2018-Oct-30 14:46 UTC  2019-Jan-28 14:46 UTC  87 days     watchtrainer.de

Your cron job is launched twice every day 07:41 and 19:41 but the problem here is that you have messed the /etc/letsencrypt/ dir structure so certbot thinks the cert located in /etc/letsencrypt/{live,archive}/watchtrainer.de/ needs to be renewed and certbot renews it but it places the resulting cert in the wrong directory, I suppose it is leaving the certs in /etc/letsencrypt/{live,archive}/watchtrainer.de-0001/ so the next time certbot runs, it can’t find the renewed cert and it starts the renew process over again. Maybe it is leaving the certicates in the right dir /etc/letsencrypt/{live,archive}/watchtrainer.de/ but as you have changed the symbolic links certbot is following them to the wrong dir.

Anyway, you should fix the dir structure and use only one renewal conf file for your domain:

Could you please show the output of the following commands?.

ls -lR "/etc/letsencrypt/{archive,live}/watchtrainer*"

cat /etc/letsencrypt/renewal/watchtrainer.de.conf

cat /etc/letsencrypt/renewal/watchtrainer.de-0001.conf

Cheers,
sahsanu

1 Like
4

hello sahsanu, hello jsha, thank you for your fast replies.
The directory structure given by sudo ls -lR /etc/letsencrypt/{archive,live}/is the following:

/etc/letsencrypt/archive/:
insgesamt 4
drwxr-xr-x 2 root root 4096 1. Nov 09:41 watchtrainer.de
drwxr-xr-x 2 root root 152 1. Nov 16:33 watchtrainer.de-0001

/etc/letsencrypt/archive/watchtrainer.de:
insgesamt 224
-rw-r--r-- 1 root root 2212 30. Okt 17:11 cert10.pem
-rw-r--r-- 1 root root 2212 30. Okt 21:41 cert11.pem
-rw-r--r-- 1 root root 2212 31. Okt 09:41 cert12.pem
-rw-r--r-- 1 root root 2212 31. Okt 21:41 cert13.pem
-rw-r--r-- 1 root root 2208 1. Nov 09:41 cert14.pem
-rw-r--r-- 1 root root 1854 20. Mai 2017 cert1.pem
-rw-r--r-- 1 root root 1854 19. Jul 2017 cert2.pem
-rw-r--r-- 1 root root 1854 17. Sep 2017 cert3.pem
-rw-r--r-- 1 root root 1854 16. Nov 2017 cert4.pem
-rw-r--r-- 1 root root 1854 15. Jan 2018 cert5.pem
-rw-r--r-- 1 root root 1854 16. Mär 2018 cert6.pem
-rw-r--r-- 1 root root 2212 15. Mai 21:41 cert7.pem
-rw-r--r-- 1 root root 2212 14. Jul 21:41 cert8.pem
-rw-r--r-- 1 root root 2212 31. Jul 10:59 cert9.pem
-rw-r--r-- 1 root root 1647 30. Okt 17:11 chain10.pem
-rw-r--r-- 1 root root 1647 30. Okt 21:41 chain11.pem
-rw-r--r-- 1 root root 1647 31. Okt 09:41 chain12.pem
-rw-r--r-- 1 root root 1647 31. Okt 21:41 chain13.pem
-rw-r--r-- 1 root root 1647 1. Nov 09:41 chain14.pem
-rw-r--r-- 1 root root 1647 20. Mai 2017 chain1.pem
-rw-r--r-- 1 root root 1647 19. Jul 2017 chain2.pem
-rw-r--r-- 1 root root 1647 17. Sep 2017 chain3.pem
-rw-r--r-- 1 root root 1647 16. Nov 2017 chain4.pem
-rw-r--r-- 1 root root 1647 15. Jan 2018 chain5.pem
-rw-r--r-- 1 root root 1647 16. Mär 2018 chain6.pem
-rw-r--r-- 1 root root 1647 15. Mai 21:41 chain7.pem
-rw-r--r-- 1 root root 1647 14. Jul 21:41 chain8.pem
-rw-r--r-- 1 root root 1647 31. Jul 10:59 chain9.pem
-rw-r--r-- 1 root root 3859 30. Okt 17:11 fullchain10.pem
-rw-r--r-- 1 root root 3859 30. Okt 21:41 fullchain11.pem
-rw-r--r-- 1 root root 3859 31. Okt 09:41 fullchain12.pem
-rw-r--r-- 1 root root 3859 31. Okt 21:41 fullchain13.pem
-rw-r--r-- 1 root root 3855 1. Nov 09:41 fullchain14.pem
-rw-r--r-- 1 root root 3501 20. Mai 2017 fullchain1.pem
-rw-r--r-- 1 root root 3501 19. Jul 2017 fullchain2.pem
-rw-r--r-- 1 root root 3501 17. Sep 2017 fullchain3.pem
-rw-r--r-- 1 root root 3501 16. Nov 2017 fullchain4.pem
-rw-r--r-- 1 root root 3501 15. Jan 2018 fullchain5.pem
-rw-r--r-- 1 root root 3501 16. Mär 2018 fullchain6.pem
-rw-r--r-- 1 root root 3859 15. Mai 21:41 fullchain7.pem
-rw-r--r-- 1 root root 3859 14. Jul 21:41 fullchain8.pem
-rw-r--r-- 1 root root 3859 31. Jul 10:59 fullchain9.pem
-rw-r--r-- 1 root root 1708 30. Okt 17:11 privkey10.pem
-rw-r--r-- 1 root root 1704 30. Okt 21:41 privkey11.pem
-rw-r--r-- 1 root root 1704 31. Okt 09:41 privkey12.pem
-rw-r--r-- 1 root root 1704 31. Okt 21:41 privkey13.pem
-rw-r--r-- 1 root root 1704 1. Nov 09:41 privkey14.pem
-rw-r--r-- 1 root root 1708 20. Mai 2017 privkey1.pem
-rw-r--r-- 1 root root 1704 19. Jul 2017 privkey2.pem
-rw-r--r-- 1 root root 1704 17. Sep 2017 privkey3.pem
-rw-r--r-- 1 root root 1704 16. Nov 2017 privkey4.pem
-rw-r--r-- 1 root root 1708 15. Jan 2018 privkey5.pem
-rw-r--r-- 1 root root 1704 16. Mär 2018 privkey6.pem
-rw-r--r-- 1 root root 1704 15. Mai 21:41 privkey7.pem
-rw-r--r-- 1 root root 1704 14. Jul 21:41 privkey8.pem
-rw-r--r-- 1 root root 1704 31. Jul 10:59 privkey9.pem

/etc/letsencrypt/archive/watchtrainer.de-0001:
insgesamt 32
-rw-r--r-- 1 root root 2155 30. Okt 16:46 cert1.pem
-rw-r--r-- 1 root root 2155 1. Nov 16:33 cert2.pem
-rw-r--r-- 1 root root 1647 30. Okt 16:46 chain1.pem
-rw-r--r-- 1 root root 1647 1. Nov 16:33 chain2.pem
-rw-r--r-- 1 root root 3802 30. Okt 16:46 fullchain1.pem
-rw-r--r-- 1 root root 3802 1. Nov 16:33 fullchain2.pem
-rw-r--r-- 1 root root 1704 30. Okt 16:46 privkey1.pem
-rw-r--r-- 1 root root 1704 1. Nov 16:33 privkey2.pem

/etc/letsencrypt/live/:
insgesamt 0
drwxr-xr-x 2 root root 110 1. Nov 16:36 watchtrainer.de
drwxr-xr-x 2 root root 88 1. Nov 16:33 watchtrainer.de-0001

/etc/letsencrypt/live/watchtrainer.de:
insgesamt 8
lrwxrwxrwx 1 root root 39 1. Nov 16:36 cert.pem -> ../../archive/watchtrainer.de/cert2.pem
lrwxrwxrwx 1 root root 40 1. Nov 16:36 chain.pem -> ../../archive/watchtrainer.de/chain2.pem
-rw-r--r-- 1 root root 424 20. Mai 2017 dhparam2048.pem
lrwxrwxrwx 1 root root 49 1. Nov 16:36 fullchain.pem -> ../../archive/watchtrainer.de-0001/fullchain2.pem
lrwxrwxrwx 1 root root 47 1. Nov 16:36 privkey.pem -> ../../archive/watchtrainer.de-0001/privkey2.pem
-rw-r--r-- 1 root root 543 20. Mai 2017 README

/etc/letsencrypt/live/watchtrainer.de-0001:
insgesamt 4
lrwxrwxrwx 1 root root 44 1. Nov 16:33 cert.pem -> ../../archive/watchtrainer.de-0001/cert2.pem
lrwxrwxrwx 1 root root 45 1. Nov 16:33 chain.pem -> ../../archive/watchtrainer.de-0001/chain2.pem
lrwxrwxrwx 1 root root 49 1. Nov 16:33 fullchain.pem -> ../../archive/watchtrainer.de-0001/fullchain2.pem
lrwxrwxrwx 1 root root 47 1. Nov 16:33 privkey.pem -> ../../archive/watchtrainer.de-0001/privkey2.pem
-rw-r--r-- 1 root root 682 30. Okt 16:46 README

cat /etc/letsencrypt/renewal/watchtrainer.de.conf gives me:

renew_before_expiry = 30 days

version = 0.25.1
archive_dir = /etc/letsencrypt/archive/watchtrainer.de
cert = /etc/letsencrypt/live/watchtrainer.de/cert.pem
privkey = /etc/letsencrypt/live/watchtrainer.de/privkey.pem
chain = /etc/letsencrypt/live/watchtrainer.de/chain.pem
fullchain = /etc/letsencrypt/live/watchtrainer.de/fullchain.pem

Options used in the renewal process

[renewalparams]
authenticator = nginx
installer = nginx
account = 4ec3f6ff4fdd6439877ed3a863f6d21a
server = https://acme-v02.api.letsencrypt.org/directory

and finally cat /etc/letsencrypt/renewal/watchtrainer.de-0001.conf outputs:

renew_before_expiry = 30 days

version = 0.25.1
archive_dir = /etc/letsencrypt/archive/watchtrainer.de-0001
cert = /etc/letsencrypt/live/watchtrainer.de-0001/cert.pem
privkey = /etc/letsencrypt/live/watchtrainer.de-0001/privkey.pem
chain = /etc/letsencrypt/live/watchtrainer.de-0001/chain.pem
fullchain = /etc/letsencrypt/live/watchtrainer.de-0001/fullchain.pem

Options used in the renewal process

[renewalparams]
authenticator = nginx
installer = None
account = 4ec3f6ff4fdd6439877ed3a863f6d21a

best regards,

Philipp

5

Hi @PhilippW,

Here is your problem:

Ok, let's try to fix this mess ;).

All below commands should be performed by root or using a user with sudo privileges:

1.- First, backup:

tar zcvf /root/backup-etc_letsencrypt_2018-Nov-5.tar.gz /etc/letsencrypt/

2.- Fix the symbolic links inside /etc/letsencrypt/live/watchtrainer.de/

cd /etc/letsencrypt/live/watchtrainer.de/
ln -sf ../../archive/watchtrainer.de/cert14.pem cert.pem
ln -sf ../../archive/watchtrainer.de/chain14.pem chain.pem
ln -sf ../../archive/watchtrainer.de/fullchain14.pem fullchain.pem
ln -sf ../../archive/watchtrainer.de/privkey14.pem privkey.pem

3.- Now restart or reload your nginx server:

service nginx restart
or
service nginx reload

Or if you are using systemd

systemctl restart nginx
or
systemctl reload nginx

Now check that you can access your site using https://watchtrainer.de and https://www.watchtrainer.de and if you can access correctly now remove the conf and dirs related to watchtrainer.de-0001

rm -f /etc/letsencrypt/renewal/watchtrainer.de-0001.conf
rm -rf /etc/letsencrypt/live/watchtrainer.de-0001/
rm -rf /etc/letsencrypt/archive/watchtrainer.de-0001/

And you should be done... I hope so :wink:

Good luck,
sahsanu

1 Like
6

Hello Sahsanu,

perfect answer ! Everything works well again and certbot certificates gives me:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

Found the following certs:
Certificate Name: watchtrainer.de
Domains: watchtrainer.de api.watchtrainer.de www.watchtrainer.de
Expiry Date: 2019-01-30 07:41:19+00:00 (VALID: 85 days)
Certificate Path: /etc/letsencrypt/live/watchtrainer.de/fullchain.pem
Private Key Path: /etc/letsencrypt/live/watchtrainer.de/privkey.pem

which is exactly what I wanted.

No I will set up a cron job to automatically renew the certificates. Therefore I thought to simply execute certbot renew in my crontab file once every week.

Best regards,

Philipp

PS: it confused me, that jsha told me about automatic certificate renewal requests, since cronjob -l tells me, that there is no cronjob configured so far for my user account. Could that mean, that there is one allready set up by another user, e.g. root ?

7

Glad it is working... but your site is still using the old cert that is covering only your main domain, did you restart nginx?.

If you installed certbot as a package provided by CentOS then you should already have a cron job added... or a systemd timer.

cat /etc/cron.d/certbot

systemctl list-timers certbot.timer

Cheers,
sahsanu

8

systemctl restart nginx
failed because the control process exited with error code. systemctl status nginx.service gives me:

â—Ź nginx.service - The nginx HTTP and reverse proxy server
Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Mo 2018-11-05 19:15:46 CET; 29s ago
Process: 3419 ExecStart=/usr/sbin/nginx (code=exited, status=1/FAILURE)
Process: 3415 ExecStartPre=/usr/sbin/nginx -t (code=exited, status=0/SUCCESS)
Process: 3413 ExecStartPre=/usr/bin/rm -f /run/nginx.pid (code=exited, status=0/SUCCESS)
Main PID: 34210 (code=killed, signal=KILL)

Nov 05 19:15:45 D-3110S44 nginx[3419]: nginx: [emerg] bind() to [::]:443 failed (98: Address already in use)
Nov 05 19:15:46 D-3110S44 nginx[3419]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
Nov 05 19:15:46 D-3110S44 nginx[3419]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
Nov 05 19:15:46 D-3110S44 nginx[3419]: nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use)
Nov 05 19:15:46 D-3110S44 nginx[3419]: nginx: [emerg] bind() to [::]:443 failed (98: Address already in use)
Nov 05 19:15:46 D-3110S44 nginx[3419]: nginx: [emerg] still could not bind()
Nov 05 19:15:46 D-3110S44 systemd[1]: nginx.service: control process exited, code=exited status=1
Nov 05 19:15:46 D-3110S44 systemd[1]: Failed to start The nginx HTTP and reverse proxy server.
Nov 05 19:15:46 D-3110S44 systemd[1]: Unit nginx.service entered failed state.
Nov 05 19:15:46 D-3110S44 systemd[1]: nginx.service failed.

I had that problem already beforehand by using certbot renew; certbot failed every time, because it wasn't able, to restart nginx subsequently. So I found myself manually killing all nginx processes, running certbot renew and eventually restarting nginx again.

9

That is really strange, could you please show the output of below commands (as root)?

ss -lptn | grep -E ':80|:443'
systemctl stop nginx
ss -lptn | grep -E ':80|:443'
10

yes sure,

the first one gives me:

':80|:443'

[sudo] Passwort fĂĽr watchtrainer:
LISTEN 0 128 *:443 : users:(("nginx",pid=34633,fd=15),("nginx",pid=10642,fd=15),("nginx",pid=10641,fd=15))
LISTEN 0 128 *:80 : users:(("nginx",pid=34633,fd=17),("nginx",pid=10642,fd=17),("nginx",pid=10641,fd=17))
LISTEN 0 1 *:8080 : users:(("java",pid=14024,fd=28))
LISTEN 0 1 :8081 : users:(("java",pid=14024,fd=30))
LISTEN 0 50 :8082 : users:(("java",pid=3455,fd=160))
LISTEN 0 128 :::443 ::: users:(("nginx",pid=34633,fd=14),("nginx",pid=10642,fd=14),("nginx",pid=10641,fd=14))
LISTEN 0 128 :::80 ::: users:(("nginx",pid=34633,fd=18),("nginx",pid=10642,fd=18),("nginx",pid=10641,fd=18))

and after stopping nginx I receive:

LISTEN 0 128 *:443 : users:(("nginx",pid=34633,fd=15),("nginx",pid=10642,fd=15),("nginx",pid=10641,fd=15))
LISTEN 0 128 *:80 : users:(("nginx",pid=34633,fd=17),("nginx",pid=10642,fd=17),("nginx",pid=10641,fd=17))
LISTEN 0 1 *:8080 : users:(("java",pid=14024,fd=28))
LISTEN 0 1 :8081 : users:(("java",pid=14024,fd=30))
LISTEN 0 50 :8082 : users:(("java",pid=3455,fd=160))
LISTEN 0 128 :::443 ::: users:(("nginx",pid=34633,fd=14),("nginx",pid=10642,fd=14),("nginx",pid=10641,fd=14))
LISTEN 0 128 :::80 ::: users:(("nginx",pid=34633,fd=18),("nginx",pid=10642,fd=18),("nginx",pid=10641,fd=18))

11

Show the output of this command:

ps -ef | grep 10641

12

the output is:

watchtr+ 6004 4605 0 19:40 pts/0 00:00:00 grep --color=auto 10641
nginx 10641 34633 0 09:41 ? 00:00:00 nginx: worker process

13

I would expect process with pid 10641 to be the master one but it is a worker. I don’t know what is the reason, we should review carefully your logs but it is not the first time I see nginx stuck on a CentOS machine but no idea what is the root cause.

Kill the nginx processes, if you have killall use it:

killall nginx

If it is still stuck after a while, use sigkill:

killall -s SIGKILL nginx

And once all nginx’s processes are stopped, start it again:

systemctl start nginx

14

Allright. I did so. But how can i figure out,
wether the actual certificate is used or not ? My way at the moment is looking at the used certificate in my browser and compare it with the output of cerbot certificates. But is there a more elegant solution ?

15

Great, I see you are using the right cert now ;).

Checking it with the browser is fine, if you prefer some online tool you could use https://www.sslshopper.com/ssl-checker.html or if you prefer the command line use this:

echo | openssl s_client -connect watchtrainer.de:443 -servername watchtrainer.de 2>/dev/null | openssl x509 -noout -text | grep -E '(DNS|Not )' | sed -e 's/^ *//g'

And you will see an output like this, showing when your cert was issued, when will expire and the domains covered by the cert:

Not Before: Nov  1 07:41:19 2018 GMT
Not After : Jan 30 07:41:19 2019 GMT
DNS:api.watchtrainer.de, DNS:watchtrainer.de, DNS:www.watchtrainer.de
16

allright i see,

thank you very much for your valuable help, Sahsanu and have a good week.

cheers,

Philipp

1 Like
17

You are welcome :wink:

Have a nice week you too.

1 Like
18

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.