Broken `conf` files contribute to renewal rate limits?

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: aperturemaps.com

I ran this command: certbot renew

It produced this output:


- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/www.aperturemaps.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate for www.aperturemaps.com
Attempting to renew cert (www.aperturemaps.com) from /etc/letsencrypt/renewal/www.aperturemaps.com.conf produced an unexpected error: urn:ietf:params:acme:error:rateLimited :: There were too many requests of a given type :: Error creating new order :: too many certificates already issued for exact set of domains: www.aperturemaps.com: see https://letsencrypt.org/docs/rate-limits/. Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/www.aperturemaps.com/fullchain.pem (failure)

My web server is (include version): apache2

The operating system my web server runs on is (include version): Linux Mint 19.2 (Ubuntu 18.04 LTS)

My hosting provider, if applicable, is: NA

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 1.10.1

Hello all,

I went to renew my certificate this morning. The first time I did so, I got several parse errors from invalid .conf files (probably left around from not setting things up right the first time, definitely my fault here, I've deleted them now) and a renew failure because there was a file in the way:

File exists: '/etc/letsencrypt/archive/www.aperturemaps.com/privkey2.pem'. Skipping.

I've renamed the file that was already in the way. Now, when I go run the certbot renew it tells me that I've hit my rate limit, and checking crt.sh (https://crt.sh/?q=www.aperturemaps.com) I can see that multiple certificates were generated this morning (as well, it would appear, as many over the last couple weeks -- autorenewals, it appears, and again, operator error on my part not trying to figure out renewal ahead of time).

From the rate limits page (https://letsencrypt.org/docs/rate-limits/) it appears that I'm out of luck for the next week, and then I'll need to be more careful and make sure I don't consume all of my renewals in a single go. Is this the case, or can I somehow retrieve the certificate that was generated the first time but couldn't be saved to use that?

Thanks!

1 Like

Something is issuing certificates for your domain on a very regular basis:

Why is that? What's the output of certbot certificates?

1 Like
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
  Certificate Name: www.aperturemaps.com
    Serial Number: <not sure if I should post this or not, 36-character hex string>
    Key Type: RSA
    Domains: www.aperturemaps.com
    Expiry Date: 2020-12-05 19:41:27+00:00 (INVALID: EXPIRED)
    Certificate Path: /etc/letsencrypt/live/www.aperturemaps.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/www.aperturemaps.com/privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 Like

That's public data in the certificate itself, not an issue.

But that's quite strange.. If certbot isn't issuing certificates then what is? Because clearly visible from the crt.sh link above, something is getting certificates regularly. Do you have other clients running? Perhaps you forgot?
How often does your certbot renew run? Do you have a cronjob? Or systemd timer? Or did you start trying to renew since November 5th manually?

Where did all those certificates go? Did you delete them in your clean-up attempts?

In any case, I believe the following certificate is the 5th in a row: https://crt.sh/?id=3719466990

And it was issued on 18:44:56 GMT. So just wait an hour and 15 minutes or so to try again.. And try not to delete that one.

1 Like

Hi @kpresler

your crt.sh output looks like you have different clients.

But that other client isn't able to install the certificate.

So every week the limit is hitted.

1 Like

But that's quite strange.. If certbot isn't issuing certificates then what is? Because clearly visible from the crt.sh link above, something is getting certificates regularly. Do you have other clients running? Perhaps you forgot? How often does your certbot renew run? Do you have a cronjob? Or systemd timer? Or did you start trying to renew since November 5th manually?

I've not done anything manually, don't believe I have any other clients, and no cronjobs. Good call on systemd, that would appear to be it:

root@Ryazan:/etc/systemd/system# cat snap.certbot.renew.timer
[Unit]
# Auto-generated, DO NOT EDIT
Description=Timer renew for snap application certbot.renew
Requires=snap-certbot-793.mount
After=snap-certbot-793.mount
X-Snappy=yes

[Timer]
Unit=snap.certbot.renew.service
OnCalendar=*-*-* 06:42
OnCalendar=*-*-* 18:06

[Install]
WantedBy=timers.target

Could this plausibly be the other client you mentioned? If not, where could I go looking for other clients?

Where did all those certificates go? Did you delete them in your clean-up attempts?

In the cleanup attempts, all I deleted were the two broken .conf files in the /renew folder (/etc/letsencrypt/renewal/aperturemaps.com.conf and /etc/letsencrypt/renewal/www.aperturemaps.com-0001.conf), both of which I think were left over from when I was stumbling through this a couple months ago (never done anything like this before, so a learning process of sorts...))

I don't think that either of those would be certificates unless I'm misunderstanding things.

I'll give the renewal another shot in two hours or so and see if it goes through. Many thanks for the help here. If not I'm sure I'll be back with more noob questions.

1 Like

That's correct, configuration files aren't the certificates. However, it might be that one of those configuration files actually was producing all those unnecessary renewals the past month. You could have a look in /etc/letsencrypt/live/www.aperturemaps.com-0001/ to see if it contains a recent/valid certificate.

Also, to be sure everything is correct now, could you paste the output of /etc/letsencrypt/renewal/www.aperturemaps.com.conf here? And check if /etc/letsencrypt/cli.ini exists. If it does, I'm interested in the contents of that file too.

1 Like

That's correct, configuration files aren't the certificates. However, it might be that one of those configuration files actually was producing all those unnecessary renewals the past month. You could have a look in /etc/letsencrypt/live/www.aperturemaps.com-0001/ to see if it contains a recent/valid certificate.

That folder actually doesn't exist any more, it appears to have gotten deleted back when I was doing the original setup (at least, it's not there, and history shows no indication it was deleted recently)

Also, to be sure everything is correct now, could you paste the output of /etc/letsencrypt/renewal/www.aperturemaps.com.conf here?

root@Ryazan:/etc/letsencrypt/live# cat /etc/letsencrypt/renewal/www.aperturemaps.com.conf
# renew_before_expiry = 30 days
version = 1.9.0
archive_dir = /etc/letsencrypt/archive/www.aperturemaps.com
cert = /etc/letsencrypt/live/www.aperturemaps.com/cert.pem
privkey = /etc/letsencrypt/live/www.aperturemaps.com/privkey.pem
chain = /etc/letsencrypt/live/www.aperturemaps.com/chain.pem
fullchain = /etc/letsencrypt/live/www.aperturemaps.com/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = 3b03ba0eb879549bc6229a20d3ff8216
authenticator = apache
installer = apache
server = https://acme-v02.api.letsencrypt.org/directory

And check if /etc/letsencrypt/cli.ini exists. If it does, I'm interested in the contents of that file too.

root@Ryazan:/etc/letsencrypt/live# cat /etc/letsencrypt/cli.ini
cat: /etc/letsencrypt/cli.ini: No such file or directory

Appears not.

Many thanks, once again!

1 Like

And in the /archive/ folder perhaps?

Also, please check the folder /archive/www.aperturemaps.com/.. Something tells me due to that "File exists" error that the certificates might all be there already, but the symbolic link in /live/ is still pointing to an old version for some reason. If the symbolic link wasn't updated for some reason, it might be the reason certbot doesn't know it renewed the cert already: every time it was thinking it's still due for renewal.

Do a ls -l to check the dates. The expired one should be from 2020-09-06, but perhaps all the others are there too.

1 Like

Seemed easier to show it all at once

root@Ryazan:/etc/letsencrypt/archive# ls -laR
.:
total 20
drwx------ 5 root root 4096 Sep  6 16:41 .
drwxr-xr-x 9 root root 4096 Dec  7 12:28 ..
drwxr-xr-x 2 root root 4096 Sep  6 16:14 aperturemaps.com
drwxr-xr-x 2 root root 4096 Dec  7 10:45 www.aperturemaps.com
drwxr-xr-x 2 root root 4096 Sep  6 16:41 www.aperturemaps.com-0001

./aperturemaps.com:
total 24
drwxr-xr-x 2 root root 4096 Sep  6 16:14 .
drwx------ 5 root root 4096 Sep  6 16:41 ..
-rw-r--r-- 1 root root 1915 Sep  6 16:14 cert1.pem
-rw-r--r-- 1 root root 1647 Sep  6 16:14 chain1.pem
-rw-r--r-- 1 root root 3562 Sep  6 16:14 fullchain1.pem
-rw------- 1 root root 1704 Sep  6 16:14 privkey1.pem

./www.aperturemaps.com:
total 40
drwxr-xr-x 2 root root 4096 Dec  7 10:45 .
drwx------ 5 root root 4096 Sep  6 16:41 ..
-rw-r--r-- 1 root root 1923 Sep  6 16:35 cert1.pem
-rw-r--r-- 1 root root 1927 Nov  5 14:43 cert2.pem
-rw-r--r-- 1 root root 1647 Sep  6 16:35 chain1.pem
-rw-r--r-- 1 root root 1647 Nov  5 14:43 chain2.pem
-rw-r--r-- 1 root root 3570 Sep  6 16:35 fullchain1.pem
-rw-r--r-- 1 root root 3574 Nov  5 14:43 fullchain2.pem
-rw------- 1 root root 1704 Sep  6 16:35 privkey1.pem
-rw------- 1 root root 1708 Nov  5 14:43 privkey2.pem.bak

./www.aperturemaps.com-0001:
total 24
drwxr-xr-x 2 root root 4096 Sep  6 16:41 .
drwx------ 5 root root 4096 Sep  6 16:41 ..
-rw-r--r-- 1 root root 1923 Sep  6 16:41 cert1.pem
-rw-r--r-- 1 root root 1647 Sep  6 16:41 chain1.pem
-rw-r--r-- 1 root root 3570 Sep  6 16:41 fullchain1.pem
-rw------- 1 root root 1704 Sep  6 16:41 privkey1.pem

privkey2.pem was the one I renamed originally based on the error in the OP.

Nov 5 would line up nicely with the 30-day autorenewal that it appears to have tried to do, and then promptly failed, on. Are those *2* files the actually-renewed certificates I can use, if I update the symlinks to point to them or replace the *1* files with them?

1 Like

It could be a valid certificate though, with a corresponding private key in now privkey2.pem.bak. Please show the output of:

ls -l /etc/letsencrypt/live/www.aperturemaps.com/

and

openssl x509 -noout -text -in /etc/letsencrypt/archive/www.aperturemaps.com/cert2.pem

It probably shows a cert with serial number 04:00:dc:a2:c0:03:f9:15:80:e5:78:3d:2b:4f:85:14:84:b3.

The above might give you a working certificate, but it's still pretty strange where all those other certificates came from and went to..?

1 Like

It could be a valid certificate though, with a corresponding private key in now privkey2.pem.bak. Please show the output of:
ls -l /etc/letsencrypt/live/www.aperturemaps.com/

root@Ryazan:/etc/letsencrypt/archive# ls -l /etc/letsencrypt/live/www.aperturemaps.com/
total 4
lrwxrwxrwx 1 root root  49 Nov  5 14:43 cert.pem -> ../../archive/www.aperturemaps.com-0001/cert1.pem
lrwxrwxrwx 1 root root  50 Nov  5 14:43 chain.pem -> ../../archive/www.aperturemaps.com-0001/chain1.pem
lrwxrwxrwx 1 root root  54 Nov  5 14:43 fullchain.pem -> ../../archive/www.aperturemaps.com-0001/fullchain1.pem
lrwxrwxrwx 1 root root  52 Nov  5 14:43 privkey.pem -> ../../archive/www.aperturemaps.com-0001/privkey1.pem
-rw-r--r-- 1 root root 692 Sep  6 16:41 README

Ah. That wouldn't look good. It looks like my symlinks are pointing to the wrong directory.

and
openssl x509 -noout -text -in /etc/letsencrypt/archive/www.aperturemaps.com/cert2.pem

root@Ryazan:/etc/letsencrypt/archive# openssl x509 -noout -text -in /etc/letsencrypt/archive/www.aperturemaps.com/cert2.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            04:00:dc:a2:c0:03:f9:15:80:e5:78:3d:2b:4f:85:14:84:b3
< lots more content > 

but it's still pretty strange where all those other certificates came from and went to..?

¯\(ツ)

Genuinely, no idea. Sorry.

For the time being, am I safe to update the symlinks under live/www.aperturemaps.com to point to the apparently-valid certificates under archive/www.aperturemaps.com/?

1 Like

Wut.. How did that happen? :laughing: Very strange.. Did you update those symlinks manually by any chance?

Yes. You can rename privkey2.pem.bak back again to privkey2.pem and point the symlinks to the correct folder. After that, you should be able to see your certificate from Nov 5th with certbot certificates. Reload any service using the files in /live/ (such as Apache/nginx/Postfix/dovecot/whatever) and you should be good to go.

Please have a look at the crt.sh page you already found the upcoming days. If something still keeps renewing certificates for some reason on a regularly basis without you knowing about it, please search your systems for any left-over ACME client you may still have running, as I believe we have fixed your certbot now.

3 Likes

Wut.. How did that happen? :laughing: Very strange.. Did you update those symlinks manually by any chance?

I don't imagine this is something that just happen so it must be PEBKAC of some sort, but I don't recall doing so.

Yes. You can rename privkey2.pem.bak back again to privkey2.pem and point the symlinks to the correct folder. After that, you should be able to see your certificate from Nov 5th with certbot certificates. Reload any service using the files in /live/ (such as Apache/nginx/Postfix/dovecot/whatever) and you should be good to go.

Done, and we're back in business. HTTPS errors are gone.

Please have a look at the crt.sh page you already found the upcoming days. If something still keeps renewing certificates for some reason on a regularly basis without you knowing about it, please search your systems for any left-over ACME client you may still have running, as I believe we have fixed your certbot now.

Will do.

Thank you so much for the help.

2 Likes

Nice :slight_smile:

You're welcome.

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.