Something is broken with rate limits

I run certbot 2 times per month: 7th and 21th. Today is 4th (so over week from last request) and certbot says rate limit reached. How is that possible? I didn’t added new certificates for over month now and i’m requesting 10 certs in total for over 9 domains (just few have more then one cert per domain). Actually 5 because another 5 don’t require renew yet.

Some failed domains: fleshki.net, codonline.net, uniqgames.com.

Hi @Nokim,

You have issued 21 certificates for fleshki.net in last 90 days so something is renewing them. What is the command you are using to renew your certs?, is it being issued by a crontab job or systemd timer?.

Cheers,
sahsanu

Just in case it is useful to you, all this started 30 days ago, June 4th and you have a crontab job or systemd timer running twice a day from 21:00 to 22:00 and from 09:00 to 10:00 (hours are UTC).

CRT ID     CERT TYPE  DOMAIN (CN)  VALID FROM             VALID TO               EXPIRES IN
570295633  Pre cert   fleshki.net  2018-Jul-02 20:25 UTC  2018-Sep-30 20:25 UTC  88 days
561653329  Pre cert   fleshki.net  2018-Jun-27 08:27 UTC  2018-Sep-25 08:27 UTC  82 days
556370783  Pre cert   fleshki.net  2018-Jun-25 20:39 UTC  2018-Sep-23 20:39 UTC  81 days
552784660  Pre cert   fleshki.net  2018-Jun-27 20:56 UTC  2018-Sep-25 20:56 UTC  83 days
551074549  Pre cert   fleshki.net  2018-Jun-26 20:01 UTC  2018-Sep-24 20:01 UTC  82 days
550348166  Pre cert   fleshki.net  2018-Jun-26 08:46 UTC  2018-Sep-24 08:46 UTC  81 days
540371030  Pre cert   fleshki.net  2018-Jun-20 20:15 UTC  2018-Sep-18 20:15 UTC  76 days
539355358  Pre cert   fleshki.net  2018-Jun-20 08:03 UTC  2018-Sep-18 08:03 UTC  75 days
538350223  Pre cert   fleshki.net  2018-Jun-19 20:53 UTC  2018-Sep-17 20:53 UTC  75 days
536733960  Pre cert   fleshki.net  2018-Jun-19 08:55 UTC  2018-Sep-17 08:55 UTC  74 days
535893062  Pre cert   fleshki.net  2018-Jun-18 20:31 UTC  2018-Sep-16 20:31 UTC  74 days
524335531  Pre cert   fleshki.net  2018-Jun-13 20:25 UTC  2018-Sep-11 20:25 UTC  69 days
523897933  Pre cert   fleshki.net  2018-Jun-13 08:17 UTC  2018-Sep-11 08:17 UTC  68 days
522631741  Pre cert   fleshki.net  2018-Jun-12 20:21 UTC  2018-Sep-10 20:21 UTC  68 days
521327330  Pre cert   fleshki.net  2018-Jun-12 08:36 UTC  2018-Sep-10 08:36 UTC  67 days
520386147  Pre cert   fleshki.net  2018-Jun-11 20:53 UTC  2018-Sep-09 20:53 UTC  67 days
510389010  Pre cert   fleshki.net  2018-Jun-06 20:15 UTC  2018-Sep-04 20:15 UTC  62 days
509513639  Pre cert   fleshki.net  2018-Jun-06 08:31 UTC  2018-Sep-04 08:31 UTC  61 days
508201152  Pre cert   fleshki.net  2018-Jun-05 20:23 UTC  2018-Sep-03 20:23 UTC  61 days
507150528  Pre cert   fleshki.net  2018-Jun-05 08:29 UTC  2018-Sep-03 08:29 UTC  60 days
506007160  Pre cert   fleshki.net  2018-Jun-04 20:00 UTC  2018-Sep-02 20:00 UTC  60 days

So you are trying to renew the certs everytime that automatic command runs so maybe you are using some kind of force renew and you shouldn’t.

Cheers,
sahsanu

2 Likes

Command is certbot renew called from crontab. Sure without forced renew. And that is only half of my domains which are so renewed. Others are OK.

This is strange. Twice a day? Can certbot call himself from some place?

UPD: Yes, certbot package installs its own crontab file /etc/cron.d/certbot with line
0 */12 * * * root test -x /usr/bin/certbot -a ! -d /run/systemd/system && perl -e ‘sleep int(rand(3600))’ && certbot -q renew
So the only question why it forces renew for those domains.

Certs for those domains where created with command like this one:
certbot
–renew-by-default
certonly
–email some@gmail.com
-d vnore.net -d www.vnore.net
-a webroot --webroot-path /var/www/html
Can certbot remember that --renew-by-default and if yes how to fix that?

Hi @nokim,

Yes, –renew-by-default is the problem. Could you please show the file /etc/letsencrypt/renewal/fleshki.net.conf ?.

Cheers,
sahsanu

Yes, here

# renew_before_expiry = 30 days
version = 0.10.2
archive_dir = /etc/letsencrypt/archive/fleshki.net
cert = /etc/letsencrypt/live/fleshki.net/cert.pem
privkey = /etc/letsencrypt/live/fleshki.net/privkey.pem
chain = /etc/letsencrypt/live/fleshki.net/chain.pem
fullchain = /etc/letsencrypt/live/fleshki.net/fullchain.pem

# Options used in the renewal process
[renewalparams]
authenticator = webroot
installer = None
account = 3e6299fd6ee2f57e1085dda5ea44da95
[[webroot_map]]
www.fleshki.net = /var/www/html
chat.fleshki.net = /var/www/html
fleshki.net = /var/www/html
test.fleshki.net = /var/www/html

That is correct, show the file /etc/letsencrypt/cli.ini

There is no such file. It existed on another server.

Show the output of these commands:

certbot certificates

certbot renew --dry-run
certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Found the following certs:
  Certificate Name: databank.uniqdir.com
    Domains: databank.uniqdir.com
    Expiry Date: 2018-09-19 20:28:52+00:00 (VALID: 76 days)
    Certificate Path: /etc/letsencrypt/live/databank.uniqdir.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/databank.uniqdir.com/privkey.pem
  Certificate Name: fleshki.net
    Domains: fleshki.net chat.fleshki.net test.fleshki.net www.fleshki.net
    Expiry Date: 2018-07-04 14:29:11+00:00 (INVALID: EXPIRED)
    Certificate Path: /etc/letsencrypt/live/fleshki.net/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/fleshki.net/privkey.pem
  Certificate Name: heroeslands.com
    Domains: heroeslands.com beta.heroeslands.com chat.beta.heroeslands.com chat.era.heroeslands.com chat.heroeslands.com chat.stage.heroeslands.com era.heroeslands.com g1.heroeslands.com g2.heroeslands.com g3.heroeslands.com game2.heroeslands.com img.heroeslands.com my.heroeslands.com stage.heroeslands.com wiki.heroeslands.com www.heroeslands.com
    Expiry Date: 2018-10-02 08:27:11+00:00 (VALID: 89 days)
    Certificate Path: /etc/letsencrypt/live/heroeslands.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/heroeslands.com/privkey.pem
  Certificate Name: inpoll.net
    Domains: inpoll.net beta.inpoll.net beta.inpoll.org inpoll.org mail.inpoll.net mail.inpoll.org www.inpoll.net www.inpoll.org
    Expiry Date: 2018-08-22 20:51:17+00:00 (VALID: 48 days)
    Certificate Path: /etc/letsencrypt/live/inpoll.net/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/inpoll.net/privkey.pem
  Certificate Name: mail.uniqdir.com
    Domains: mail.uniqdir.com
    Expiry Date: 2018-07-04 14:29:33+00:00 (INVALID: EXPIRED)
    Certificate Path: /etc/letsencrypt/live/mail.uniqdir.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/mail.uniqdir.com/privkey.pem
  Certificate Name: codonline.net
    Domains: codonline.net chat.codonline.net g1.codonline.net g2.codonline.net g3.codonline.net img.codonline.net my.codonline.net wiki.codonline.net www.codonline.net
    Expiry Date: 2018-07-04 14:28:55+00:00 (INVALID: EXPIRED)
    Certificate Path: /etc/letsencrypt/live/codonline.net/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/codonline.net/privkey.pem
  Certificate Name: uniqgames.com
    Domains: uniqgames.com runes.uniqgames.com www.uniqgames.com
    Expiry Date: 2018-07-04 14:30:03+00:00 (INVALID: EXPIRED)
    Certificate Path: /etc/letsencrypt/live/uniqgames.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/uniqgames.com/privkey.pem
  Certificate Name: dav.uniqdir.com
    Domains: dav.uniqdir.com
    Expiry Date: 2018-09-02 20:01:21+00:00 (VALID: 59 days)
    Certificate Path: /etc/letsencrypt/live/dav.uniqdir.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/dav.uniqdir.com/privkey.pem
  Certificate Name: oca.com.ua
    Domains: oca.com.ua oca.guru oca.uniqdir.com www.oca.com.ua www.oca.guru
    Expiry Date: 2018-08-09 20:04:48+00:00 (VALID: 35 days)
    Certificate Path: /etc/letsencrypt/live/oca.com.ua/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/oca.com.ua/privkey.pem
  Certificate Name: vnore.net
    Domains: vnore.net www.vnore.net
    Expiry Date: 2018-07-04 14:29:22+00:00 (INVALID: EXPIRED)
    Certificate Path: /etc/letsencrypt/live/vnore.net/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/vnore.net/privkey.pem
-------------------------------------------------------------------------------
certbot renew --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/databank.uniqdir.com.conf
-------------------------------------------------------------------------------
Cert not due for renewal, but simulating renewal for dry run
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for databank.uniqdir.com
Waiting for verification...
Cleaning up challenges
Generating key (2048 bits): /etc/letsencrypt/keys/0347_key-certbot.pem
Creating CSR: /etc/letsencrypt/csr/0347_csr-certbot.pem

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/fleshki.net.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for fleshki.net
http-01 challenge for chat.fleshki.net
http-01 challenge for test.fleshki.net
http-01 challenge for www.fleshki.net
Waiting for verification...
Cleaning up challenges
Generating key (2048 bits): /etc/letsencrypt/keys/0348_key-certbot.pem
Creating CSR: /etc/letsencrypt/csr/0348_csr-certbot.pem

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/heroeslands.com.conf
-------------------------------------------------------------------------------
Cert not due for renewal, but simulating renewal for dry run
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for heroeslands.com
http-01 challenge for beta.heroeslands.com
http-01 challenge for chat.beta.heroeslands.com
http-01 challenge for chat.era.heroeslands.com
http-01 challenge for chat.heroeslands.com
http-01 challenge for chat.stage.heroeslands.com
http-01 challenge for era.heroeslands.com
http-01 challenge for g1.heroeslands.com
http-01 challenge for g2.heroeslands.com
http-01 challenge for g3.heroeslands.com
http-01 challenge for game2.heroeslands.com
http-01 challenge for img.heroeslands.com
http-01 challenge for my.heroeslands.com
http-01 challenge for stage.heroeslands.com
http-01 challenge for wiki.heroeslands.com
http-01 challenge for www.heroeslands.com
Waiting for verification...
Cleaning up challenges
Generating key (2048 bits): /etc/letsencrypt/keys/0349_key-certbot.pem
Creating CSR: /etc/letsencrypt/csr/0349_csr-certbot.pem

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/inpoll.net.conf
-------------------------------------------------------------------------------
Cert not due for renewal, but simulating renewal for dry run
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for inpoll.net
http-01 challenge for beta.inpoll.net
http-01 challenge for beta.inpoll.org
http-01 challenge for inpoll.org
http-01 challenge for mail.inpoll.net
http-01 challenge for mail.inpoll.org
http-01 challenge for www.inpoll.net
http-01 challenge for www.inpoll.org
Waiting for verification...
Cleaning up challenges
Generating key (2048 bits): /etc/letsencrypt/keys/0350_key-certbot.pem
Creating CSR: /etc/letsencrypt/csr/0350_csr-certbot.pem

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/mail.uniqdir.com.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for mail.uniqdir.com
Waiting for verification...
Cleaning up challenges
Generating key (2048 bits): /etc/letsencrypt/keys/0351_key-certbot.pem
Creating CSR: /etc/letsencrypt/csr/0351_csr-certbot.pem

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/codonline.net.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for codonline.net
http-01 challenge for chat.codonline.net
http-01 challenge for g1.codonline.net
http-01 challenge for g2.codonline.net
http-01 challenge for g3.codonline.net
http-01 challenge for img.codonline.net
http-01 challenge for my.codonline.net
http-01 challenge for wiki.codonline.net
http-01 challenge for www.codonline.net
Waiting for verification...
Cleaning up challenges
Generating key (2048 bits): /etc/letsencrypt/keys/0352_key-certbot.pem
Creating CSR: /etc/letsencrypt/csr/0352_csr-certbot.pem

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/uniqgames.com.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for uniqgames.com
http-01 challenge for runes.uniqgames.com
http-01 challenge for www.uniqgames.com
Waiting for verification...
Cleaning up challenges
Generating key (2048 bits): /etc/letsencrypt/keys/0353_key-certbot.pem
Creating CSR: /etc/letsencrypt/csr/0353_csr-certbot.pem

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/dav.uniqdir.com.conf
-------------------------------------------------------------------------------
Cert not due for renewal, but simulating renewal for dry run
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for dav.uniqdir.com
Waiting for verification...
Cleaning up challenges
Generating key (2048 bits): /etc/letsencrypt/keys/0354_key-certbot.pem
Creating CSR: /etc/letsencrypt/csr/0354_csr-certbot.pem

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/oca.com.ua.conf
-------------------------------------------------------------------------------
Cert not due for renewal, but simulating renewal for dry run
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for oca.com.ua
http-01 challenge for oca.guru
http-01 challenge for oca.uniqdir.com
http-01 challenge for www.oca.com.ua
http-01 challenge for www.oca.guru
Waiting for verification...
Cleaning up challenges
Generating key (2048 bits): /etc/letsencrypt/keys/0355_key-certbot.pem
Creating CSR: /etc/letsencrypt/csr/0355_csr-certbot.pem

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/vnore.net.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for vnore.net
http-01 challenge for www.vnore.net
Waiting for verification...
Cleaning up challenges
Generating key (2048 bits): /etc/letsencrypt/keys/0356_key-certbot.pem
Creating CSR: /etc/letsencrypt/csr/0356_csr-certbot.pem
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

Congratulations, all renewals succeeded. The following certs have been renewed:
  /etc/letsencrypt/live/databank.uniqdir.com/fullchain.pem (success)
  /etc/letsencrypt/live/fleshki.net/fullchain.pem (success)
  /etc/letsencrypt/live/heroeslands.com/fullchain.pem (success)
  /etc/letsencrypt/live/inpoll.net/fullchain.pem (success)
  /etc/letsencrypt/live/mail.uniqdir.com/fullchain.pem (success)
  /etc/letsencrypt/live/codonline.net/fullchain.pem (success)
  /etc/letsencrypt/live/uniqgames.com/fullchain.pem (success)
  /etc/letsencrypt/live/dav.uniqdir.com/fullchain.pem (success)
  /etc/letsencrypt/live/oca.com.ua/fullchain.pem (success)
  /etc/letsencrypt/live/vnore.net/fullchain.pem (success)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)

IMPORTANT NOTES:
 - Your account credentials have been saved in your Certbot
   configuration directory at /etc/letsencrypt. You should make a
   secure backup of this folder now. This configuration directory will
   also contain certificates and private keys obtained by Certbot so
   making regular backups of this folder is ideal.

That is strange, your cert for fleshki.net and the others that are having issues are expired so certbot is trying to renew them every time it runs so the problem is really that, did you modify manually any of the dirs, files or symlinks inside /etc/letsencrypt?.

Show the output of these commands

ls -la /etc/letsencrypt/live/fleshki.net/
ls -la /etc/letsencrypt/archive/fleshki.net/

Yes, I messed them up somewhere in March after moving to mew server but after that April and May all was OK.
ls -la /etc/letsencrypt/live/fleshki.net/
total 12
drwxr-xr-x 2 root root 4096 лип 5 00:16 .
drwx------ 13 1000 www-data 4096 кві 5 18:32 ..
-rw-r--r-- 1 root root 543 кві 5 18:29 README
lrwxrwxrwx 1 root root 40 лип 5 00:16 cert.pem -> ../../archive/fleshki.net-0001/cert1.pem
lrwxrwxrwx 1 root root 41 лип 5 00:16 chain.pem -> ../../archive/fleshki.net-0001/chain1.pem
lrwxrwxrwx 1 root root 45 лип 5 00:16 fullchain.pem -> ../../archive/fleshki.net-0001/fullchain1.pem
lrwxrwxrwx 1 root root 43 лип 5 00:16 privkey.pem -> ../../archive/fleshki.net-0001/privkey1.pem
ls -la /etc/letsencrypt/archive/fleshki.net/
total 24
drwxr-xr-x 2 root root 4096 чер 5 00:00 .
drwx------ 17 root root 4096 кві 5 18:42 ..
-rw-r--r-- 1 root root 2216 лип 5 00:16 cert2.pem
-rw-r--r-- 1 root root 1647 лип 5 00:16 chain2.pem
-rw-r--r-- 1 root root 3863 лип 5 00:16 fullchain2.pem
-rw-r--r-- 1 root root 1704 лип 5 00:16 privkey2.pem
So filenames changed but symlinks stayed old?
ls -la /etc/letsencrypt/archive/fleshki.net-0001/
total 24
drwxr-xr-x 2 root root 4096 кві 5 18:29 .
drwx------ 17 root root 4096 кві 5 18:42 ..
-rw-r--r-- 1 root root 2216 кві 5 18:29 cert1.pem
-rw-r--r-- 1 root root 1647 кві 5 18:29 chain1.pem
-rw-r--r-- 1 root root 3863 кві 5 18:29 fullchain1.pem
-rw-r--r-- 1 root root 1704 кві 5 18:29 privkey1.pem

ok, here is the problem :wink:

Firs of all, made a backup:

tar zcvf /root/backup-etc-letsencrypt_2018-Jul-5.tar.gz /etc/letsencrypt/

Now, let’s go to fix the mess:

rm -f /etc/letsencrypt/archive/fleshki.net/*
cp -p /etc/letsencrypt/archive/fleshki.net-0001/* /etc/letsencrypt/archive/fleshki.net/
cd /etc/letsencrypt/live/fleshki.net/
rm *.pem
ln -s ../../archive/fleshki.net/cert1.pem cert.pem
ln -s ../../archive/fleshki.net/chain1.pem chain.pem
ln -s ../../archive/fleshki.net/fullchain1.pem fullchain.pem
ln -s ../../archive/fleshki.net/privkey1.pem privkey.pem
rm -rf /etc/letsencrypt/archive/fleshki.net-0001/

In case /etc/letsencrypt/live/fleshki.net-0001/ dir exists, remove it too:

rm -rf /etc/letsencrypt/live/fleshki.net-0001/

And run certbot certificates again to see whether it sees the right cert for fleshki.net domain.

2 Likes

Done.
Certificate Name: fleshki.net
Domains: fleshki.net chat.fleshki.net test.fleshki.net www.fleshki.net
Expiry Date: 2018-07-04 14:29:11+00:00 (INVALID: EXPIRED)
Certificate Path: /etc/letsencrypt/live/fleshki.net/fullchain.pem
Private Key Path: /etc/letsencrypt/live/fleshki.net/privkey.pem

Sorry, it was my fault, we copied the wrong certificates, fortunately we have a backup :wink:

cd /root/
tar zxvf backup-etc-letsencrypt_2018-Jul-5.tar.gz
cp ./etc/letsencrypt/archive/fleshki.net/cert2.pem /etc/letsencrypt/archive/fleshki.net/cert1.pem
cp ./etc/letsencrypt/archive/fleshki.net/chain2.pem /etc/letsencrypt/archive/fleshki.net/chain1.pem
cp ./etc/letsencrypt/archive/fleshki.net/fullchain2.pem /etc/letsencrypt/archive/fleshki.net/fullchain1.pem
cp ./etc/letsencrypt/archive/fleshki.net/privkey2.pem /etc/letsencrypt/archive/fleshki.net/privkey1.pem

And again, run certbot certificates

1 Like

Success!

  Certificate Name: fleshki.net
    Domains: fleshki.net chat.fleshki.net test.fleshki.net www.fleshki.net
    Expiry Date: 2018-10-02 20:16:54+00:00 (VALID: 89 days)
    Certificate Path: /etc/letsencrypt/live/fleshki.net/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/fleshki.net/privkey.pem

Great, now you should do the same for the other domains that are in the same situation, the other domains could have different numbers in their file names.

Here the fixed steps:

ls -l /etc/letsencrypt/archive/domain/*

Now, check what is the last number in files, it could be cert2.pem or cert3.pem, etc.

cd /etc/letsencrypt/live/domain/
rm *.pem

And now you need to create the right symbolic links, if we saw that the last number in archive dir was for example cert3.pem we will use it to create the links.

ln -s ../../archive/domain/cert3.pem cert.pem
ln -s ../../archive/domain/chain3.pem chain.pem
ln -s ../../archive/domain/fullchain3.pem fullchain.pem
ln -s ../../archive/domain/privkey3.pem privkey.pem

rm -rf /etc/letsencrypt/archive/domain-0001/

And that should solve all the issues.

4 Likes

Thank you! Problem seems solved.

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.