Renew error rate limited

Hi guys,
thank you for your work and your support, you’re great!!!

Since 10 days I can’t issue the renewal for example.org here the error:

Attempting to renew cert from /etc/letsencrypt/renewal/example.org.conf produced an unexpected error: urn:acme:error:rateLimited :: There were too many requests of a given type :: Error creating new cert :: too many certificates already issued for exact set of domains: example.org. Skipping.

here my crontab
# Run every day at 5:30
30 5 * * * service nginx stop && certbot renew -n --agree-tos ;; service nginx start

thank you so much for your help!

Andrea

Over the last 13 days, 12 certificates have been issued for “example.org”, only one of them at 12:30 UTC. (Plus one today for “example.org, www.example.org”.)

Check if other computers are issuing certificates.

Check if there’s another cron job or systemd timer, and if so, what it does.

Check /etc/letsencrypt/cli.ini and if it does --force-renewal or --renew-by-default. (It shouldn’t.)

And the same for the /etc/letsencrypt/renewal/ file(s).

And run “certbot certificates” to see if you have multiple duplicate certificates being managed, and when they were issued?

And fill out the questions below?


The operating system my web server runs on is (include version):

My hosting provider, if applicable, is:

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):

Hi mnordhoff, thank you so much.
Here my answers:

Over the last 13 days, 12 certificates1 have been issued for “example.org”, only one of them at 12:30 UTC. (Plus one today for "example.org, www.example.org”.) --> but all seems expired!

Check if other computers are issuing certificates. --> no at all

Check if there’s another cron job or systemd timer, and if so, what it does. --> no at all

Check /etc/letsencrypt/cli.ini and if it does --force-renewal or --renew-by-default. (It shouldn’t.) --> I don’t have cli.ini in that dir

And the same for the /etc/letsencrypt/renewal/ file(s). --> seems all right

And run “certbot certificates” to see if you have multiple duplicate certificates being managed, and when they were issued? -->

root@icarus:/srv# certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Revocation status for /etc/letsencrypt/live/example.org/cert.pem is unknown

-------------------------------------------------------------------------------
Found the following certs:
  Certificate Name: www.example.com
    Domains: www.example.com
    Expiry Date: 2017-11-17 17:39:00+00:00 (VALID: 89 days)
    Certificate Path: /etc/letsencrypt/live/www.example.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/www.example.com/privkey.pem
  Certificate Name: example.org
    Domains: example.org
    Expiry Date: 2017-08-06 13:19:00+00:00 (INVALID: EXPIRED)
    Certificate Path: /etc/letsencrypt/live/example.org/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/example.org/privkey.pem
  Certificate Name: example.com
    Domains: example.com
    Expiry Date: 2017-11-17 17:39:00+00:00 (VALID: 89 days)
    Certificate Path: /etc/letsencrypt/live/example.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/example.com/privkey.pem
-------------------------------------------------------------------------------

Operating system: Ubuntu 16.04
Hosting provider: digital ocean
Control panel: no

Thanks!!!

I can’t understand the output of crt.sh … it seems that I’m renewing the certificate rightly, but the files on my server aren’t updated.

Any hints?

Can you look in /var/log/letsencrypt to see if Certbot is being run at other times?

Mmm nice catch schoen.

root@icarus:/var/log/letsencrypt# ll -t
total 2064
drwx------ 2 root root 4096 Aug 20 19:45 ./
-rw-r–r-- 1 root root 30836 Aug 20 12:33 letsencrypt.log
-rw-r–r-- 1 root root 27526 Aug 20 11:35 letsencrypt.log.1
-rw-r–r-- 1 root root 5638 Aug 20 11:34 letsencrypt.log.2
-rw-r–r-- 1 root root 7290 Aug 20 11:33 letsencrypt.log.3
-rw-r–r-- 1 root root 30876 Aug 20 11:22 letsencrypt.log.4
-rw-r–r-- 1 root root 7290 Aug 20 10:50 letsencrypt.log.5
-rw-r–r-- 1 root root 7290 Aug 20 10:45 letsencrypt.log.6
drwxrwxr-x 16 root syslog 4096 Aug 20 06:25 …/
-rw-r–r-- 1 root root 29054 Aug 20 00:22 letsencrypt.log.7
-rw-r–r-- 1 root root 29097 Aug 19 18:38 letsencrypt.log.8
-rw-r–r-- 1 root root 66098 Aug 19 18:37 letsencrypt.log.9
-rw-r–r-- 1 root root 29093 Aug 19 18:06 letsencrypt.log.10
-rw-r–r-- 1 root root 37827 Aug 19 18:05 letsencrypt.log.11
-rw-r–r-- 1 root root 17184 Aug 19 18:04 letsencrypt.log.12
-rw-r–r-- 1 root root 18707 Aug 19 18:04 letsencrypt.log.13
-rw-r–r-- 1 root root 30872 Aug 19 18:01 letsencrypt.log.14
-rw-r–r-- 1 root root 30837 Aug 19 12:44 letsencrypt.log.15
-rw-r–r-- 1 root root 11020 Aug 19 05:30 letsencrypt.log.16
-rw-r–r-- 1 root root 30833 Aug 19 00:14 letsencrypt.log.17
-rw-r–r-- 1 root root 30837 Aug 18 12:30 letsencrypt.log.18
-rw-r–r-- 1 root root 11020 Aug 18 05:30 letsencrypt.log.19
-rw-r–r-- 1 root root 30834 Aug 18 00:51 letsencrypt.log.20
-rw-r–r-- 1 root root 30831 Aug 17 12:07 letsencrypt.log.21
-rw-r–r-- 1 root root 11020 Aug 17 05:30 letsencrypt.log.22
-rw-r–r-- 1 root root 30832 Aug 17 00:43 letsencrypt.log.23
-rw-r–r-- 1 root root 32876 Aug 16 12:41 letsencrypt.log.24
-rw-r–r-- 1 root root 11020 Aug 16 05:30 letsencrypt.log.25
-rw-r–r-- 1 root root 32872 Aug 16 00:26 letsencrypt.log.26
-rw-r–r-- 1 root root 32875 Aug 15 12:29 letsencrypt.log.27
-rw-r–r-- 1 root root 11020 Aug 15 05:30 letsencrypt.log.28
-rw-r–r-- 1 root root 32873 Aug 15 00:27 letsencrypt.log.29
-rw-r–r-- 1 root root 32874 Aug 14 12:12 letsencrypt.log.30
-rw-r–r-- 1 root root 11020 Aug 14 05:30 letsencrypt.log.31
-rw-r–r-- 1 root root 32868 Aug 14 00:09 letsencrypt.log.32
-rw-r–r-- 1 root root 30835 Aug 13 12:49 letsencrypt.log.33
-rw-r–r-- 1 root root 11020 Aug 13 05:30 letsencrypt.log.34
-rw-r–r-- 1 root root 30832 Aug 13 00:30 letsencrypt.log.35
-rw-r–r-- 1 root root 30837 Aug 12 12:49 letsencrypt.log.36
-rw-r–r-- 1 root root 11020 Aug 12 05:30 letsencrypt.log.37
-rw-r–r-- 1 root root 30834 Aug 12 00:47 letsencrypt.log.38
-rw-r–r-- 1 root root 30837 Aug 11 12:57 letsencrypt.log.39
-rw-r–r-- 1 root root 11020 Aug 11 05:30 letsencrypt.log.40
-rw-r–r-- 1 root root 30834 Aug 11 00:56 letsencrypt.log.41
-rw-r–r-- 1 root root 30836 Aug 10 12:38 letsencrypt.log.42
-rw-r–r-- 1 root root 11020 Aug 10 05:30 letsencrypt.log.43
-rw-r–r-- 1 root root 30833 Aug 10 00:57 letsencrypt.log.44
-rw-r–r-- 1 root root 30833 Aug 9 12:53 letsencrypt.log.45
-rw-r–r-- 1 root root 11017 Aug 9 05:30 letsencrypt.log.46
-rw-r–r-- 1 root root 30831 Aug 9 00:50 letsencrypt.log.47
-rw-r–r-- 1 root root 30803 Aug 8 12:08 letsencrypt.log.48
-rw-r–r-- 1 root root 30831 Aug 8 12:06 letsencrypt.log.49
-rw-r–r-- 1 root root 37469 Aug 8 12:03 letsencrypt.log.50
-rw-r–r-- 1 root root 34776 Aug 8 11:58 letsencrypt.log.51
-rw-r–r-- 1 root root 5659 Aug 8 11:57 letsencrypt.log.52
-rw-r–r-- 1 root root 28836 Aug 8 11:54 letsencrypt.log.53
-rw-r–r-- 1 root root 28795 Aug 8 00:25 letsencrypt.log.54
-rw-r–r-- 1 root root 28796 Aug 7 12:34 letsencrypt.log.55
-rw-r–r-- 1 root root 28795 Aug 7 00:58 letsencrypt.log.56
-rw-r–r-- 1 root root 28809 Aug 6 20:36 letsencrypt.log.57
-rw-r–r-- 1 root root 27845 Aug 6 20:35 letsencrypt.log.58
-rw-r–r-- 1 root root 5855 Aug 6 20:34 letsencrypt.log.59
-rw-r–r-- 1 root root 9147 Aug 6 20:33 letsencrypt.log.60
-rw-r–r-- 1 root root 30881 Aug 6 20:33 letsencrypt.log.61
-rw-r–r-- 1 root root 33256 Aug 6 20:32 letsencrypt.log.62
-rw-r–r-- 1 root root 46111 Aug 6 20:26 letsencrypt.log.63
-rw-r–r-- 1 root root 46183 Aug 6 20:20 letsencrypt.log.64
-rw-r–r-- 1 root root 47211 Aug 6 20:19 letsencrypt.log.65
-rw-r–r-- 1 root root 90124 Aug 6 20:18 letsencrypt.log.66
-rw-r–r-- 1 root root 59985 May 26 13:05 letsencrypt.log.67
-rw-r–r-- 1 root root 753 May 26 13:02 letsencrypt.log.68
-rw-r–r-- 1 root root 2821 May 8 14:21 letsencrypt.log.69
-rw-r–r-- 1 root root 2821 May 8 14:21 letsencrypt.log.70
-rw-r–r-- 1 root root 3974 May 8 14:20 letsencrypt.log.71
-rw-r–r-- 1 root root 44057 May 8 14:18 letsencrypt.log.72
-rw-r–r-- 1 root root 16024 May 8 14:18 letsencrypt.log.73

as you can see, seesm that certbot is executed three-times a day every day. One at 00:30, one at 5:30 (that is the only one that I would expect, since i’ve setted it in the crontab) and one at 12:30…

But well, I can’t get why!
If I list all cron job for all user I get this:

root@icarus:/var/log/letsencrypt# for user in $(cut -f1 -d: /etc/passwd); do crontab -u $user -l; done
# Edit this file to introduce tasks to be run by cron.
# 
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
# 
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').# 
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
# 
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
# 
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
# 
# For more information see the manual pages of crontab(5) and cron(8)
# 
# m h  dom mon dow   command

# Run every day at 5:30
30 5 * * * service nginx stop && certbot renew -n --agree-tos ;; service nginx start
no crontab for daemon
no crontab for bin
no crontab for sys
no crontab for sync
no crontab for games
no crontab for man
no crontab for lp
no crontab for mail
no crontab for news
no crontab for uucp
no crontab for proxy
# Edit this file to introduce tasks to be run by cron.
# 
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
# 
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').# 
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
# 
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
# 
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
# 
# For more information see the manual pages of crontab(5) and cron(8)
# 
# m h  dom mon dow   command
0 * * * * /srv/venvs/leadplatform/bin/python /srv/www/leadplatform/manage.py videotutorial_preload 
no crontab for backup
no crontab for list
no crontab for irc
no crontab for gnats
no crontab for nobody
no crontab for systemd-timesync
no crontab for systemd-network
no crontab for systemd-resolve
no crontab for systemd-bus-proxy
no crontab for syslog
no crontab for _apt
no crontab for messagebus
no crontab for lxd
no crontab for dnsmasq
no crontab for sshd
no crontab for pollinate
no crontab for uuidd
no crontab for nginx
no crontab for postfix
no crontab for andrea
no crontab for ntp
no crontab for dovecot
no crontab for dovenull
no crontab for postgres
no crontab for redis
no crontab for statd
no crontab for epmd
no crontab for rabbitmq
no crontab for svrable

and that’s perfectly normal… so I really don’t know why it runs 3 times :confused: any thoughts ?

Thank you so much

Mmm further digging on systemd timer shows:

root@icarus:/var/log/letsencrypt# systemctl list-timers --all
NEXT                         LEFT          LAST                         PASSED       UNIT                         ACTIVATES
Sun 2017-08-20 21:28:38 UTC  1h 25min left Sun 2017-08-20 14:50:55 UTC  5h 12min ago snapd.refresh.timer          snapd.refresh.service
Sun 2017-08-20 23:08:52 UTC  3h 5min left  Sun 2017-08-20 15:50:58 UTC  4h 12min ago apt-daily.timer              apt-daily.service
Mon 2017-08-21 00:31:41 UTC  4h 28min left Sun 2017-08-20 12:32:57 UTC  7h ago       certbot.timer                certbot.service
Mon 2017-08-21 14:32:57 UTC  18h left      Sun 2017-08-20 14:32:57 UTC  5h 30min ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
n/a                          n/a           n/a                          n/a          ureadahead-stop.timer        ureadahead-stop.service

5 timers listed.

the questions now is: why the execution trigger rate limiting

Ok I’ve just discovered

# /etc/cron.d/certbot: crontab entries for the certbot package
#
# Upstream recommends attempting renewal twice a day
#
# Eventually, this will be an opportunity to validate certificates
# haven't been revoked, etc.  Renewal will only occur if expiration
# is within 30 days.
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

0 */12 * * * root test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(3600))' && certbot -q renew

but I can’t get why it triggers the rate limits…

Did you happen to make a cli.ini file?

Otherwise, could you show what ls -l /etc/letsencrypt/*/example.org/ looks like?

I also think that it seems not to be saving the renewed certificates for some reason, which can be revealed in the content of those log files in /var/log/letsencrypt (although they’re likely to be pretty long and mostly irrelevant).

Probably because it is issuing a new cert almost daily.

Dates from certs matched to entries from your logs:
Not Before: Aug 13 23:11:00 2017 GMT = -rw-r–r-- 1 root root 32868 Aug 14 00:09 letsencrypt.log.32
Not Before: Aug 14 11:14:00 2017 GMT = -rw-r–r-- 1 root root 32874 Aug 14 12:12 letsencrypt.log.30
Not Before: Aug 14 23:29:00 2017 GMT = -rw-r–r-- 1 root root 32873 Aug 15 00:27 letsencrypt.log.29
Not Before: Aug 15 11:31:00 2017 GMT = -rw-r–r-- 1 root root 32875 Aug 15 12:29 letsencrypt.log.27
Not Before: Aug 15 23:27:00 2017 GMT = -rw-r–r-- 1 root root 32872 Aug 16 00:26 letsencrypt.log.26
Not Before: Aug 16 11:43:00 2017 GMT = -rw-r–r-- 1 root root 32876 Aug 16 12:41 letsencrypt.log.24
Not Before: Aug 19 17:07:00 2017 GMT = -rw-r–r-- 1 root root 29093 Aug 19 18:06 letsencrypt.log.10

schoen: I’ve no cli.ini at all. Here the paste of that command:

root@icarus:/srv#  ls -l /etc/letsencrypt/*/example.org/
/etc/letsencrypt/archive/example.org/:
total 32
-rw-r--r-- 1 root root 1797 Oct 17  2016 cert1.pem
-rw-r--r-- 1 root root 1777 Aug 21 00:48 cert2.pem
-rw-r--r-- 1 root root 1647 Oct 17  2016 chain1.pem
-rw-r--r-- 1 root root 1647 Aug 21 00:48 chain2.pem
-rw-r--r-- 1 root root 3444 Oct 17  2016 fullchain1.pem
-rw-r--r-- 1 root root 3424 Aug 21 00:48 fullchain2.pem
-rw-r--r-- 1 root root 1704 Oct 17  2016 privkey1.pem
-rw-r--r-- 1 root root 1704 Aug 21 00:48 privkey2.pem

/etc/letsencrypt/live/example.org/:
total 0
lrwxrwxrwx 1 root root 37 Aug 21 00:48 cert.pem -> ../../archive/example.org-0001/cert1.pem
lrwxrwxrwx 1 root root 38 Aug 21 00:48 chain.pem -> ../../archive/example.org-0001/chain1.pem
lrwxrwxrwx 1 root root 42 Aug 21 00:48 fullchain.pem -> ../../archive/example.org-0001/fullchain1.pem
lrwxrwxrwx 1 root root 40 Aug 21 00:48 privkey.pem -> ../../archive/example.org-0001/privkey1.pem

rg305: yeah it makes sense, but why it is issuing a new cert!?

If you guys could have some spare minutes, I’ve uploaded the entire content of /var/log/letsencrypt directory in that tar archive on dropbox: https://www.dropbox.com/s/tdcoh6uuqxmg4cf/letsencrypt.tar?dl=0

I’m driving crazy to resolve this issue, anyway thank you so so much!!

Andrea

Here is an abbreviation of one of the logs:
2017-08-20 00:22:10,037:INFO:certbot.renewal:Cert not yet due for renewal
2017-08-20 00:22:10,042:INFO:certbot.renewal:Cert is due for renewal, auto-renewing…
2017-08-20 00:22:15,329:WARNING:certbot.renewal:Attempting to renew cert from /etc/letsencrypt/renewal/example.org.conf produced an unexpected error: urn:acme:error:rateLimited :: There were too many requests of a given type :: Error creating new cert :: too many certificates already issued for exact set of domains: example.org. Skipping.
2017-08-20 00:22:15,345:INFO:certbot.renewal:Cert not yet due for renewal
2017-08-20 00:22:15,345:DEBUG:certbot.log:Exiting abnormally:

Which seems to indicate that certbot is trying to renew multiple certs (3 by my count).
Some of which it thinks are not yet due for renewal and some that it does but then can’t because “too many certs…”.

I think this indicates that there is a misconfiguration between what certbot thinks is installed and what is actually installed.
To help clarify that, please show the certs on the system (certbot certificates) and show the contents of “cli.ini” or “certbot.cli”.

rg305: I’ve no certbot.ini nor cli.ini.
Here my certbot certificates

    > root@icarus:/srv# certbot certificates
    > Saving debug log to /var/log/letsencrypt/letsencrypt.log
    > Revocation status for /etc/letsencrypt/live/example.org/cert.pem is unknown
    > 
    > -------------------------------------------------------------------------------
    > Found the following certs:
    >   Certificate Name: www.example.com
    >     Domains: www.example.com
    >     Expiry Date: 2017-11-17 17:39:00+00:00 (VALID: 89 days)
    >     Certificate Path: /etc/letsencrypt/live/www.example.com/fullchain.pem
    >     Private Key Path: /etc/letsencrypt/live/www.example.com/privkey.pem
    >   Certificate Name: example.org
    >     Domains: example.org
    >     Expiry Date: 2017-08-06 13:19:00+00:00 (INVALID: EXPIRED)
    >     Certificate Path: /etc/letsencrypt/live/example.org/fullchain.pem
    >     Private Key Path: /etc/letsencrypt/live/example.org/privkey.pem
    >   Certificate Name: example.com
    >     Domains: eample.com
    >     Expiry Date: 2017-11-17 17:39:00+00:00 (VALID: 89 days)
    >     Certificate Path: /etc/letsencrypt/live/example.com/fullchain.pem
    >     Private Key Path: /etc/letsencrypt/live/example.com/privkey.pem
    > -------------------------------------------------------------------------------

:confused:

for comparison:
ls -l /etc/letsencrypt/live/www.example.com/
ls -l /etc/letsencrypt/live/example.org/

Then if not similar access, make similar or
delete cert example.org then recreate it or
delete /etc/letsencrypt/live/example.org/fullchain.pem and retry renewal.

The symlinks are pointing to the wrong directory and Certbot is getting confused.

Not sure best way to fix. “certbot update_symlinks” should do it, i think. You may have to delete them first, but i don’t think so. (You can also delete them and create the correct ones by hand, of course.)

2 Likes

@mnordhoff: wow, great catch. Now I’ve to wait 1 week, then we’ll see.
Thank you so much all

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.