Hi guys,
thank you for your work and your support, you’re great!!!
Since 10 days I can’t issue the renewal for example.org here the error:
Attempting to renew cert from /etc/letsencrypt/renewal/example.org.conf produced an unexpected error: urn:acme:error:rateLimited :: There were too many requests of a given type :: Error creating new cert :: too many certificates already issued for exact set of domains: example.org. Skipping.
here my crontab
# Run every day at 5:30
30 5 * * * service nginx stop && certbot renew -n --agree-tos ;; service nginx start
Over the last 13 days, 12 certificates have been issued for “example.org”, only one of them at 12:30 UTC. (Plus one today for “example.org, www.example.org”.)
Check if other computers are issuing certificates.
Check if there’s another cron job or systemd timer, and if so, what it does.
Check /etc/letsencrypt/cli.ini and if it does --force-renewal or --renew-by-default. (It shouldn’t.)
And the same for the /etc/letsencrypt/renewal/ file(s).
And run “certbot certificates” to see if you have multiple duplicate certificates being managed, and when they were issued?
And fill out the questions below?
The operating system my web server runs on is (include version):
My hosting provider, if applicable, is:
I’m using a control panel to manage my site (no, or provide the name and version of the control panel):
Over the last 13 days, 12 certificates1 have been issued for “example.org”, only one of them at 12:30 UTC. (Plus one today for "example.org, www.example.org”.) --> but all seems expired!
Check if other computers are issuing certificates. --> no at all
Check if there’s another cron job or systemd timer, and if so, what it does. --> no at all
Check /etc/letsencrypt/cli.ini and if it does --force-renewal or --renew-by-default. (It shouldn’t.) --> I don't have cli.ini in that dir
And the same for the /etc/letsencrypt/renewal/ file(s). --> seems all right
And run “certbot certificates” to see if you have multiple duplicate certificates being managed, and when they were issued? -->
root@icarus:/var/log/letsencrypt# ll -t
total 2064
drwx------ 2 root root 4096 Aug 20 19:45 ./
-rw-r--r-- 1 root root 30836 Aug 20 12:33 letsencrypt.log
-rw-r--r-- 1 root root 27526 Aug 20 11:35 letsencrypt.log.1
-rw-r--r-- 1 root root 5638 Aug 20 11:34 letsencrypt.log.2
-rw-r--r-- 1 root root 7290 Aug 20 11:33 letsencrypt.log.3
-rw-r--r-- 1 root root 30876 Aug 20 11:22 letsencrypt.log.4
-rw-r--r-- 1 root root 7290 Aug 20 10:50 letsencrypt.log.5
-rw-r--r-- 1 root root 7290 Aug 20 10:45 letsencrypt.log.6
drwxrwxr-x 16 root syslog 4096 Aug 20 06:25 ../
-rw-r--r-- 1 root root 29054 Aug 20 00:22 letsencrypt.log.7
-rw-r--r-- 1 root root 29097 Aug 19 18:38 letsencrypt.log.8
-rw-r--r-- 1 root root 66098 Aug 19 18:37 letsencrypt.log.9
-rw-r--r-- 1 root root 29093 Aug 19 18:06 letsencrypt.log.10
-rw-r--r-- 1 root root 37827 Aug 19 18:05 letsencrypt.log.11
-rw-r--r-- 1 root root 17184 Aug 19 18:04 letsencrypt.log.12
-rw-r--r-- 1 root root 18707 Aug 19 18:04 letsencrypt.log.13
-rw-r--r-- 1 root root 30872 Aug 19 18:01 letsencrypt.log.14
-rw-r--r-- 1 root root 30837 Aug 19 12:44 letsencrypt.log.15
-rw-r--r-- 1 root root 11020 Aug 19 05:30 letsencrypt.log.16
-rw-r--r-- 1 root root 30833 Aug 19 00:14 letsencrypt.log.17
-rw-r--r-- 1 root root 30837 Aug 18 12:30 letsencrypt.log.18
-rw-r--r-- 1 root root 11020 Aug 18 05:30 letsencrypt.log.19
-rw-r--r-- 1 root root 30834 Aug 18 00:51 letsencrypt.log.20
-rw-r--r-- 1 root root 30831 Aug 17 12:07 letsencrypt.log.21
-rw-r--r-- 1 root root 11020 Aug 17 05:30 letsencrypt.log.22
-rw-r--r-- 1 root root 30832 Aug 17 00:43 letsencrypt.log.23
-rw-r--r-- 1 root root 32876 Aug 16 12:41 letsencrypt.log.24
-rw-r--r-- 1 root root 11020 Aug 16 05:30 letsencrypt.log.25
-rw-r--r-- 1 root root 32872 Aug 16 00:26 letsencrypt.log.26
-rw-r--r-- 1 root root 32875 Aug 15 12:29 letsencrypt.log.27
-rw-r--r-- 1 root root 11020 Aug 15 05:30 letsencrypt.log.28
-rw-r--r-- 1 root root 32873 Aug 15 00:27 letsencrypt.log.29
-rw-r--r-- 1 root root 32874 Aug 14 12:12 letsencrypt.log.30
-rw-r--r-- 1 root root 11020 Aug 14 05:30 letsencrypt.log.31
-rw-r--r-- 1 root root 32868 Aug 14 00:09 letsencrypt.log.32
-rw-r--r-- 1 root root 30835 Aug 13 12:49 letsencrypt.log.33
-rw-r--r-- 1 root root 11020 Aug 13 05:30 letsencrypt.log.34
-rw-r--r-- 1 root root 30832 Aug 13 00:30 letsencrypt.log.35
-rw-r--r-- 1 root root 30837 Aug 12 12:49 letsencrypt.log.36
-rw-r--r-- 1 root root 11020 Aug 12 05:30 letsencrypt.log.37
-rw-r--r-- 1 root root 30834 Aug 12 00:47 letsencrypt.log.38
-rw-r--r-- 1 root root 30837 Aug 11 12:57 letsencrypt.log.39
-rw-r--r-- 1 root root 11020 Aug 11 05:30 letsencrypt.log.40
-rw-r--r-- 1 root root 30834 Aug 11 00:56 letsencrypt.log.41
-rw-r--r-- 1 root root 30836 Aug 10 12:38 letsencrypt.log.42
-rw-r--r-- 1 root root 11020 Aug 10 05:30 letsencrypt.log.43
-rw-r--r-- 1 root root 30833 Aug 10 00:57 letsencrypt.log.44
-rw-r--r-- 1 root root 30833 Aug 9 12:53 letsencrypt.log.45
-rw-r--r-- 1 root root 11017 Aug 9 05:30 letsencrypt.log.46
-rw-r--r-- 1 root root 30831 Aug 9 00:50 letsencrypt.log.47
-rw-r--r-- 1 root root 30803 Aug 8 12:08 letsencrypt.log.48
-rw-r--r-- 1 root root 30831 Aug 8 12:06 letsencrypt.log.49
-rw-r--r-- 1 root root 37469 Aug 8 12:03 letsencrypt.log.50
-rw-r--r-- 1 root root 34776 Aug 8 11:58 letsencrypt.log.51
-rw-r--r-- 1 root root 5659 Aug 8 11:57 letsencrypt.log.52
-rw-r--r-- 1 root root 28836 Aug 8 11:54 letsencrypt.log.53
-rw-r--r-- 1 root root 28795 Aug 8 00:25 letsencrypt.log.54
-rw-r--r-- 1 root root 28796 Aug 7 12:34 letsencrypt.log.55
-rw-r--r-- 1 root root 28795 Aug 7 00:58 letsencrypt.log.56
-rw-r--r-- 1 root root 28809 Aug 6 20:36 letsencrypt.log.57
-rw-r--r-- 1 root root 27845 Aug 6 20:35 letsencrypt.log.58
-rw-r--r-- 1 root root 5855 Aug 6 20:34 letsencrypt.log.59
-rw-r--r-- 1 root root 9147 Aug 6 20:33 letsencrypt.log.60
-rw-r--r-- 1 root root 30881 Aug 6 20:33 letsencrypt.log.61
-rw-r--r-- 1 root root 33256 Aug 6 20:32 letsencrypt.log.62
-rw-r--r-- 1 root root 46111 Aug 6 20:26 letsencrypt.log.63
-rw-r--r-- 1 root root 46183 Aug 6 20:20 letsencrypt.log.64
-rw-r--r-- 1 root root 47211 Aug 6 20:19 letsencrypt.log.65
-rw-r--r-- 1 root root 90124 Aug 6 20:18 letsencrypt.log.66
-rw-r--r-- 1 root root 59985 May 26 13:05 letsencrypt.log.67
-rw-r--r-- 1 root root 753 May 26 13:02 letsencrypt.log.68
-rw-r--r-- 1 root root 2821 May 8 14:21 letsencrypt.log.69
-rw-r--r-- 1 root root 2821 May 8 14:21 letsencrypt.log.70
-rw-r--r-- 1 root root 3974 May 8 14:20 letsencrypt.log.71
-rw-r--r-- 1 root root 44057 May 8 14:18 letsencrypt.log.72
-rw-r--r-- 1 root root 16024 May 8 14:18 letsencrypt.log.73
as you can see, seesm that certbot is executed three-times a day every day. One at 00:30, one at 5:30 (that is the only one that I would expect, since i've setted it in the crontab) and one at 12:30...
But well, I can't get why!
If I list all cron job for all user I get this:
root@icarus:/var/log/letsencrypt# for user in $(cut -f1 -d: /etc/passwd); do crontab -u $user -l; done
# Edit this file to introduce tasks to be run by cron.
#
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
#
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').#
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
#
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
#
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
#
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h dom mon dow command
# Run every day at 5:30
30 5 * * * service nginx stop && certbot renew -n --agree-tos ;; service nginx start
no crontab for daemon
no crontab for bin
no crontab for sys
no crontab for sync
no crontab for games
no crontab for man
no crontab for lp
no crontab for mail
no crontab for news
no crontab for uucp
no crontab for proxy
# Edit this file to introduce tasks to be run by cron.
#
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
#
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').#
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
#
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
#
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
#
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h dom mon dow command
0 * * * * /srv/venvs/leadplatform/bin/python /srv/www/leadplatform/manage.py videotutorial_preload
no crontab for backup
no crontab for list
no crontab for irc
no crontab for gnats
no crontab for nobody
no crontab for systemd-timesync
no crontab for systemd-network
no crontab for systemd-resolve
no crontab for systemd-bus-proxy
no crontab for syslog
no crontab for _apt
no crontab for messagebus
no crontab for lxd
no crontab for dnsmasq
no crontab for sshd
no crontab for pollinate
no crontab for uuidd
no crontab for nginx
no crontab for postfix
no crontab for andrea
no crontab for ntp
no crontab for dovecot
no crontab for dovenull
no crontab for postgres
no crontab for redis
no crontab for statd
no crontab for epmd
no crontab for rabbitmq
no crontab for svrable
and that's perfectly normal.. so I really don't know why it runs 3 times any thoughts ?
root@icarus:/var/log/letsencrypt# systemctl list-timers --all
NEXT LEFT LAST PASSED UNIT ACTIVATES
Sun 2017-08-20 21:28:38 UTC 1h 25min left Sun 2017-08-20 14:50:55 UTC 5h 12min ago snapd.refresh.timer snapd.refresh.service
Sun 2017-08-20 23:08:52 UTC 3h 5min left Sun 2017-08-20 15:50:58 UTC 4h 12min ago apt-daily.timer apt-daily.service
Mon 2017-08-21 00:31:41 UTC 4h 28min left Sun 2017-08-20 12:32:57 UTC 7h ago certbot.timer certbot.service
Mon 2017-08-21 14:32:57 UTC 18h left Sun 2017-08-20 14:32:57 UTC 5h 30min ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
n/a n/a n/a n/a ureadahead-stop.timer ureadahead-stop.service
5 timers listed.
the questions now is: why the execution trigger rate limiting
# /etc/cron.d/certbot: crontab entries for the certbot package
#
# Upstream recommends attempting renewal twice a day
#
# Eventually, this will be an opportunity to validate certificates
# haven't been revoked, etc. Renewal will only occur if expiration
# is within 30 days.
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
0 */12 * * * root test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(3600))' && certbot -q renew
I also think that it seems not to be saving the renewed certificates for some reason, which can be revealed in the content of those log files in /var/log/letsencrypt (although they’re likely to be pretty long and mostly irrelevant).
schoen: I've no cli.ini at all. Here the paste of that command:
root@icarus:/srv# ls -l /etc/letsencrypt/*/example.org/
/etc/letsencrypt/archive/example.org/:
total 32
-rw-r--r-- 1 root root 1797 Oct 17 2016 cert1.pem
-rw-r--r-- 1 root root 1777 Aug 21 00:48 cert2.pem
-rw-r--r-- 1 root root 1647 Oct 17 2016 chain1.pem
-rw-r--r-- 1 root root 1647 Aug 21 00:48 chain2.pem
-rw-r--r-- 1 root root 3444 Oct 17 2016 fullchain1.pem
-rw-r--r-- 1 root root 3424 Aug 21 00:48 fullchain2.pem
-rw-r--r-- 1 root root 1704 Oct 17 2016 privkey1.pem
-rw-r--r-- 1 root root 1704 Aug 21 00:48 privkey2.pem
/etc/letsencrypt/live/example.org/:
total 0
lrwxrwxrwx 1 root root 37 Aug 21 00:48 cert.pem -> ../../archive/example.org-0001/cert1.pem
lrwxrwxrwx 1 root root 38 Aug 21 00:48 chain.pem -> ../../archive/example.org-0001/chain1.pem
lrwxrwxrwx 1 root root 42 Aug 21 00:48 fullchain.pem -> ../../archive/example.org-0001/fullchain1.pem
lrwxrwxrwx 1 root root 40 Aug 21 00:48 privkey.pem -> ../../archive/example.org-0001/privkey1.pem
rg305: yeah it makes sense, but why it is issuing a new cert!?
If you guys could have some spare minutes, I've uploaded the entire content of /var/log/letsencrypt directory in that tar archive on dropbox: Dropbox - Error - Simplify your life
I'm driving crazy to resolve this issue, anyway thank you so so much!!
Here is an abbreviation of one of the logs:
2017-08-20 00:22:10,037:INFO:certbot.renewal:Cert not yet due for renewal
2017-08-20 00:22:10,042:INFO:certbot.renewal:Cert is due for renewal, auto-renewing…
2017-08-20 00:22:15,329:WARNING:certbot.renewal:Attempting to renew cert from /etc/letsencrypt/renewal/example.org.conf produced an unexpected error: urn:acme:error:rateLimited :: There were too many requests of a given type :: Error creating new cert :: too many certificates already issued for exact set of domains: example.org. Skipping.
2017-08-20 00:22:15,345:INFO:certbot.renewal:Cert not yet due for renewal
2017-08-20 00:22:15,345:DEBUG:certbot.log:Exiting abnormally:
Which seems to indicate that certbot is trying to renew multiple certs (3 by my count).
Some of which it thinks are not yet due for renewal and some that it does but then can’t because “too many certs…”.
I think this indicates that there is a misconfiguration between what certbot thinks is installed and what is actually installed.
To help clarify that, please show the certs on the system (certbot certificates) and show the contents of “cli.ini” or “certbot.cli”.
for comparison:
ls -l /etc/letsencrypt/live/www.example.com/
ls -l /etc/letsencrypt/live/example.org/
Then if not similar access, make similar or
delete cert example.org then recreate it or
delete /etc/letsencrypt/live/example.org/fullchain.pem and retry renewal.
The symlinks are pointing to the wrong directory and Certbot is getting confused.
Not sure best way to fix. "certbot update_symlinks" should do it, i think. You may have to delete them first, but i don't think so. (You can also delete them and create the correct ones by hand, of course.)
any thoughts ?