OK good, by the way you probably don’t need cert.pem in that list you’re handing to “cat”, because the certificate from cert.pem is already at the start of the next file, fullchain.pem, but if it’s not hurting anything, why change it.
If you want to check you fixed the unwanted renewals problem, you could wait a few days and check https://crt.sh/?q=gitlab.typoworx.de to see if there are new certificates still being issued for that name.
Thanks for the hint about combined.pem. I will check later if skipping cert.pem works fine for HaProxy. With the refreshed combined.pem the given domain also runs like a charm now.
I’ve completly rewritten the cron.d/certbot delivered originally by the deb-package. Sine yesterday it seems to run fine without bugging the cert-server with daily renewals. I hope the automated renewal for combined.pem also works.
Thanks. This can be closed now.
For what it’s worth some process is continuing to issue new certificates for gitlab.typoworx.de more or less every day
That is really crappy. What could went wrong here? I am running the cron every day, but isn’t it certbots task to check which of the already created cert needs a renewal using it as described above?
Yes, I think there is a flag named --force-renewal which overrides the default behaviour, but unless that’s used Certbot’s renew mode is supposed to check whether a certificate needs renewing (is 60+ days old) and if not, ignore it.
To understand what’s going wrong I recommend ensuring you will receive the output of the scheduled job (e.g. by email or in a text file) and perhaps also increasing the verbosity (-v on the command line) to give more details of what it’s doing and why. This might give you (or someone reviewing here on the community site) an insight.
A further things to consider, especially if the reviewed output shows nothing is being renewed, and yet a CT monitor like crt.sh shows certificates have been issued anyway, is whether there might be some other “rogue” cron job running as well as the intended one, e.g. in someone’s personal account or in a backup file mistakenly placed in the cron directory. Of course there must be some reason this job doesn’t succeed correctly and stop issuing more and more certificates, but if it’s a rogue job rather than the one you expect, it might have some permission problem or even a left over use of the “force-renewal” flag from some earlier experiment.
Hello @gkaufmann,
As @tialaramex said, seems you have another cron job out there renewing your cert every day. As you installed certbot using the debian package it is worth to check this post I wrote a few minutes ago because maybe you have an active systemd timer that is trying to renew your cert… but well, it should not renew it if it is not due to renewal yet…
Cheers,
sahsanu
Thank you Sahsanu,
that was the culprit at all! I didn’t know about the systemd-timer at all! I think this is a real problem with the debian-package …
I’ve recently tried to disable the systemd-timer for certbot like this and will keep an eye on it if this resolves the problem at all. I decided not to manually delete the systemd-files as this could cause a re-activation by an upgrade for the certbot-package.
$> systemctl list-timers --all
$> systemctl disable certbot.service
$> systemctl stop certbot.service
(last one already confirms that this service has been disabled yet!)
Hi @gkaufmann,
You should stop/disable the timer not the service. No matter that you have certbot.service stopped and disabled, certbot.timer is still able to execute it so you should stop, disable and even mask the certbot.timer.
systemctl stop certbot.timer
systemctl disable certbot.timer
And to be sure that in case a new debian certbot update doesn’t activate certbot.timer again you could mask the certbot.timer.
systemctl mask certbot.timer
This mask creates a symlink from /etc/systemd/system/certbot.timer to /dev/null so this timer won’t run anything.
Cheers,
sahsanu
Thanks! I will so that.
Every so often I am reminded to see if this got fixed, it seems not, still five new certificates per week.
Hello,
sorry for keeping you waiting. I was in vacation for the last days. I currently have no idea what else I could do. May be I am going better by replacing the debian-package based certbot with the one offered for manual installation.
I already did every of the mentioned steps to avoid automatic renewal when it’s not ready for renewal yet. My cron-skript always gives me a positive looking feedback on that:
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Cert not yet due for renewal
No renewals attempted, so not running post-hook
Processing /etc/letsencrypt/renewal/gitlab.typoworx.de.conf
The following certs are not due for renewal yet:
/etc/letsencrypt/live/gitlab.typoworx.de/fullchain.pem (skipped)
No renewals were attempted.
I simply don’t know what else I could do. Any ideas or suggestions?
I re-applied the mentioned way of sahsanu above in hope this may fix it (in case it did not work on my first attempt)
systemctl stop certbot.timer
systemctl disable certbot.timer
systemctl mask certbot.timer
Hello @gkaufmann,
This is pretty strange because a renew command shouldn’t renew your certs if they are not close to expire, anyway, it doesn’t happen only with gitlab.typoworx.de but also with your other domains (pop3, imap, kunden, etc.) in a random way from 6:30 am to 7:05 am CET so something is launching a command to issue these certs waiting a random wait of minutes.
Please, show the output of the following commands:
grep -Eril "(certbot|letsencrypt)" /etc/cron*
grep -Eril "(certbot|letsencrypt)" /var/spool/
If the above commands show files, please issue a cat to those files to view the content.
Show also the content of /etc/letsencrypt/renewal/gitlab.typoworx.de.conf and the output of:
ls -lart /etc/letsencrypt/live/gitlab.typoworx.de/
ls -lart /etc/letsencrypt/archive/gitlab.typoworx.de/
I forgot to say that usually, you don’t issue certs on monday, maybe because your server is down… who knows, I say it because it could be some kind of clue ;). Also, are you sure you only used certbot to issue certs?, I mean, is it possible you tested some other client like acme.sh, getssl, dehydrated, etc.?.
Let’s see if we can find the culprit 
Cheers,
sahsanu
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.
Sorry,
had to re-open issue that automatically was closed due to that I was very busy in the last few weeks.
Initially opened as
@sahsanu
Pleased me to show the output of the following commands:
Today I tried to move the cron-job "certbot" to cron.weekly hopefully to see some results in the next few days.
On machine #1 using Certbot
$> grep -Eril "(certbot|letsencrypt)" /etc/cron*
grep -Eril "(certbot|letsencrypt)" /etc/cron*
/etc/cron.weekly/certbot
grep -Eril "(certbot|letsencrypt)" /var/spool/
(no result)
On machine #2 using Certbot
Note: This one is still running Ubuntu 13.04 without SystemD Scheduler.
$> grep -Eril "(certbot|letsencrypt)" /etc/cron*
/etc/cron.monthly/letsencrypt
/etc/cron.weekly/certbot
grep -Eril "(certbot|letsencrypt)" /var/spool/
(no result)
No other issuing client should be running. Machine #1 has been recently setup (Debian 8) and is using only Certbot (debian package "Certbot 0.9.3-1~bpo8+1").
@gkaufmann: I reopened the original thread and moved your post there.
I’d be curious about the output these commands as well:
ls -lart /etc/letsencrypt/live/gitlab.typoworx.de/
ls -lart /etc/letsencrypt/archive/gitlab.typoworx.de/
I’ve occasionally seen behaviour like this when the symlinks in that directory are changed; the commands should show if this is the case.
Hello @pfg
here we go (I currently have to machines running certbot):
On Machine #1
ls -lart /etc/letsencrypt/live/gitlab.typoworx.de/
insgesamt 24
drwxrwx— 3 root root 4096 Aug 24 11:28 …
lrwxrwxrwx 1 root root 45 Okt 31 12:30 privkey.pem -> …/…/archive/gitlab.typoworx.de/privkey2.pem
lrwxrwxrwx 1 root root 47 Okt 31 12:30 fullchain.pem -> …/…/archive/gitlab.typoworx.de/fullchain2.pem
lrwxrwxrwx 1 root root 43 Okt 31 12:30 chain.pem -> …/…/archive/gitlab.typoworx.de/chain2.pem
lrwxrwxrwx 1 root root 42 Okt 31 12:30 cert.pem -> …/…/archive/gitlab.typoworx.de/cert2.pem
-rw-r–r-- 1 root root 5156 Dez 1 11:13 combined.pem~
drwxrwx— 2 root root 4096 Dez 1 11:13 .
-rw-r–r-- 1 root root 6961 Dez 1 11:15 combined.pem
ls -lart /etc/letsencrypt/archive/gitlab.typoworx.de/
insgesamt 40
drwxrwx— 3 root root 4096 Aug 24 11:28 …
-rwxrwx— 1 root root 1704 Aug 24 11:28 privkey1.pem
-rwxrwx— 1 root root 3452 Aug 24 11:28 fullchain1.pem
-rwxrwx— 1 root root 1647 Aug 24 11:28 chain1.pem
-rwxrwx— 1 root root 1805 Aug 24 11:28 cert1.pem
-rw-r–r-- 1 root root 1704 Okt 31 12:30 privkey2.pem
-rw-r–r-- 1 root root 3452 Okt 31 12:30 fullchain2.pem
-rw-r–r-- 1 root root 1647 Okt 31 12:30 chain2.pem
-rw-r–r-- 1 root root 1805 Okt 31 12:30 cert2.pem
drwxrwx— 2 root root 4096 Okt 31 12:30 .
On Machine #2
ls -lart /etc/letsencrypt/live/gitlab.typoworx.de/
total 8
drwx------ 16 root root 4096 Aug 8 2016 …
lrwxrwxrwx 1 root root 47 Feb 10 07:48 privkey.pem -> …/…/archive/gitlab.typoworx.de/privkey150.pem
lrwxrwxrwx 1 root root 49 Feb 10 07:48 fullchain.pem -> …/…/archive/gitlab.typoworx.de/fullchain150.pem
lrwxrwxrwx 1 root root 45 Feb 10 07:48 chain.pem -> …/…/archive/gitlab.typoworx.de/chain150.pem
lrwxrwxrwx 1 root root 44 Feb 10 07:48 cert.pem -> …/…/archive/gitlab.typoworx.de/cert150.pem
drwxr-xr-x 2 root root 4096 Feb 10 07:48 .
ls -lart /etc/letsencrypt/archive/gitlab.typoworx.de/
total 2428
[…] a lot of files more!!!
-rw-r–r-- 1 root root 1647 Jan 30 07:38 chain141.pem
-rw-r–r-- 1 root root 1805 Jan 30 07:38 cert141.pem
-rw-r–r-- 1 root root 1708 Jan 31 07:59 privkey142.pem
-rw-r–r-- 1 root root 3452 Jan 31 07:59 fullchain142.pem
-rw-r–r-- 1 root root 1647 Jan 31 07:59 chain142.pem
-rw-r–r-- 1 root root 1805 Jan 31 07:59 cert142.pem
-rw-r–r-- 1 root root 1704 Feb 2 08:01 privkey143.pem
-rw-r–r-- 1 root root 3452 Feb 2 08:01 fullchain143.pem
-rw-r–r-- 1 root root 1647 Feb 2 08:01 chain143.pem
-rw-r–r-- 1 root root 1805 Feb 2 08:01 cert143.pem
-rw-r–r-- 1 root root 1708 Feb 3 08:06 privkey144.pem
-rw-r–r-- 1 root root 3452 Feb 3 08:06 fullchain144.pem
-rw-r–r-- 1 root root 1647 Feb 3 08:06 chain144.pem
-rw-r–r-- 1 root root 1805 Feb 3 08:06 cert144.pem
-rw-r–r-- 1 root root 1708 Feb 4 07:58 privkey145.pem
-rw-r–r-- 1 root root 3452 Feb 4 07:58 fullchain145.pem
-rw-r–r-- 1 root root 1647 Feb 4 07:58 chain145.pem
-rw-r–r-- 1 root root 1805 Feb 4 07:58 cert145.pem
-rw-r–r-- 1 root root 1704 Feb 5 07:58 privkey146.pem
-rw-r–r-- 1 root root 3452 Feb 5 07:58 fullchain146.pem
-rw-r–r-- 1 root root 1647 Feb 5 07:58 chain146.pem
-rw-r–r-- 1 root root 1805 Feb 5 07:58 cert146.pem
-rw-r–r-- 1 root root 1704 Feb 6 07:55 privkey147.pem
-rw-r–r-- 1 root root 3452 Feb 6 07:55 fullchain147.pem
-rw-r–r-- 1 root root 1647 Feb 6 07:55 chain147.pem
-rw-r–r-- 1 root root 1805 Feb 6 07:55 cert147.pem
-rw-r–r-- 1 root root 1708 Feb 7 07:52 privkey148.pem
-rw-r–r-- 1 root root 3452 Feb 7 07:52 fullchain148.pem
-rw-r–r-- 1 root root 1647 Feb 7 07:52 chain148.pem
-rw-r–r-- 1 root root 1805 Feb 7 07:52 cert148.pem
-rw-r–r-- 1 root root 1708 Feb 9 07:49 privkey149.pem
-rw-r–r-- 1 root root 1805 Feb 9 07:49 cert149.pem
-rw-r–r-- 1 root root 3452 Feb 9 07:49 fullchain149.pem
-rw-r–r-- 1 root root 1647 Feb 9 07:49 chain149.pem
-rw-r–r-- 1 root root 1704 Feb 10 07:48 privkey150.pem
-rw-r–r-- 1 root root 3452 Feb 10 07:48 fullchain150.pem
-rw-r–r-- 1 root root 1647 Feb 10 07:48 chain150.pem
-rw-r–r-- 1 root root 1805 Feb 10 07:48 cert150.pem
drwxr-xr-x 2 root root 20480 Feb 10 07:48 .
After changing the cron-jobs from @daily to @weekly I have not noticed any more “flooding” requests while having an eye on crt.sh.
I was running certbot renewal today manually (same command as used in cron-task) and it seems to run fine and smooth.
I’ll still having a look on this and if this solves all my problems now.
Is it still continuing to issue unnecessary certificates once a week?
Did the renew command you tried to day issue another unnecessary certificate?
Slowing down to 1-2 unnecessary certificates per week stays far below the rate limits, but i wouldn’t call it good…
I had to renew a bunch of certs manually today as a lot of them where unable to renew since the past few weeks. As far as I noticed everything worked fine without problems.
I think it will get interesting again for the next 1-2 next scheduled weekly renewals.