Let's Encrypt creates new config files

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: eec.de

I ran this command: certbot cron job

It produced this output: none

My web server is (include version): Apache 2.4.18

The operating system my web server runs on is (include version): Ubuntu 16.04 LTS

My hosting provider, if applicable, is: myself

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): Virtualmin

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): 0.23.0

At some stage my system started creating new config files whenever a new renewal was scheduled. My system obviously had some configuration issues (see Technical questions - config problems), but now I do not have any errors any longer, but only this “new config file issue”.

From what I can see, the new versions only differ by the version number, the rest is identical.

My current version of certbot is 0.23.0, the latest config files show 0.34.2, the latest version in the ppa is 0.31.0.

I refrained from activating the ppa in my software packages setup, as the version provided in the repository is still lower than the one in the config files. Moreover I fear that using the packages from the ppa may have side effects on my system.

Any idea, how I could keep certbot from creating new versions every month / two months?

2

Hi @Vince42

what's your cron job command?

If you use Virtualmin, it's normally not a good idea to run an own Certbot.

1 Like
3

The cron job command is test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot -q renew.

I do not run an own certbot aside from the certbot that has been configured by / in Virtualmin. I only use the certbot tool to query information and to remove obsolent configs etc.

4

I removed all additional config files, kept the most recent version and requested a new certificate for that version. certbot keeps creating new config files on each request.

@JuergenAuer How can I track down this issue? When is usually a new config file created?

5

What exactly do you mean by “new config files”?

Can you post “sudo ls -alR /etc/letsencrypt/” (or whatever directory it’s in)?

Edit:

What does “sudo certbot certificates” show?

What version of Certbot is currently installed?

Edit:

Change ‘What does “sudo certbot certificates show”?’ to ‘What does “sudo certbot certificates” show?’. Sorry for the mistake.

6

Then ignore the problem.

May be a Virtualmin configuration. Virtualmin may use explicit definitions, so every time a config file with the next number is created.

That's not a problem, ignore it.

7

If something is creating duplicate certificate lineages in Certbot’s configuration, there might be a problem. Especially if excessive duplicate certificates are being issued.

8

@mnordhoff
With “new config files” I mean, that I have one original config file domain.tld.conf and on each renewal new config files named domain.tld-0001.conf and so forth are created.

I could post the listing of /etc/letsencrypt, but it is very long. Is there anything specific, which you are interested in?

The certbot version is 0.23.0 and it lists

  • A lot of Attempting to parse the version 0.35.1 renewal configuration file found at /etc/letsencrypt/renewal/domain.tld-0001.conf with version 0.23.0 of Certbot. This might not work.
  • Some Revocation status for /etc/letsencrypt/live/domain.tld/cert.pem is unknown
  • And then lists all certificates and their superfluous duplicates.
9

"certbot renew" shouldn't ever do that. I'd be surprised if it can do that, but I can't say for sure.

Not really. I'm curious what exists and if anything looks damaged.

There must be second installation of Certbot -- probably certbot-auto.

Despite the warning that "this might not work", it should almost definitely work. Certainly, this shouldn't happen.

Typically happens if the certificate is expired. (CAs only have to maintain revocation information for current certificates. It saves a lot of money.)

Edit: I meant to add some more speculation:

I wonder if there is a cron job or systemd timer running some sort of different Certbot command periodically.

10

Checking the CT log there was a mess ( https://check-your-website.server-daten.de/?q=eec.de#ct-logs )

Issuer not before not after Domain names LE-Duplicate next LE
Let’s Encrypt Authority X3 2019-06-18 2019-09-16 autoconfig.eec.de, autodiscover.eec.de, eec.de, www.eec.de
4 entries
Let’s Encrypt Authority X3 2019-06-18 2019-09-16 autoconfig.eec.de, autodiscover.eec.de, eec.de, www.eec.de
4 entries
Let’s Encrypt Authority X3 2019-05-19 2019-08-17 autoconfig.eec.de, autodiscover.eec.de, eec.de, www.eec.de
4 entries
Let’s Encrypt Authority X3 2019-04-19 2019-07-18 autoconfig.eec.de, autodiscover.eec.de, eec.de, www.eec.de
4 entries
Let’s Encrypt Authority X3 2019-03-20 2019-06-18 autoconfig.eec.de, autodiscover.eec.de, eec.de, www.eec.de
4 entries
Let’s Encrypt Authority X3 2019-03-14 2019-06-12 autoconfig.eec.de, autodiscover.eec.de, eec.de, www.eec.de, x3.eec.de
5 entries
Let’s Encrypt Authority X3 2019-03-14 2019-06-12 eec.de, www.eec.de
2 entries
Let’s Encrypt Authority X3 2019-03-14 2019-06-12 autoconfig.eec.de, autodiscover.eec.de, eec.de, www.eec.de, x3.eec.de
5 entries
Let’s Encrypt Authority X3 2019-03-14 2019-06-12 autoconfig.eec.de, autodiscover.eec.de, eec.de, www.eec.de, x3.eec.de
5 entries
Let’s Encrypt Authority X3 2019-03-14 2019-06-12 eec.de, www.eec.de
2 entries
Let’s Encrypt Authority X3 2019-03-14 2019-06-12 eec.de, www.eec.de
2 entries
Let’s Encrypt Authority X3 2019-03-14 2019-06-12 autoconfig.eec.de, autodiscover.eec.de, eec.de, www.eec.de, x3.eec.de
5 entries
Let’s Encrypt Authority X3 2019-03-14 2019-06-12 autoconfig.eec.de, autodiscover.eec.de, eec.de, www.eec.de, x3.eec.de
5 entries

15 certificates created 2019-03-14 (not all copied), but that has stopped.

The two newest entries:

2019-06-18 20:47:58.
2019-06-18 19:06:09.

Looks like you run Certbot as Cronjob, then your VirtualMin updates the certificate.

It’s wrong to use VirtualMin and an own certbot.

11

certbot renew did that trick in the past. I only noticed that new config files are created, when an attempt to renew the certificates failed due to too many certification requests.

I uploaded listing etc_letsencrypt.txt (498.9 KB) for your curiosity. :wink:

I only installed certbot once through the Virtualmin interface and refrained from installing certbot-auto. Finding certbot on my server delivers the following:
apt list --installed | grep certbot
certbot/xenial-updates,xenial-updates,now 0.23.0-1~ubuntu16.04.1 all [installed,automatic]
python-certbot/xenial-updates,xenial-updates,now 0.23.0-1~ubuntu16.04.1 all [installed,automatic]

/usr/bin/certbot is used as command.

The only command run is the one in my second post.

12

In march I ran into the “too-many-certificate-requests-problem” described above. I manually removed all additional configs except for one. On each renewal a new config file is created for each existing config file.

The two newest entries should, for example, be just one entry.

You are probably right, that Certbot is run as cron job and VirtualMin takes care of the rest.

As it is wrong to use VirtualMin and an own certbot, I refrained from installing anything outside of Virtualmin. It all worked like charm, but back some months ago, the system started creating new config files without having changed anything; I would really like to understand, when a new config file is created and who does that: certbot or Virtualmin.

Without knowing the cause for the newly created config files, I cannot track down the root of the problem.

13

Hm. I don't know about Virtualmin, so I can only speculate about what it might be doing. :confounded:

Thanks!

Preliminary thoughts:

  • Wow, 3441 CSRs and keys! That's quite a lot of certificates, or attempts to create certificates.

    (If you try to issue a certificate, but it fails due to validation or rate limits or something, it can or will fail after Certbot has created the CSR and key.)

  • Certbot will, by default, renew certificates every 60 days.

  • The default Certbot systemd timer -- or, if you have systemd disabled, cron job -- runs at a random time of the day, as you posted earlier. You have some certificates issued at 23:02, 23:06, 00:01 or 00:06.

  • Interesting that stuff seems to happen March 20, April 19, May 20, June 19.

  • Looking at the eec.de certificates, it seems like something creates new certificate lineages every month, shortly before or after midnight. Then something else renews them every 60 days at random times of the day.

  • The file permissions are different. This might imply that the renewal certificates are being created by an older version of Certbot than the original certificates.

  • I haven't looked super closely, but nothing looks damaged, except that an impossible number of modification times are set to March 19 at 09:17.

  • It looks like "certbot renew" is trying to renew 6 or so certificates every time it runs. Yikes.

  • Sometimes other stuff happens, e.g. attempts to issue 4 certificates on July 6 at 02:12, 02:17, 02:19 and 02:20, which resulted in 3 domain1.tld certificate lineages.

Hypothesis:

The Ubuntu Certbot package and its "certbot renew" systemd timer (/cron job) are behaving normally.

There is a second installation of Certbot, and it's doing other stuff. I'm inclined to point the finger at Virtualmin, or one or more cron jobs or timers that are running Certbot commands other than renew.

Can you poke around in /var/log/letsencrypt/ for stuff from June 19 and July 6? It should log both the version number and command line options -- but not the full command. E.g. if you run "certbot renew -q", it only logs "['-q']".

Each date should have two runs of "certbot renew -q", and also some mysterious other stuff.

It's possible that some logs are being directed elsewhere.

Can you check systemctl list-timers for Certbot related stuff? There should be one service.

And do something like "grep -r certbot /etc/cron* to look for cron jobs? (Those files aren't the only place cron jobs are stored, though.)

1 Like
14

First of all: thank you very much that you are taking the time to dive deeper into this phenomenon!

I do not know what Virtualmin is doing either, but it used to work for months without any problem - that’s all I can say.

Regarding your preliminary thoughts:

  • I think that the 3,441 CSRs result from the duplicity of config files (used to be something like 40 per domain, after the weirdness started).

  • I used to switch all my domains to a 30 day renewal period. Especially as long as this problem occurs, I want to see on a regular basis, whether everything is okay or not.

  • The certificates that do not fall into the time window of the default certbot renewal process, were manually requested by me through the Virtualmin interface. This happened, when I tested, whether still new config files are created or not.

  • Your finding about the 30 and 60 day renewals is interesting! This does indeed look like a second certbot, but as the package list shows there are only two packages with “certbot” in their names - certbot and python-certbot - and those were automatically installed. I assumed that both packages are needed.

  • I could try to reapply the recommended file permissions - do you think that this could change something?

  • The modification times result from an attempt to start from scratch. After finding that this attempt failed, I copied the old directory back to its original location on March 19th.

  • I currently have 11 certificates managed by certbot. Shouldn’t certbot try to renew all 11 certificates?

  • The multiple certificate issues all seem to come from the multiple config files, I assume.

I checked /var/log/letsencrypt/letsencrypt.log and found the following lines regarding the domain eec.de. I hope that I grabbed all the lines that are related to the domain. letsencrypt.log.txt (29.7 KB).

systemctl list-timers
NEXT LEFT LAST PASSED UNIT ACTIVATES
Wed 2019-07-10 22:39:00 CEST 29min left Wed 2019-07-10 22:09:01 CEST 32s ago phpsessionclean.timer phpsessionclean.service
Thu 2019-07-11 00:48:39 CEST 2h 39min left Wed 2019-07-10 00:48:39 CEST 21h ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Thu 2019-07-11 01:57:18 CEST 3h 47min left Wed 2019-07-10 14:29:55 CEST 7h ago apt-daily.timer apt-daily.service
Thu 2019-07-11 06:08:00 CEST 7h left Wed 2019-07-10 06:04:21 CEST 16h ago apt-daily-upgrade.timer apt-daily-upgrade.service
Thu 2019-07-11 10:09:14 CEST 11h left Wed 2019-07-10 18:29:56 CEST 3h 39min ago certbot.timer certbot.service
n/a n/a Sat 2019-07-06 00:34:55 CEST 4 days ago ureadahead-stop.timer ureadahead-stop.service
6 timers listed.

grep -r certbot /etc/cron*
/etc/cron.d/certbot:# /etc/cron.d/certbot: crontab entries for the certbot package
/etc/cron.d/certbot:0 */12 * * * root test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot -q renew

I hope you will develop more ideas - I am becoming more and more clueless. :sweat_smile:

15

This is an interesting problem. I do think that Virtualmin is probably somehow calling Certbot in an incorrect way which is resulting in this duplicative stuff.

It would be helpful to have access to more complete logs from /var/log/letsencrypt, and also the output of certbot certificates.

16

Ah.

How did you switch it?

Right. The certbot and python-certbot packages both contain different parts of the Certbot installation.

But Certbot can also be installed in other ways.

No, it doesn't have an effect on things. I was just interested because it's a way to tell apart files created by 0.23.0 and 0.35.1, because it was a change that happened at some point in between.

Yes.

(To be excessively precise, you can configure Certbot not to renew certain certificates, but there's no reason to think that's happening here.)

MissingCommandlineFlag: Missing command line flag or config entry for this setting:
Input the webroot for www.eec.de:

That's caused by an unfortunate bug that was introduced around 0.31.0 and fixed around 0.35.0. When issuing certificates, under some circumstances, buggy versions of Certbot would leave out the web root setting from the renewal configuration file, which would make renewal fail next time.

If you're only running 0.23.0 and 0.35.1, no more incorrect configuration files will be produced, but those that already exist need to be fixed. (It takes a couple minutes of running commands or editing files.)

This may explain why renewal is failing -- at least for that certificate; there could be other reasons too -- but does not explain why duplicate certificates are being produced. :confused:

Does Virtualmin have any cron jobs or timers to periodically do stuff like issue certificates?

Edit:

I meant to say that the timers and cron job you pasted look normal, at least as far as Certbot goes. :slightly_smiling_face:

But it wasn't a complete check, and it's still possible for Certbot to be run periodically in other ways.

1 Like
17

@schoen
I just downloaded the complete current log file: letsencrypt-anonymized.log.txt (552.0 KB)
If I should provide other log files as well, please let me know.

This is the output of certbot certificates: certbot-certificates.txt (20.7 KB)

18

@mnordhoff

  • Virtualmin has an option to specify the renewal interval.
  • All certbot binaries (just grepped for bin) are installed under /opt/eff.org/certbot. Besides * I am only using apt and I doubt that Virtualmin would extract some tar.gz and provide binaries anywhere else, as such a system would almost be unmaintainable. This being said: I am quite sure, that I am only running one version (certbot 0.23.0) and nothing else.
  • I am also quite sure that certbot is only run periodically by the afore-mentioned cron job.

Would it be a good idea to switch the repository for certbot to something, where the latest versions are provided earlier?

19

Could you look for certbot-auto anywhere on the system?

20

The only location of certbot-auto on the system is /opt/eff.org/certbot/venv/certbot-auto-bootstrap-version.txt.