First renewal not successful. HTTP service stopped


#1

Hello all,

Thanks to the people at Let’s Encrypt for the free certificates and certbot for making it simple to setup.

Not realizing I had entered the 30-days-before-expiry period for the four domains I have Let’s Encrypt certificates for, I noticed a few days ago that my self-hosted websites were all offline. I logged into my server and noticed that the httpd service was not running, so I started it manually.

When I noticed the websites were all offline again this evening, I logged into the server once more and started the httpd service. I happened to notice that it was almost time for my twice-daily certbot cron job to run. I had just checked that the websites were back up and running and decided to check them again after my certbot cron job had run. Sure enough the httpd service had stopped again. I checked the httpd logs for each of the sites and they each contain gaps where the last access was shortly before the time the certbot cron job was scheduled to run.

I ran ‘certbot renew’ (as root) from the command line to see what was the matter, but the certificates renewed successfully.

My hunch is that SELinux is the responsible as these denials started appearing in the audit log at the same time the first gaps in the httpd logs appear (and the certbot cron job was scheduled to run):

type=AVC msg=audit(1524976981.708:16075): avc: denied { write } for pid=7556 comm=“httpd” path="/var/lib/letsencrypt/.certbot.lock" dev=“dm-0” ino=54526412 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:cron_var_lib_t:s0 tclass=file
type=AVC msg=audit(1524976981.708:16075): avc: denied { write } for pid=7556 comm=“httpd” path="/var/log/letsencrypt/.certbot.lock" dev=“dm-0” ino=54526085 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:cron_log_t:s0 tclass=file
type=AVC msg=audit(1524976981.708:16075): avc: denied { write } for pid=7556 comm=“httpd” path="/etc/letsencrypt/.certbot.lock" dev=“dm-0” ino=3932467 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:etc_t:s0 tclass=file

Thanks for reading my post. If anyone has suggestions for how I might make the next renewal go smoother, I’d appreciate them.

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:

I ran this command: certbot renew (from cron)

It produced this output: excerpt from /var/log/messages:

May 8 00:43:01 server3 systemd: Created slice User Slice of root.
May 8 00:43:01 server3 systemd: Starting User Slice of root.
May 8 00:43:01 server3 systemd: Started Session 572 of user root.
May 8 00:43:01 server3 systemd: Starting Session 572 of user root.
May 8 00:43:04 server3 httpd: AH00526: Syntax error on line 10 of /etc/httpd/conf.d/le_tls_sni_01_cert_challenge.conf:
May 8 00:43:04 server3 httpd: SSLCertificateFile: file ‘/var/lib/letsencrypt/vGUiLNHXgx0Rd6Ek2XiGJB_mnRX1pP7OVteVIyfnW0E.crt’ does not exist or is empty
May 8 00:43:04 server3 systemd: httpd.service: control process exited, code=exited status=1
May 8 00:43:04 server3 systemd: Reload failed for The Apache HTTP Server.
May 8 00:43:04 server3 systemd: Stopping The Apache HTTP Server…
May 8 00:43:05 server3 systemd: Unit httpd.service entered failed state.
May 8 00:43:05 server3 systemd: httpd.service failed.
May 8 00:43:05 server3 systemd: Starting The Apache HTTP Server…
May 8 00:43:05 server3 httpd: AH00526: Syntax error on line 10 of /etc/httpd/conf.d/le_tls_sni_01_cert_challenge.conf:
May 8 00:43:05 server3 httpd: SSLCertificateFile: file ‘/var/lib/letsencrypt/vGUiLNHXgx0Rd6Ek2XiGJB_mnRX1pP7OVteVIyfnW0E.crt’ does not exist or is empty
May 8 00:43:05 server3 systemd: httpd.service: main process exited, code=exited, status=1/FAILURE
May 8 00:43:05 server3 kill: kill: cannot find process “”
May 8 00:43:05 server3 systemd: httpd.service: control process exited, code=exited status=1
May 8 00:43:05 server3 systemd: Failed to start The Apache HTTP Server.
May 8 00:43:05 server3 systemd: Unit httpd.service entered failed state.
May 8 00:43:05 server3 systemd: httpd.service failed.
May 8 00:43:05 server3 systemd: Unit httpd.service cannot be reloaded because it is inactive.
May 8 00:43:05 server3 systemd: Starting The Apache HTTP Server…
May 8 00:43:05 server3 systemd: Started The Apache HTTP Server.
May 8 00:43:07 server3 httpd: AH00526: Syntax error on line 10 of /etc/httpd/conf.d/le_tls_sni_01_cert_challenge.conf:
May 8 00:43:07 server3 httpd: SSLCertificateFile: file ‘/var/lib/letsencrypt/20N6Yuxa165aBWVHu_mSCPME_uCGmvzKhYnzoGHU7ZY.crt’ does not exist or is empty
May 8 00:43:07 server3 systemd: httpd.service: control process exited, code=exited status=1
May 8 00:43:07 server3 systemd: Reload failed for The Apache HTTP Server.
May 8 00:43:07 server3 systemd: Stopping The Apache HTTP Server…
May 8 00:43:08 server3 systemd: Unit httpd.service entered failed state.
May 8 00:43:08 server3 systemd: httpd.service failed.
May 8 00:43:08 server3 systemd: Starting The Apache HTTP Server…
May 8 00:43:08 server3 httpd: AH00526: Syntax error on line 10 of /etc/httpd/conf.d/le_tls_sni_01_cert_challenge.conf:
May 8 00:43:08 server3 httpd: SSLCertificateFile: file ‘/var/lib/letsencrypt/20N6Yuxa165aBWVHu_mSCPME_uCGmvzKhYnzoGHU7ZY.crt’ does not exist or is empty
May 8 00:43:08 server3 systemd: httpd.service: main process exited, code=exited, status=1/FAILURE
May 8 00:43:08 server3 kill: kill: cannot find process “”
May 8 00:43:08 server3 systemd: httpd.service: control process exited, code=exited status=1
May 8 00:43:08 server3 systemd: Failed to start The Apache HTTP Server.
May 8 00:43:08 server3 systemd: Unit httpd.service entered failed state.
May 8 00:43:08 server3 systemd: httpd.service failed.
May 8 00:43:08 server3 systemd: Unit httpd.service cannot be reloaded because it is inactive.
May 8 00:43:08 server3 systemd: Starting The Apache HTTP Server…
May 8 00:43:08 server3 systemd: Started The Apache HTTP Server.
May 8 00:43:11 server3 httpd: AH00526: Syntax error on line 10 of /etc/httpd/conf.d/le_tls_sni_01_cert_challenge.conf:
May 8 00:43:11 server3 httpd: SSLCertificateFile: file ‘/var/lib/letsencrypt/12BtcgE5wEH9uWXJQgbYqvJT8zaMGCCnQ_ZVNUW_XF0.crt’ does not exist or is empty
May 8 00:43:11 server3 systemd: httpd.service: control process exited, code=exited status=1
May 8 00:43:11 server3 systemd: Reload failed for The Apache HTTP Server.
May 8 00:43:11 server3 systemd: Stopping The Apache HTTP Server…
May 8 00:43:12 server3 systemd: Unit httpd.service entered failed state.
May 8 00:43:12 server3 systemd: httpd.service failed.
May 8 00:43:12 server3 systemd: Starting The Apache HTTP Server…
May 8 00:43:12 server3 httpd: AH00526: Syntax error on line 10 of /etc/httpd/conf.d/le_tls_sni_01_cert_challenge.conf:
May 8 00:43:12 server3 httpd: SSLCertificateFile: file ‘/var/lib/letsencrypt/12BtcgE5wEH9uWXJQgbYqvJT8zaMGCCnQ_ZVNUW_XF0.crt’ does not exist or is empty
May 8 00:43:12 server3 systemd: httpd.service: main process exited, code=exited, status=1/FAILURE
May 8 00:43:12 server3 kill: kill: cannot find process “”
May 8 00:43:12 server3 systemd: httpd.service: control process exited, code=exited status=1
May 8 00:43:12 server3 systemd: Failed to start The Apache HTTP Server.
May 8 00:43:12 server3 systemd: Unit httpd.service entered failed state.
May 8 00:43:12 server3 systemd: httpd.service failed.
May 8 00:43:12 server3 systemd: Unit httpd.service cannot be reloaded because it is inactive.
May 8 00:43:12 server3 systemd: start request repeated too quickly for httpd.service
May 8 00:43:12 server3 systemd: Failed to start The Apache HTTP Server.
May 8 00:43:12 server3 systemd: httpd.service failed.
May 8 00:43:14 server3 systemd: Unit httpd.service cannot be reloaded because it is inactive.
May 8 00:43:14 server3 systemd: start request repeated too quickly for httpd.service
May 8 00:43:14 server3 systemd: Failed to start The Apache HTTP Server.
May 8 00:43:14 server3 systemd: httpd.service failed.
May 8 00:43:14 server3 systemd: Unit httpd.service cannot be reloaded because it is inactive.
May 8 00:43:14 server3 systemd: start request repeated too quickly for httpd.service
May 8 00:43:14 server3 systemd: Failed to start The Apache HTTP Server.
May 8 00:43:14 server3 systemd: httpd.service failed.

My web server is (include version): Apache 2.4.6

The operating system my web server runs on is (include version): CentOS 7.4.1708

My hosting provider, if applicable, is: self-hosted

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no


#2

Firstly, I would check the cronjob to ensure it no longer stops your web service - or remove it for now.

Have you tried
sudo certbot renew

Please show:
/etc/letsencrypt/renewal/<your.domain>.conf


#3

Thanks for the very quick reply.

Yes. I ran ‘certbot renew’ as root and the certificates renewed successfully.

Now that the certificates are not due for renewal, I don’t think the problem will re-occur until I reach 30 days before expiry again.

Here are the contents of one of the domains in /etc/letsencrypt/renewal/*.conf

renew_before_expiry = 30 days

version = 0.23.0
archive_dir = /etc/letsencrypt/archive/DOMAIN
cert = /etc/letsencrypt/live/DOMAIN/cert.pem
privkey = /etc/letsencrypt/live/DOMAIN/privkey.pem
chain = /etc/letsencrypt/live/DOMAIN/chain.pem
fullchain = /etc/letsencrypt/live/DOMAIN/fullchain.pem

Options used in the renewal process

[renewalparams]
authenticator = apache
installer = apache
account = ACCOUNT
/etc/letsencrypt/renewal/DOMAIN.conf (END)


#4

Yes, and I’m glad to hear that; but it will repeat the bad process in 60 days and shut down your web service again.
You should take the time to fix the process now so that it will work without any manually intervention.


#5

The renewal .conf file looks good.

Please show your cron job.
crontab -l


#6

Contents of /etc/cron.d/certbot:

MAILTO=""
43 0,12 * * * root certbot renew

Running ‘crontab -l’ as root gives: ‘no crontab for root’


#7

Mine is more like:
43 0,12 * * * /usr/bin/certbot renew --nginx
and it works flawlessly.

You could try it with Apache plugin, like:
43 0,12 * * * /usr/bin/certbot renew --apache
or
43 0,12 * * * root /usr/bin/certbot renew --apache

Try this now - let’s see if it throws any errors:
/usr/bin/certbot renew --apache

I’m on Ubuntu
This may be related: Certbot via cron writes files unreadable by apache (SELinux/CentOS 7)

There doesn’t seem to be a lot of support for SELinux…


#8

Are you running SELinux?


#9

No errors upon running ‘certbot renew --apache’ (mine is in /bin) but it does not attempt to renew as it’s not due for renewal now.

Yes, Thanks.


#10

Fingers crossed.
I was going to say “let us know in 60 days”
But this thread would close after 30 days of non-activity…

So, maybe we can speed things up a bit.
modify this:
renew_before_expiry = 30 days
to this (for the first renewal test):
renew_before_expiry = 80 days
Which should then start to try to renew in 10 days.
If it works well then just put it back to 30 days.
Either way, please update the ticket for future readers benefit.


#11

Thanks for the link.

Messages in the RedHat bug report suggest using systemd rather than cron.

Here are the contents of /usr/share/doc/certbot-0.23.0/README.fedora:

Automated renewal of certificates

The Fedora certbot package includes an optional systemd timer to handle renewals.

This timer is set to run daily, with a random fudge factor of a 6 hours applied.

To enable the timer based renewals:

systemctl enable --now certbot-renew.timer

The timer makes use of /etc/sysconfig/certbot to customise the behaviour.

Unless there is a plugin that automates restarts (eg the apache plugin) it is
important to configure a command to restart anything that uses the certificates
(END)


#12

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.