Centos7/apache/certbot broken (apachectl)

Client dev
1

The command “% certbot renew --dry-run” fails on Centos 7 due to problems running apachectl.

First, I noticed that certbot is attempting to use apachectl -k start (and later stop), which produce errors (“Passing arguments to httpd using apachectl is no longer supported. …”). Changing pre_hook and post_hook in /etc/letsencrypt/renewal/{domain name}.conf to use systemctl (apparently the centos7 ways of managing apache httpd) helps:
#pre_hook = apachectl -k stop
pre_hook = systemctl stop httpd
#post_hook = apachectl -k start
post_hook = systemctl start httpd

However, additional references to apachectl are buried inside the certbot code, which invokes “apachectl graceful” during the renew procedure as seen in the command’s output:

% certbot renew --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/{domain name}.conf

Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator standalone, Installer apache
Starting new HTTPS connection (1): acme-staging.api.letsencrypt.org
Running pre-hook command: systemctl stop httpd
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for {domain name}
Waiting for verification…
Cleaning up challenges
Error while running apachectl graceful.

Job for httpd.service invalid.

Attempting to renew cert ({domain name}) from /etc/letsencrypt/renewal/{domain nam}.conf produced an unexpected error: Error while running apachectl graceful.

Job for httpd.service invalid.
. Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/{domain name}/fullchain.pem (failure)

** DRY RUN: simulating ‘certbot renew’ close to cert expiry
** (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/{domain name}.com/fullchain.pem (failure)
** DRY RUN: simulating ‘certbot renew’ close to cert expiry
** (The test certificates above have not been saved.)

Running post-hook command: systemctl start httpd
1 renew failure(s), 0 parse failure(s)

OK. I patched this by editing the certbot code (/usr/lib/python2.7/site-packages/certbot_apache/override_centos.py):
# restart_cmd=[‘apachectl’, ‘graceful’],
restart_cmd=[‘systemctl’, ‘restart’, ‘httpd’],

This change appears to fix certbot (renew --dry-run finishes successfully), but potentially at the expense of graceful shutdown of existing connections.

I don’t know if my changes are correct and I worry that a certbot update will override the modified code, but I do know that something is wrong with the way the installation process and the current code interact with systemctl managed apache httpd.

2

@joohoi @bmw, do you think there’s an indication that the Apache override command for CentOS needs to be changed? Or just documentation about what hooks to pass?

3

What do you mean by “the Apache override command for CentOS?”

My understanding is that CentOS 7 (like its RHEL parent OS) uses systemd to control services, including Apache httpd, and that the correct way to manage services is via systemctl (and related) commands. Apachectl is still present, runs, and affects the server processes, but may be confusing systemd or may not do what we expect. Certbot is configured to use apachectl. I don’t know apachectl and systemctl well enough to know if they are inherently incompatible, or just being used incorrectly.

Also, I see that certbot attempts to stop the web server (pre_hook = apachectl -k stop), then gracefully restart (apachectl graceful), then start the service again (post_hook -= apachectl -k start). This sequence seems odd since you can’t gracefully restart a stopped server, nor can you start an already running server. Note that I might be overlooking intermediate steps, I don’t have time to fully investigate certbot’s behavior. I also notice that the command line switch “-k” is ignored by apachectl on some systems and causes an error on CentOS7 (it appears to be a switch for httpd that carried over to apachectl).

Again, I don’t know the details of certbot’s operation, but I wonder why it attempts to stop and start httpd at all. It seems better to issue the graceful reload command each time the server configuration is changed. In the case of systemd on CentOS7, that command seems to be “systemctl reload httpd”. Note I’ve been using “httpd” but the full service name appears to be “httpd.service.” I’ve been using the shorter because it is easier to type.

4

The command defined in override_centos.py (that you changed to use systemd, which may be a more correct solution, as you describe here).

The use of a particular pre_hook and post_hook are not Certbot defaults at all—they're not part of the upstream shipped package—and were presumably specified by whoever originally obtained the certificate at the time it was obtained. Normally those particular values (stopping and starting Apache) are used only with --standalone, which you are using, so perhaps you got them from a particular online tutorial?

--standalone creates its own web server so it can only be used on systems that don't already have one, or where the existing one is temporarily stopped. There are other methods intended for use on systems that have an existing web server, which don't have this requirement.

5

(Most Apache users can use --apache; most nginx users can use --nginx; most users of other web servers can use --webroot, sometimes with a configuration change to allow static files to be served out of /.well-known/acme-challenge.)

6

AHA! Thank you. Repairing my configuration solved the problem and the original certbot (change removed) runs correctly.

My error was following instructions [at https://certbot.eff.org/all-instructions/] for standalone authentication for the wrong OS. The command was:

sudo certbot --authenticator standalone --installer apache --pre-hook “apachectl -k stop” --post-hook “apachectl -k start”

Removing the hooks and setting authenticator to apache in the domain’s letsencrypt conf file got me to a nearly working state. From there, I just needed to add a VirtualHost on port 80 to the httpd config to enable certbot to share the server.

Glad to have fixed this!

1 Like
7

I’m just now getting to this. Sorry it took two weeks.

Glad you got things working. We’re aware of this problem and it’s being tracked by https://github.com/certbot/certbot/issues/5439.

8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.