Httpd reload fails after loading challenges

AFAIK when changes to httpd conf files are made, like those to load challenges, the httpd needs to be restarted or reloaded. I am never prompted to perform this restart or reload when I run certbot, so I assume that certbot attempts to do it by itself, automatically. Apparently it does not work - see below. If I add --debug-challenges option that forces certbot to wait for confirmation:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Challenges loaded. Press continue to submit to CA.

The following URLs should be accessible from the internet and return the value
mentioned:

URL:
http://letsencrypt.transmodus.net/.well-known/acme-challenge/Ha8CcWuyJMifoa8gnmeXmAMyM_jDWijzsfrVMWzMIzQ
Expected value:
Ha8CcWuyJMifoa8gnmeXmAMyM_jDWijzsfrVMWzMIzQ.t3ntf5XroI-oL7AocX5UC5rUw-N5GiXZqeKy0Mfta4M
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Press Enter to Continue

then I can open another console, restart or reload httpd by hand, which loads the updated configuration files with Includes, then hit the Enter in the first console, and receive the certificates. I am sure something is not working as intended here, but cannot find any references to certbot restarting/reloading httpd. Please help.

Standard responses below.

My domain is:r3test.paycosmos.com

I ran this command:
certbot --apache --apache-server-root /opt/rh/jbcs-httpd24/root/etc/httpd-wtl --apache-logs-root /opt/rh/jbcs-httpd24/root/var/log/httpd-wtl --apache-challenge-location /opt/rh/jbcs-httpd24/root/var/www/html --apache-ctl /opt/rh/jbcs-httpd24/root/usr/sbin/apachectl

It produced this output:

Challenge failed for domain r3test.paycosmos.com
http-01 challenge for r3test.paycosmos.com
Notifying user:
Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
  Domain: r3test.paycosmos.com
  Type:   unauthorized
  Detail: 129.146.194.11: Invalid response from https://r3test.paycosmos.com/.well-known/acme-challenge/51_-Cjn6Q4GpvQtD7rt_-5UCmChEjbC9BDtO6qKzquo: 404

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

My web server is (include version):jbcs-httpd24-httpd-2.4.37-64.jbcs.el7.x86_64

The operating system my web server runs on is (include version):RHEL 7.9

My hosting provider, if applicable, is:none

I can login to a root shell on my machine (yes or no, or I don't know):yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):certbot 1.26.0

Do you happen to have a second apachectl in your PATH, other than /opt/rh/jbcs-httpd24/root/usr/sbin/apachectl?

I'm reading through the CentOS overrides in Certbot (which are used for RHEL too) and what it looks like is that Certbot might be trying to call apachectl graceful first, and only if that fails, does it try to call /opt/rh/jbcs-httpd24/root/usr/sbin/apachectl restart.

I'm not 100% sure about this though, I could be misreading the code. I will try to test it when I get a chance.

2 Likes

I would try using --webroot and provide a --deploy-hook to handle the reload.
[avoiding the term "apache" as much as possible]

2 Likes

My /sbin/apachectl is a symbolic link to /opt/rh/jbcs-httpd24/root/usr/sbin/apachectl:

lrwxrwxrwx 1 root root 44 Mar 8 08:32 /sbin/apachectl -> /opt/rh/jbcs-httpd24/root/usr/sbin/apachectl

But where in the certbot files/scripts will I even see an attempt to do a reload/restart??? And by what means? That's my question.... it would help me troubleshoot what is happening.

Would it be too much to ask for the command rewrite based on the options you mention here? The command is here:

certbot --apache \ 
--apache-server-root /opt/rh/jbcs-httpd24/root/etc/httpd-wtl \
--apache-logs-root /opt/rh/jbcs-httpd24/root/var/log/httpd-wtl \
--apache-challenge-location /opt/rh/jbcs-httpd24/root/var/www/html \
--apache-ctl /opt/rh/jbcs-httpd24/root/usr/sbin/apachectl

And how would I avoid the term apache when I am running httpd?

Where will I find them?

Hi @transmodus,

The logs from Certbot should be in /var/log/letsencrypt; maybe you can find a specific reason there that it's trying to do the wrong thing (e.g. restarting the wrong instance of Apache).

The "overrides" refer to OS-specific defaults that override the original Certbot defaults. The file @_az is talking about is

These defaults are not really meant to be edited by end users; the command-line options that indicate a different location for apachectl should have priority over the OS-specific defaults ("overrides"), I would think.

3 Likes

What I was worried about here was that restart_cmd_alt gets overriden, but restart_cmd doesn't. So even if you passed a different --apache-ctl, it would not use it for restart_cmd.

But I think at the end of the day, this is not an issue, because _prepare_options in the parent class should take care of that.

I'm not sure what's wrong here, I will still try to test later on, assuming I can find this httpd package.

3 Likes

certbot doesn't need to know which web server is being used, nor does it need to modify it.
You could use --webroot instead.
Something like:

certbot \
--webroot -w /your/sites/root/path \
-d r3test.paycosmos.com \
--deploy-hook /some/script/that/reloads/apache
3 Likes