Restarting apache after changing conf - please help!

I need something very basic explained to me. Please forgive my ignorance. My server is httpd on linux.

To point the web server to the temporary location that contains the challenge file certbot modifies the httpd.conf file.

writing a pre config file with text:
         RewriteEngine on
        RewriteRule ^/\.well-known/acme-challenge/([A-Za-z0-9-_=]+)$ /var/lib/letsencrypt/http_challenges/$1 [END]

writing a post config file with text:
         <Directory /var/lib/letsencrypt/http_challenges>
            Require all granted
        </Directory>
        <Location /.well-known/acme-challenge>
            Require all granted
        </Location>

AFAIU this modification can only be enabled after the httpd.conf file is re-read by httpd, which can be done in three ways: stop/start, restart, or reload. For whatever reason certbot fails to perform this action. There are no mentions of any failures even with verbosity on -vvv, and I cannot find the mechanism that performs the restart/reload. All I get is:

Certbot failed to authenticate some domains 
Invalid response : 404

which is not surprising, since the httpd does not know anything about the configuration change without restart/reload.

I can confirm that the challenge file is created successfully, and the httpd.conf changes are also made successfully, because if I add debug-challenge option to the command line, I can wait for the prompt, then open another terminal window and manually perform the httpd restart, then press Enter in the certbot window, and the process completes successfully. But this defies the certbot's usefulness as an automation tool.

What is missing in this picture?
Please help.

1 Like

A complete Certbot log file.

1 Like

Please include the complete certbot command being run.

1 Like

@transmodus, Certbot tries to detect your OS and then tries to issue an appropriate command to restart your Apache server. There are some builtin ways that it will do this, but it could get it wrong, especially if your Apache is in a nonstandard location for your OS or was installed in an unusual way. I agree with the other commenters that seeing the command you ran and the log file would be helpful.

The detected OSes are listed in

If you don't see yours there, or you know that your httpd isn't where it typically is on that OS, then you may have to do something extra here.

1 Like

Here is the command:

/bin/certbot -vvv --apache --apache-server-root /opt/rh/jbcs-httpd24/root/etc/httpd-wtl --apache-logs-root /opt/rh/jbcs-httpd24/root/var/log/httpd-wtl --apache-challenge-location /opt/rh/jbcs-httpd24/root/etc/httpd-wtl/conf.d --apache-ctl /opt/rh/jbcs-httpd24/root/usr/sbin/apachectl --apache-bin /opt/rh/jbcs-httpd24/root/usr/sbin/httpd --force-renewal -d xxx.pmt-tiny.io

And the log is attached.
xxx.pmt-tiny.io.failure.log.txt (22.8 KB)

Please don't use that option.

The log indeed does not mentioning anything about reloading Apache. I'm wondering if that is logged even if everything goes correctly though.

2 Likes

I just took a look really quickly and it (surprisingly) looks like it might not be logged if Certbot thinks it succeeded, only if it thinks it failed. I would think it would be helpful to have it log the reload command it tried to use.

2 Likes

Yeah from what I see it seems that certbot is convinced that everything worked...
So there IS a reload command somewhere that is supposed to reload httpd? Where is it? Can it be changed to "restart" instead?

      "error": {
        "type": "urn:ietf:params:acme:error:unauthorized",
        "detail": "144.24.42.68: Invalid response from https://xxx.pmt-tiny.io/.well-known/acme-challenge/1rYb492yHJpjDusoyh--F5BEHQnkqhW5-6Rn6BBq220: 404",
        "status": 403

That seems to imply that the HTTP request did NOT match and was redirected to HTTPS.

You should review your Apache configuration.
OR
Switch to --webroot authentication.

1 Like

As mentioned above the EXACT same configuration works fine when I stop and reload the conf files before proceeding. Doesn't seem to me that the http/https has anything to do with that. And the response is 404, file not found, so the connection is made successfully over https.

Looking at the code for the reloading part:

It indeed looks like the plugin is only logging stuff when reloading does not succeed.

It defaults to apache2ctl graceful (certbot/configurator.py at 295fc5e33a68c945d2f62e84ed8e6aaecfe93102 · certbot/certbot · GitHub), but as @schoen said it can be overridden by specific distro configurations.

I'm interested to see the log if you'd add some logging to configurator.py at line 2462.

1 Like

Yes, I'm game. Tell me what and where to add.
BTW, maybe this is the problem?

[root@cmp-l2p-web-qa-04 wtl.d]# which apache2ctl
/usr/bin/which: no apache2ctl in (/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/var/lib/snapd/snap/bin:/root/bin)
[root@cmp-l2p-web-qa-04 wtl.d]# which apachectl
/sbin/apachectl

I'll make a symlink for apache2ctl and try the renewal again.

[root@cmp-l2p-web-qa-04 wtl.d]# ls -l /sbin/apachectl
lrwxrwxrwx 1 root root 44 Mar  8 08:32 /sbin/apachectl -> /opt/rh/jbcs-httpd24/root/usr/sbin/apachectl
[root@cmp-l2p-web-qa-04 wtl.d]# ln -s /opt/rh/jbcs-httpd24/root/usr/sbin/apachectl /sbin/apache2ctl
[root@cmp-l2p-web-qa-04 wtl.d]# ls -l /sbin/apache*
lrwxrwxrwx 1 root root 44 Jun  6 23:41 /sbin/apache2ctl -> /opt/rh/jbcs-httpd24/root/usr/sbin/apachectl
lrwxrwxrwx 1 root root 44 Mar  8 08:32 /sbin/apachectl -> /opt/rh/jbcs-httpd24/root/usr/sbin/apachectl

Still interested in replacing "reload" with "restart" - there are other issues at play.

I'm not sure that would fix it, as it would log an error if the command couldn't be found.

I'm also interested to see what the Apache log would show. Usually I'd use tail -f to open the log and show the lines being added in realtime and then run e.g. Certbot to see what Apache does.

2 Likes

I'll try that tomorrow with the apache2ctl link in place. But I agree about the error.

creating apache2ctl link did not help.

Here is where I think information about reloading apache should appear:

With ' --debug-challenges' :

2022-06-07 10:24:16,542:DEBUG:certbot.reverter:Creating backup of /opt/rh/jbcs-httpd24/root/etc/httpd/conf.d/mod_cluster.conf
2022-06-07 10:24:16,543:DEBUG:certbot.reverter:Creating backup of /opt/rh/jbcs-httpd24/root/etc/httpd-wtl/wtl.d/r3test.paycosmos.com-le-ssl.conf
2022-06-07 10:24:16,543:DEBUG:certbot.reverter:Creating backup of /opt/rh/jbcs-httpd24/root/etc/httpd-wtl/conf.d/mod_cluster.conf
2022-06-07 10:24:16,544:DEBUG:certbot.reverter:Creating backup of /opt/rh/jbcs-httpd24/root/etc/httpd-wtl/wtl.d/r3test.paycosmos.com.conf
2022-06-07 10:24:19,828:DEBUG:certbot._internal.display.obj:Notifying user: Challenges loaded. Press continue to submit to CA.

The following URLs should be accessible from the internet and return the value
mentioned:

URL:
http://r3test.paycosmos.com/.well-known/acme-challenge/suUWNtCPWRb3BfbAwX78v8FE6jwYr2HkmSMV6HPfjGw
Expected value:
suUWNtCPWRb3BfbAwX78v8FE6jwYr2HkmSMV6HPfjGw.tnPLcddM602NajLZJBpYWVPVn3E93ubwf0htgR_QWRQ
<I PRESS ENTER, POSSIBLY RESTARTING HTTPD IN A SEPARATE TERMINAL>:
2022-06-07 10:24:20,700:DEBUG:acme.client:JWS payload:
b'{}'

The same operation without ' --debug-challenges' :

2022-06-07 10:12:24,189:DEBUG:certbot.reverter:Creating backup of /opt/rh/jbcs-httpd24/root/etc/httpd-wtl/wtl.d/r3test.paycosmos.com-le-ssl.conf
2022-06-07 10:12:24,190:DEBUG:certbot.reverter:Creating backup of /opt/rh/jbcs-httpd24/root/etc/httpd-wtl/conf.d/mod_cluster.conf
2022-06-07 10:12:24,190:DEBUG:certbot.reverter:Creating backup of /opt/rh/jbcs-httpd24/root/etc/httpd-wtl/wtl.d/r3test.paycosmos.com.conf
2022-06-07 10:12:24,190:DEBUG:certbot.reverter:Creating backup of /opt/rh/jbcs-httpd24/root/etc/httpd/conf.d/mod_cluster.conf
<THE HTTPD SHOULD GET RELOADED/RESTARTED AT THIS POINT>
2022-06-07 10:12:27,489:DEBUG:acme.client:JWS payload:
b'{}'
2022-06-07 10:12:27,492:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/chall-v3/117136456736/VecRZw:

Yes, probably. But I'm more interested in the Apache logs around that time. Not sure how to continue debugging this further without modifying the code to the certbot-apache plugin, but that would have to be done on your computer.

3 Likes

Will it require recompilation?
I am ready to do whatever you suggest, it is a test environment and I have full control there.

Python is an interpreted language, so: no.

It's kinda hard to tell someone through a forum to do this, do that. Something that one would test out in a few minutes, could take hours unfortunately.

Best way is probably to know how you installed your Apache including its configuration and to try to reproduce it locally.

Also, looking at your own Apache logs doesn't require anything but you looking.

1 Like